Add Perseus data

[scott-fleischman]

Load XML from https://github.com/PerseusDL

Data we may want to extract:

• Author
• Work
• Date?
• A way to reference a part of the work: chapter / paragraph / line number
• Greek text
  • Attested form
  • Variant readings?
  • Distinguish between editor additions and MSS forms (anything else?)

I imagine we want to be able to access the data:

• Access all works as a giant list of works. I think a work or book is best top-level division.
  • Allow access to author, date, etc. so the list can be filtered on those parameters
• Each work should at least provide a list of "words", with metadata on the word.
  • Words can be broken down into a list of Tokens (a data type representing letter, case and mark information)
  • It may be useful to know punctuation at the beginning or end of a word (period; Greek question mark, semicolon, etc.). Some editions use punctuation to indicate a variant reading (found in an apparatus).
  • Each word should have a reference (author, work, line/chapter number) so we can trace where it came from.

LSJ: Is the full work available here? If so, we will want to extract more precise information from it for lemmas, principal parts, dialect forms, etc.

[scott-fleischman]

I'd be curious to see what an SML development of these ideas would look like. Perhaps modules would give a good abstraction for each level that we want to interact with the language (e.g., letter, syllable, word, clause, etc.)?

[scott-fleischman]

We probably should add the Perseus repos as submodules. Adding a test to verify successful loading of the data would be helpful.

[jonsterling]

@scott-fleischman Thanks for the added detail!

As you mention, ML modules might be a good fit for the kind of abstractions you're talking about. On the other hand, it feels like Haskell's laziness could be a big win when considering "the whole Perseus corpus" as a big datastructure that we want to query into...

[scott-fleischman]

I wonder if it would be worthwhile to create a separate Haskell library that provides access to the Perseus data.

[jonsterling]

It may be a good idea; I'm afraid that during the next week, I will probably not have time to look into this. So if anyone else wants to take a look, you won't be stepping on my toes.

[scott-fleischman]

I won't be getting to it in the next week either; maybe in the next month or two it will become useful as we work on our own from-scratch morphological analysis.

[scott-fleischman]

One thing I would like to find out sooner than later is whether LSJ is available in the data.