Sentence tokenization

We initially relied on the sentence divisions in our source corpora:

• MHG texts (from ReM) and ENHG texts (from ReF) indicate the end of a sentence with the annotation $MSBI (for "moderne satzbeendende Interpunktion"). Not being part of the original text, we annotate such punctuation as (CODE <.>), (CODE <?>), etc.
• Modern German texts in the DTA are divided into sentences, each with a sentence ID number.

However, the Penn annotation system specifies that each sentence token consist of at most one matrix clause (with a small number of exceptions). The following structures require that we split what is considered a single sentence in the source corpus into additional sentence tokens in our corpus.

The following are taken to indicate a new main clause. (Early in the project, rather than creating a new sentence token, we flagged such clauses as IP-MAT-CONJ, which helped train the parser to identify compound sentences. In version 0_9, these were split into separate sentence tokens.)

• What seems semantically and syntactically like a second-conjunct main clause, with or without a conjunction:

• The first sentence of a direct quotation (as in HeliPaD, but unlike CHLG and PPCHE, where the first sentence of a quote is the complement of the matrix verb of saying):

New main clauses will be kept within a sentence token if:

• The clause is parenthetical and embedded in another main clause. These are annotated as IP-MAT-PRN. Frequently these are inquits.
• There is clearly coordination of main clauses, with elided material in one of the clauses that can't be explained away as pro-drop or auxiliary-drop. The conjunct clause with elided material is annotated IPX-MAT.
• The clause can reasonably be interpreted as a relative clause (begins with der or welcher, sole purpose seems to be to elaborate on an NP in the main clause). This is to prevent potential verb-early relative clauses from being inaccessible to a query.