Concepts
The document tree
Analysis returns one nested value. Each level owns the level below it and carries its own data.
| Level | Type | Carries |
|---|---|---|
| document | Document | the section list, plus vocabulary_ttr, nominalization_ratio, passive_ratio |
| section | Section | heading, level, and its paragraphs in document order |
| paragraph | Paragraph | its span, text and kind, its sentences, plus readability_grade, lexical_density, compression_ratio |
| sentence | Sentence | its text, id-sorted tokens, and the structural primitive fields |
| token | Token | the ten CoNLL-U columns (text, lemma, pos, xpos, features, head, dep, deps, misc, plus id) and the derived is_punct |
Paragraph.text is the text the parser reads, a markdown paragraph's inline markup removed. Sentence.text is rebuilt from the tokens, so its spacing can differ from the source. Both are normalized, and each carries a span, in UTF-8 bytes, saying where in the analysed text it came from. Each paragraph that is not a quote is parsed on its own, so a sentence always belongs to exactly one paragraph, rather than being matched back by text.
Inside a sentence, exactly one token has head equal to 0: the root. Every other token names its governor in head and the relation in dep. Those two columns are what turn a flat token list into a tree.
Words keep their reading order left to right; height is depth in the tree. built is the root, everything else hangs off it directly or through one more hop, so its tree_depth is 2.
The same two columns, read from matra's output for five short sentences. The table is what a Sentence holds; the arcs below it draw each head and dep. The first sentence carries a negation, the second a modal, the third a clausal complement, the fourth a root adverbial, and the fifth two Hearst pairs: the structural primitives described further down.
Dependency parse
1The committee did not approve the budget.
- The det of committee
- committee nsubj of approve
- did aux of approve
- not advmod of approve
- approve root
- the det of budget
- budget obj of approve
- . punct of approve
show the data
| # | Word | Lemma | POS | Head | Relation |
|---|---|---|---|---|---|
| 1 | The | the | DET | 2committee | det |
| 2 | committee | committee | NOUN | 5approve | nsubj |
| 3 | did | do | AUX | 5approve | aux |
| 4 | not | not | PART | 5approve | advmod |
| 5 | approve | approve | VERB | 0root | root |
| 6 | the | the | DET | 7budget | det |
| 7 | budget | budget | NOUN | 5approve | obj |
| 8 | . | . | PUNCT | 5approve | punct |
- Core argument
- Modifier
- Function word
- Other
Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.
Input: Five sentences, one structural primitive each: a negation, a modal, a reporting, a root adverbial, a Hearst pair, matra contributors. MIT, as part of the matra repository.
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: primitives/parse.json.
The four output tiers
The pipeline produces them in this order, and each is a different kind of result.
- Structure, read and parsed by
Engine::analyze. Sections, paragraphs, sentences, tokens, and the structural primitive fields. Derived from the parse, checkable against the source bytes. - Measures, filled by
Engine::analyzeas its last step. SixOption<f64>slots filled by the metric suite: three per paragraph, two per document, plus the materializedpassive_ratio.Nonemeans not computed, which is distinct from a computed zero. - Outcomes, from free functions over the
Document:summarizeandkeyphrases, with_withforms that name the method. They returnSummaryandKeyphrasesvalues, each carrying the document's provenance with the method and its parameters, never fields on the tree. - Semantic, with the reference embedder behind the
model2vecfeature.clustersreturns aClustersvalue whose provenance carries the embedder's identity and the threshold you supplied. This tier depends on the model, so by design it never becomes a field onDocumentorSentence.
Structural primitives
A structural primitive is a construction read off the dependency graph and reported as a field. Six of them are computed once and travel with the sentence into every language binding: five at Sentence construction, and hearst_pairs as the engine parses each paragraph, because its detector lives outside the domain (a Sentence built by hand has an empty hearst_pairs). They name an arc shape. They never name a judgment.
| Field | The arc shape | Grounded example |
|---|---|---|
negations | not, never, no, neither, nor on an advmod, det, or cc arc | It was never reviewed. gives one Negation: cue never, head reviewed |
modals | one of ten modal lemmas on an aux arc or with the AUX part of speech | You must complete the form by Friday. gives one Modal: must on complete |
clausal_complements | a verb governing a clausal complement (ccomp), with its subject when the parse has one | Smith reported that the effect vanished. gives one ClausalComplement: verb report, complement head vanished, subject Smith |
root_adverbials | every advmod arc into the root | Reportedly, the deal closed. gives one RootAdverbial: reportedly |
hearst_pairs | the six Hearst (1992) lexico-syntactic patterns, matched as dependency arcs | Animals such as dogs and cats need daily care. gives two HearstPair values, both tagged such_as, hypernym animal, hyponyms dog and cat |
root_mood | the Mood feature of the root clause, on the root or on its cop, aux, or aux:pass child, with the root's token id | The committee approved it. gives Ind on approved; It might have been done. gives none, because neither the participle root nor its auxiliaries carry a mood |
Two consequences of reporting arcs rather than readings:
root_mood consults neither negation nor modals. The detector reads the root and its finiteness carriers. Nothing else. The sky is not blue reports Ind and also carries a negation; an indicative root that no modal governs is Ind with no modal whose head_id is the root's id. Combining the facts is your code's job.
Overlap is intentional. never in the example above is both a Negation and a RootAdverbial, because it satisfies both arc shapes. Neither field is assigning it a meaning, so neither has to win.
Passive voice is the seventh construction, a flag rather than a list of arcs: Sentence::passive is true when a token carries nsubj:pass, nsubjpass, or aux:pass, and its document-level share is the passive_ratio field of DocumentMetrics.
Reporting verbs and root adverbials are open word classes, so matra ships no lexicon for either. Sentence::clausal_complements_in and Sentence::root_adverbials_in filter by a list you supply.
Where to go next
Situation model covers what this output is for. Programming model covers the surface that produces it.
This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 13 paragraphs, in the margin. the data