Concepts

on this page

The document tree

grade6.5density0.59compressionnone

Analysis returns one nested value. Each level owns the level below it and carries its own data.

LevelTypeCarries
documentDocumentthe section list, plus vocabulary_ttr, nominalization_ratio, passive_ratio
sectionSectionheading, level, and its paragraphs in document order
paragraphParagraphits span, text and kind, its sentences, plus readability_grade, lexical_density, compression_ratio
sentenceSentenceits text, id-sorted tokens, and the structural primitive fields
tokenTokenthe ten CoNLL-U columns (text, lemma, pos, xpos, features, head, dep, deps, misc, plus id) and the derived is_punct
grade7.1density0.49compression0.55

Paragraph.text is the text the parser reads, a markdown paragraph's inline markup removed. Sentence.text is rebuilt from the tokens, so its spacing can differ from the source. Both are normalized, and each carries a span, in UTF-8 bytes, saying where in the analysed text it came from. Each paragraph that is not a quote is parsed on its own, so a sentence always belongs to exactly one paragraph, rather than being matched back by text.

grade6.4density0.56compressionnone

Inside a sentence, exactly one token has head equal to 0: the root. Every other token names its governor in head and the relation in dep. Those two columns are what turn a flat token list into a tree.

Dependency tree for: The system was built by the team root depth 1 depth 2 root, head = 0 nsubj:pass aux:pass obl det case det The system was built by the team 1 DET 2 NOUN 3 AUX 4 VERB 5 ADP 6 DET 7 NOUN heavier lines are the arcs that make passive true
grade5.3density0.53compressionnone

Words keep their reading order left to right; height is depth in the tree. built is the root, everything else hangs off it directly or through one more hop, so its tree_depth is 2.

grade9.1density0.60compression0.58

The same two columns, read from matra's output for five short sentences. The table is what a Sentence holds; the arcs below it draw each head and dep. The first sentence carries a negation, the second a modal, the third a clausal complement, the fourth a root adverbial, and the fifth two Hearst pairs: the structural primitives described further down.

Dependency parse

1The committee did not approve the budget.

  1. The det of committee
  2. committee nsubj of approve
  3. did aux of approve
  4. not advmod of approve
  5. approve root
  6. the det of budget
  7. budget obj of approve
  8. . punct of approve
Dependency arcs for: The committee did not approve the budget.rootdetadvmoddetauxobjnsubjpunctTheDETcommitteeNOUNdidAUXnotPARTapproveVERBtheDETbudgetNOUN.PUNCT
show the data
#WordLemmaPOSHeadRelation
1ThetheDET2committeedet
2committeecommitteeNOUN5approvensubj
3diddoAUX5approveaux
4notnotPART5approveadvmod
5approveapproveVERB0rootroot
6thetheDET7budgetdet
7budgetbudgetNOUN5approveobj
8..PUNCT5approvepunct
  • Core argument
  • Modifier
  • Function word
  • Other

Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.

Input: Five sentences, one structural primitive each: a negation, a modal, a reporting, a root adverbial, a Hearst pair, matra contributors. MIT, as part of the matra repository. Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: primitives/parse.json.

The four output tiers

grade8.4density0.40compressionnone

The pipeline produces them in this order, and each is a different kind of result.

  1. Structure, read and parsed by Engine::analyze. Sections, paragraphs, sentences, tokens, and the structural primitive fields. Derived from the parse, checkable against the source bytes.
  2. Measures, filled by Engine::analyze as its last step. Six Option<f64> slots filled by the metric suite: three per paragraph, two per document, plus the materialized passive_ratio. None means not computed, which is distinct from a computed zero.
  3. Outcomes, from free functions over the Document: summarize and keyphrases, with _with forms that name the method. They return Summary and Keyphrases values, each carrying the document's provenance with the method and its parameters, never fields on the tree.
  4. Semantic, with the reference embedder behind the model2vec feature. clusters returns a Clusters value whose provenance carries the embedder's identity and the threshold you supplied. This tier depends on the model, so by design it never becomes a field on Document or Sentence.

Structural primitives

grade9.6density0.55compression0.52

A structural primitive is a construction read off the dependency graph and reported as a field. Six of them are computed once and travel with the sentence into every language binding: five at Sentence construction, and hearst_pairs as the engine parses each paragraph, because its detector lives outside the domain (a Sentence built by hand has an empty hearst_pairs). They name an arc shape. They never name a judgment.

FieldThe arc shapeGrounded example
negationsnot, never, no, neither, nor on an advmod, det, or cc arcIt was never reviewed. gives one Negation: cue never, head reviewed
modalsone of ten modal lemmas on an aux arc or with the AUX part of speechYou must complete the form by Friday. gives one Modal: must on complete
clausal_complementsa verb governing a clausal complement (ccomp), with its subject when the parse has oneSmith reported that the effect vanished. gives one ClausalComplement: verb report, complement head vanished, subject Smith
root_adverbialsevery advmod arc into the rootReportedly, the deal closed. gives one RootAdverbial: reportedly
hearst_pairsthe six Hearst (1992) lexico-syntactic patterns, matched as dependency arcsAnimals such as dogs and cats need daily care. gives two HearstPair values, both tagged such_as, hypernym animal, hyponyms dog and cat
root_moodthe Mood feature of the root clause, on the root or on its cop, aux, or aux:pass child, with the root's token idThe committee approved it. gives Ind on approved; It might have been done. gives none, because neither the participle root nor its auxiliaries carry a mood
gradenonedensity0.75compressionnone

Two consequences of reporting arcs rather than readings:

grade6.5density0.61compression0.62

root_mood consults neither negation nor modals. The detector reads the root and its finiteness carriers. Nothing else. The sky is not blue reports Ind and also carries a negation; an indicative root that no modal governs is Ind with no modal whose head_id is the root's id. Combining the facts is your code's job.

grade8.4density0.48compressionnone

Overlap is intentional. never in the example above is both a Negation and a RootAdverbial, because it satisfies both arc shapes. Neither field is assigning it a meaning, so neither has to win.

grade18.6density0.57compressionnone

Passive voice is the seventh construction, a flag rather than a list of arcs: Sentence::passive is true when a token carries nsubj:pass, nsubjpass, or aux:pass, and its document-level share is the passive_ratio field of DocumentMetrics.

grade13.8density0.64compressionnone

Reporting verbs and root adverbials are open word classes, so matra ships no lexicon for either. Sentence::clausal_complements_in and Sentence::root_adverbials_in filter by a list you supply.

Where to go next

grade8.2density0.56compressionnone

Situation model covers what this output is for. Programming model covers the surface that produces it.

This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 13 paragraphs, in the margin. the data