Domain types

on this page
grade10.8density0.63compressionnone

matra::domain holds every type the library hands back. The module depends on serde, thiserror, and the standard library, and on nothing else, so a caller can depend on these types without inheriting a C++ toolchain.

grade10.6density0.53compressionnone

It is not a copy of the generated item-level documentation. For that, run cargo doc --no-deps --all-features.

The type graph

grade2.6density0.73compressionnone

Five types nest, each owning the next. Nothing is owned twice.

Containment from Document to Token, stored fields against computed methods stored fields computed on call Document Section Paragraph Sentence Token provenance · sections · metrics (DocumentMetrics) · skipped heading · level · span · paragraphs span · text · kind · sentences · metrics (ParagraphMetrics) word_count · sentence_count span · text · tokens · word_count · tree_depth · passive · six structural fields root_token · head_of · children_of · subtree · content_tokens the ten CoNLL-U columns · span · is_punct
grade11.6density0.54compression0.53

The left column is what the pipeline writes into the struct. The right column is recomputed from the tree every time you ask, and an empty right column means the type only holds. So there is no second list of paragraphs anywhere in Document: the flat iterators walk the section tree. Every number a reader needs from a document is stored, in DocumentMetrics and ParagraphMetrics and in a sentence's word_count, tree_depth and passive, because a field crosses the language boundary and a method does not.

grade16.9density0.57compression0.52

The derived structural types (Negation, Modal, ClausalComplement, RootAdverbial, RootMood, HearstPair with its HearstSpan and HearstPattern) sit inside the nest, owned by Sentence, and the metric records DocumentMetrics and ParagraphMetrics are owned by Document and Paragraph. Provenance, with its Identity and ProvenanceSource, is owned by Document, and so is skipped, a list of Skipped records, each a SkippedKind and a Span. Every heading, paragraph, sentence and token holds a Span, and a paragraph its ParagraphKind. The other types sit outside it:

TypeRole
Blocka heading or paragraph a caller supplies, the input to Input::blocks; never returned
Summary, SummarySentenceoutput of summarize and summarize_with
Keyphrases, Keyphraseoutput of keyphrases and keyphrases_with
Clusters, Cluster, SimilarityEdgeoutput of clusters, described in Semantic clusters
Parameterswhat shaped an outcome, recorded in its provenance
Formatwhich reader a document needs
Errorevery failure the library can return
ErrorDetails, DetailValuethe values a caller acts on in an error, by name: what Error::details returns
DocumentErrorone document's failure, an Err item of analyze_all's stream
Embeddingone vector from an Embedder, the input to clustering
ProvisionNoticewhat a model fetch is about to download, handed to the _with_notice constructors' callback, described in Errors
grade7.6density0.38compressionnone

Where each of those enters and leaves the pipeline is drawn in Architecture.

Attributes shared across the module

TypeDerives#[non_exhaustive]
TokenDebug, Clone, Serialize, Deserializeyes
TokenBuildernoneno
SentenceDebug, Clone, Serialize, Deserializeyes
ParagraphDebug, Clone, Serialize, Deserializeyes
SectionDebug, Clone, Serialize, Deserializeyes
DocumentDebug, Clone, Serialize, Deserializeyes
DocumentMetrics, ParagraphMetricsDebug, Clone, Default, PartialEq, Serialize, Deserializeyes
ProvenanceDebug, Clone, PartialEq, Serialize, Deserializeyes
ParametersDebug, Clone, Default, PartialEq, Serialize, Deserializeyes
Identity, ProvenanceSourceDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
Summary, SummarySentence, Keyphrases, KeyphraseDebug, Clone, PartialEq, Serialize, Deserializeyes
Clusters, Cluster, SimilarityEdgeDebug, Clone, PartialEq, Serialize, Deserializeyes
FormatDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
SpanDebug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserializeyes
Skipped, SkippedKind, ParagraphKindDebug, Clone, Copy, PartialEq, Eq, Serialize, Deserializeyes
BlockDebug, Clone, PartialEq, Eqno
ErrorDebug, thiserror::Error; Serialize by handyes
ErrorDetailsDebug, Clone, Default, PartialEq, Eq; Serialize by handno
DetailValueDebug, Clone, PartialEq, Eq, Serializeyes
Negation, Modal, ClausalComplement, RootAdverbialDebug, Clone, Serialize, Deserializeyes
RootMoodDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
HearstPair, HearstSpanDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
HearstPatternDebug, Clone, Copy, PartialEq, Eq, Serialize, Deserializeyes
DocumentErrorDebug; Serialize by handyes
EmbeddingDebug, Clone, PartialEq, Serialize, Deserializeno
ProvisionNoticeDebug, Clone, PartialEq, Eqyes
grade5.8density0.55compressionnone

Block is the one input type. It has no serde derives and no public fields, so it needs no #[non_exhaustive]: a caller builds one with Block::heading or Block::paragraph and only matra reads it. Its JSON form is the blocks format, whose schema is spec/schema/v2/blocks.schema.json.

grade14.7density0.60compression0.50

No type in the module derives Hash, and the parse tree derives no equality: two Token, Sentence, Paragraph, Section or Document values cannot be compared with ==. Format derives PartialEq and Eq, which is what the reader table keys on, and the metric records, the provenance types, Span, Skipped, the kind enums, Block, RootMood, the Hearst types, the outcome types, Embedding, ProvisionNotice, ErrorDetails and DetailValue derive PartialEq. Error has no Clone, no equality and no serde derives; its Serialize is written by hand, as DocumentError's and ErrorDetails' are. ErrorDetails has no public fields, like Block, so it needs no #[non_exhaustive]. Embedding is the one public type with public fields and without #[non_exhaustive]: on a tuple struct the attribute would make the constructor private, and an Embedder implemented outside the crate has to build one.

grade10.3density0.63compression0.62

#[non_exhaustive] has two effects outside the crate. Struct literal syntax is unavailable, so you build values through the associated constructors listed below. The outcome types have no public constructor: matra produces them, and serde reads them back. Matches on Format, ParagraphKind, SkippedKind and Error need a catch-all arm, because variants can be added in a minor release.

grade7.6density0.48compressionnone

Every field on these types is pub and readable. TokenBuilder and Block are the exceptions: their fields are private, and you reach them through the builder's setters and the block's constructors.

Token

grade9.4density0.92compressionnone

One parsed token carrying the full CoNLL-U annotation set plus one derived flag.

FieldTypeCoNLL-U columnContents
idusize1Position within the sentence, 1-based
spanSpannoneWhere the surface form lies in the analysed text, in UTF-8 bytes
textString2Surface form
lemmaString3Dictionary form
posString4Universal POS tag, for example NOUN, VERB, ADJ
xposString5Language-specific POS tag
featuresBTreeMap<String, String>6Morphological features as a map from feature name to value
headusize7Id of the governing token; 0 marks the root
depString8Dependency relation to the head, for example nsubj, obj, aux:pass
depsString9Enhanced dependency graph
miscString10Miscellaneous annotations, for example SpaceAfter=No
is_punctboolderivedWhether the token is punctuation
grade12.3density0.64compressionnone

Provider-specific notes for the UDPipe adapter, the one adapter that ships:

  • is_punct is set when the UPOS tag is PUNCT.
  • deps is always the string _, the CoNLL-U empty marker. The underlying udpipe-rs binding does not surface column 9.
  • xpos and misc carry whatever the model emits, and can be _ or empty. features is converted from the column: Mood=Ind|Tense=Pres becomes two entries, a multi-valued feature (Case=Nom,Acc) keeps its value unsplit, and _ becomes an empty map.
grade12.9density0.48compressionnone

A token id or head that does not fit in usize is rejected at the adapter with Error::ParseFailed rather than being silently coerced.

grade17.1density0.53compression0.50

span comes from aligning each form with the paragraph text the parser was given, then mapping it into the analysed text. A form the adapter cannot find where the text says it should be is Error::ParseFailed naming the token, never a guessed span, and the engine checks every parser's spans by the same rule, so a span from any parser that does not line up with its form is Error::ParseFailed with reason token_form_mismatch. In a markdown paragraph whose markup the reader removed, a token's span can cover markup inside it (foo**bar** is one token from the f to the last r), and a token read from an escape or an entity (&amp;) covers all of it.

Construction

let token = matra::domain::Token::builder(
        1,                       // id
        "committee".to_string(), // text
        "committee".to_string(), // lemma
        "NOUN".to_string(),      // pos
        3,                       // head
        "nsubj".to_string(),     // dep
    )
    .xpos("NN".to_string())
    .feats("Number=Sing".to_string())
    .is_punct(false)
    .build();
grade10.3density0.71compression0.54

Token::builder(id, text, lemma, pos, head, dep) -> TokenBuilder takes the six CoNLL-U essentials. TokenBuilder carries one setter per optional field (xpos, deps, misc, each taking String; feats, taking the CoNLL-U column 6 string and converting it into features; is_punct taking bool; span taking a Span), each consuming and returning the builder. build() returns the Token. Unset string fields default to empty, features to an empty map, is_punct to false, and span to the empty span at 0.

Methods

SignatureReturnsBehavior
feat(&self, key: &str)Option<&str>The value of one feature in features, by exact name. None when the feature is absent
grade9.6density0.55compression0.64

feat("Mood") returns Some("Ind") when the token carries Mood=Ind. The value is what the provider emitted, so a multi-valued feature (Case=Nom,Acc) comes back unsplit: matra does not normalise it. A token whose column 6 was _ or empty has no features, so every lookup on it returns None.

grade8.7density0.53compressionnone

feat is Rust-only by design. features crosses FFI as a map, so a lookup over it adds no information to the wire; a Python or TypeScript caller indexes the same map (RFC-0009).

Sentence

FieldTypeContents
spanSpanWhere the sentence lies in the analysed text: from its first token's start to its last token's end, derived at construction
textStringSentence text rebuilt from the token forms, normalized, not a slice of the input
tokensVec<Token>Tokens in ascending id order
word_countusizeNon-punctuation tokens, derived at construction
tree_depthOption<usize>Longest path from any token to the root, the root at depth 0, derived at construction. None when the head references form a cycle
passiveboolWhether any token's dep is nsubj:pass, nsubjpass, or aux:pass, derived at construction
negationsVec<Negation>Negation cues, derived at construction
modalsVec<Modal>Modal auxiliaries, derived at construction
clausal_complementsVec<ClausalComplement>Verbs governing a clausal complement (ccomp), each with its subject when the parse has one, derived at construction
root_adverbialsVec<RootAdverbial>Adverbial modifiers attached to the root, derived at construction
hearst_pairsVec<HearstPair>Candidate hypernymy pairs from the six Hearst (1992) patterns, filled by the pipeline at the annotate stage
root_moodOption<RootMood>The root clause's Mood feature with the root's token id, derived at construction. None when the root clause carries no mood
grade13.6density0.60compression0.54

Invariants that downstream code relies on: tokens are id-sorted; exactly one token has head == 0; every other head names a token in the same sentence; following head never loops. Every sentence the engine builds holds them, because it checks what the parser returns before it builds one and fails the document with Error::ParseFailed naming the check otherwise. A sentence built by hand with Sentence::new, which trusts its caller, or read back from JSON can break any of them, and the tree walks stay safe when the last one breaks (see tree_depth below).

grade9.3density0.55compression0.59

tree_depth is computed in time linear in token count with a visited set, and is None when the head references form a cycle, which reports a malformed parse as a value a reader must handle instead of truncating it or returning a number that reads as a very deep tree. An empty sentence has depth Some(0). word_count counts tokens the NLP provider identified, minus punctuation. It is not a whitespace split, and the two counts differ on the same text.

grade11.9density0.56compression0.56

The engine builds text by joining the token surface forms a parser returned, inserting a space unless the preceding token carries SpaceAfter=No in misc. The string is a reconstruction, not a slice of the input, so whitespace can differ from the source text; span is where the sentence came from.

Derived structural fields

grade14.8density0.57compression0.55

The fields after tokens are computed once from the dependency graph and serialized with the sentence, so every language binding and the CLI's JSON read one Rust detection as data (RFC-0008). Most are computed at Sentence::new; hearst_pairs is the exception, filled by the pipeline at the annotate stage because its detector lives in the hearst module outside the domain, so a hand-built Sentence carries an empty vector until the caller runs hearst::hypernymy_pairs. Each field reports structure only; the reading is the caller's.

grade11.3density0.58compression0.59

Negation reports one cue by token id: cue_id, cue_lemma (not, never, no, neither, nor), and head_id, the token the cue attaches to. A cue fires only on its carrying relation (advmod, det, cc), so nothing as a subject does not fire and tokenizer-split cannot fires exactly once, on the not token. Whether the negation reverses the sentence's claim is not matra's call.

grade12.2density0.56compression0.50

Modal reports one modal auxiliary by token id: aux_id, aux_lemma, and head_id, the token the auxiliary attaches to (0 when the modal is itself the root). The closed class is the ten lemmas the UD English treebank tags MD: can, could, may, might, must, ought, shall, should, will, would, matched by lemma on the aux relation or the AUX part of speech. The part-of-speech arm catches the shapes the model does not park on aux: VP ellipsis promotes the modal to root ("Yes, you should.") and coordinated auxiliaries parse as root plus aux or conj ("He can and will succeed."). Precision holds because the model's only other AUX lemmas (be, have, do, get) are outside the class. Contracted forms fire on their lemma (wo in won't lemmatizes to will), the noun will never fires because it never carries aux or AUX, and a multi-auxiliary chain (might have been done) reports its modal with head_id on the content verb. English modals are ambiguous across readings, and resolving must into obligation or inference takes context matra does not model, so no epistemic, deontic, or dynamic category is assigned: the auxiliary and its arc are reported, the reading is the caller's.

grade8.2density0.57compression0.51

RootMood reports the grammatical mood of the root clause: root_id, the root's token id, and mood, the Mood feature as the model emitted it (Ind, Imp, and an open set beyond). The feature is read off the root token or, when the root carries none, off its first cop, aux, or aux:pass child that does, because the model parks finiteness on the auxiliary in copular ("The sky is blue."), do-support ("He did leave.") and passive clauses. A modal carries VerbForm=Fin and no mood, so a clause whose root a modal governs ("You must complete the form.") usually has no root mood; whether a modal governs the root is the separate fact in modals. It reads the root clause only, so a mood in a subordinate clause does not reach it. The bare assertoric form, an indicative root no modal governs, is mood == "Ind" with no modal whose head_id is root_id.

grade8.4density0.57compression0.48

ClausalComplement reports one verb governing a clausal complement by token id: verb_id and verb_lemma for the verb, ccomp_id for the head of the complement, and subject_id plus subject_lemma for the verb's subject (nsubj, nsubj:pass) when the parse has one in the same sentence. The construction fires for every verb that fills it ("Smith reported that the effect vanished.", "These results suggest that the mechanism is shared.", "We show that the method works." all match), because verbs taking a clausal complement are an open class and any list matra shipped would be incomplete while looking authoritative. Which verb lemmas count as reporting or evidential is the caller's lexicon, supplied as a parameter to clausal_complements_in or applied to the field directly in any language. A ccomp under a non-verb head ("I am sure that it works." parses the complement under the adjective) is outside the construction by decision. The subject is optional by observed fact: UDPipe splits "Smith et al. reported that ..." into two sentences at the period in "et al.", stranding the attribution in the previous sentence while the verb keeps its ccomp. That upstream segmentation defect is recorded here and in the test suite rather than papered over.

grade12.0density0.59compression0.50

RootAdverbial reports one adverbial modifier attached to the root by token id: adv_id and adv_lemma. The root-attached advmod arc is where sentence-scope adverbs land ("Reportedly, the deal closed."), but the dependency graph does not distinguish sentence scope from manner ("The team quickly shipped." puts quickly on the same arc), so every root-attached adverbial is reported and the caller's lexicon selects the evidential ones, via root_adverbials_in or a filter over the field. Negation particles on the same arc ("not" in "The plan is not ready.") appear in both negations and root_adverbials; both report arcs, neither assigns a reading, and the overlap is intentional.

grade18.3density0.61compression0.46

HearstPair reports one candidate hypernymy pair from the six lexico-syntactic patterns of Hearst (1992), detected as dependency-arc patterns rather than regex over surface text: NP such as NP, such NP as NP, NP, including NP, NP, especially NP, NP and other NP, NP or other NP. The pattern field carries a HearstPattern tag naming the construction that matched (such_as, such_np_as, including, especially, and_other, or_other on the wire), and hypernym and hyponym each carry a HearstSpan: head_id and head_lemma for the noun phrase head, plus first_id and last_id for the contiguous token range covering the head and its adjacent nominal modifiers, with the pattern's own marker words (such, other) excluded, so traceability holds against tokens. Precision is favoured over recall: each detector requires the full verified arc shape, so a clausal coordination ("She invited John, and other guests brought gifts."), a definite contrastive coordination ("The teacher and the other students laughed."), or an emphasized plain coordination ("France and especially Spain") does not fire, and a sentence the model misparses is missed rather than misreported. The pair is a candidate by design: matra reports that the sentence used a construction which conventionally signals hypernymy, and does not build a taxonomy or assert the relation is true.

Methods

grade8.8density0.57compressionnone

Most methods below are walks over the head and dep columns; the two _in filters are views over the derived fields against a caller-supplied lexicon. Concepts draws one parsed sentence as the tree these walk.

SignatureReturnsBehavior
new(text: String, tokens: Vec<Token>)SentenceConstructor. Derives span and the structural fields from tokens. Does not validate the invariants
content_tokens(&self)Vec<&Token>Tokens where is_punct is false
root_token(&self)Option<&Token>The token with head == 0, if there is one
children_of(&self, id: usize)Vec<&Token>Tokens whose head equals id. Empty for an unknown id
head_of(&self, id: usize)Option<&Token>The governing token. None for the root and for an unknown id
subtree(&self, id: usize)Vec<&Token>The token and all its descendants, sorted by id. Empty for an unknown id
clausal_complements_in(&self, lexicon: &[&str])Vec<&ClausalComplement>Clausal complements whose verb_lemma is in the caller-supplied lexicon
root_adverbials_in(&self, lexicon: &[&str])Vec<&RootAdverbial>Root adverbials whose adv_lemma is in the caller-supplied lexicon
grade9.1density0.64compressionnone

subtree carries a visited set, so it terminates on cyclic input.

Paragraph

FieldTypeContents
spanSpanWhere the paragraph lies in the analysed text, trailing whitespace excluded. For markdown it covers the quote and list markers on its lines
textStringThe text the parser reads, normalized: for markdown the inline text with the markup removed (emphasis, link and image syntax, code span backticks, quote and list markers), soft line breaks as \n, escapes and entities decoded; for plain text and block input a verbatim slice
kindParagraphKindParagraph, Quote (inside a block quote at any depth) or ListItem; paragraph, quote, list_item on the wire
sentencesVec<Sentence>Sentences from parsing this paragraph
metricsParagraphMetricsThe paragraph's three measures
gradenonedensity0.75compressionnone

ParagraphMetrics holds the three:

FieldTypeContents
readability_gradeOption<f64>Flesch-Kincaid grade level, when the paragraph holds more than ten words
lexical_densityOption<f64>Content-word ratio, 0.0 to 1.0, when the paragraph holds a word
compression_ratioOption<f64>Brotli compressed size over original size, when the paragraph holds more than fifty words
grade16.4density0.59compressionnone

Each slot holds None until Engine::compose fills it, and stays None when the paragraph does not meet the metric's applicability condition; no outer Option wraps the record, because each slot gates on its own.

grade13.5density0.60compressionnone

Engine::annotate skips quotes when it parses, so a paragraph whose kind is ParagraphKind::Quote reaches compose with no sentences and keeps all three slots at None.

Methods

SignatureReturnsBehavior
new(text: String, kind: ParagraphKind, span: Span)ParagraphConstructor. Empty sentences, every metric None
word_count(&self)usizeNon-punctuation tokens summed across this paragraph's sentences
sentence_count(&self)usizeNumber of sentences

Section

FieldTypeContents
headingOption<String>Heading text, its markup removed
levelusizeHeading depth
spanOption<Span>Where the heading lies in the analysed text, markers included; None with no heading
paragraphsVec<Paragraph>Paragraphs in document order
grade7.5density0.53compression0.48

The markdown reader sets level to the heading's level (the number of # characters, or 1 and 2 for a setext heading) and heading to its text. Content before the first heading becomes a section with heading: None and level: 0. Plain text produces at most one section, always with heading: None and level: 0. Block input opens a section at each Block::heading, at the level it names.

grade12.2density0.61compressionnone

Section::new(heading: Option<String>, level: usize, span: Option<Span>, paragraphs: Vec<Paragraph>) -> Section is the constructor. There are no other methods.

Document

gradenonedensity0.40compressionnone

The output of the pipeline.

FieldTypeContents
provenanceProvenanceWhere the document came from and what produced it, recorded by Engine::annotate
sectionsVec<Section>The section tree, which owns every paragraph, sentence, and token
metricsDocumentMetricsThe document's aggregates, filled by Engine::compose
skippedVec<Skipped>What the reader saw and did not analyse, in document order
grade9.9density0.60compressionnone

DocumentMetrics holds what the command line's table prints and what a binding would otherwise rebuild:

FieldTypeContents
total_wordsusizeNon-punctuation tokens across every sentence
total_sentencesusizeSentences across every paragraph
paragraph_countusizeParagraphs, quotes included
mean_sentence_lengthOption<f64>total_words over total_sentences. None with no sentences
sentence_length_stdOption<f64>Sample standard deviation of sentence length in words. None with fewer than two sentences, where it is undefined
vocabulary_ttrOption<f64>Type-token ratio over lemmas. Falls as text grows, so documents of different lengths are not comparable on it
nominalization_ratioOption<f64>Share of nominalizing nouns
passive_ratioOption<f64>Passive sentences over total sentences. None with no sentences
grade13.1density0.55compressionnone

Before the document is measured every count is 0 and every ratio None; None after it means the document did not meet the metric's condition, never a computed zero. Engine::analyze always measures, so this holds for a document you build by hand with Document::new.

grade11.2density0.55compression0.49

Provenance holds matra, the version that produced the document; schema, the schema version as "major.minor" (SCHEMA_VERSION, "2.0"), which appears nowhere else; parser, the parser's Identity; format, the Format the input was read as; and source, a ProvenanceSource with the input's path (None for anything but a file), and the length in bytes and the sha256 of the analysed text every span points into, computed once at annotation. For markdown and plain text that is the input; for block input, the blocks joined. On a document, method and parameters are None and absent from the JSON; an outcome derived from the document carries a copy of its provenance with both set (see Summary). There is no timestamp, so two analyses of the same input by the same build are identical. Identity names a model-backed component: name (udpipe), model (english-ewt-ud-2.5-191206), sha256 of the model artifact, and license. The UDPipe adapter derives it from the bytes it loaded, so a model matra does not pin reports its digest with model and license set to None; an embedder reports one in the same shape (model2vec for Model2Vec), recorded in a cluster result's parameters.embedder. Provenance::new(parser, format, source) fills matra and schema from the build.

Methods

SignatureReturnsBehavior
new(provenance: Provenance, sections: Vec<Section>, skipped: Vec<Skipped>)DocumentConstructor. An empty metric record
paragraphs(&self)impl Iterator<Item = &Paragraph>Every paragraph, in document order
paragraphs_mut(&mut self)impl Iterator<Item = &mut Paragraph>The mutable form, which the engine uses as it parses and measures
sentences(&self)impl Iterator<Item = &Sentence>Every sentence across every paragraph
tokens(&self)impl Iterator<Item = &Token>Every token across every sentence

Summary

grade12.3density0.58compressionnone

What summarize and summarize_with return. The selector is matra::SummaryMethod, at the crate root (TfIdf, the default, and TextRank, #[non_exhaustive]).

FieldTypeContents
provenanceProvenanceThe document's, with method (tfidf or textrank, an open set) and parameters (n)
sentencesVec<SummarySentence>At most n sentences, in document order
gradenonedensity0.71compressionnone

SummarySentence is one sentence a summary keeps:

FieldTypeContents
indexusizeThe sentence's position among the document's sentences (Document::sentences), counting from 0
spanSpanWhere it lies in the analysed text, as on the Sentence
textStringIts text, as on the Sentence
scoref64The method's score, higher is more representative
grade9.8density0.51compressionnone

Both summarizers select the top n by score and then return them in ascending index, so the result reads in document order while score carries the ranking. A score compares only with another from the same method.

grade15.5density0.53compression0.57

Parameters records what shaped an outcome, each field present only when the method takes it: n (Option<usize>) for a summary and keyphrases, threshold (Option<f64>) and embedder (Option<Identity>, the identity the embedder reports, the same shape as the parser's) for clusters. In JSON an absent parameter is an absent key, so a summary's is {"n": 3}.

Keyphrases

grade13.3density0.60compressionnone

What keyphrases and keyphrases_with return. The selector is matra::KeyphraseMethod (Rake, the default, and Yake, #[non_exhaustive]).

FieldTypeContents
provenanceProvenanceThe document's, with method (rake or yake, an open set) and parameters (n)
phrasesVec<Keyphrase>At most n phrases, highest score first; equal scores in the order the phrases are first met, the same on every run
gradenonedensity0.80compressionnone

Keyphrase is one ranked phrase:

FieldTypeContents
phraseStringThe phrase, built from lowercased lemmas joined by single spaces
scoref64Relevance score, higher is more relevant
occurrencesVec<Span>Every stretch of the analysed text it was found in, in document order, from its first word's start to its last word's end

Clusters

grade14.2density0.60compressionnone

What clusters returns: provenance (the document's, with method cosine and parameters threshold and embedder), clusters (Vec<Cluster>, each members, sentence indices) and edges (Vec<SimilarityEdge>, each a, b and score: f64). Semantic clusters describes each field and what co-membership does and does not mean.

Format

gradenonedensity0.60compressionnone

Which reader a document needs.

VariantWire nameAssigned to
MarkdownmarkdownExtension .md or .markdown
PlainTextplain_textEvery other extension, and a file with no extension
BlocksblocksNo extension: a caller names it, and Input::blocks takes it
PdfpdfExtension .pdf, or bytes opening with %PDF
DocxdocxExtension .docx, or a zip archive holding [Content_Types].xml
Other(String)the name it holdsNever detected: a caller names it, built with Format::named
grade10.7density0.49compression0.60

Format::from_path(path: impl AsRef<Path>) -> Format is the extension column, and it is the only copy of it. Reading a file with Input::file applies it, and the command line reads it for the name given to --stdin-filename when --format names none, so a name means the same thing on both routes. When the bytes are read, a PDF or DOCX signature outranks both the extension and a format the caller names.

grade12.9density0.53compression0.51

Pdf, Docx and an Other have no reader that ships, so analyzing such a document returns Error::UnsupportedFormat with that format, unless the engine was given a reader for it with Engine::with_reader. Other names a format matra has no variant for, the one a caller's own reader reads: Format::named("html"). Format::named gives the variant matra has for a name it knows, so Format::named("markdown") is Markdown, and a name read back from JSON does the same, an unknown one reading as Other rather than failing. Input::text and Input::file_as take the format as an argument; Input::bytes takes one optionally; Input::file detects one. Format derives PartialEq and Eq, which is what the reader table keys on.

Block

grade10.3density0.57compressionnone

A heading or a paragraph a caller supplies, for a document matra has no reader for. Input only: built by Block::heading(level: u8, text: impl Into<String>) and Block::paragraph(text: impl Into<String>), handed to Input::blocks, and never returned. It has no public fields.

grade4.5density0.52compression0.56

The analysed text of block input is the blocks' texts joined by a blank line ("\n\n"), in order, and every span in the result points into it. A heading opens a section at its level; a level of 0 is refused with Error::InvalidInput when the input is analysed. The same blocks as JSON are the blocks format: an array of {"kind": "heading", "level": 1, "text": "..."} and {"kind": "paragraph", "text": "..."} objects, with an unknown kind or field refused by name.

Span

FieldTypeContents
byte_startusizeThe first byte
byte_endusizeOne past the last byte
grade7.3density0.55compression0.54

A stretch of the analysed text in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is the stretch. Bytes rather than characters: a caller in a language that indexes strings by character converts. The unit is in the field names because a byte span used to slice a character-indexed string, a Python str, is right on ASCII and wrong, with no error, after the first character outside it. Span::new(byte_start, byte_end) is the constructor.

Skipped

FieldTypeContents
kindSkippedKindWhat the construct is
spanSpanWhere it lies in the analysed text, trailing whitespace excluded
grade10.8density0.68compression0.61

One construct the reader saw and did not analyse. SkippedKind names it: Table, Code, FrontMatter, Html, ThematicBreak, LinkDefinition (table, code, front_matter, html, thematic_break, link_definition on the wire), a set a reader treats as open. Skipping is specified behaviour beside a correct result, not an error. Skipped::new(kind, span) is the constructor.

grade16.6density0.61compressionnone

Together with the heading and paragraph spans, skipped accounts for the input: every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span.

DocumentError

FieldTypeContents
pathOption<PathBuf>Where the failure occurred, None for input that is not a file
errorErrorWhat went wrong
grade10.6density0.76compression0.60

One per-document failure: the Err item Engine::analyze_all yields in the failed input's place, so documents plus errors equals inputs consumed. Display prints path: error when a path exists and the bare error otherwise; source() returns the wrapped Error. DocumentError::new(path, error) is the constructor. Not Clone, because Error wraps std::io::Error.

grade5.6density0.53compressionnone

It serializes as its error's form plus the path, {"path", "kind", "message", "details"}, with path null for input that is not a file and written lossily where it is not UTF-8. The JSON Schema is spec/schema/v2/error.schema.json.

grade14.5density0.56compressionnone

matra collects nothing across documents: a stream is the collection, and how to combine documents is the caller's decision (a count sums; vocabulary_ttr does not).

Constants, aliases, and the error type

ItemDefinitionRole
limitsmodule of usize constantsEvery cap and metric minimum, below
SCHEMA_VERSION&str, "2.0"The document schema version this build emits, as "major.minor", written into Provenance::schema. The JSON Schema for each major is in spec/schema/
Result<T>std::result::Result<T, Error>Return type of every fallible library function
Errorenum, #[non_exhaustive]Every failure the library returns. See Errors
ErrorDetailsordered names and values; get(key), iter(), len(), is_empty()What Error::details returns: the values a caller acts on, by name. Serializes as an object. See kinds and details
DetailValueenum, #[non_exhaustive]: Text(String), Integer(u64), Bool(bool)One value in an ErrorDetails. Serializes as the bare value

Limits

grade10.5density0.54compression0.51

domain::limits holds every bound matra holds its work to, each defined once and read where it applies. They are published so you can size work before a call instead of learning a bound from a refusal: split an input under MAX_INPUT_BYTES, or chunk a document's sentences under the cap of the method you select, each named for that method as SummaryMethod and KeyphraseMethod name it. They are constants, not settings: no configuration changes them. They are this release's values and may change in any 0.x release, so read them from domain::limits rather than copying the numbers. A cap that is exceeded returns InputTooLarge, whose limit detail is the constant and whose what detail names the gate; a paragraph outside a metric's bounds keeps that metric None. Each constant's rustdoc carries the cost model behind its value.

ConstantValueBounds
MAX_INPUT_BYTES8,388,608 (8 MiB)Text one input may hold; the "input" and "file_read" gates
MAX_CONFIG_FILE_BYTES65,536 (64 KiB)The configuration file; the "config_file_read" gate
MAX_DOWNLOAD_BYTES67,108,864 (64 MiB)Any one model download; the "parser_model_download" and "embedding_model_download" gates
MAX_TFIDF_SENTENCES2,000Sentences a TF-IDF summary ranks; "tfidf"
MAX_TEXTRANK_SENTENCES2,000Sentences a TextRank summary ranks; "textrank"
MAX_CLUSTER_SENTENCESMAX_TEXTRANK_SENTENCESSentences or vectors clustered; "clusters". Defined as TextRank's, since both build the same quadratic graph
MAX_RAKE_TOKENS200,000Tokens RAKE reads, punctuation included; "rake"
MAX_YAKE_TOKENS200,000Tokens YAKE reads, punctuation included; "yake"
MAX_COMPRESSION_PARAGRAPH_BYTES262,144 (256 KiB)The largest paragraph given a compression_ratio
MIN_READABILITY_WORDS11The fewest words for a readability_grade (more than 10)
MIN_COMPRESSION_WORDS51The fewest words for a compression_ratio (more than 50)
grade10.5density0.47compressionnone

Two caps with the same value stay two constants where their costs differ, so tuning one does not move the other: TF-IDF's 2,000 is not TextRank's, and YAKE's 200,000 is not RAKE's.

What crosses the language boundary

grade6.8density0.50compressionnone

The Python surface serializes values with pythonize. Fields have a serde representation and cross with their names unchanged. Methods have none, so there is nothing for them to cross with.

Fields cross the FFI boundary, methods stop at it Rust Python Document Document stored fields dict keys provenance sections metrics skipped provenance sections metrics skipped methods paragraphs() sentences() tokens() FFI boundary pythonize + serde a Python caller walks these in the sections it already has
grade15.1density0.53compression0.55

Token::feat, Document::sentences, Sentence::subtree, and every other method in the tables above are available to Rust callers only. The two methods on Error, kind and details, are different: their values are computed into the error's serde form and onto the Python exception, so they cross as fields. Nothing a reader needs is method-only: every aggregate is a field of DocumentMetrics or ParagraphMetrics, and a sentence carries its word_count, tree_depth and passive, so each crosses with the record that holds it.

Rust typePython shape
TokenToken
SentenceSentence
ParagraphParagraph
SectionSection
Documentmatra.Document, read-only, whose to_dict() is a DocumentDict
DocumentMetrics, ParagraphMetricsDocumentMetrics, ParagraphMetrics
Provenance, Identity, ProvenanceSourceProvenance, Identity, ProvenanceSource
ClausalComplement, RootMoodClausalComplement, RootMood
Span, SkippedSpan, Skipped
ParametersParameters
Summary, SummarySentenceSummary, SummarySentence
Keyphrases, KeyphraseKeyphrases, Keyphrase
Clusters, Cluster, SimilarityEdgeClusters, Cluster, SimilarityEdge
DocumentErrorDocumentError
Errormatra.ErrorInfo, and the exception classes under matra.MatraError
ErrorDetailsErrorDetails
BlockBlockDict, a HeadingBlock or a ParagraphBlock
Formatits name, a FormatName
grade9.4density0.59compression0.50

Document and DocumentError are the two items Python's Engine.analyze_all yields. A Document stays a Rust value behind a read-only handle, and every read of it crosses through serde like every shape above it, with one field replaced: its provenance.source.path is decoded with the platform's filesystem decoding (os.fsdecode on Unix), so a name that is not valid UTF-8 crosses exactly instead of as serde's lossy string. A DocumentError crosses through serde too, as {"path", "kind", "message", "details"}, with its path decoded the same way. kind is Error::kind on the domain type, pinned across bindings by spec/tests/corpus/items.json and exported to Python as matra.ERROR_KINDS; message is the Rust error's own Display text; details is Error::details. An Error raised rather than returned crosses as a Python exception carrying the same three, kind, str(exc) and details: a method is computed into a value before it crosses, as an aggregate is. A future binding takes the same serde form.

grade10.7density0.58compressionnone

The Python shapes are TypedDict declarations available at runtime. They are declared in matra.types and re-exported from the package root, so either import works:

from matra.types import DocumentDict, Keyphrases, Summary
from matra import DocumentDict, Keyphrases, Summary
grade8.2density0.84compressionnone

Field names and nesting match the Rust types exactly. Option<f64> becomes float | None; Option<String> becomes str | None; Vec<T> becomes list[T]; usize becomes int.

This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 63 paragraphs, in the margin. the data