{
  "generator": {
    "matra": "0.3.0",
    "udpipe_model": {
      "name": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957"
    }
  },
  "page": "reference/domain-types.md",
  "paragraphs": [
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6286,
      "readability_grade": 10.7893,
      "sentences": 2,
      "text": "matra::domain holds every type the library hands back. The module depends on serde, thiserror, and the standard library, and on nothing else, so a caller can depend on these types without inheriting a C++ toolchain.",
      "words": 37
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5294,
      "readability_grade": 10.6309,
      "sentences": 2,
      "text": "It is not a copy of the generated item-level documentation. For that, run cargo doc --no-deps --all-features.",
      "words": 20
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.7273,
      "readability_grade": 2.6459,
      "sentences": 2,
      "text": "Five types nest, each owning the next. Nothing is owned twice.",
      "words": 11
    },
    {
      "compression_ratio": 0.5268,
      "kind": "paragraph",
      "lexical_density": 0.5412,
      "readability_grade": 11.5775,
      "sentences": 4,
      "text": "The left column is what the pipeline writes into the struct. The right column is recomputed from the tree every time you ask, and an empty right column means the type only holds. So there is no second list of paragraphs anywhere in Document: the flat iterators walk the section tree. Every number a reader needs from a document is stored, in DocumentMetrics and ParagraphMetrics and in a sentence's word_count, tree_depth and passive, because a field crosses the language boundary and a method does not.",
      "words": 86
    },
    {
      "compression_ratio": 0.5199,
      "kind": "paragraph",
      "lexical_density": 0.575,
      "readability_grade": 16.935,
      "sentences": 4,
      "text": "The derived structural types (Negation, Modal, ClausalComplement, RootAdverbial, RootMood, HearstPair with its HearstSpan and HearstPattern) sit inside the nest, owned by Sentence, and the metric records DocumentMetrics and ParagraphMetrics are owned by Document and Paragraph. Provenance, with its Identity and ProvenanceSource, is owned by Document, and so is skipped, a list of Skipped records, each a SkippedKind and a Span. Every heading, paragraph, sentence and token holds a Span, and a paragraph its ParagraphKind. The other types sit outside it:",
      "words": 80
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.3846,
      "readability_grade": 7.6338,
      "sentences": 1,
      "text": "Where each of those enters and leaves the pipeline is drawn in Architecture.",
      "words": 13
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5455,
      "readability_grade": 5.8102,
      "sentences": 3,
      "text": "Block is the one input type. It has no serde derives and no public fields, so it needs no #[non_exhaustive]: a caller builds one with Block::heading or Block::paragraph and only matra reads it. Its JSON form is the blocks format, whose schema is spec/schema/v2/blocks.schema.json.",
      "words": 47
    },
    {
      "compression_ratio": 0.5029,
      "kind": "paragraph",
      "lexical_density": 0.6045,
      "readability_grade": 14.6754,
      "sentences": 4,
      "text": "No type in the module derives Hash, and the parse tree derives no equality: two Token, Sentence, Paragraph, Section or Document values cannot be compared with ==. Format derives PartialEq and Eq, which is what the reader table keys on, and the metric records, the provenance types, Span, Skipped, the kind enums, Block, RootMood, the Hearst types, the outcome types, Embedding, ProvisionNotice, ErrorDetails and DetailValue derive PartialEq. Error has no Clone, no equality and no serde derives; its Serialize is written by hand, as DocumentError's and ErrorDetails' are. ErrorDetails has no public fields, like Block, so it needs no #[non_exhaustive]. Embedding is the one public type with public fields and without #[non_exhaustive]: on a tuple struct the attribute would make the constructor private, and an Embedder implemented outside the crate has to build one.",
      "words": 138
    },
    {
      "compression_ratio": 0.6178,
      "kind": "paragraph",
      "lexical_density": 0.6316,
      "readability_grade": 10.2552,
      "sentences": 4,
      "text": "#[non_exhaustive] has two effects outside the crate. Struct literal syntax is unavailable, so you build values through the associated constructors listed below. The outcome types have no public constructor: matra produces them, and serde reads them back. Matches on Format, ParagraphKind, SkippedKind and Error need a catch-all arm, because variants can be added in a minor release.",
      "words": 59
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4839,
      "readability_grade": 7.584,
      "sentences": 2,
      "text": "Every field on these types is pub and readable. TokenBuilder and Block are the exceptions: their fields are private, and you reach them through the builder's setters and the block's constructors.",
      "words": 33
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.9231,
      "readability_grade": 9.4492,
      "sentences": 1,
      "text": "One parsed token carrying the full CoNLL-U annotation set plus one derived flag.",
      "words": 14
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6364,
      "readability_grade": 12.3,
      "sentences": 1,
      "text": "Provider-specific notes for the UDPipe adapter, the one adapter that ships:",
      "words": 12
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.5556,
      "readability_grade": null,
      "sentences": 1,
      "text": "is_punct is set when the UPOS tag is PUNCT.",
      "words": 9
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.5789,
      "readability_grade": 5.5045,
      "sentences": 2,
      "text": "deps is always the string _, the CoNLL-U empty marker. The underlying udpipe-rs binding does not surface column 9.",
      "words": 21
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.6053,
      "readability_grade": 12.0042,
      "sentences": 3,
      "text": "xpos and misc carry whatever the model emits, and can be _ or empty. features is converted from the column: Mood=Ind|Tense=Pres becomes two entries, a multi-valued feature (Case=Nom,Acc) keeps its value unsplit, and _ becomes an empty map.",
      "words": 45
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4783,
      "readability_grade": 12.8757,
      "sentences": 1,
      "text": "A token id or head that does not fit in usize is rejected at the adapter with Error::ParseFailed rather than being silently coerced.",
      "words": 24
    },
    {
      "compression_ratio": 0.5,
      "kind": "paragraph",
      "lexical_density": 0.5259,
      "readability_grade": 17.0883,
      "sentences": 3,
      "text": "span comes from aligning each form with the paragraph text the parser was given, then mapping it into the analysed text. A form the adapter cannot find where the text says it should be is Error::ParseFailed naming the token, never a guessed span, and the engine checks every parser's spans by the same rule, so a span from any parser that does not line up with its form is Error::ParseFailed with reason token_form_mismatch. In a markdown paragraph whose markup the reader removed, a token's span can cover markup inside it (foo**bar** is one token from the f to the last r), and a token read from an escape or an entity (&amp;) covers all of it.",
      "words": 124
    },
    {
      "compression_ratio": 0.5367,
      "kind": "paragraph",
      "lexical_density": 0.7143,
      "readability_grade": 10.3071,
      "sentences": 4,
      "text": "Token::builder(id, text, lemma, pos, head, dep) -> TokenBuilder takes the six CoNLL-U essentials. TokenBuilder carries one setter per optional field (xpos, deps, misc, each taking String; feats, taking the CoNLL-U column 6 string and converting it into features; is_punct taking bool; span taking a Span), each consuming and returning the builder. build() returns the Token. Unset string fields default to empty, features to an empty map, is_punct to false, and span to the empty span at 0.",
      "words": 82
    },
    {
      "compression_ratio": 0.6434,
      "kind": "paragraph",
      "lexical_density": 0.5532,
      "readability_grade": 9.6009,
      "sentences": 3,
      "text": "feat(\"Mood\") returns Some(\"Ind\") when the token carries Mood=Ind. The value is what the provider emitted, so a multi-valued feature (Case=Nom,Acc) comes back unsplit: matra does not normalise it. A token whose column 6 was _ or empty has no features, so every lookup on it returns None.",
      "words": 54
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5313,
      "readability_grade": 8.7188,
      "sentences": 2,
      "text": "feat is Rust-only by design. features crosses FFI as a map, so a lookup over it adds no information to the wire; a Python or TypeScript caller indexes the same map (RFC-0009).",
      "words": 34
    },
    {
      "compression_ratio": 0.5369,
      "kind": "paragraph",
      "lexical_density": 0.6022,
      "readability_grade": 13.629,
      "sentences": 3,
      "text": "Invariants that downstream code relies on: tokens are id-sorted; exactly one token has head == 0; every other head names a token in the same sentence; following head never loops. Every sentence the engine builds holds them, because it checks what the parser returns before it builds one and fails the document with Error::ParseFailed naming the check otherwise. A sentence built by hand with Sentence::new, which trusts its caller, or read back from JSON can break any of them, and the tree walks stay safe when the last one breaks (see tree_depth below).",
      "words": 97
    },
    {
      "compression_ratio": 0.5889,
      "kind": "paragraph",
      "lexical_density": 0.55,
      "readability_grade": 9.32,
      "sentences": 4,
      "text": "tree_depth is computed in time linear in token count with a visited set, and is None when the head references form a cycle, which reports a malformed parse as a value a reader must handle instead of truncating it or returning a number that reads as a very deep tree. An empty sentence has depth Some(0). word_count counts tokens the NLP provider identified, minus punctuation. It is not a whitespace split, and the two counts differ on the same text.",
      "words": 81
    },
    {
      "compression_ratio": 0.5627,
      "kind": "paragraph",
      "lexical_density": 0.56,
      "readability_grade": 11.86,
      "sentences": 2,
      "text": "The engine builds text by joining the token surface forms a parser returned, inserting a space unless the preceding token carries SpaceAfter=No in misc. The string is a reconstruction, not a slice of the input, so whitespace can differ from the source text; span is where the sentence came from.",
      "words": 51
    },
    {
      "compression_ratio": 0.5491,
      "kind": "paragraph",
      "lexical_density": 0.5732,
      "readability_grade": 14.7846,
      "sentences": 3,
      "text": "The fields after tokens are computed once from the dependency graph and serialized with the sentence, so every language binding and the CLI's JSON read one Rust detection as data (RFC-0008). Most are computed at Sentence::new; hearst_pairs is the exception, filled by the pipeline at the annotate stage because its detector lives in the hearst module outside the domain, so a hand-built Sentence carries an empty vector until the caller runs hearst::hypernymy_pairs. Each field reports structure only; the reading is the caller's.",
      "words": 87
    },
    {
      "compression_ratio": 0.5886,
      "kind": "paragraph",
      "lexical_density": 0.5806,
      "readability_grade": 11.3119,
      "sentences": 3,
      "text": "Negation reports one cue by token id: cue_id, cue_lemma (not, never, no, neither, nor), and head_id, the token the cue attaches to. A cue fires only on its carrying relation (advmod, det, cc), so nothing as a subject does not fire and tokenizer-split cannot fires exactly once, on the not token. Whether the negation reverses the sentence's claim is not matra's call.",
      "words": 66
    },
    {
      "compression_ratio": 0.4996,
      "kind": "paragraph",
      "lexical_density": 0.5556,
      "readability_grade": 12.1797,
      "sentences": 8,
      "text": "Modal reports one modal auxiliary by token id: aux_id, aux_lemma, and head_id, the token the auxiliary attaches to (0 when the modal is itself the root). The closed class is the ten lemmas the UD English treebank tags MD: can, could, may, might, must, ought, shall, should, will, would, matched by lemma on the aux relation or the AUX part of speech. The part-of-speech arm catches the shapes the model does not park on aux: VP ellipsis promotes the modal to root (\"Yes, you should.\") and coordinated auxiliaries parse as root plus aux or conj (\"He can and will succeed.\"). Precision holds because the model's only other AUX lemmas (be, have, do, get) are outside the class. Contracted forms fire on their lemma (wo in won't lemmatizes to will), the noun will never fires because it never carries aux or AUX, and a multi-auxiliary chain (might have been done) reports its modal with head_id on the content verb. English modals are ambiguous across readings, and resolving must into obligation or inference takes context matra does not model, so no epistemic, deontic, or dynamic category is assigned: the auxiliary and its arc are reported, the reading is the caller's.",
      "words": 203
    },
    {
      "compression_ratio": 0.5077,
      "kind": "paragraph",
      "lexical_density": 0.5695,
      "readability_grade": 8.1818,
      "sentences": 6,
      "text": "RootMood reports the grammatical mood of the root clause: root_id, the root's token id, and mood, the Mood feature as the model emitted it (Ind, Imp, and an open set beyond). The feature is read off the root token or, when the root carries none, off its first cop, aux, or aux:pass child that does, because the model parks finiteness on the auxiliary in copular (\"The sky is blue.\"), do-support (\"He did leave.\") and passive clauses. A modal carries VerbForm=Fin and no mood, so a clause whose root a modal governs (\"You must complete the form.\") usually has no root mood; whether a modal governs the root is the separate fact in modals. It reads the root clause only, so a mood in a subordinate clause does not reach it. The bare assertoric form, an indicative root no modal governs, is mood == \"Ind\" with no modal whose head_id is root_id.",
      "words": 156
    },
    {
      "compression_ratio": 0.4759,
      "kind": "paragraph",
      "lexical_density": 0.5729,
      "readability_grade": 8.4403,
      "sentences": 11,
      "text": "ClausalComplement reports one verb governing a clausal complement by token id: verb_id and verb_lemma for the verb, ccomp_id for the head of the complement, and subject_id plus subject_lemma for the verb's subject (nsubj, nsubj:pass) when the parse has one in the same sentence. The construction fires for every verb that fills it (\"Smith reported that the effect vanished.\", \"These results suggest that the mechanism is shared.\", \"We show that the method works.\" all match), because verbs taking a clausal complement are an open class and any list matra shipped would be incomplete while looking authoritative. Which verb lemmas count as reporting or evidential is the caller's lexicon, supplied as a parameter to clausal_complements_in or applied to the field directly in any language. A ccomp under a non-verb head (\"I am sure that it works.\" parses the complement under the adjective) is outside the construction by decision. The subject is optional by observed fact: UDPipe splits \"Smith et al. reported that ...\" into two sentences at the period in \"et al.\", stranding the attribution in the previous sentence while the verb keeps its ccomp. That upstream segmentation defect is recorded here and in the test suite rather than papered over.",
      "words": 202
    },
    {
      "compression_ratio": 0.4978,
      "kind": "paragraph",
      "lexical_density": 0.5882,
      "readability_grade": 11.9792,
      "sentences": 4,
      "text": "RootAdverbial reports one adverbial modifier attached to the root by token id: adv_id and adv_lemma. The root-attached advmod arc is where sentence-scope adverbs land (\"Reportedly, the deal closed.\"), but the dependency graph does not distinguish sentence scope from manner (\"The team quickly shipped.\" puts quickly on the same arc), so every root-attached adverbial is reported and the caller's lexicon selects the evidential ones, via root_adverbials_in or a filter over the field. Negation particles on the same arc (\"not\" in \"The plan is not ready.\") appear in both negations and root_adverbials; both report arcs, neither assigns a reading, and the overlap is intentional.",
      "words": 107
    },
    {
      "compression_ratio": 0.462,
      "kind": "paragraph",
      "lexical_density": 0.6127,
      "readability_grade": 18.2622,
      "sentences": 6,
      "text": "HearstPair reports one candidate hypernymy pair from the six lexico-syntactic patterns of Hearst (1992), detected as dependency-arc patterns rather than regex over surface text: NP such as NP, such NP as NP, NP, including NP, NP, especially NP, NP and other NP, NP or other NP. The pattern field carries a HearstPattern tag naming the construction that matched (such_as, such_np_as, including, especially, and_other, or_other on the wire), and hypernym and hyponym each carry a HearstSpan: head_id and head_lemma for the noun phrase head, plus first_id and last_id for the contiguous token range covering the head and its adjacent nominal modifiers, with the pattern's own marker words (such, other) excluded, so traceability holds against tokens. Precision is favoured over recall: each detector requires the full verified arc shape, so a clausal coordination (\"She invited John, and other guests brought gifts.\"), a definite contrastive coordination (\"The teacher and the other students laughed.\"), or an emphasized plain coordination (\"France and especially Spain\") does not fire, and a sentence the model misparses is missed rather than misreported. The pair is a candidate by design: matra reports that the sentence used a construction which conventionally signals hypernymy, and does not build a taxonomy or assert the relation is true.",
      "words": 208
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5714,
      "readability_grade": 8.7664,
      "sentences": 2,
      "text": "Most methods below are walks over the head and dep columns; the two _in filters are views over the derived fields against a caller-supplied lexicon. Concepts draws one parsed sentence as the tree these walk.",
      "words": 36
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6364,
      "readability_grade": 9.0818,
      "sentences": 1,
      "text": "subtree carries a visited set, so it terminates on cyclic input.",
      "words": 11
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.75,
      "readability_grade": null,
      "sentences": 1,
      "text": "ParagraphMetrics holds the three:",
      "words": 4
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5882,
      "readability_grade": 16.4112,
      "sentences": 2,
      "text": "Each slot holds None until Engine::compose fills it, and stays None when the paragraph does not meet the metric's applicability condition; no outer Option wraps the record, because each slot gates on its own.",
      "words": 36
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6,
      "readability_grade": 13.512,
      "sentences": 1,
      "text": "Engine::annotate skips quotes when it parses, so a paragraph whose kind is ParagraphKind::Quote reaches compose with no sentences and keeps all three slots at None.",
      "words": 27
    },
    {
      "compression_ratio": 0.4816,
      "kind": "paragraph",
      "lexical_density": 0.5294,
      "readability_grade": 7.5253,
      "sentences": 6,
      "text": "The markdown reader sets level to the heading's level (the number of # characters, or 1 and 2 for a setext heading) and heading to its text. Content before the first heading becomes a section with heading: None and level: 0. Plain text produces at most one section, always with heading: None and level: 0. Block input opens a section at each Block::heading, at the level it names.",
      "words": 70
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6111,
      "readability_grade": 12.1756,
      "sentences": 3,
      "text": "Section::new(heading: Option<String>, level: usize, span: Option<Span>, paragraphs: Vec<Paragraph>) -> Section is the constructor. There are no other methods.",
      "words": 25
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4,
      "readability_grade": null,
      "sentences": 1,
      "text": "The output of the pipeline.",
      "words": 5
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6,
      "readability_grade": 9.9267,
      "sentences": 1,
      "text": "DocumentMetrics holds what the command line's table prints and what a binding would otherwise rebuild:",
      "words": 16
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5455,
      "readability_grade": 13.1036,
      "sentences": 2,
      "text": "Before the document is measured every count is 0 and every ratio None; None after it means the document did not meet the metric's condition, never a computed zero. Engine::analyze always measures, so this holds for a document you build by hand with Document::new.",
      "words": 47
    },
    {
      "compression_ratio": 0.492,
      "kind": "paragraph",
      "lexical_density": 0.5528,
      "readability_grade": 11.1519,
      "sentences": 7,
      "text": "Provenance holds matra, the version that produced the document; schema, the schema version as \"major.minor\" (SCHEMA_VERSION, \"2.0\"), which appears nowhere else; parser, the parser's Identity; format, the Format the input was read as; and source, a ProvenanceSource with the input's path (None for anything but a file), and the length in bytes and the sha256 of the analysed text every span points into, computed once at annotation. For markdown and plain text that is the input; for block input, the blocks joined. On a document, method and parameters are None and absent from the JSON; an outcome derived from the document carries a copy of its provenance with both set (see Summary). There is no timestamp, so two analyses of the same input by the same build are identical. Identity names a model-backed component: name (udpipe), model (english-ewt-ud-2.5-191206), sha256 of the model artifact, and license. The UDPipe adapter derives it from the bytes it loaded, so a model matra does not pin reports its digest with model and license set to None; an embedder reports one in the same shape (model2vec for Model2Vec), recorded in a cluster result's parameters.embedder. Provenance::new(parser, format, source) fills matra and schema from the build.",
      "words": 205
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5789,
      "readability_grade": 12.3361,
      "sentences": 2,
      "text": "What summarize and summarize_with return. The selector is matra::SummaryMethod, at the crate root (TfIdf, the default, and TextRank, #[non_exhaustive]).",
      "words": 21
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.7143,
      "readability_grade": null,
      "sentences": 1,
      "text": "SummarySentence is one sentence a summary keeps:",
      "words": 7
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5135,
      "readability_grade": 9.8034,
      "sentences": 2,
      "text": "Both summarizers select the top n by score and then return them in ascending index, so the result reads in document order while score carries the ranking. A score compares only with another from the same method.",
      "words": 37
    },
    {
      "compression_ratio": 0.5726,
      "kind": "paragraph",
      "lexical_density": 0.5273,
      "readability_grade": 15.5168,
      "sentences": 3,
      "text": "Parameters records what shaped an outcome, each field present only when the method takes it: n (Option<usize>) for a summary and keyphrases, threshold (Option<f64>) and embedder (Option<Identity>, the identity the embedder reports, the same shape as the parser's) for clusters. In JSON an absent parameter is an absent key, so a summary's is {\"n\": 3}.",
      "words": 61
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6,
      "readability_grade": 13.295,
      "sentences": 3,
      "text": "What keyphrases and keyphrases_with return. The selector is matra::KeyphraseMethod (Rake, the default, and Yake, #[non_exhaustive]).",
      "words": 17
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.8,
      "readability_grade": null,
      "sentences": 1,
      "text": "Keyphrase is one ranked phrase:",
      "words": 5
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5952,
      "readability_grade": 14.2333,
      "sentences": 2,
      "text": "What clusters returns: provenance (the document's, with method cosine and parameters threshold and embedder), clusters (Vec<Cluster>, each members, sentence indices) and edges (Vec<SimilarityEdge>, each a, b and score: f64). Semantic clusters describes each field and what co-membership does and does not mean.",
      "words": 45
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6,
      "readability_grade": null,
      "sentences": 1,
      "text": "Which reader a document needs.",
      "words": 5
    },
    {
      "compression_ratio": 0.6035,
      "kind": "paragraph",
      "lexical_density": 0.4857,
      "readability_grade": 10.7043,
      "sentences": 3,
      "text": "Format::from_path(path: impl AsRef<Path>) -> Format is the extension column, and it is the only copy of it. Reading a file with Input::file applies it, and the command line reads it for the name given to --stdin-filename when --format names none, so a name means the same thing on both routes. When the bytes are read, a PDF or DOCX signature outranks both the extension and a format the caller names.",
      "words": 75
    },
    {
      "compression_ratio": 0.5142,
      "kind": "paragraph",
      "lexical_density": 0.5315,
      "readability_grade": 12.9473,
      "sentences": 5,
      "text": "Pdf, Docx and an Other have no reader that ships, so analyzing such a document returns Error::UnsupportedFormat with that format, unless the engine was given a reader for it with Engine::with_reader. Other names a format matra has no variant for, the one a caller's own reader reads: Format::named(\"html\"). Format::named gives the variant matra has for a name it knows, so Format::named(\"markdown\") is Markdown, and a name read back from JSON does the same, an unknown one reading as Other rather than failing. Input::text and Input::file_as take the format as an argument; Input::bytes takes one optionally; Input::file detects one. Format derives PartialEq and Eq, which is what the reader table keys on.",
      "words": 120
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.575,
      "readability_grade": 10.26,
      "sentences": 3,
      "text": "A heading or a paragraph a caller supplies, for a document matra has no reader for. Input only: built by Block::heading(level: u8, text: impl Into<String>) and Block::paragraph(text: impl Into<String>), handed to Input::blocks, and never returned. It has no public fields.",
      "words": 47
    },
    {
      "compression_ratio": 0.5597,
      "kind": "paragraph",
      "lexical_density": 0.5185,
      "readability_grade": 4.5274,
      "sentences": 3,
      "text": "The analysed text of block input is the blocks' texts joined by a blank line (\"\\n\\n\"), in order, and every span in the result points into it. A heading opens a section at its level; a level of 0 is refused with Error::InvalidInput when the input is analysed. The same blocks as JSON are the blocks format: an array of {\"kind\": \"heading\", \"level\": 1, \"text\": \"...\"} and {\"kind\": \"paragraph\", \"text\": \"...\"} objects, with an unknown kind or field refused by name.",
      "words": 81
    },
    {
      "compression_ratio": 0.5351,
      "kind": "paragraph",
      "lexical_density": 0.5507,
      "readability_grade": 7.2694,
      "sentences": 5,
      "text": "A stretch of the analysed text in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is the stretch. Bytes rather than characters: a caller in a language that indexes strings by character converts. The unit is in the field names because a byte span used to slice a character-indexed string, a Python str, is right on ASCII and wrong, with no error, after the first character outside it. Span::new(byte_start, byte_end) is the constructor.",
      "words": 76
    },
    {
      "compression_ratio": 0.6083,
      "kind": "paragraph",
      "lexical_density": 0.68,
      "readability_grade": 10.761,
      "sentences": 4,
      "text": "One construct the reader saw and did not analyse. SkippedKind names it: Table, Code, FrontMatter, Html, ThematicBreak, LinkDefinition (table, code, front_matter, html, thematic_break, link_definition on the wire), a set a reader treats as open. Skipping is specified behaviour beside a correct result, not an error. Skipped::new(kind, span) is the constructor.",
      "words": 53
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6061,
      "readability_grade": 16.5891,
      "sentences": 1,
      "text": "Together with the heading and paragraph spans, skipped accounts for the input: every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span.",
      "words": 34
    },
    {
      "compression_ratio": 0.596,
      "kind": "paragraph",
      "lexical_density": 0.7551,
      "readability_grade": 10.6202,
      "sentences": 4,
      "text": "One per-document failure: the Err item Engine::analyze_all yields in the failed input's place, so documents plus errors equals inputs consumed. Display prints path: error when a path exists and the bare error otherwise; source() returns the wrapped Error. DocumentError::new(path, error) is the constructor. Not Clone, because Error wraps std::io::Error.",
      "words": 54
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5278,
      "readability_grade": 5.62,
      "sentences": 2,
      "text": "It serializes as its error's form plus the path, {\"path\", \"kind\", \"message\", \"details\"}, with path null for input that is not a file and written lossily where it is not UTF-8. The JSON Schema is spec/schema/v2/error.schema.json.",
      "words": 38
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.56,
      "readability_grade": 14.456,
      "sentences": 1,
      "text": "matra collects nothing across documents: a stream is the collection, and how to combine documents is the caller's decision (a count sums; vocabulary_ttr does not).",
      "words": 26
    },
    {
      "compression_ratio": 0.5133,
      "kind": "paragraph",
      "lexical_density": 0.5368,
      "readability_grade": 10.468,
      "sentences": 6,
      "text": "domain::limits holds every bound matra holds its work to, each defined once and read where it applies. They are published so you can size work before a call instead of learning a bound from a refusal: split an input under MAX_INPUT_BYTES, or chunk a document's sentences under the cap of the method you select, each named for that method as SummaryMethod and KeyphraseMethod name it. They are constants, not settings: no configuration changes them. They are this release's values and may change in any 0.x release, so read them from domain::limits rather than copying the numbers. A cap that is exceeded returns InputTooLarge, whose limit detail is the constant and whose what detail names the gate; a paragraph outside a metric's bounds keeps that metric None. Each constant's rustdoc carries the cost model behind its value.",
      "words": 142
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4688,
      "readability_grade": 10.5338,
      "sentences": 1,
      "text": "Two caps with the same value stay two constants where their costs differ, so tuning one does not move the other: TF-IDF's 2,000 is not TextRank's, and YAKE's 200,000 is not RAKE's.",
      "words": 36
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5,
      "readability_grade": 6.7967,
      "sentences": 3,
      "text": "The Python surface serializes values with pythonize. Fields have a serde representation and cross with their names unchanged. Methods have none, so there is nothing for them to cross with.",
      "words": 30
    },
    {
      "compression_ratio": 0.5465,
      "kind": "paragraph",
      "lexical_density": 0.5316,
      "readability_grade": 15.1433,
      "sentences": 3,
      "text": "Token::feat, Document::sentences, Sentence::subtree, and every other method in the tables above are available to Rust callers only. The two methods on Error, kind and details, are different: their values are computed into the error's serde form and onto the Python exception, so they cross as fields. Nothing a reader needs is method-only: every aggregate is a field of DocumentMetrics or ParagraphMetrics, and a sentence carries its word_count, tree_depth and passive, so each crosses with the record that holds it.",
      "words": 84
    },
    {
      "compression_ratio": 0.5015,
      "kind": "paragraph",
      "lexical_density": 0.5882,
      "readability_grade": 9.3577,
      "sentences": 6,
      "text": "Document and DocumentError are the two items Python's Engine.analyze_all yields. A Document stays a Rust value behind a read-only handle, and every read of it crosses through serde like every shape above it, with one field replaced: its provenance.source.path is decoded with the platform's filesystem decoding (os.fsdecode on Unix), so a name that is not valid UTF-8 crosses exactly instead of as serde's lossy string. A DocumentError crosses through serde too, as {\"path\", \"kind\", \"message\", \"details\"}, with its path decoded the same way. kind is Error::kind on the domain type, pinned across bindings by spec/tests/corpus/items.json and exported to Python as matra.ERROR_KINDS; message is the Rust error's own Display text; details is Error::details. An Error raised rather than returned crosses as a Python exception carrying the same three, kind, str(exc) and details: a method is computed into a value before it crosses, as an aggregate is. A future binding takes the same serde form.",
      "words": 166
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5833,
      "readability_grade": 10.7233,
      "sentences": 3,
      "text": "The Python shapes are TypedDict declarations available at runtime. They are declared in matra.types and re-exported from the package root, so either import works:",
      "words": 24
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.84,
      "readability_grade": 8.165,
      "sentences": 2,
      "text": "Field names and nesting match the Rust types exactly. Option<f64> becomes float | None; Option<String> becomes str | None; Vec<T> becomes list[T]; usize becomes int.",
      "words": 29
    }
  ]
}
