{
  "generator": {
    "matra": "0.3.0",
    "udpipe_model": {
      "name": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957"
    }
  },
  "page": "explanation/programming-model.md",
  "paragraphs": [
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5,
      "readability_grade": null,
      "sentences": 2,
      "text": "Input is one document. Engine is what happens to it.",
      "words": 10
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.56,
      "readability_grade": 12.568,
      "sentences": 1,
      "text": "Source variation lives in the data, not in the function namespace, which is why Engine::analyze is one function rather than one per source and format.",
      "words": 26
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6,
      "readability_grade": null,
      "sentences": 1,
      "text": "Two ways to run it:",
      "words": 5
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4,
      "readability_grade": 8.14,
      "sentences": 1,
      "text": "And what to ask of the result, with free functions over the Document that need no engine and no parser:",
      "words": 20
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5625,
      "readability_grade": 15.6963,
      "sentences": 1,
      "text": "The method is optional and never in a function's name; every result records the method that ran and its parameters in its provenance, because which method ran decides what a score means.",
      "words": 33
    },
    {
      "compression_ratio": 0.6011,
      "kind": "paragraph",
      "lexical_density": 0.6452,
      "readability_grade": 10.741,
      "sentences": 3,
      "text": "Reads are lazy. An Input reads nothing until the engine reaches it, and Inputs::dir lists entries up front, a listing failure being its Err, but reads no file until the iterator is pulled. A per-file failure (unreadable, oversized, a symlink, bytes that are not text) arrives as an Err item carrying the path, so one bad file cannot abort a directory walk.",
      "words": 65
    },
    {
      "compression_ratio": 0.5874,
      "kind": "paragraph",
      "lexical_density": 0.5882,
      "readability_grade": 16.0641,
      "sentences": 2,
      "text": "Behind analyze, the input side is cut by decision, one crate-private module each: fetching the bytes (with the symlink and size guards), detecting the format (a PDF or DOCX signature, then the format you named, then the extension), reading one format into blocks with their spans, building the section tree from blocks, and parsing. A new format is a reader and a registration line; no published signature moves.",
      "words": 68
    },
    {
      "compression_ratio": 0.5935,
      "kind": "paragraph",
      "lexical_density": 0.5373,
      "readability_grade": 12.4931,
      "sentences": 4,
      "text": "Inside the engine, analysis is two stages: reading and parsing (annotate), then measuring (compose). Both are crate-private, since a public parse-only call would publish a second shape of Document with every metric empty, and the library's tests pin that analysis is exactly the one followed by the other. Measuring is total: it reads what is attached and skips what is not, so it has no failure path.",
      "words": 69
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6364,
      "readability_grade": 6.9364,
      "sentences": 1,
      "text": "One short Markdown document at each stage, as matra produces them:",
      "words": 11
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5217,
      "readability_grade": 5.3124,
      "sentences": 2,
      "text": "The engine needs to know where the UDPipe model lives. Config answers that question once so no caller has to invent a path.",
      "words": 23
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6296,
      "readability_grade": 16.7644,
      "sentences": 3,
      "text": "Engine::new() resolves the locations from the environment alone and reads no configuration file; Engine::from_config(&Config::resolve()?) reads the file as well. An explicit directory, Config::with_model_dir, wins over everything below.",
      "words": 32
    },
    {
      "compression_ratio": 0.5685,
      "kind": "paragraph",
      "lexical_density": 0.5854,
      "readability_grade": 8.6977,
      "sentences": 4,
      "text": "Config resolves locations and defaults, never behavior. It carries the model directory, the embedding model's directory name, and the default counts, methods and cluster threshold the command line uses, which are all things a caller could pass as arguments instead. It does not carry which metrics run or how output is shaped. A key 0.3.0 renamed or removed ([summarize], algorithm, [semantic], models.udpipe) is refused with Error::InvalidInput naming what replaced it, so a config file from 0.2 fails to load rather than half-load.",
      "words": 85
    },
    {
      "compression_ratio": 0.5335,
      "kind": "paragraph",
      "lexical_density": 0.5588,
      "readability_grade": 11.5165,
      "sentences": 3,
      "text": "Resolution order, per key: an explicit argument, then the environment, then the config file, then the defaults compiled into the crate from config/default.toml. The argument rung has one producer, Config::with_model_dir, which returns the same configuration with the model directory replaced and that one key's source changed. It is the rung a command line's --model-dir lands on, so the flag reaches the adapter through Config rather than past it.",
      "words": 73
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6,
      "readability_grade": null,
      "sentences": 1,
      "text": "Paths follow the XDG conventions on Linux and on macOS.",
      "words": 10
    },
    {
      "compression_ratio": 0.5526,
      "kind": "paragraph",
      "lexical_density": 0.5625,
      "readability_grade": 10.26,
      "sentences": 4,
      "text": "One exception to the model directory rule: if the data root has no models subdirectory yet, and ~/.matra/models exists and is not empty, that older location is used. matra never creates ~/.matra. When an existing, non-empty legacy cache is selected it is used as the model directory, downloads and re-downloads included, because the resolved directory is the one handed to the UDPipe adapter and that is where it writes. Create the new location, or set MATRA_MODEL_DIR, to move off it.",
      "words": 82
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.55,
      "readability_grade": 6.895,
      "sentences": 3,
      "text": "An empty ~/.matra/models is not selected. There is no cache in it to keep working, and picking it would send every later download into a directory matra would otherwise never create. A fresh install therefore lands in the new place.",
      "words": 42
    },
    {
      "compression_ratio": 0.5791,
      "kind": "paragraph",
      "lexical_density": 0.55,
      "readability_grade": 10.1067,
      "sentences": 3,
      "text": "A missing config file is not an error; the built-in defaults stand. A malformed one is Error::InvalidInput naming the file and the line, and an algorithm name the build does not know is rejected the same way at resolve time rather than at call time. Every resolved value records which rung it came from, readable through Config::sources as a ValueSource.",
      "words": 63
    },
    {
      "compression_ratio": 0.5551,
      "kind": "paragraph",
      "lexical_density": 0.5455,
      "readability_grade": 14.2205,
      "sentences": 3,
      "text": "The engine's reading stage, annotate, runs the size check on every Input, then hands each paragraph that is not a quote to the parser (NlpProvider::parse, a crate-private trait), and maps the token and sentence spans the parser reports against the paragraph's text into the analysed text. Nothing else in the library calls the parser. That is what makes the 8 MiB cap a property of the pipeline rather than of each entry point: no text over MAX_INPUT_BYTES can reach the parser, whichever call or constructor you started from.",
      "words": 91
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5357,
      "readability_grade": 6.5929,
      "sentences": 4,
      "text": "It is also why the outcomes are cheap. Analyze once, and hand the same document to every outcome function: none of them parses anything. Nothing is parsed twice.",
      "words": 28
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6087,
      "readability_grade": 7.8776,
      "sentences": 2,
      "text": "The pipeline depends on port traits, and adapters implement them. The domain depends on neither: the ports import the domain, never the reverse.",
      "words": 23
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6667,
      "readability_grade": 9.2508,
      "sentences": 4,
      "text": "Features are additive. Enabling udpipe adds UDPipe; disabling it removes UDPipe and nothing else. cargo check --no-default-features compiles. The other two features are cli (the command line, which implies udpipe) and python (the PyO3 bindings, which implies cli so that a wheel carries the command).",
      "words": 47
    },
    {
      "compression_ratio": 0.5061,
      "kind": "paragraph",
      "lexical_density": 0.5492,
      "readability_grade": 12.6899,
      "sentences": 5,
      "text": "Every port is open to your own adapter. clusters takes any &dyn Embedder, and with your own implementation it needs no feature flag at all. Engine::with_parser takes a parser of your own (NlpProvider), in any build; with_reader registers a reader (Decomposer) for a format, a Format::Other you name included; and with_source reads the paths inputs name from somewhere else (Source). The engine checks what a parser or a reader returns before it reaches a Document, so a part of yours cannot hand the outcomes a tree with a cycle or a span outside its text. Without implementing anything, bytes you fetch yourself go in through Input::bytes, and a format matra has no reader for goes in as headings and paragraphs through Input::blocks.",
      "words": 129
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5714,
      "readability_grade": 8.1771,
      "sentences": 4,
      "text": "Every gate returns a typed variant of Error. There is no Result<T, String> anywhere in the library, and library code does not panic: UDPipe panics at the C boundary are caught in nlp/udpipe.rs and converted.",
      "words": 37
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5,
      "readability_grade": 14.3744,
      "sentences": 1,
      "text": "The what discriminator names which gate fired, so a caller can route a document-too-big differently from a corpus-too-big.",
      "words": 22
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5909,
      "readability_grade": 13.3718,
      "sentences": 1,
      "text": "Cycle safety is a value, not a ceiling: Sentence::tree_depth is None on a malformed parse containing a cycle rather than silently truncated.",
      "words": 23
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4688,
      "readability_grade": 5.1638,
      "sentences": 3,
      "text": "Values reach Python through serde and pythonize. A field has a serde representation and crosses with its name unchanged. A method has none, so there is nothing for it to cross with.",
      "words": 32
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.8077,
      "readability_grade": 25.4115,
      "sentences": 1,
      "text": "Rust-only, therefore: the Document iterators (paragraphs, sentences, tokens); the Sentence tree walks and filters (root_token, head_of, children_of, subtree, content_tokens, clausal_complements_in, root_adverbials_in); Paragraph::word_count and sentence_count; and Token::feat.",
      "words": 29
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5111,
      "readability_grade": 12.065,
      "sentences": 2,
      "text": "When an aggregate needs to be visible cross-language it is materialized as a field. The document's counts, sentence-length statistics and ratios are the fields of DocumentMetrics, which the metric suite fills, and a sentence's word_count, tree_depth and passive are fields set when it is built.",
      "words": 49
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6098,
      "readability_grade": 10.5367,
      "sentences": 3,
      "text": "The same words Rust has, in Python's syntax: Engine, the read-only Document, the outcome functions, Model2Vec, and the entry point the matra console script launches. A Rust Input constructor is a Python keyword, and a Rust _with form is Python's method=.",
      "words": 45
    },
    {
      "compression_ratio": 0.5261,
      "kind": "paragraph",
      "lexical_density": 0.584,
      "readability_grade": 13.7952,
      "sentences": 5,
      "text": "A stream crosses item by item: analyze_all yields a Document or a DocumentError dict per input, so a failure on one file is one item rather than an exception that ends the walk. And clusters takes any object with embed and identity: the Embedder port was always the extension point, and the binding now reaches it. Model2Vec carries both methods too, so it satisfies the same protocol a caller's own object does rather than sitting beside it as a second accepted type. In the extension module that arm needs no embedding adapter compiled in. The matra package itself still requires an extension built with model2vec: its __init__ imports Model2Vec and raises ImportError naming the feature when it is absent, and every published wheel includes it.",
      "words": 127
    },
    {
      "compression_ratio": 0.5691,
      "kind": "paragraph",
      "lexical_density": 0.5833,
      "readability_grade": 11.7048,
      "sentences": 4,
      "text": "An outcome takes a document you already analyzed, in both languages, so a text is parsed once however many outcomes are asked of it. Python's outcome functions take only a Document, never a dict: a dict could carry provenance no matra run produced. Parsing is the one thing no caller substitutes, in either language, because the size cap and the panic boundary live behind it. The Python binding lives in src/python/ and uses only the public surface, which boundary rule 7 holds it to.",
      "words": 86
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6786,
      "readability_grade": 7.57,
      "sentences": 3,
      "text": "Engine is unsendable in Python, because the UDPipe model holds C-side state that is not thread safe. Multi-process Python is fine; sharing one engine across threads is not.",
      "words": 29
    }
  ]
}
