{
  "generator": {
    "matra": "0.3.0",
    "udpipe_model": {
      "name": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957"
    }
  },
  "page": "guides/rust.md",
  "paragraphs": [
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4286,
      "readability_grade": null,
      "sentences": 1,
      "text": "You already have matra in your Cargo.toml.",
      "words": 7
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5625,
      "readability_grade": 9.655,
      "sentences": 2,
      "text": "Every snippet below assumes it runs inside a function returning matra::domain::Result<()>, which is why ? works.",
      "words": 18
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6667,
      "readability_grade": null,
      "sentences": 1,
      "text": "One line, no arguments, no model on disk yet:",
      "words": 9
    },
    {
      "compression_ratio": 0.5775,
      "kind": "paragraph",
      "lexical_density": 0.5345,
      "readability_grade": 12.3306,
      "sentences": 4,
      "text": "Engine::new locates the model directory from the environment, downloads the English UDPipe model into it if it is not already there, and wires the readers this build ships. The directory is MATRA_MODEL_DIR if you set it, otherwise the models subdirectory of $XDG_DATA_HOME/matra, which defaults to ~/.local/share/matra. It reads no configuration file. Programming model has the full resolution order.",
      "words": 61
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.45,
      "readability_grade": 9.32,
      "sentences": 1,
      "text": "When you want your configuration file read, or the resolved values but not all of them, hold the Config yourself:",
      "words": 20
    },
    {
      "compression_ratio": 0.5674,
      "kind": "paragraph",
      "lexical_density": 0.5088,
      "readability_grade": 10.8763,
      "sentences": 3,
      "text": "with_model_dir puts the directory on the Argument rung, which outranks the environment and the defaults, and cfg.sources() reports where every other value came from. Config is also what Model2Vec::from_config reads. Engine::from_config_with_notice takes a closure called once before each model download, with what is about to arrive and its licence, so a caller that renders can say so.",
      "words": 58
    },
    {
      "compression_ratio": 0.5179,
      "kind": "paragraph",
      "lexical_density": 0.5102,
      "readability_grade": 13.6964,
      "sentences": 4,
      "text": "The parser is UDPipe with one pinned English model, english-ewt-ud-2.5-191206, and the engine builds it for you, so there is no parser to construct unless you bring your own (below). Name a directory you own: provisioning trusts what it finds there, and a shared one such as /tmp is writable by anybody on the machine. The engine creates the directory if it is missing, downloads the model (about 16 MB) on first use, verifies it against a pinned SHA-256 hash, and loads from the cached file on every call after that. The verified bytes are written to a temporary subdirectory whose name is unique to the call (the process id, the clock in nanoseconds, and a counter) and moved into place with a single rename, so two processes pointed at the same directory cannot corrupt each other's file, even in containers where every process is pid 1.",
      "words": 151
    },
    {
      "compression_ratio": 0.5909,
      "kind": "paragraph",
      "lexical_density": 0.5167,
      "readability_grade": 13.45,
      "sentences": 3,
      "text": "The path is passed to std::fs::create_dir_all unchanged, and ~ in it is taken literally: neither Rust nor matra expands it, so with_model_dir(\"~/models\") creates a directory literally named ~ under your current working directory. The same holds for MATRA_MODEL_DIR and the other variables when the shell has not already expanded them. Pass an absolute path, or expand the home directory yourself.",
      "words": 62
    },
    {
      "compression_ratio": 0.5745,
      "kind": "paragraph",
      "lexical_density": 0.5065,
      "readability_grade": 7.4264,
      "sentences": 5,
      "text": "Construction is the expensive step: it loads and validates the model once. Build one engine and reuse it for every document. An Engine is Send but not Sync: it moves to another thread, but it is not shared between two, and a stream borrowed from it stays on the thread that holds it. That is a compile error, not a runtime surprise. To parallelize, build one engine on each worker thread, or serialize calls behind a mutex.",
      "words": 77
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6957,
      "readability_grade": 7.3646,
      "sentences": 2,
      "text": "matra's public Rust surface is one pipeline assembled from two values. Input says what one document is; Engine says what happens to it.",
      "words": 24
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5556,
      "readability_grade": 5.8454,
      "sentences": 2,
      "text": "Input has five constructors. None of them reads anything or fails: the read, the format and the size cap are settled when the engine reaches the input.",
      "words": 27
    },
    {
      "compression_ratio": 0.5489,
      "kind": "paragraph",
      "lexical_density": 0.5789,
      "readability_grade": 9.0755,
      "sentences": 5,
      "text": ".md and .markdown read as markdown; any other extension, or none, reads as plain text when the bytes are UTF-8. Format::Blocks names the same headings and paragraphs as a JSON array, for Input::text, Input::bytes or Input::file_as, since no extension implies it, with the schema in spec/schema/v2/blocks.schema.json. Inputs::dir(path) lists one level of a directory, up front, and is an iterator of Input::file; its Err is the listing failing. Input::file on a directory fails with Error::InvalidInput naming Inputs::dir.",
      "words": 85
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.7143,
      "readability_grade": null,
      "sentences": 1,
      "text": "Engine exposes the pipeline at two grains:",
      "words": 7
    },
    {
      "compression_ratio": 0.6024,
      "kind": "paragraph",
      "lexical_density": 0.5818,
      "readability_grade": 15.3023,
      "sentences": 2,
      "text": "The grains always agree: analyze_all is analyze mapped over the inputs, for every Input constructor. Inside, analyze reads and parses the document, then measures it; those two stages are the engine's own, and the library's test suite pins that analysis is exactly the one followed by the other, so the grains cannot drift apart silently.",
      "words": 58
    },
    {
      "compression_ratio": 0.562,
      "kind": "paragraph",
      "lexical_density": 0.5395,
      "readability_grade": 10.9174,
      "sentences": 4,
      "text": "The readers this build ships are markdown, plain text and block input. Format::Pdf and Format::Docx are recognised, by extension and by signature, and have no reader: analysing one returns Error::UnsupportedFormat with that format, and so do bytes that are not UTF-8 and carry no signature matra recognises (with format: None). The signature outranks the extension and the format you name, so a PDF renamed notes.txt is still refused as a PDF rather than read as text.",
      "words": 80
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5143,
      "readability_grade": 9.1036,
      "sentences": 2,
      "text": "Seven caps guard the pipeline and the functions over its output. Each one returns Error::InputTooLarge, and the what field names which gate fired so you can route on the label instead of guessing from context.",
      "words": 36
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5714,
      "readability_grade": 9.4407,
      "sentences": 2,
      "text": "Three more labels guard what matra reads that is not your text: \"parser_model_download\" and \"embedding_model_download\" cap a model download at 64 MiB, and \"config_file_read\" caps the config file at 64 KiB. Errors lists all ten.",
      "words": 35
    },
    {
      "compression_ratio": 0.5748,
      "kind": "paragraph",
      "lexical_density": 0.4769,
      "readability_grade": 9.38,
      "sentences": 3,
      "text": "A document from disk passes the gates in sequence: the metadata check when the file is read, then the byte and text checks before it is parsed. Block input is checked on its analysed text, the blocks joined, so blocks each under the cap that join to more than it are refused. A file at exactly the cap is accepted; one byte over is rejected.",
      "words": 65
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5429,
      "readability_grade": 8.0511,
      "sentences": 2,
      "text": "The outcome caps are counted over the parsed document, so a document under 8 MiB can still exceed the TF-IDF sentence cap. Check doc.metrics.total_sentences before calling if you need to decide between summarizing and chunking.",
      "words": 37
    },
    {
      "compression_ratio": 0.5706,
      "kind": "paragraph",
      "lexical_density": 0.5439,
      "readability_grade": 13.5355,
      "sentences": 3,
      "text": "The parse inside analyze is the expensive step: it runs the parser once per paragraph. The outcome functions are free functions over the Document it returns, and never parse anything themselves, so they need no engine and run in a build without UDPipe. Analyze once, and hand the same document to as many outcomes as you need:",
      "words": 57
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5588,
      "readability_grade": 8.74,
      "sentences": 2,
      "text": "Each outcome reads the same document; nothing re-parses. The sentences it reads are the ones the reader kept, so markdown headings, code, tables and quotes never reach a summary as if they were prose.",
      "words": 34
    },
    {
      "compression_ratio": 0.5705,
      "kind": "paragraph",
      "lexical_density": 0.6522,
      "readability_grade": 12.0013,
      "sentences": 3,
      "text": "The method is optional. summarize runs TF-IDF and keyphrases runs RAKE, the defaults config/default.toml ships; summarize_with and keyphrases_with take a SummaryMethod or KeyphraseMethod, both #[non_exhaustive]. Every result is a record carrying the document's provenance plus the method that ran (tfidf, textrank, rake, yake) and the parameters it ran with (n), because the method decides what a score means: TF-IDF and TextRank, or RAKE and YAKE, score on unrelated scales.",
      "words": 73
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5294,
      "readability_grade": 13.9818,
      "sentences": 1,
      "text": "Keyphrases with equal scores come back in the same order on every run, the order the phrases are first met, so a tie at the requested count is decided the same way each time.",
      "words": 34
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5385,
      "readability_grade": 11.8697,
      "sentences": 3,
      "text": "Every ratio in ParagraphMetrics and DocumentMetrics is an Option, and each metric has its own threshold. A None means the metric declined to run, not that it computed nothing. The counts in DocumentMetrics (total_words, total_sentences, paragraph_count) are plain integers.",
      "words": 39
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6471,
      "readability_grade": 12.9047,
      "sentences": 2,
      "text": "\"Words\" here means Paragraph::word_count(), the non-punctuation token count summed over the paragraph's attached sentences. A paragraph whose sentences vector is empty has a word count of zero and therefore no per-paragraph metrics at all.",
      "words": 36
    },
    {
      "compression_ratio": 0.584,
      "kind": "paragraph",
      "lexical_density": 0.5833,
      "readability_grade": 10.1067,
      "sentences": 3,
      "text": "Quotes are never parsed. A paragraph inside a block quote, at any depth, has kind ParagraphKind::Quote; the parser never sees it, so its sentences stays empty, its metric slots stay None, and it contributes nothing to total_words or passive_ratio, though it counts in paragraph_count. It stays in the section tree with its text, and its span covers the > markers.",
      "words": 62
    },
    {
      "compression_ratio": 0.5864,
      "kind": "paragraph",
      "lexical_density": 0.561,
      "readability_grade": 10.5367,
      "sentences": 3,
      "text": "A markdown document (by format argument or by .md extension) is read as CommonMark, with tables, strikethrough, task lists and front matter. Headings, paragraphs, list items and quotes become the section tree; a # with no space after it, as in #hashtag, is text, and ## References is a heading like any other. Each paragraph's text is what the parser reads, with the inline markup removed (emphasis, link and image syntax, the backticks of a code span) and escapes and entities decoded.",
      "words": 83
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.45,
      "readability_grade": 12.27,
      "sentences": 1,
      "text": "What it does not analyse it records in Document::skipped, each entry a SkippedKind and a span, never dropping it silently:",
      "words": 21
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.4286,
      "readability_grade": 5.8843,
      "sentences": 1,
      "text": "FrontMatter: YAML between --- lines, or TOML between +++ lines, at the very start",
      "words": 13
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.8333,
      "readability_grade": null,
      "sentences": 1,
      "text": "Code: fenced and indented code blocks",
      "words": 6
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.6667,
      "readability_grade": null,
      "sentences": 1,
      "text": "Table: a table, header and rows",
      "words": 6
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.75,
      "readability_grade": null,
      "sentences": 1,
      "text": "Html: an HTML block",
      "words": 4
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.1818,
      "readability_grade": null,
      "sentences": 1,
      "text": "ThematicBreak: ---, *** or ___ on a line of its own",
      "words": 8
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.8571,
      "readability_grade": null,
      "sentences": 1,
      "text": "LinkDefinition: a link reference definition, [label]: https://example.com",
      "words": 7
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6471,
      "readability_grade": 10.8224,
      "sentences": 2,
      "text": "Every byte of the input that is not whitespace lies in exactly one heading span, paragraph span or skipped span. The test suite checks that over every conformance fixture and thousands of generated documents.",
      "words": 34
    },
    {
      "compression_ratio": 0.5317,
      "kind": "paragraph",
      "lexical_density": 0.5543,
      "readability_grade": 7.251,
      "sentences": 4,
      "text": "Every heading (Section::span), paragraph, skipped construct, sentence and token carries a Span { byte_start, byte_end } into the analysed text, in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is what it came from. For markdown and plain text the analysed text is the input; for block input it is the blocks' texts joined by a blank line. The text fields are normalized (a sentence's text is rebuilt from its tokens, a paragraph's has its markup removed); the span is the authority on where a value came from. provenance.source records the analysed text's length and SHA-256.",
      "words": 100
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5814,
      "readability_grade": 20.1149,
      "sentences": 2,
      "text": "analyze and the outcome functions return domain::Result<T>, an alias for Result<T, domain::Error>. The stream wraps the same error with its path: analyze_all yields DocumentError, whose path field is Some for a file input and whose error field is the domain::Error to match on:",
      "words": 49
    },
    {
      "compression_ratio": 0.5833,
      "kind": "paragraph",
      "lexical_density": 0.5652,
      "readability_grade": 12.7046,
      "sentences": 3,
      "text": "Error::ParseFailed also carries panics that crossed the UDPipe FFI boundary. The adapter wraps the C-side call in catch_unwind and converts a panic into ParseFailed with the payload message, so a bug inside UDPipe surfaces as a matchable error instead of aborting your process. It is also what the parser returns when it cannot place a token in the text it was given, rather than reporting a span it guessed.",
      "words": 72
    },
    {
      "compression_ratio": 0.5187,
      "kind": "paragraph",
      "lexical_density": 0.5783,
      "readability_grade": 7.4838,
      "sentences": 5,
      "text": "To pass an error on to another program, use what every channel carries rather than the message. e.kind() is a stable string such as \"input_too_large\". e.details() holds the values to act on by name (every dynamic value in the message, and on invalid_input the reason naming the check that failed), so e.details().get(\"limit\") reads a cap without matching the variant. serde_json::to_value(&e) is {\"kind\", \"message\", \"details\"}, and a DocumentError serializes the same way with its path beside them. Errors lists the keys for every kind.",
      "words": 88
    },
    {
      "compression_ratio": 0.6,
      "kind": "paragraph",
      "lexical_density": 0.52,
      "readability_grade": 13.276,
      "sentences": 2,
      "text": "domain::Error is #[non_exhaustive], so a trailing _ => (or a named catch-all binding, as above) is required even once you have named every variant that exists today. A future variant becomes a compile requirement for code that matches without a wildcard, and a routed catch-all for code that has one.",
      "words": 54
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4634,
      "readability_grade": 2.3287,
      "sentences": 3,
      "text": "The domain structs carry #[non_exhaustive] for the same reason. From outside the crate you cannot write Token { .. } or Sentence { .. } as struct literals. Use the constructors, which matters most when you are building fixtures for tests:",
      "words": 37
    },
    {
      "compression_ratio": 0.5797,
      "kind": "paragraph",
      "lexical_density": 0.5833,
      "readability_grade": 9.7381,
      "sentences": 4,
      "text": "Every string argument is an owned String, not a &str, so a literal needs .to_string() on the way in. TokenBuilder takes the remaining CoNLL-U columns through xpos, feats, deps and misc, each also a String, plus is_punct for the derived flag and span for where the token lies in its text, and build() hands back the Token. Sentence::new derives the sentence's span from its first and last token. feats takes column 6 as CoNLL-U writes it and converts it into the token's features map:",
      "words": 90
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.7333,
      "readability_grade": 9.9267,
      "sentences": 1,
      "text": "Inputs::dir lists one level of a directory, and analyze_all streams it, one result per file:",
      "words": 16
    },
    {
      "compression_ratio": 0.5913,
      "kind": "paragraph",
      "lexical_density": 0.5395,
      "readability_grade": 10.2117,
      "sentences": 3,
      "text": "Inputs::dir is Err only when the listing itself fails, such as the directory not existing. Everything else is an Err item carrying its path: an unreadable file as Io, a file over the cap as InputTooLarge with what = \"file_read\", a non-UTF-8 file, a PDF or a DOCX (by signature or by name) as UnsupportedFormat, and a parser failure as ParseFailed. Documents plus errors always equals inputs consumed, and each document names its file in provenance.source.path.",
      "words": 78
    },
    {
      "compression_ratio": 0.5476,
      "kind": "paragraph",
      "lexical_density": 0.5443,
      "readability_grade": 13.3509,
      "sentences": 3,
      "text": "Two kinds of entry are skipped without an error entry, so they will not appear in either list: subdirectories, because the walk is one level deep, and symlinks, which are filtered out with symlink_metadata so a link cannot redirect the read elsewhere. Files are attempted in sorted path order. How to aggregate across documents is yours to decide: a count such as total_words sums, and a ratio such as vocabulary_ttr does not, since it falls as a document grows.",
      "words": 82
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4242,
      "readability_grade": 8.0086,
      "sentences": 2,
      "text": "You do not have to collect. The stream from analyze_all is lazy, so a loop that renders each document as it completes holds one document in memory at a time, not the corpus.",
      "words": 33
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.7407,
      "readability_grade": 16.3548,
      "sentences": 4,
      "text": "Document holds a sections: Vec<Section> tree; sections hold paragraphs: Vec<Paragraph>; paragraphs hold sentences: Vec<Sentence>. Flat iterators cut across the tree when you do not need the structure:",
      "words": 31
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5556,
      "readability_grade": null,
      "sentences": 1,
      "text": "doc.paragraphs() and doc.tokens() flatten at the other two levels.",
      "words": 9
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5814,
      "readability_grade": 18.4684,
      "sentences": 4,
      "text": "A Sentence carries its dependency tree in tokens: Vec<Token>, where each token's head field points at another token's id in the same sentence and head == 0 marks the root. Navigate it with the methods Sentence provides rather than walking tokens by hand:",
      "words": 46
    },
    {
      "compression_ratio": 0.5889,
      "kind": "paragraph",
      "lexical_density": 0.629,
      "readability_grade": 9.9797,
      "sentences": 2,
      "text": "Token fields carry the CoNLL-U columns under Rust names: text is the surface form, pos is the universal POS tag, xpos is the language-specific tag, and dep is the dependency relation. features is the FEATS column as a map from feature name to value (token.feat(\"Mood\") reads one), deps and misc hold their raw column strings, and is_punct is set by the parser.",
      "words": 65
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5682,
      "readability_grade": 20.8791,
      "sentences": 2,
      "text": "subtree(id) collects every token reachable from a given token, sorted back into document order, which is the tool for pulling a phrase out from under a dependency relation rather than a single token. Pulling the direct object out of every sentence that has one:",
      "words": 45
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6087,
      "readability_grade": 8.4348,
      "sentences": 3,
      "text": "matra reports the dependency label UDPipe assigned (\"obj\" here). It does not interpret what the phrase means. That interpretation is your application's job.",
      "words": 24
    },
    {
      "compression_ratio": 0.5737,
      "kind": "paragraph",
      "lexical_density": 0.6071,
      "readability_grade": 14.2943,
      "sentences": 2,
      "text": "The remaining navigation methods: head_of(id) returns the head token, or None for the root and for an id that is not in the sentence; content_tokens() gives you the non-punctuation view. Three counts are fields, computed when the sentence is built: word_count, the non-punctuation token count; passive, whether any token carries nsubj:pass, nsubjpass, or aux:pass; and tree_depth.",
      "words": 59
    },
    {
      "compression_ratio": 0.5788,
      "kind": "paragraph",
      "lexical_density": 0.5362,
      "readability_grade": 10.4814,
      "sentences": 3,
      "text": "tree_depth is the longest root-to-leaf path in the sentence, an Option<usize>. On a malformed parse where tokens form a cycle it is None rather than a depth truncated at some arbitrary ceiling, so a broken parse is a value the compiler makes you handle rather than a number that reads as a deep tree. subtree and children_of stay safe on the same malformed input; both use a visited set.",
      "words": 73
    },
    {
      "compression_ratio": 0.5805,
      "kind": "paragraph",
      "lexical_density": 0.5254,
      "readability_grade": 15.315,
      "sentences": 2,
      "text": "The engine is assembled from three parts, and each is a trait you can implement: the parser (matra::nlp::NlpProvider), a reader for one format (matra::decompose::Decomposer), and the source that reads the paths inputs name (matra::source::Source). Plug one in and the rest of the pipeline, every metric and every outcome function, runs on what it produces, with no fork of matra.",
      "words": 63
    },
    {
      "compression_ratio": 0.5928,
      "kind": "paragraph",
      "lexical_density": 0.6111,
      "readability_grade": 13.7326,
      "sentences": 3,
      "text": "A parser takes one paragraph's text and returns its sentences, each a Vec<Token>. It does not build Sentence values: the engine builds them from your tokens, text, span and every derived field included, so passive, negations and mood mean the same thing whichever parser ran. Engine::with_parser takes it, in any build, udpipe or not:",
      "words": 58
    },
    {
      "compression_ratio": 0.5199,
      "kind": "paragraph",
      "lexical_density": 0.5556,
      "readability_grade": 7.5548,
      "sentences": 5,
      "text": "The contract is on the trait. In short: ids run 1..=n in each sentence; every head is 0 or a token of the sentence, and exactly one is 0; the heads form no cycle; and every span is where the token's form lies in the paragraph text you were handed, in UTF-8 bytes relative to that text, in token order. Set is_punct for punctuation and SpaceAfter=No in misc where no space follows, since the sentence text is rebuilt from the forms. identity names your parser and its model, and lands in every document's provenance.parser. examples/custom_parser.rs is a runnable version that splits sentences too: cargo run --example custom_parser --no-default-features.",
      "words": 115
    },
    {
      "compression_ratio": 0.5122,
      "kind": "paragraph",
      "lexical_density": 0.5211,
      "readability_grade": 12.0879,
      "sentences": 4,
      "text": "A reader takes a format's bytes and returns a Read: its blocks, built with ReadBlock::heading(level, text, span) and ReadBlock::paragraph(kind, text, span), and the constructs it skipped. A paragraph's text is the analysed text at its span. For a format that is not text, extract the text and name it with Read::with_text, and every span points into that. Register a reader for a format with with_reader, which composes with the standard constructors:",
      "words": 79
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5,
      "readability_grade": 8.9027,
      "sentences": 3,
      "text": "A PDF or DOCX is detected by its bytes, so a registered reader for it runs on Input::file with no format named. A Format::Other is never detected: name it on the input, as above. Replacing a shipped reader, say Format::Markdown, works the same way.",
      "words": 47
    },
    {
      "compression_ratio": 0.5521,
      "kind": "paragraph",
      "lexical_density": 0.5385,
      "readability_grade": 8.2131,
      "sentences": 3,
      "text": "A source takes a path and returns the bytes there. with_source swaps it in for the local filesystem, so Input::file(\"bucket/notes.md\") reads through it, the extension still decides the format, and the path is still what provenance.source.path records. Inputs::dir still lists the local filesystem, and Input::bytes and Input::blocks need no source at all.",
      "words": 59
    },
    {
      "compression_ratio": 0.4882,
      "kind": "paragraph",
      "lexical_density": 0.5,
      "readability_grade": 10.8341,
      "sentences": 6,
      "text": "The engine trusts none of the three. What a reader returns is checked before the section tree is built, and what the parser returns is checked for each paragraph before a sentence is built from it. A break fails that document with Error::ParseFailed, whose reason names the check (head_out_of_range, token_span_order, block_span_invalid, and the rest Errors lists) and whose cause names the paragraph, sentence and token, or the block. That is also how to test an implementation: run your own texts through Engine::analyze, and it conforms when no document fails with one of those reasons. tests/providers.rs in the repository holds a broken parser and a broken reader for each check. A failure of your own is ParseFailed with reason None, or the kind the trait lists for it, and never a panic: a parser wrapping native code catches a panic at its own boundary, as the UDPipe adapter does.",
      "words": 150
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.375,
      "readability_grade": 1.5,
      "sentences": 2,
      "text": "These traits are Rust only in 0.3.0. Python takes an embedder of your own, and not yet a parser, a reader or a source.",
      "words": 24
    }
  ]
}
