Every page of this documentation, in reading order.
matra
matra parses English text into sections, paragraphs, sentences and words, and returns measures, summaries and keyphrases over them, from Rust, Python or the command line.
Quick start
Install matra, save one sentence, and run matra on it.
Rust
1Install
$ cargo new quickstart && cd quickstart
$ cargo add matra@0.3 serde_json 2Save the text
$ echo 'The committee approved the budget on Tuesday.' > note.txt 3Run: save this as src/main.rs,
then run it
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("note.txt"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} $ cargo run Python
1Install
$ pip install 'matra>=0.3' 2Save the text
$ echo 'The committee approved the budget on Tuesday.' > note.txt 3Run: save this as quickstart.py,
then run it
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="note.txt") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) $ python quickstart.py CLI
1Install
$ pip install 'matra>=0.3' # or: cargo install matra --version '^0.3' --features cli 2Save the text
$ echo 'The committee approved the budget on Tuesday.' > note.txt 3Run
$ matra analyze --json note.txt It prints JSON; here is the first word:
[
{
"id": 1,
"span": {
"byte_start": 0,
"byte_end": 3
},
"text": "The",
"lemma": "the",
"pos": "DET",
"xpos": "DT",
"features": {
"Definite": "Def",
"PronType": "Art"
},
"head": 2,
"dep": "det",
"deps": "_",
"misc": "",
"is_punct": false
}
// … 7 more
] The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
Every word, drawn. Each arc runs from a word to the word that depends on it:
Dependency parse
1The committee approved the budget on Tuesday.
show the data
| # | Word | Lemma | POS | Head | Relation |
|---|---|---|---|---|---|
| 1 | The | the | DET | 2committee | det |
| 2 | committee | committee | NOUN | 3approved | nsubj |
| 3 | approved | approve | VERB | 0root | root |
| 4 | the | the | DET | 5budget | det |
| 5 | budget | budget | NOUN | 3approved | obj |
| 6 | on | on | ADP | 7Tuesday | case |
| 7 | Tuesday | Tuesday | PROPN | 5budget | nmod |
| 8 | . | . | PUNCT | 3approved | punct |
- Core argument
- Modifier
- Function word
- Other
Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.
Input: The committee approved the budget on Tuesday., matra contributors. MIT, as part of the matra repository.
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: quickstart/parse.json.
Where to go next
- Tutorial: run matra on two short paragraphs and find each part of what it returns. About ten minutes.
- How-to guides: one task each on real texts: summarize a document, extract keyphrases, find negations and modals, compare readability, cluster sentences by meaning.
- Explanation: why the output has the shape it has, with figures you can step through.
- Reference: every type and field, the Rust, Python and command-line surfaces, the methods behind each measure, and every error.
Your first analysis
You will run matra on two short paragraphs of meeting minutes, then find four things in what it returns: the document's structure, the grammar it read off each sentence, the measures of each paragraph, and the measures of the whole document. You need matra installed; Installation covers every route.
Save the text
The calls read it as minutes.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/first-analysis/minutes.txt The committee did not approve the budget on Tuesday. The chair said that a second vote may follow next week.
Three amendments were submitted by the working group. Each one narrows the scope of the plan.
Run matra
Pick the language you will use. Each tab installs matra and runs the same analysis.
Rust
1Install
$ cargo new first-analysis && cd first-analysis
$ cargo add matra@0.3 serde_json 2Run: save this as src/main.rs,
then run it
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("minutes.txt"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} $ cargo run Python
1Install
$ pip install 'matra>=0.3' 2Run: save this as first_analysis.py,
then run it
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="minutes.txt") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) $ python first_analysis.py CLI
1Install
$ pip install 'matra>=0.3' # or: cargo install matra --version '^0.3' --features cli 2Run
$ matra analyze --json minutes.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "analyze",
"input": "minutes.txt",
"result": /* the output below */,
"error": null
}Read the output
Every tab prints the same value. The words of each sentence are shown here as a count; the next section finds them.
Trimmed: tokens shown as a count, long strings shortened.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "plain_text",
"source": {
"path": "minutes.txt",
"bytes": 204,
"sha256": "b4f55f24335ecfae3b641caec0db70fb9566f76143bba54401a99869e29185d2"
}
},
"sections": [
{
"heading": null,
"level": 0,
"span": null,
"paragraphs": [
{
"span": {
"byte_start": 0,
"byte_end": 108
},
"text": "The committee did not approve the budget on Tuesday. The chair said tha…",
"kind": "paragraph",
"sentences": [
{
"span": {
"byte_start": 0,
"byte_end": 52
},
"text": "The committee did not approve the budget on Tuesday.",
"tokens": [ /* 10 items */ ],
"word_count": 9,
"tree_depth": 2,
"passive": false,
"negations": [
{
"cue_id": 4,
"cue_lemma": "not",
"head_id": 5
}
],
"modals": [],
"clausal_complements": [],
"root_adverbials": [
{
"adv_id": 4,
"adv_lemma": "not"
}
],
"hearst_pairs": [],
"root_mood": {
"root_id": 5,
"mood": "Ind"
}
},
{
"span": {
"byte_start": 53,
"byte_end": 108
},
"text": "The chair said that a second vote may follow next week.",
"tokens": [ /* 12 items */ ],
"word_count": 11,
"tree_depth": 3,
"passive": false,
"negations": [],
"modals": [
{
"aux_id": 8,
"aux_lemma": "may",
"head_id": 9
}
],
"clausal_complements": [
{
"verb_id": 3,
"verb_lemma": "say",
"ccomp_id": 9,
"subject_id": 2,
"subject_lemma": "chair"
}
],
"root_adverbials": [],
"hearst_pairs": [],
"root_mood": {
"root_id": 3,
"mood": "Ind"
}
}
],
"metrics": {
"readability_grade": 3.650000000000002,
"lexical_density": 0.55,
"compression_ratio": null
}
},
{
"span": {
"byte_start": 110,
"byte_end": 203
},
"text": "Three amendments were submitted by the working group. Each one narrows …",
"kind": "paragraph",
"sentences": [
{
"span": {
"byte_start": 110,
"byte_end": 163
},
"text": "Three amendments were submitted by the working group.",
"tokens": [ /* 9 items */ ],
"word_count": 8,
"tree_depth": 2,
"passive": true,
"negations": [],
"modals": [],
"clausal_complements": [],
"root_adverbials": [],
"hearst_pairs": [],
"root_mood": {
"root_id": 4,
"mood": "Ind"
}
},
{
"span": {
"byte_start": 164,
"byte_end": 203
},
"text": "Each one narrows the scope of the plan.",
"tokens": [ /* 9 items */ ],
"word_count": 8,
"tree_depth": 3,
"passive": false,
"negations": [],
"modals": [],
"clausal_complements": [],
"root_adverbials": [],
"hearst_pairs": [],
"root_mood": {
"root_id": 3,
"mood": "Ind"
}
}
],
"metrics": {
"readability_grade": 3.7550000000000026,
"lexical_density": 0.5625,
"compression_ratio": null
}
}
]
}
],
"metrics": {
"total_words": 36,
"total_sentences": 4,
"paragraph_count": 2,
"mean_sentence_length": 9,
"sentence_length_std": 1.4142135623730951,
"vocabulary_ttr": 0.8611111111111112,
"nominalization_ratio": 0,
"passive_ratio": 0.25
},
"skipped": []
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
Find the structure
The output is a tree. The document holds sections; this one has a single section, because the text has no headings. The section holds paragraphs, two of them, one for each block of text separated by a blank line. Each paragraph holds its sentences, and each sentence its tokens.
The tokens are shown above only as a count. Each one carries its text, its lemma, its part of speech (pos), the id of the word it depends on (head) and the relation between them (dep). The quick start shows one sentence's tokens in full.
Find the grammar
matra reads a few facts off each sentence's parse and stores them on the sentence.
- In the first sentence,
negationsholds one entry. Itscue_lemmaisnot, and itshead_idis 5:notnegates the fifth word,approve. - In the second sentence,
modalsholdsmay, over word 9,follow.clausal_complementsholdssay: the chair (subject_lemma) said something, and what was said is the clause headed by word 9. - The third sentence is passive: the amendments were submitted, and its
passiveistrue. Every sentence also carries itsword_count, itstree_depth, and itsroot_mood: the grammatical mood of its main clause,Indfor a plain statement.
Find the paragraph measures
Each paragraph carries three measures in its metrics. readability_grade is a grade level, and lexical_density is the share of content words. compression_ratio is null for both paragraphs: matra computes it only for a paragraph of more than 50 words, and these are shorter. A null means the measure declined to run, never that it came out as zero.
Find the document measures
The document's own metrics describe the whole of it: the counts (total_words, total_sentences, paragraph_count), the mean and spread of sentence length, and three ratios. passive_ratio is 0.25: one sentence of the four is passive. vocabulary_ttr is the share of distinct lemmas among all of them. nominalization_ratio is the share of words that are nouns ending in a suffix such as -tion or -ment. It is 0 here: the one candidate, amendments, ends in an s, and the test reads the word as written (Methodology has the rule).
Where to go next
- How-to guides: the same calls on longer, real texts, one task each.
- Concepts: why the output is a tree, and what each part of it records.
- Domain model: every field named above, with its type.
Install matra
Install matra as a Rust library, a command-line binary or a Python package, then check that it works. Each route is one command; take the one for the language you call matra from, or more than one. Platforms and models lists what each route needs and installs, and One core, three install routes explains why the commands look as they do.
To check a release file before you install it, follow Verifying a release in the security policy.
Requirements
For the Rust library and the CLI, and for the Python package on a platform with no wheel, install Rust 1.88 or later (check with rustc --version) and a C++ compiler. A C compiler on its own is not enough: without C++ the build stops with error occurred in cc-rs: failed to find tool "c++".
| Platform | Package |
|---|---|
| Debian, Ubuntu | apt install build-essential |
| Fedora, RHEL, Rocky, Alma | dnf install gcc-c++ |
| Alpine | apk add g++ |
| Arch | pacman -S base-devel |
| macOS | xcode-select --install |
For the Python package, use Python 3.12 or later (check with python3 --version, or python --version inside an activated virtual environment; stock macOS has only python3). Python wheels lists the platforms with a prebuilt wheel; on any other, and on a free-threaded interpreter, pip builds from source and needs the Rust toolchain and the C++ compiler above.
The Rust library
cargo add matra@0.3
The default features include udpipe, so matra::Engine::new() builds the pinned UDPipe parser without a feature flag. The UDPipe adapter itself is private to the crate: you reach it through Engine, and a parser of your own reaches it through Engine::with_parser.
The CLI binary
cargo install matra --version '^0.3' --features cli
--features cli is required: the binary is gated behind it. This compiles from source and places the binary in your cargo bin directory, which can take a minute or more. Confirm it landed:
matra --version
Expected output:
matra 0.3.0
features: udpipe cli
The second line names the features this build was compiled with, so the Python package's banner below is longer.
The Python package
pip install 'matra>=0.3' # or: uv add 'matra>=0.3'
This installs the library and the matra command together; the command is the same program as the CLI binary above, so uvx 'matra>=0.3' analyze essay.md and the installed binary do the same thing. Keep the version floor in the line: why the install lines name a version says what an unpinned one can give you.
On a platform with a wheel this installs in seconds. Its version banner reads:
matra 0.3.0
features: udpipe model2vec python cli
The verify step below doubles as the check: if import matra fails there, the package did not install correctly.
The English model
There is nothing to install. The library, the CLI and the Python package each download the English UDPipe model (about 16 MB) on first use, verify it against a pinned hash, and cache it in the model directory, by default ~/.local/share/matra/models.
To keep it somewhere else, set MATRA_MODEL_DIR, pass --model-dir to the CLI, or pass the directory through Config::with_model_dir in Rust. Python reads MATRA_MODEL_DIR. The model directory gives the full resolution order, and the model download what happens when a download fails its check.
Model licenses
matra's code is MIT, and the two models it downloads are separate works with licenses of their own. The English parsing model comes from LINDAT under CC BY-NC-SA 4.0, which does not permit commercial use; read it before you use matra commercially. The embedding model, potion-base-8M, comes from Hugging Face under MIT per its model card, and is used for semantic clusters only. Model licenses has both in a table, with links to the license texts.
The command line, the config file, Rust and Python always use this model. No surface loads a different UDPipe model file: the model is a pin.
Verify the install
Run this once. No arguments and no environment: it resolves the model directory, downloads and caches the model on this first run, then parses a sentence through it and prints two results:
import matra
engine = matra.Engine()
doc = engine.analyze("The committee approved the proposal without debate.")
print("sections:", len(doc["sections"]))
print("vocabulary_ttr:", doc["metrics"]["vocabulary_ttr"])
To name the directory, set MATRA_MODEL_DIR before the call. Use a real path: the value is used unchanged, so ~ is not expanded.
Expected output:
sections: 1
vocabulary_ttr: 0.8571428571428571
That first run fetches the model from a university server in Prague, and before it does, matra.Engine() writes two lines to standard error: what is downloading, its size and where to, then the model's licence. They are the lines the command line prints for the same download. Cold starts measured on a fast connection ranged from 3 to 35 seconds; on a slow network it takes longer, and the call is waiting on the network rather than working. Every run after that loads the cached file and touches no network, in about a second.
If matra.Engine() raises OSError (a matra.IoError), either the download never arrived or the model directory could not be written, and the message says which by naming the URL or the path, as details["url"] or details["path"] do. Check your network connection first; then run matra config show to see which directory matra resolved and check the permissions on it. If it raises RuntimeError (a matra.ModelInvalidError), bytes did arrive and then failed the pinned hash check; run the snippet again.
What you have
- matra installed as a Rust crate, a CLI binary, a Python package, or some combination, all from the same 0.3.x core.
- The English UDPipe model cached in the resolved model directory, verified against a pinned hash.
- A confirmed working call from
matra.Engine()throughanalyze()to a result.
Next: Your first analysis walks through what that result holds, or go to Rust, Python or CLI for the surface you are calling from.
Migrate from 0.2 to 0.3
0.3.0 changes Rust, Python, the command line, the JSON and the config file at once. Nothing keeps its old name. There are no deprecated aliases, no hidden flags, no fields written under both names and no setting that brings back the 0.2 output. A renamed or removed call fails at once in 0.3 rather than running with a different meaning: a compile error in Rust, an AttributeError or TypeError in Python, an unknown key in the config file, and a new format_version and schema major in the JSON. A few results do change for the same call, and the last section lists them.
This page lists every change a 0.2 caller meets, with the 0.2 form and the 0.3 form side by side. Work through the section for each surface you use.
Update the version
Move the version requirement first. The old requirement keeps you on 0.2, because cargo reads a change in the first non-zero component as breaking.
cargo add matra@0.3 # was matra@0.2
cargo install matra --version '^0.3' --features cli # was '^0.2'
pip install 'matra>=0.3' # was 'matra>=0.2'
uvx 'matra>=0.3' --skill # the skill for 0.3
Every rename at a glance
| 0.2 | 0.3 | Where |
|---|---|---|
Engine::new(nlp, decomposers), Engine::with_defaults() | Engine::new(), Engine::from_config(&Config::resolve()?) | Rust |
Ingest::text, Ingest::path | Input::text, Input::file, Inputs::dir | Rust |
Engine::analyze(Ingest), Engine::analyze_one | Engine::analyze(Input), Engine::analyze_all(inputs) | Rust |
Engine::annotate, Engine::compose | none: analyze is the call | Rust |
Ingested, RawDocument, CorpusEntry, Corpus, CorpusResult | none: Document, its path in provenance.source.path | Rust |
extraction::tfidf_summarize, textrank_summarize | summarize, summarize_with(.., SummaryMethod::TextRank) | Rust |
extraction::rake_keyphrases, yake_keyphrases | keyphrases, keyphrases_with(.., KeyphraseMethod::Yake) | Rust |
embed_and_cluster, extraction::semantic_clusters | clusters | Rust |
ScoredSentence, SemanticClusters, SemanticCluster, SemanticEdge | SummarySentence in a Summary, Clusters, Cluster, SimilarityEdge | Rust, Python types |
domain::MAX_INPUT_BYTES | domain::limits::MAX_INPUT_BYTES | Rust |
Error::InvalidInput(message) | Error::InvalidInput { reason, message } | Rust |
Error::ParseFailed(message) | Error::ParseFailed { reason, cause } | Rust |
Embedder::identity() -> &str | Embedder::identity() -> Identity | Rust |
Config::summarize_n, summarize_algorithm, keyphrases_algorithm | Config::summary_n, summary_method, keyphrases_method | Rust |
Config::udpipe_model, Config::semantic_threshold | none; Config::clusters_threshold is new | Rust |
Engine::new(Box::new(parser), decomposers) | Engine::with_parser(parser), then .with_reader(format, reader) per reader | Rust |
NlpProvider::parse -> Vec<Sentence> | NlpProvider::parse -> Vec<Vec<Token>>, one list of tokens per sentence | Rust |
Decomposer::decompose(&str) -> Vec<Section> | Decomposer::decompose(&[u8]) -> Result<Read>, built from ReadBlock::heading and ReadBlock::paragraph | Rust |
Source::read -> Vec<RawDocument>, Source::accepts | Source::read -> Vec<u8>; accepts is gone | Rust |
Udpipe, Decomposers, MarkdownDecomposer, PlainTextDecomposer, FileSource, DirectorySource, standard_decomposers, metrics, hearst, extraction | private to the crate | Rust |
Matra.english(), Matra.from_path(path) | matra.Engine() | Python |
Matra.analyze(text), analyze_markdown(text) | engine.analyze(text), engine.analyze(text, format="markdown") | Python |
Matra.analyze_path(path) | engine.analyze(file=path), engine.analyze_all(dir=path) | Python |
Matra.tfidf_summarize(text, n), textrank_summarize | matra.summarize(doc, n, method=...) | Python |
Matra.rake_keyphrases(text, max_phrases), yake_keyphrases | matra.keyphrases(doc, n, method=...) | Python |
Matra.semantic_clusters(text, threshold, model) | matra.clusters(doc, model, threshold) | Python |
matra.semantic_clusters(vectors, threshold, model_hash) | none: a cosine over the vectors you hold | Python |
an embedder's identity() -> str, Model2Vec.identity() | identity() returning an Identity dict, {name, model, sha256, license} | Python |
Model2Vec.potion_base_8m() reading the config file | Model2Vec.from_config() | Python |
Document, CorpusEntry, CorpusItem (types) | matra.Document (a class), DocumentDict, and Document or DocumentError items | Python |
paragraph["readability_grade"] and its siblings | paragraph["metrics"]["readability_grade"] | JSON, Python |
doc["vocabulary_ttr"] and its siblings | doc["metrics"]["vocabulary_ttr"] | JSON, Python |
Paragraph.in_blockquote | Paragraph.kind ("quote") | all |
Sentence.reportings, Reporting, reportings_in | Sentence.clausal_complements, ClausalComplement, clausal_complements_in | all |
Sentence.bare_assertion | Sentence.root_mood | all |
Token.feats (a string) | Token.features (a map) | all |
Sentence.tree_depth() returning usize::MAX on a cycle | Sentence.tree_depth, None on a cycle | all |
SemanticClusters.model_hash | Clusters.provenance.parameters.embedder.sha256 | all |
InputTooLarge what: file_source, config_file, udpipe_download, embedding_download | file_read, config_file_read, parser_model_download, embedding_model_download | all |
invalid_input details.reason, the message text | details.reason, the name of the check that failed | all |
model_invalid, parse_failed and io details.reason, the error's text | details.cause | all |
Format serialized as "Markdown", "PlainText" | "markdown", "plain_text", "blocks" | Rust serde, JSON |
[summarize], algorithm, [semantic] threshold, [models] udpipe | [summary], method, [clusters] threshold, none | config file |
--json envelope format_version: 1 | format_version: 2, with an error key | command line |
Rust
Build the engine
Engine::new takes no arguments and builds the pinned parser itself. It reads no configuration file. Engine::with_defaults is gone, because it read one under a name that did not say so.
// 0.2
let engine = Engine::with_defaults()?;
let engine = Engine::new(Box::new(Udpipe::english(dir)?), standard_decomposers());
// 0.3
let engine = Engine::new()?;
let engine = Engine::from_config(&Config::resolve()?)?; // reads the config file
let engine = Engine::from_config(&Config::resolve()?.with_model_dir(dir))?; // a model directory you name
There is no provider to construct and no reader table to pass for the standard pipeline. No surface loads a UDPipe model file of your choosing: the model is a pin.
A parser, reader or source of your own still plugs in, through the engine rather than a table:
// 0.2
let engine = Engine::new(Box::new(MyParser), standard_decomposers().with(Format::Markdown, Box::new(MyReader)));
// 0.3
let engine = Engine::with_parser(MyParser).with_reader(Format::Markdown, MyReader);
let engine = Engine::new()?.with_reader(Format::Pdf, MyPdfReader); // UDPipe, plus a reader of yours
let engine = Engine::new()?.with_source(MyStore); // paths read from somewhere else
The traits changed shape. NlpProvider::parse returns one Vec<Token> per sentence instead of Sentence values: the engine builds each sentence from the tokens, its text and every derived field included, so drop your Sentence::new call and return the tokens. Spans are required now, relative to the paragraph text parse was handed. Decomposer::decompose takes the format's bytes and returns a Result<Read> of ReadBlock values with their spans, where it returned sections; a text reader decodes the bytes itself. Source::read returns the document's bytes, and accepts is gone, since the engine never asked it. Whatever a parser or a reader returns is checked against its trait's contract, and a break is Error::ParseFailed naming the check (Errors); 0.2 passed it through unchecked. Bring your own parser or reader has the contracts.
Analyze one document
Input is one document. Engine::analyze returns its Document or the error. There is no stream to unwrap and no CorpusEntry around the document.
// 0.2
let analysis = engine
.analyze(Ingest::path("essay.md")?)
.next()
.unwrap()
.map_err(|e| e.error)?
.analysis;
let doc = engine
.analyze_one(RawDocument::new(text, None, Format::Markdown))
.map_err(|e| e.error)?
.analysis;
// 0.3
let doc = engine.analyze(Input::file("essay.md"))?;
let doc = engine.analyze(Input::text(text, Format::Markdown))?;
Input has five constructors: text, bytes, file, file_as (a file read as the format you name) and blocks (headings and paragraphs you supply for a format matra has no reader for). Input::file on a directory fails with InvalidInput, naming Inputs::dir.
Analyze a directory
Ingest::path on a directory is Inputs::dir, and analyze_all streams one result per file. A failed file is an Err item carrying its path. The path of a document that analyzed is doc.provenance.source.path.
// 0.2
for item in engine.analyze(Ingest::path("docs/")?) {
match item {
Ok(entry) => println!("{:?}: {}", entry.path, entry.analysis.total_words()),
Err(e) => eprintln!("{:?}: {}", e.path, e.error),
}
}
// 0.3
for item in engine.analyze_all(Inputs::dir("docs/")?) {
match item {
Ok(doc) => println!("{:?}: {}", doc.provenance.source.path, doc.metrics.total_words),
Err(e) => eprintln!("{e}"),
}
}
Corpus and CorpusResult are gone, with Corpus::total_words, passive_ratio and mean_readability. Collect the stream yourself. A count such as total_words sums across documents. A ratio such as vocabulary_ttr does not, because it falls as a document grows.
Read the measures
The aggregate methods are fields now, so they reach Python and the JSON too. The paragraph measures moved into Paragraph.metrics, and the document measures into Document.metrics.
// 0.2
doc.total_words();
doc.passive_ratio(); // 0.0 on a document with no sentences
doc.vocabulary_ttr;
paragraph.readability_grade;
sentence.word_count();
sentence.is_passive();
sentence.tree_depth(); // usize::MAX on a cycle
// 0.3
doc.metrics.total_words;
doc.metrics.passive_ratio; // None on a document with no sentences
doc.metrics.vocabulary_ttr;
paragraph.metrics.readability_grade;
sentence.word_count;
sentence.passive;
sentence.tree_depth; // None on a cycle
DocumentMetrics also carries total_sentences, paragraph_count, mean_sentence_length and sentence_length_std. In 0.2 these were methods, so they never reached Python or the JSON.
Read the record facts
The facts on a sentence are named for what the detector finds. ClausalComplement is a verb governing a clausal complement. Which of those verbs report speech is your lexicon, as before.
// 0.2
sentence.reportings;
sentence.reportings_in(&["say", "claim"]);
sentence.bare_assertion;
token.feats; // "Mood=Ind|Tense=Past"
// 0.3
sentence.clausal_complements;
sentence.clausal_complements_in(&["say", "claim"]);
sentence.root_mood; // Option<RootMood { root_id, mood }>
token.features; // BTreeMap: {"Mood": "Ind", "Tense": "Past"}
The 0.2 bare_assertion flag is root_mood.mood == "Ind" with no modal whose head_id is root_mood.root_id. Token::feat("Mood") reads one feature, as before. Paragraph.in_blockquote is Paragraph.kind: in_blockquote: true is ParagraphKind::Quote, and a list item is now ParagraphKind::ListItem.
Summaries, keyphrases and clusters
The outcomes are free functions at the crate root. They take the analysed &Document, not a slice of its sentences, and need no engine, so they run in a build without UDPipe. The method is optional and never in the name. Each returns a record whose provenance names the method that ran and its parameters.
// 0.2
let sentences: Vec<Sentence> = doc.sentences().cloned().collect();
let top = extraction::tfidf_summarize(&sentences, 3)?; // Vec<ScoredSentence>
let ranked = extraction::textrank_summarize(&sentences, 3)?;
let phrases = extraction::yake_keyphrases(&sentences, 10)?; // Vec<Keyphrase>
let groups = embed_and_cluster(&doc, &model, 0.85)?; // SemanticClusters
// 0.3
let top = matra::summarize(&doc, 3)?; // Summary
let ranked = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Yake)?; // Keyphrases
let groups = matra::clusters(&doc, &model, 0.85)?; // Clusters
What moved inside the results:
ScoredSentence { text, score, position }isSummarySentence { index, span, text, score }, inSummary.sentences.positionisindex.Keyphrasegainsoccurrences, every span the phrase was found in, and sits inKeyphrases.phrases.SemanticClusters { model_hash, threshold, clusters }isClusters { provenance, clusters, edges }. The embedder's identity isprovenance.parameters.embedder, anIdentitylike the parser's (the old hash is itssha256), and the thresholdprovenance.parameters.threshold. The edges moved out of each cluster into one list, ordered byaand thenb. EachClusterholds onlymembers.- The threshold and every score are
f64, notf32. Model2Vec::potion_base_8m(dir)takes the models directory, the oneMATRA_MODEL_DIRnames, and keeps the three files indir/potion-base-8M/. In 0.2dirwas the model's own directory. Pass the parent:potion_base_8m(cfg.model_dir())where 0.2 tookpotion_base_8m(cfg.model_dir().join("potion-base-8M")). Python'sModel2Vec.potion_base_8m(dir)changed the same way.
Errors
Error keeps its variants and kind(). Three shapes changed.
// 0.2
Error::UnsupportedFormat(format)
Error::InvalidInput(message)
Error::ParseFailed(message)
domain::MAX_INPUT_BYTES
// 0.3
Error::UnsupportedFormat { format, feature } // both Option
Error::InvalidInput { reason, message } // reason: "threshold_not_finite", ...
Error::ParseFailed { reason, cause } // reason: None, or "head_out_of_range", ...
domain::limits::MAX_INPUT_BYTES
InvalidInput's reason names the check that failed, a stable name the errors reference lists, and details["reason"] carries it, where 0.2 put the message text there. On model_invalid, parse_failed and io the error's own text moves from details["reason"] to details["cause"], so reason names a check wherever it appears. Four InputTooLarge labels name what you read rather than matra's modules: file_source is file_read, config_file is config_file_read, udpipe_download is parser_model_download and embedding_download is embedding_model_download. A match on the old strings no longer fires.
Error::details() is new. It returns every value in the message by name, so no caller parses a message. Error and DocumentError serialize as {kind, message, details}, and DocumentError adds path. A failed read now names its path: io error: cannot read essay.md: No such file or directory (os error 2). A file whose bytes are not UTF-8 is UnsupportedFormat, not Io. Every cap is in domain::limits, with the same values as 0.2. Errors lists every kind and its details.
What left the public surface
These modules and items are private to the crate in 0.3: the Udpipe adapter, metrics with Metric, default_suite, run_suite and readability::flesch_kincaid_grade, hearst, extraction, the readers MarkdownDecomposer and PlainTextDecomposer with the Decomposers table, FileSource and DirectorySource, and standard_decomposers. The port modules nlp, decompose and source stay public with their traits, reshaped as above. The measures stay public as fields of the Document. For another measure, write a function over &Document. Embedder stays public and keeps its shape. Its identity returns a domain::Identity, the shape the parser reports, where it returned &str: Identity::new("my-service", Some("v3".into()), None, None) in place of "my-service/v3". Return ModelInvalid, Io or InputTooLarge for your own failures, as the trait documents.
Configuration
Config reads the renamed sections, so its accessors follow them.
// 0.2
cfg.summarize_n();
cfg.summarize_algorithm();
cfg.keyphrases_algorithm();
cfg.semantic_threshold(); // f32
cfg.udpipe_model();
// 0.3
cfg.summary_n();
cfg.summary_method();
cfg.keyphrases_method();
cfg.clusters_threshold(); // f64
Config::from_file(path) is new: the configuration with a file you name, where a missing file is an error.
Python
Build the engine
Matra is Engine. A model directory is named by MATRA_MODEL_DIR, or by a config file you pass to Engine.from_config. Matra.from_path is gone: no surface loads a model file of your choosing.
# 0.2
from matra import Matra
m = Matra.english()
m = Matra.english("/opt/matra/models")
# 0.3
import matra
engine = matra.Engine() # MATRA_MODEL_DIR, else the default
engine = matra.Engine.from_config("matra.toml") # a config file you name
engine = matra.Engine.from_config() # the config file the environment names
Analyze
engine.analyze takes exactly one input. A str alone is text, never a path, and reads as plain text unless you name a format. It returns a matra.Document, not a dict.
# 0.2
doc = m.analyze(text)
doc = m.analyze_markdown(text)
items = m.analyze_path("essay.md") # a list of one
items = m.analyze_path("docs/") # a list, built in full
# 0.3
doc = engine.analyze(text)
doc = engine.analyze(text, format="markdown")
doc = engine.analyze(file="essay.md") # read as its extension and bytes say
for item in engine.analyze_all(dir="docs/"): # lazy, one item per file
...
analyze_path returned CorpusEntry dicts ({"path", "analysis"}) and DocumentError dicts ({"path", "error": {"kind", "message"}}). analyze_all yields a matra.Document or a DocumentError dict, {"path", "kind", "message", "details"}. Tell them apart with isinstance(item, matra.Document). The path of a document is item["provenance"]["source"]["path"].
# 0.2
for item in m.analyze_path("docs/"):
if "error" in item:
print(item["path"], item["error"]["kind"])
else:
print(item["path"], item["analysis"]["passive_ratio"])
# 0.3
for item in engine.analyze_all(dir="docs/"):
if isinstance(item, matra.Document):
print(item["provenance"]["source"]["path"], item["metrics"]["passive_ratio"])
else:
print(item["path"], item["kind"])
Read a document
A matra.Document reads like the dict it replaced: doc["metrics"]["total_words"], list(doc) and len(doc) work. It is read-only. Call doc.to_dict() where your code needs a dict, changes the result, or pickles it. The keys follow the schema 2.0 record, so the measures moved:
# 0.2
doc["passive_ratio"]
doc["sections"][0]["paragraphs"][0]["readability_grade"]
doc["sections"][0]["paragraphs"][0]["in_blockquote"]
# 0.3
doc["metrics"]["passive_ratio"]
doc["sections"][0]["paragraphs"][0]["metrics"]["readability_grade"]
doc["sections"][0]["paragraphs"][0]["kind"] == "quote"
Summaries, keyphrases and clusters
The four algorithm-named methods and semantic_clusters are module functions over a Document. A text is parsed once however many outcomes you ask of it, and in the format you analysed it in: 0.2's methods read every text as plain text, so a README's headings could land in its summary. Analyse a markdown file with file=, or a markdown string with format="markdown", and its headings stay headings. max_phrases is n. Each returns a dict whose provenance names the method and its parameters.
# 0.2
top = m.tfidf_summarize(text, 3) # [{"text", "score", "position"}]
ranked = m.textrank_summarize(text, 3)
phrases = m.yake_keyphrases(text, max_phrases=10) # [{"phrase", "score"}]
groups = m.semantic_clusters(text, 0.85, model)
# 0.3
doc = engine.analyze(text)
top = matra.summarize(doc, 3) # {"provenance", "sentences": [{"index", "span", "text", "score"}]}
ranked = matra.summarize(doc, 3, method="textrank")
phrases = matra.keyphrases(doc, 10, method="yake") # {"provenance", "phrases": [{"phrase", "score", "occurrences"}]}
groups = matra.clusters(doc, model, 0.85)
The vectors-in matra.semantic_clusters(vectors, threshold, model_hash) is gone, because Rust never had it. To compare vectors you already hold, compute the cosine yourself, as the semantic clusters guide shows.
Exceptions
Every exception is a matra.MatraError subclass with kind and details. Each also derives from the builtin class 0.2 raised, so except ValueError and except OSError catch what they caught.
# 0.2
try:
m.analyze(text)
except ValueError as e: # which ValueError: read the message
...
# 0.3
try:
engine.analyze(text)
except matra.InputTooLargeError as e: # also a ValueError
print(e.kind, e.details["limit"], e.details["actual"])
A missing file was a bare OSError in 0.2. It is now matra.IoNotFoundError, which is a FileNotFoundError and an OSError, so except FileNotFoundError now catches it. A permission failure is matra.IoPermissionError, a PermissionError.
Types
matra.types follows the record. ScoredSentence is SummarySentence inside Summary; SemanticClusters, SemanticCluster and SemanticEdge are Clusters, Cluster and SimilarityEdge; Reporting is ClausalComplement; CorpusEntry and CorpusItem are gone. The TypedDict that was named Document is DocumentDict, the shape to_dict() returns, and matra.Document is the class.
Command line
The config file
A config file from 0.2 does not load. Every command that reads one fails with invalid_input, naming the file and each key's replacement. Edit the file, or write a fresh one with matra config init --force.
# 0.2
[models]
udpipe = "english-ewt-ud-2.5-191206"
embedding = "potion-base-8M"
[semantic]
threshold = 0.85
[summarize]
n = 3
algorithm = "tfidf"
[keyphrases]
n = 10
algorithm = "rake"
# 0.3
[models]
embedding = "potion-base-8M"
[summary]
n = 3
method = "tfidf"
[keyphrases]
n = 10
method = "rake"
[clusters]
threshold = 0.85
models.udpipe goes without a replacement, because the parsing model is pinned in the source beside its digest.
The JSON envelope
format_version is 2, and the envelope gains a fifth key, error. All five are always present, and exactly one of result and error is null. A consumer pinned to format_version: 1 should fail, and then move.
{ "format_version": 1, "command": "analyze", "input": "essay.md", "result": {} }
{ "format_version": 2, "command": "analyze", "input": "essay.md", "result": {}, "error": null }
result changed too:
analyze: a schema 2.0Document. See The JSON record below.summarize: aSummaryrecord,{provenance, sentences}, not a bare list of{text, score, position}.keyphrases: aKeyphrasesrecord,{provenance, phrases}, not a bare list of{phrase, score}.
# 0.2
matra summarize essay.md --json | jq '.result[0].text'
matra keyphrases essay.md --json | jq '.result[].phrase'
# 0.3
matra summarize essay.md --json | jq '.result.sentences[0].text'
matra keyphrases essay.md --json | jq '.result.phrases[].phrase'
Failures under --json
In 0.2 a failure under --json wrote nothing to stdout. In 0.3 it writes the envelope there, with result null and error holding the kind, message and details. It still writes matra: <message> to stderr and still exits 2, so a script that reads only the exit code is unchanged. A usage error under --json is an envelope too.
# 0.3
matra analyze missing.md --json | jq -r '.error.kind' # io
The message for a missing file changed with it: io error: cannot read missing.md: ... where 0.2 printed no such file: missing.md. Branch on error.kind and read error.details, never the message.
Standard input
Standard input is read as bytes, and its format is detected as a file's is. --stdin-filename still names it, and its extension still picks the format. Bytes that are not UTF-8 fail as unsupported_format, as a file's do, where 0.2 failed with a read error that had no kind.
What is new
These do not break a 0.2 command, and are listed so you know they exist: matra analyze DIR, one result per file; matra clusters; --format on every document command; --depth on analyze --json; and matra --schema. The CLI guide has each.
The JSON record
The same record is the Rust Document, the Python matra.Document and analyze --json's result. It now names its own schema version in provenance.schema, "2.0". The schema is published: matra --schema document prints it.
| 0.2 | 0.3 |
|---|---|
| no provenance | provenance: the matra version, the schema, the parser and its model's SHA-256, the format, and the input's path, and the size and SHA-256 of the analysed text |
vocabulary_ttr, nominalization_ratio, passive_ratio | metrics.vocabulary_ttr, metrics.nominalization_ratio, metrics.passive_ratio |
| absent | metrics.total_words, total_sentences, paragraph_count, mean_sentence_length, sentence_length_std |
| absent | skipped: each construct the reader saw and did not analyse, with its kind and span |
sections[].heading, level, paragraphs | the same, and span (the heading's, null with no heading) |
paragraph.readability_grade, lexical_density, compression_ratio | paragraph.metrics.readability_grade, and so on |
paragraph.in_blockquote | paragraph.kind: paragraph, quote or list_item |
| absent | span on every paragraph, sentence and token, in UTF-8 bytes of the analysed text |
| absent | sentence.word_count, tree_depth (null on a cycle), passive |
sentence.reportings | sentence.clausal_complements |
sentence.bare_assertion | sentence.root_mood: {root_id, mood}, or null |
token.feats: one string of pipe-separated features | token.features: {"Mood": "Ind", "Tense": "Past"} |
Read the record as the schema asks: ignore a field you do not recognize, treat every string set you receive as open, and reject a major you do not know.
Results that change with no rename
A few results differ for the same input, and nothing about the call tells you so.
- Markdown is read by a CommonMark parser. 0.2 dropped tables without a trace, ended the document at a
## Referencesline, read#hashtagas a heading, and passed inline markup such as**and link syntax to the parser as words. In 0.3 tables, code, front matter and HTML areskippedentries, a## Referencesheading is a heading like any other, a heading needs a space after its#, list items are paragraphs of kindlist_item, and inline markup is removed from the text the parser reads. The counts and measures of a markdown document can change. - Plain text splits paragraphs at a line of only whitespace, so
\r\nendings and a line of spaces separate paragraphs as an empty line does. - Keyphrases with equal scores come in the same order on every run, by first occurrence, where 0.2 followed a hash map's order.
- TF-IDF scores are the same to the last bit on every run.
- A binary file is
unsupported_format, naming the format when it is a PDF or DOCX. In 0.2 it was an I/O error. - The size cap's message reads
input too large, where 0.2 readinput input too large. Model2Vec.potion_base_8m(dir)looks indir/potion-base-8M/, where 0.2 looked indiritself. A 0.2 call that passed the model's own directory still runs, and downloads a second copy one level down. Pass the models directory instead.Model2Vec.potion_base_8m()with no argument reads no config file. It finds the model directory from the environment, asEngine()does, so amodel_dirormodels.embeddingset only in the config file no longer moves it.Model2Vec.from_config()reads the file.- Python's constructors announce a download.
Engine(),Engine.from_config()andModel2Vec.potion_base_8m()write two lines tosys.stderrbefore they download a model, the ones the command line prints. Once the model is on disk they write nothing. - An empty result says so.
matra summarize,keyphrasesandclustersstill exit 1 when they find nothing, and now also write one line to stderr saying so, unless--jsonor--quietis given.
How-to guides
Each guide does one task on one real text. Six show the input, the same call in Rust, Python and the command line, and what matra prints; the three calls print the same result, and CI runs all three and compares what they print with the page. The seventh has no command-line call, because the CLI has no command for it.
| Guide | Input | What it reads |
|---|---|---|
| Parse a sentence into its dependency tree | Jane Austen, one sentence | every word's lemma, part of speech, head and relation |
| Find negations and modals | the Constitution, Article I (opening) | the negations and modal verbs of each sentence |
| Compare readability across paragraphs | The Federalist No. 10 | the three measures of every paragraph |
| Summarize a long document | Darwin, 2,214 words | the three sentences TextRank ranks highest |
| Extract a document's keyphrases | the same Darwin | the ten phrases RAKE ranks highest |
| Read a Markdown document's structure | a short Markdown document | sections, headings, and the quote left unparsed |
| Cluster sentences by meaning | your own text | groups of sentences that say the same thing, Rust and Python only |
Before you start
Install matra the way you mean to use it: the library, the CLI, or the Python package (Install matra). The Rust calls also need serde_json to print, which is the first line of each one: cargo add matra@0.3 serde_json.
Each call reads its input from the directory it runs in, under the file name the page gives. Each page has the command that downloads it. The first call on a machine downloads the English model (about 16 MB), and every later call reads it from disk.
What is not here
Analyzing a directory as a corpus has no worked example here, because each example pins the output of one input. The CLI, Rust and Python guides cover it.
Parse a sentence into its dependency tree
The first sentence of Pride and Prejudice, parsed: every word with its lemma, its part of speech, the word it depends on, and the relation between them.
Input
The calls read it as austen-first-sentence.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/parse-a-sentence/austen-first-sentence.txt or download austen-first-sentence.txt.
It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("austen-first-sentence.txt"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="austen-first-sentence.txt") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) CLI
$ matra analyze --json austen-first-sentence.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "analyze",
"input": "austen-first-sentence.txt",
"result": /* the output below */,
"error": null
}Output
Trimmed: only result.sections[0].paragraphs[0].sentences[0].tokens, lists cut to the first 3.
[
{
"id": 1,
"span": {
"byte_start": 0,
"byte_end": 2
},
"text": "It",
"lemma": "it",
"pos": "PRON",
"xpos": "PRP",
"features": {
"Case": "Nom",
"Gender": "Neut",
"Number": "Sing",
"Person": "3",
"PronType": "Prs"
},
"head": 4,
"dep": "nsubj",
"deps": "_",
"misc": "",
"is_punct": false
},
{
"id": 2,
"span": {
"byte_start": 3,
"byte_end": 5
},
"text": "is",
"lemma": "be",
"pos": "AUX",
"xpos": "VBZ",
"features": {
"Mood": "Ind",
"Number": "Sing",
"Person": "3",
"Tense": "Pres",
"VerbForm": "Fin"
},
"head": 4,
"dep": "cop",
"deps": "_",
"misc": "",
"is_punct": false
},
{
"id": 3,
"span": {
"byte_start": 6,
"byte_end": 7
},
"text": "a",
"lemma": "a",
"pos": "DET",
"xpos": "DT",
"features": {
"Definite": "Ind",
"PronType": "Art"
},
"head": 4,
"dep": "det",
"deps": "_",
"misc": "",
"is_punct": false
}
// … 22 more
] The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
The same 25 tokens as a table, then as arcs drawn from each word's head:
Dependency parse
1It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.
show the data
| # | Word | Lemma | POS | Head | Relation |
|---|---|---|---|---|---|
| 1 | It | it | PRON | 4truth | nsubj |
| 2 | is | be | AUX | 4truth | cop |
| 3 | a | a | DET | 4truth | det |
| 4 | truth | truth | NOUN | 0root | root |
| 5 | universally | universally | ADV | 6acknowledged | advmod |
| 6 | acknowledged | acknowledge | VERB | 4truth | conj |
| 7 | , | , | PUNCT | 6acknowledged | punct |
| 8 | that | that | PRON | 21want | obj |
| 9 | a | a | DET | 11man | det |
| 10 | single | single | ADJ | 11man | amod |
| 11 | man | man | NOUN | 21want | nsubj |
| 12 | in | in | ADP | 13possession | case |
| 13 | possession | possession | NOUN | 11man | nmod |
| 14 | of | of | ADP | 17fortune | case |
| 15 | a | a | DET | 17fortune | det |
| 16 | good | good | ADJ | 17fortune | amod |
| 17 | fortune | fortune | NOUN | 13possession | nmod |
| 18 | must | must | AUX | 21want | aux |
| 19 | be | be | AUX | 21want | cop |
| 20 | in | in | ADP | 21want | case |
| 21 | want | want | NOUN | 6acknowledged | ccomp |
| 22 | of | of | ADP | 24wife | case |
| 23 | a | a | DET | 24wife | det |
| 24 | wife | wife | NOUN | 21want | nmod |
| 25 | . | . | PUNCT | 4truth | punct |
- Core argument
- Modifier
- Function word
- Other
Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.
Input: Pride and Prejudice, the first sentence, Jane Austen. Public domain (first published 1813).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: austen-first-sentence/parse.json.
What to notice
- A document is sections of paragraphs of sentences, even when it is one sentence, so the tokens are at
sections[0].paragraphs[0].sentences[0].tokens. In Rust,doc.sentences()walks every sentence in document order. - Each token carries its CoNLL-U columns under matra's names:
id(from 1 within the sentence),text,lemma,pos(the Universal Dependencies tag),head(theidof the word it depends on, 0 for the root) anddep(the relation). Heretruthis the root, andItis itsnsubj. lemmais the dictionary form:isbecomesbe, andacknowledgedbecomesacknowledge.- The parse is the model's, reported as the model returned it. The model reads
that(token 8) as a pronoun and the object ofwant, where a grammar would call it the conjunction that opens the clause. matra does not correct the parser; Methodology names the model and its limits. - The whole output also carries the structural primitives read off this tree: a modal (
must, governingwant) and a clausal complement (acknowledge, whose clause is headed bywant). The next example reads those across a document.
Find negations and modals
Which sentences of a legal text are negated, and which carry a modal verb, read off each sentence's dependency tree.
Input
The calls read it as constitution-article-one.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/negations-and-modals/constitution-article-one.txt or download constitution-article-one.txt.
The text, 3 paragraphs
We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.
All legislative Powers herein granted shall be vested in a Congress of the United States, which shall consist of a Senate and House of Representatives.
The House of Representatives shall be composed of Members chosen every second Year by the People of the several States, and the Electors in each State shall have the Qualifications requisite for Electors of the most numerous Branch of the State Legislature. No Person shall be a Representative who shall not have attained to the Age of twenty five Years, and been seven Years a Citizen of the United States, and who shall not, when elected, be an Inhabitant of that State in which he shall be chosen.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("constitution-article-one.txt"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="constitution-article-one.txt") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) CLI
$ matra analyze --json constitution-article-one.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "analyze",
"input": "constitution-article-one.txt",
"result": /* the output below */,
"error": null
}Output
Trimmed: only result.sections[0].paragraphs[2].sentences, tokens shown as a count, lists cut to the first 2, long strings shortened.
[
{
"span": {
"byte_start": 482,
"byte_end": 739
},
"text": "The House of Representatives shall be composed of Members chosen every …",
"tokens": [ /* 44 items */ ],
"word_count": 42,
"tree_depth": 6,
"passive": true,
"negations": [],
"modals": [
{
"aux_id": 5,
"aux_lemma": "shall",
"head_id": 7
},
{
"aux_id": 28,
"aux_lemma": "shall",
"head_id": 29
}
],
"clausal_complements": [],
"root_adverbials": [],
"hearst_pairs": [],
"root_mood": null
},
{
"span": {
"byte_start": 740,
"byte_end": 982
},
"text": "No Person shall be a Representative who shall not have attained to the …",
"tokens": [ /* 51 items */ ],
"word_count": 46,
"tree_depth": 7,
"passive": true,
"negations": [
{
"cue_id": 1,
"cue_lemma": "no",
"head_id": 2
},
{
"cue_id": 9,
"cue_lemma": "not",
"head_id": 11
}
// … 1 more
],
"modals": [
{
"aux_id": 3,
"aux_lemma": "shall",
"head_id": 6
},
{
"aux_id": 8,
"aux_lemma": "shall",
"head_id": 11
}
// … 2 more
],
"clausal_complements": [],
"root_adverbials": [],
"hearst_pairs": [],
"root_mood": null
}
] The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
Every primitive of every sentence, marked on the word it is read from and the word it attaches to:
Structural primitives
- 1
We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.
- 2
All legislative Powers herein granted shall be vested in a Congress of the United States, which shall consist of a Senate and House of Representatives.
- 3
The House of Representatives shall be composed of Members chosen every second Year by the People of the several States, and the Electors in each State shall have the Qualifications requisite for Electors of the most numerous Branch of the State Legislature.
- 4
No Person shall be a Representative who shall not have attained to the Age of twenty five Years, and been seven Years a Citizen of the United States, and who shall not, when elected, be an Inhabitant of that State in which he shall be chosen.
show the data
| # | Primitive | Read from | Attaches to | Field and detail |
|---|---|---|---|---|
| 1 | no primitive | |||
| 2 | modal | 6shall | 8vested | modals |
| 2 | modal | 18shall | 19consist | modals |
| 3 | modal | 5shall | 7composed | modals |
| 3 | modal | 28shall | 29have | modals |
| 4 | negation | 1No | 2Person | negations |
| 4 | negation | 9not | 11attained | negations |
| 4 | negation | 34not | 33shall | negations |
| 4 | modal | 3shall | 6Representative | modals |
| 4 | modal | 8shall | 11attained | modals |
| 4 | modal | 33shall | 41Inhabitant | modals |
| 4 | modal | 48shall | 50chosen | modals |
- negation
- modal
Each primitive is a field matra fills on Sentence from its dependency tree: the tinted
word is the one it is read from, the underlined word the one it attaches to. The root mood is read
off the whole root clause: the Mood feature on the root or its auxiliary. What any of
it means for a text is the reader's call; matra reports the construction.
Input: The Constitution of the United States: the Preamble, and Article I, Sections 1 and 2 (opening), The Constitutional Convention, 1787. Public domain (a founding document of the United States, 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: constitution-article-one/primitives.json.
What to notice
- Every sentence has a
negationslist and amodalslist, empty when there is none. A negation names its cue (cue_lemma, at tokencue_id) and the word it attaches to (head_id). A modal names its auxiliary (aux_lemma, ataux_id) and the word it governs (head_id). The ids are token ids within the sentence. - The second sentence of Section 2 has three negations,
noonPersonandnottwice, and four modals, eachshall. To the parser the Constitution'sshallis a modal like any other. - One
notattaches toshallrather than to a verb: in "who shall not, when elected, be an Inhabitant", the parser hung the negation on the auxiliary.head_idis what the tree says, and that is all it says. - To collect the negated sentences, keep those whose
negationsis not empty. What a negation means for your purpose is your code's decision; matra reports where it is.
Compare readability across paragraphs
The three paragraph measures of The Federalist No. 10, to see where the essay is harder going and where it eases.
Input
The calls read it as federalist-10.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/readability-by-paragraph/federalist-10.txt or download federalist-10.txt.
The text, 23 paragraphs
AMONG the numerous advantages promised by a well constructed Union, none deserves to be more accurately developed than its tendency to break and control the violence of faction. The friend of popular governments never finds himself so much alarmed for their character and fate, as when he contemplates their propensity to this dangerous vice. He will not fail, therefore, to set a due value on any plan which, without violating the principles to which he is attached, provides a proper cure for it. The instability, injustice, and confusion introduced into the public councils, have, in truth, been the mortal diseases under which popular governments have everywhere perished; as they continue to be the favorite and fruitful topics from which the adversaries to liberty derive their most specious declamations. The valuable improvements made by the American constitutions on the popular models, both ancient and modern, cannot certainly be too much admired; but it would be an unwarrantable partiality, to contend that they have as effectually obviated the danger on this side, as was wished and expected. Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority. However anxiously we may wish that these complaints had no foundation, the evidence, of known facts will not permit us to deny that they are in some degree true. It will be found, indeed, on a candid review of our situation, that some of the distresses under which we labor have been erroneously charged on the operation of our governments; but it will be found, at the same time, that other causes will not alone account for many of our heaviest misfortunes; and, particularly, for that prevailing and increasing distrust of public engagements, and alarm for private rights, which are echoed from one end of the continent to the other. These must be chiefly, if not wholly, effects of the unsteadiness and injustice with which a factious spirit has tainted our public administrations.
By a faction, I understand a number of citizens, whether amounting to a majority or a minority of the whole, who are united and actuated by some common impulse of passion, or of interest, adversed to the rights of other citizens, or to the permanent and aggregate interests of the community.
There are two methods of curing the mischiefs of faction: the one, by removing its causes; the other, by controlling its effects.
There are again two methods of removing the causes of faction: the one, by destroying the liberty which is essential to its existence; the other, by giving to every citizen the same opinions, the same passions, and the same interests.
It could never be more truly said than of the first remedy, that it was worse than the disease. Liberty is to faction what air is to fire, an aliment without which it instantly expires. But it could not be less folly to abolish liberty, which is essential to political life, because it nourishes faction, than it would be to wish the annihilation of air, which is essential to animal life, because it imparts to fire its destructive agency.
The second expedient is as impracticable as the first would be unwise. As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed. As long as the connection subsists between his reason and his self-love, his opinions and his passions will have a reciprocal influence on each other; and the former will be objects to which the latter will attach themselves. The diversity in the faculties of men, from which the rights of property originate, is not less an insuperable obstacle to a uniformity of interests. The protection of these faculties is the first object of government. From the protection of different and unequal faculties of acquiring property, the possession of different degrees and kinds of property immediately results; and from the influence of these on the sentiments and views of the respective proprietors, ensues a division of the society into different interests and parties.
The latent causes of faction are thus sown in the nature of man; and we see them everywhere brought into different degrees of activity, according to the different circumstances of civil society. A zeal for different opinions concerning religion, concerning government, and many other points, as well of speculation as of practice; an attachment to different leaders ambitiously contending for pre-eminence and power; or to persons of other descriptions whose fortunes have been interesting to the human passions, have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to co-operate for their common good. So strong is this propensity of mankind to fall into mutual animosities, that where no substantial occasion presents itself, the most frivolous and fanciful distinctions have been sufficient to kindle their unfriendly passions and excite their most violent conflicts. But the most common and durable source of factions has been the various and unequal distribution of property. Those who hold and those who are without property have ever formed distinct interests in society. Those who are creditors, and those who are debtors, fall under a like discrimination. A landed interest, a manufacturing interest, a mercantile interest, a moneyed interest, with many lesser interests, grow up of necessity in civilized nations, and divide them into different classes, actuated by different sentiments and views. The regulation of these various and interfering interests forms the principal task of modern legislation, and involves the spirit of party and faction in the necessary and ordinary operations of the government.
No man is allowed to be a judge in his own cause, because his interest would certainly bias his judgment, and, not improbably, corrupt his integrity. With equal, nay with greater reason, a body of men are unfit to be both judges and parties at the same time; yet what are many of the most important acts of legislation, but so many judicial determinations, not indeed concerning the rights of single persons, but concerning the rights of large bodies of citizens? And what are the different classes of legislators but advocates and parties to the causes which they determine? Is a law proposed concerning private debts? It is a question to which the creditors are parties on one side and the debtors on the other. Justice ought to hold the balance between them. Yet the parties are, and must be, themselves the judges; and the most numerous party, or, in other words, the most powerful faction must be expected to prevail. Shall domestic manufactures be encouraged, and in what degree, by restrictions on foreign manufactures? are questions which would be differently decided by the landed and the manufacturing classes, and probably by neither with a sole regard to justice and the public good. The apportionment of taxes on the various descriptions of property is an act which seems to require the most exact impartiality; yet there is, perhaps, no legislative act in which greater opportunity and temptation are given to a predominant party to trample on the rules of justice. Every shilling with which they overburden the inferior number, is a shilling saved to their own pockets.
It is in vain to say that enlightened statesmen will be able to adjust these clashing interests, and render them all subservient to the public good. Enlightened statesmen will not always be at the helm. Nor, in many cases, can such an adjustment be made at all without taking into view indirect and remote considerations, which will rarely prevail over the immediate interest which one party may find in disregarding the rights of another or the good of the whole.
The inference to which we are brought is, that the CAUSES of faction cannot be removed, and that relief is only to be sought in the means of controlling its EFFECTS.
If a faction consists of less than a majority, relief is supplied by the republican principle, which enables the majority to defeat its sinister views by regular vote. It may clog the administration, it may convulse the society; but it will be unable to execute and mask its violence under the forms of the Constitution. When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens. To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed. Let me add that it is the great desideratum by which this form of government can be rescued from the opprobrium under which it has so long labored, and be recommended to the esteem and adoption of mankind.
By what means is this object attainable? Evidently by one of two only. Either the existence of the same passion or interest in a majority at the same time must be prevented, or the majority, having such coexistent passion or interest, must be rendered, by their number and local situation, unable to concert and carry into effect schemes of oppression. If the impulse and the opportunity be suffered to coincide, we well know that neither moral nor religious motives can be relied on as an adequate control. They are not found to be such on the injustice and violence of individuals, and lose their efficacy in proportion to the number combined together, that is, in proportion as their efficacy becomes needful.
From this view of the subject it may be concluded that a pure democracy, by which I mean a society consisting of a small number of citizens, who assemble and administer the government in person, can admit of no cure for the mischiefs of faction. A common passion or interest will, in almost every case, be felt by a majority of the whole; a communication and concert result from the form of government itself; and there is nothing to check the inducements to sacrifice the weaker party or an obnoxious individual. Hence it is that such democracies have ever been spectacles of turbulence and contention; have ever been found incompatible with personal security or the rights of property; and have in general been as short in their lives as they have been violent in their deaths. Theoretic politicians, who have patronized this species of government, have erroneously supposed that by reducing mankind to a perfect equality in their political rights, they would, at the same time, be perfectly equalized and assimilated in their possessions, their opinions, and their passions.
A republic, by which I mean a government in which the scheme of representation takes place, opens a different prospect, and promises the cure for which we are seeking. Let us examine the points in which it varies from pure democracy, and we shall comprehend both the nature of the cure and the efficacy which it must derive from the Union.
The two great points of difference between a democracy and a republic are: first, the delegation of the government, in the latter, to a small number of citizens elected by the rest; secondly, the greater number of citizens, and greater sphere of country, over which the latter may be extended.
The effect of the first difference is, on the one hand, to refine and enlarge the public views, by passing them through the medium of a chosen body of citizens, whose wisdom may best discern the true interest of their country, and whose patriotism and love of justice will be least likely to sacrifice it to temporary or partial considerations. Under such a regulation, it may well happen that the public voice, pronounced by the representatives of the people, will be more consonant to the public good than if pronounced by the people themselves, convened for the purpose. On the other hand, the effect may be inverted. Men of factious tempers, of local prejudices, or of sinister designs, may, by intrigue, by corruption, or by other means, first obtain the suffrages, and then betray the interests, of the people. The question resulting is, whether small or extensive republics are more favorable to the election of proper guardians of the public weal; and it is clearly decided in favor of the latter by two obvious considerations:
In the first place, it is to be remarked that, however small the republic may be, the representatives must be raised to a certain number, in order to guard against the cabals of a few; and that, however large it may be, they must be limited to a certain number, in order to guard against the confusion of a multitude. Hence, the number of representatives in the two cases not being in proportion to that of the two constituents, and being proportionally greater in the small republic, it follows that, if the proportion of fit characters be not less in the large than in the small republic, the former will present a greater option, and consequently a greater probability of a fit choice.
In the next place, as each representative will be chosen by a greater number of citizens in the large than in the small republic, it will be more difficult for unworthy candidates to practice with success the vicious arts by which elections are too often carried; and the suffrages of the people being more free, will be more likely to centre in men who possess the most attractive merit and the most diffusive and established characters.
It must be confessed that in this, as in most other cases, there is a mean, on both sides of which inconveniences will be found to lie. By enlarging too much the number of electors, you render the representatives too little acquainted with all their local circumstances and lesser interests; as by reducing it too much, you render him unduly attached to these, and too little fit to comprehend and pursue great and national objects. The federal Constitution forms a happy combination in this respect; the great and aggregate interests being referred to the national, the local and particular to the State legislatures.
The other point of difference is, the greater number of citizens and extent of territory which may be brought within the compass of republican than of democratic government; and it is this circumstance principally which renders factious combinations less to be dreaded in the former than in the latter. The smaller the society, the fewer probably will be the distinct parties and interests composing it; the fewer the distinct parties and interests, the more frequently will a majority be found of the same party; and the smaller the number of individuals composing a majority, and the smaller the compass within which they are placed, the more easily will they concert and execute their plans of oppression. Extend the sphere, and you take in a greater variety of parties and interests; you make it less probable that a majority of the whole will have a common motive to invade the rights of other citizens; or if such a common motive exists, it will be more difficult for all who feel it to discover their own strength, and to act in unison with each other. Besides other impediments, it may be remarked that, where there is a consciousness of unjust or dishonorable purposes, communication is always checked by distrust in proportion to the number whose concurrence is necessary.
Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic,--is enjoyed by the Union over the States composing it. Does the advantage consist in the substitution of representatives whose enlightened views and virtuous sentiments render them superior to local prejudices and schemes of injustice? It will not be denied that the representation of the Union will be most likely to possess these requisite endowments. Does it consist in the greater security afforded by a greater variety of parties, against the event of any one party being able to outnumber and oppress the rest? In an equal degree does the increased variety of parties comprised within the Union, increase this security. Does it, in fine, consist in the greater obstacles opposed to the concert and accomplishment of the secret wishes of an unjust and interested majority? Here, again, the extent of the Union gives it the most palpable advantage.
The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States. A religious sect may degenerate into a political faction in a part of the Confederacy; but the variety of sects dispersed over the entire face of it must secure the national councils against any danger from that source. A rage for paper money, for an abolition of debts, for an equal division of property, or for any other improper or wicked project, will be less apt to pervade the whole body of the Union than a particular member of it; in the same proportion as such a malady is more likely to taint a particular county or district, than an entire State.
In the extent and proper structure of the Union, therefore, we behold a republican remedy for the diseases most incident to republican government. And according to the degree of pleasure and pride we feel in being republicans, ought to be our zeal in cherishing the spirit and supporting the character of Federalists.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("federalist-10.txt"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="federalist-10.txt") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) CLI
$ matra analyze --json federalist-10.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "analyze",
"input": "federalist-10.txt",
"result": /* the output below */,
"error": null
}Output
Trimmed: sentences shown as a count, lists cut to the first 3, long strings shortened.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "plain_text",
"source": {
"path": "federalist-10.txt",
"bytes": 17880,
"sha256": "e074108c9cdf3d26db71dd52b22ab9f1e1af47b94618e3707daee74673b79a00"
}
},
"sections": [
{
"heading": null,
"level": 0,
"span": null,
"paragraphs": [
{
"span": {
"byte_start": 0,
"byte_end": 2351
},
"text": "AMONG the numerous advantages promised by a well constructed Union, non…",
"kind": "paragraph",
"sentences": [ /* 9 items */ ],
"metrics": {
"readability_grade": 20.724682332463008,
"lexical_density": 0.4699738903394256,
"compression_ratio": 0.4564015312632922
}
},
{
"span": {
"byte_start": 2353,
"byte_end": 2644
},
"text": "By a faction, I understand a number of citizens, whether amounting to a…",
"kind": "paragraph",
"sentences": [ /* 1 item */ ],
"metrics": {
"readability_grade": 24.8921568627451,
"lexical_density": 0.43137254901960786,
"compression_ratio": 0.563573883161512
}
},
{
"span": {
"byte_start": 2646,
"byte_end": 2775
},
"text": "There are two methods of curing the mischiefs of faction: the one, by r…",
"kind": "paragraph",
"sentences": [ /* 1 item */ ],
"metrics": {
"readability_grade": 10.690000000000001,
"lexical_density": 0.45454545454545453,
"compression_ratio": null
}
}
// … 20 more
]
}
],
"metrics": {
"total_words": 3000,
"total_sentences": 88,
"paragraph_count": 23,
"mean_sentence_length": 34.09090909090909,
"sentence_length_std": 17.36930597256701,
"vocabulary_ttr": 0.25233333333333335,
"nominalization_ratio": 0.042333333333333334,
"passive_ratio": 0.42045454545454547
},
"skipped": []
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
Each measure over the essay's 23 paragraphs, one panel each:
Paragraph measures across one document
- Mean readability
- 18.20
- Vocabulary TTR
- 0.25
- Nominalization ratio
- 0.04
- Passive ratio
- 0.42
show the data
| # | Opens with | Words | Grade | Density | Compression |
|---|---|---|---|---|---|
| 1 | AMONG the numerous advantages promised by a well… | 384 | 20.72 | 0.47 | 0.46 |
| 2 | By a faction, I understand a number of… | 51 | 24.89 | 0.43 | 0.56 |
| 3 | There are two methods of curing the mischiefs… | 22 | 10.69 | 0.45 | none |
| 4 | There are again two methods of removing the… | 40 | 18.89 | 0.40 | none |
| 5 | It could never be more truly said than… | 79 | 12.90 | 0.46 | 0.52 |
| 6 | The second expedient is as impracticable as the… | 157 | 14.59 | 0.44 | 0.45 |
| 7 | The latent causes of faction are thus sown… | 263 | 18.68 | 0.53 | 0.42 |
| 8 | No man is allowed to be a judge… | 267 | 12.92 | 0.47 | 0.46 |
| 9 | It is in vain to say that enlightened… | 80 | 13.54 | 0.47 | 0.55 |
| 10 | The inference to which we are brought is,… | 32 | 12.87 | 0.35 | none |
| 11 | If a faction consists of less than a… | 172 | 16.10 | 0.43 | 0.45 |
| 12 | By what means is this object attainable? Evidently… | 121 | 13.74 | 0.47 | 0.50 |
| 13 | From this view of the subject it may… | 179 | 21.57 | 0.45 | 0.48 |
| 14 | A republic, by which I mean a government… | 61 | 13.91 | 0.44 | 0.58 |
| 15 | The two great points of difference between a… | 50 | 23.03 | 0.48 | none |
| 16 | The effect of the first difference is, on… | 175 | 20.22 | 0.49 | 0.48 |
| 17 | In the first place, it is to be… | 123 | 26.53 | 0.45 | 0.42 |
| 18 | In the next place, as each representative will… | 76 | 32.06 | 0.42 | 0.55 |
| 19 | It must be confessed that in this, as… | 103 | 17.28 | 0.46 | 0.52 |
| 20 | The other point of difference is, the greater… | 216 | 24.70 | 0.44 | 0.45 |
| 21 | Hence, it clearly appears, that the same advantage… | 169 | 14.14 | 0.49 | 0.48 |
| 22 | The influence of factious leaders may kindle a… | 128 | 19.86 | 0.48 | 0.51 |
| 23 | In the extent and proper structure of the… | 52 | 14.75 | 0.48 | 0.54 |
One panel per paragraph measure, paragraphs in document order along the bottom. A cross marks a
paragraph matra declined to measure: the field is None because the paragraph falls
outside the measure's applicability conditions above, and the line breaks there rather than
inventing a value. The dashed line is the document's mean readability: the mean of the grades
of the paragraphs matra measured. matra computes no document-level lexical density or
compression ratio, so those panels have no line.
Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: federalist-10/metrics.json.
What to notice
- Each paragraph carries
readability_grade,lexical_densityandcompression_ratioin itsmetrics. The document's ownmetrics, beside itssections, carry the counts, the sentence-length statistics,vocabulary_ttr,nominalization_ratioandpassive_ratio. - The grade runs from 10.69 (paragraph 3) to 32.06 (paragraph 18). It is the Flesch-Kincaid formula over sentence length and syllables per word (Methodology), so an eighteenth-century paragraph of a few long sentences scores far above any school grade. Compare paragraphs with each other, not with a grade band.
compression_ratioisnullfor four paragraphs, the ones of 50 words or fewer.nullmeans the measure declined to run, not that it measured zero; which measure needs what is in the Rust guide.
Summarize a long document
The three sentences TextRank ranks highest in the opening of Darwin's chapter on the struggle for existence, 2,214 words.
Input
The calls read it as origin-struggle.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/summarize/origin-struggle.txt or download origin-struggle.txt.
The text, 10 paragraphs
Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.
Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.
We will now discuss in a little more detail the struggle for existence. In my future work this subject shall be treated, as it well deserves, at much greater length. The elder De Candolle and Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. In regard to plants, no one has treated this subject with more spirit and ability than W. Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.
I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.
A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.
There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.
But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.
In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.
The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.
In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, SummaryMethod};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let engine = Engine::new()?;
let text = std::fs::read_to_string("origin-struggle.txt")?;
let doc = engine.analyze(Input::text(text, Format::PlainText))?;
// Parse once; a summary is a function of the document.
let summary = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
println!("{}", serde_json::to_string_pretty(&summary)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
doc = engine.analyze(f.read()) # read as plain text
# Parse once; a summary is a function of the document.
summary = matra.summarize(doc, n=3, method="textrank")
print(json.dumps(summary, indent=2, ensure_ascii=False)) CLI
$ matra summarize --method textrank -n 3 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "summarize",
"input": "origin-struggle.txt",
"result": /* the output below */,
"error": null
}Output
Trimmed: long strings shortened.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "plain_text",
"source": {
"path": null,
"bytes": 12808,
"sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
},
"method": "textrank",
"parameters": {
"n": 3
}
},
"sentences": [
{
"index": 32,
"span": {
"byte_start": 6000,
"byte_end": 6328
},
"text": "Every being, which during its natural lifetime produces several eggs or…",
"score": 0.029215311359903445
},
{
"index": 33,
"span": {
"byte_start": 6329,
"byte_end": 6591
},
"text": "Hence, as more individuals are produced than can possibly survive, ther…",
"score": 0.03081881474196398
},
{
"index": 58,
"span": {
"byte_start": 12049,
"byte_end": 12444
},
"text": "In looking at Nature, it is most necessary to keep the foregoing consid…",
"score": 0.030089482622730077
}
]
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
Every sentence's score in document order, with the three the summary keeps marked:
TextRank: the summary and the scores it came from
- 33Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product.
- 34Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life.
- 59In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals.
show the data
| # | ¶ | Score | Summary | Sentence |
|---|---|---|---|---|
| 1 | 1 | 0.0155 | Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. | |
| 2 | 1 | 0.0145 | It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. | |
| 3 | 1 | 0.0182 | It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. | |
| 4 | 1 | 0.0180 | But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. | |
| 5 | 1 | 0.0148 | How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? | |
| 6 | 1 | 0.0155 | We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world. | |
| 7 | 2 | 0.0123 | Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? | |
| 8 | 2 | 0.0089 | How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? | |
| 9 | 2 | 0.0168 | All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. | |
| 10 | 2 | 0.0240 | Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. | |
| 11 | 2 | 0.0153 | The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. | |
| 12 | 2 | 0.0102 | I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. | |
| 13 | 2 | 0.0192 | We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. | |
| 14 | 2 | 0.0113 | But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art. | |
| 15 | 3 | 0.0114 | We will now discuss in a little more detail the struggle for existence. | |
| 16 | 3 | 0.0060 | In my future work this subject shall be treated, as it well deserves, at much greater length. | |
| 17 | 3 | 0.0025 | The elder De Candolle and | |
| 18 | 3 | 0.0055 | Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. | |
| 19 | 3 | 0.0103 | In regard to plants, no one has treated this subject with more spirit and ability than W. | |
| 20 | 3 | 0.0046 | Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. | |
| 21 | 3 | 0.0121 | Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. | |
| 22 | 3 | 0.0123 | Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. | |
| 23 | 3 | 0.0237 | We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year. | |
| 24 | 4 | 0.0202 | I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. | |
| 25 | 4 | 0.0152 | Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. | |
| 26 | 4 | 0.0161 | But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. | |
| 27 | 4 | 0.0237 | A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. | |
| 28 | 4 | 0.0121 | The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. | |
| 29 | 4 | 0.0153 | But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. | |
| 30 | 4 | 0.0216 | As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. | |
| 31 | 4 | 0.0120 | In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence. | |
| 32 | 5 | 0.0197 | A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. | |
| 33 | 5 | 0.0292 | yes | Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. |
| 34 | 5 | 0.0308 | yes | Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. |
| 35 | 5 | 0.0129 | It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. | |
| 36 | 5 | 0.0154 | Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them. | |
| 37 | 6 | 0.0142 | There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. | |
| 38 | 6 | 0.0098 | Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. | |
| 39 | 6 | 0.0173 | Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. | |
| 40 | 6 | 0.0152 | The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair. | |
| 41 | 7 | 0.0199 | But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. | |
| 42 | 7 | 0.0154 | Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. | |
| 43 | 7 | 0.0153 | So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. | |
| 44 | 7 | 0.0093 | Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. | |
| 45 | 7 | 0.0198 | In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. | |
| 46 | 7 | 0.0112 | The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. | |
| 47 | 7 | 0.0109 | In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes. | |
| 48 | 8 | 0.0213 | In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. | |
| 49 | 8 | 0.0245 | Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. | |
| 50 | 8 | 0.0205 | Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of. | |
| 51 | 9 | 0.0179 | The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. | |
| 52 | 9 | 0.0115 | The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. | |
| 53 | 9 | 0.0173 | One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. | |
| 54 | 9 | 0.0182 | A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. | |
| 55 | 9 | 0.0237 | But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. | |
| 56 | 9 | 0.0247 | If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. | |
| 57 | 9 | 0.0180 | It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. | |
| 58 | 9 | 0.0225 | So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds. | |
| 59 | 10 | 0.0301 | yes | In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. |
| 60 | 10 | 0.0183 | Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. | |
| 61 | 10 | 0.0120 | The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force. |
- in the summary (3 sentences, matra's default)
- not picked
TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.
Input: On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: origin-struggle/textrank.json.
What to notice
- The summary carries the text's
provenancewith what made it:methodistextrankandparametersis{"n": 3}. Each of itssentenceshasindex, the sentence's place among the document's sentences counting from 0, itsspanin the input, itstextand itsscore. The three come back in document order (indices 32, 33 and 58), not by score: the highest score, 0.0308, is the second. The figure numbers sentences from 1, so there they are 33, 34 and 59. - All three calls hand matra the text rather than the file, so the provenance records no
path: the Python method takes text, and the three must record the same input. - The summary is extractive: the sentences are as Darwin wrote them, and nothing is rephrased.
- matra has a second summarizer, TF-IDF, and it is the default:
matra::summarizein Rust, and the command line'ssummary.methodas shipped. This example names TextRank in all three calls (SummaryMethod::TextRank,method="textrank",--method textrank), so they agree whatever a configuration file says.
Extract a document's keyphrases
The ten phrases RAKE ranks highest in the same Darwin text as the summary example.
Input
The calls read it as origin-struggle.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/keyphrases/origin-struggle.txt or download origin-struggle.txt.
The text, 10 paragraphs
Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.
Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.
We will now discuss in a little more detail the struggle for existence. In my future work this subject shall be treated, as it well deserves, at much greater length. The elder De Candolle and Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. In regard to plants, no one has treated this subject with more spirit and ability than W. Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.
I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.
A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.
There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.
But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.
In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.
The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.
In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, KeyphraseMethod};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let engine = Engine::new()?;
let text = std::fs::read_to_string("origin-struggle.txt")?;
let doc = engine.analyze(Input::text(text, Format::PlainText))?;
// Parse once; keyphrases are a function of the document.
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Rake)?;
println!("{}", serde_json::to_string_pretty(&phrases)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
doc = engine.analyze(f.read()) # read as plain text
# Parse once; keyphrases are a function of the document.
phrases = matra.keyphrases(doc, n=10, method="rake")
print(json.dumps(phrases, indent=2, ensure_ascii=False)) CLI
$ matra keyphrases --method rake -n 10 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "keyphrases",
"input": "origin-struggle.txt",
"result": /* the output below */,
"error": null
}Output
Trimmed: lists cut to the first 5.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "plain_text",
"source": {
"path": null,
"bytes": 12808,
"sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
},
"method": "rake",
"parameters": {
"n": 10
}
},
"phrases": [
{
"phrase": "larger domestic animal tend",
"score": 12.636363636363637,
"occurrences": [
{
"byte_start": 10132,
"byte_end": 10161
}
]
},
{
"phrase": "elder de candolle",
"score": 9,
"occurrences": [
{
"byte_start": 3228,
"byte_end": 3245
}
]
},
{
"phrase": "fulmar petrel lay",
"score": 9,
"occurrences": [
{
"byte_start": 10791,
"byte_end": 10809
}
]
},
{
"phrase": "mere theoretical calculation",
"score": 8.5,
"occurrences": [
{
"byte_start": 8016,
"byte_end": 8045
}
]
},
{
"phrase": "great horticultural knowledge",
"score": 8.2,
"occurrences": [
{
"byte_start": 3501,
"byte_end": 3530
}
]
}
// … 5 more
]
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
RAKE against YAKE on the same text, as ranks:
RAKE and YAKE on the same text
show the data
| Phrase | RAKE rank | RAKE score | YAKE rank | YAKE score |
|---|---|---|---|---|
| immaterial whether multitude | not ranked | 1 | 45079.3 | |
| larger domestic animal tend | 1 | 12.636 | not ranked | |
| elder de candolle | 2 | 9.000 | 1659 | 0.548 |
| fulmar petrel lay | 2 | 9.000 | 2421 | 0.241 |
| sub-specy varietie rank | not ranked | 2 | 26693.1 | |
| enter subject | not ranked | 3 | 8987.0 | |
| enter subject chapter | not ranked | 4 | 8544.3 | |
| mere theoretical calculation | 4 | 8.500 | 2245 | 0.352 |
| enter | not ranked | 5 | 2233.5 | |
| great horticultural knowledge | 5 | 8.200 | 1919 | 0.459 |
| probable minimum rate | 6 | 7.600 | 2307 | 0.323 |
| whether multitude doubtful | not ranked | 6 | 1735.7 | |
| astonishingly rapid increase | 7 | 7.357 | 2176 | 0.377 |
| cling hair quadruped | not ranked | 7 | 1679.6 | |
| great destruction fal | 8 | 6.914 | 2379 | 0.281 |
| hair quadruped feather | not ranked | 8 | 1615.5 | |
| slow -breeding man | 9 | 6.667 | 1810 | 0.496 |
| structure beetle dive | not ranked | 9 | 1383.8 | |
| beetle dive water | not ranked | 10 | 1334.5 | |
| numerous record case | 10 | 5.972 | 2361 | 0.292 |
| domestic animal | 11 | 4.636 | 1547 | 0.590 |
| immaterial whether | not ranked | 11 | 1313.1 | |
| rapid increase | 12 | 4.357 | 1612 | 0.561 |
| whether multitude | not ranked | 12 | 1214.9 |
- in RAKE's top 12
- in YAKE's top 12
- only one method produces the phrase
Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 321 RAKE and 2430 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.
Input: On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: origin-struggle/keyphrases.json.
What to notice
- The result carries the text's
provenancewithmethodrakeandparameters{"n": 10}. Each of itsphraseshasphrase,scoreandoccurrences, the span of every place in the input it was found, highest score first, and a higher score means more relevant. RAKE's score sums each word's co-occurrence degree over its frequency. - Phrases are built from lemmas, so the parser's lemma errors show through.
great destruction falis "great destruction falling", withfallingread asfal, andslow -breeding manis "slow-breeding man" split at the hyphen. RAKE takes the words between stopwords and punctuation, which is whylarger domestic animal tendruns "animals tends" together. - Two phrases tie at 9.0. Equal scores come back in the order the phrases first occur, the same on every run, so a cut that falls inside a tie is decided by position in the text, not by merit. This example asks for ten, where the tenth scores 5.972 and no phrase outside the list ties it, so no tie decides the list.
- The figure shows how little RAKE and YAKE agree on a text like this one. Neither method is right; they rank by different evidence (Methodology).
Read a Markdown document's structure
A short Markdown document with two headings and a quote, read into sections and paragraphs.
Input
The calls read it as field-notes.md in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/markdown-structure/field-notes.md # Field notes The survey team reached the northern ridge on the third day. Snow had closed the lower pass, so the route ran along the eastern face instead. > Radio contact was lost for six hours. ## Findings Three of the five markers were intact and matched their recorded positions. The fourth had been moved about ten metres downhill by a rockfall, and its base plate was cracked. The fifth could not be found, although the team searched the slope below it for most of an afternoon. A replacement marker will be set on the next visit, once the pass is open.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("field-notes.md"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="field-notes.md") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) CLI
$ matra analyze --json field-notes.md It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "analyze",
"input": "field-notes.md",
"result": /* the output below */,
"error": null
}Output
Trimmed: sentences shown as a count, long strings shortened.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "markdown",
"source": {
"path": "field-notes.md",
"bytes": 564,
"sha256": "bbf7f1c703c0bf2b998b1a8fc61e1fe0f0e6aeb670b169fd5e2da372c62fe55f"
}
},
"sections": [
{
"heading": "Field notes",
"level": 1,
"span": {
"byte_start": 0,
"byte_end": 13
},
"paragraphs": [
{
"span": {
"byte_start": 15,
"byte_end": 156
},
"text": "The survey team reached the northern ridge on the third day. Snow had c…",
"kind": "paragraph",
"sentences": [ /* 2 items */ ],
"metrics": {
"readability_grade": 4.91076923076923,
"lexical_density": 0.6538461538461539,
"compression_ratio": null
}
},
{
"span": {
"byte_start": 158,
"byte_end": 197
},
"text": "Radio contact was lost for six hours.",
"kind": "quote",
"sentences": [],
"metrics": {
"readability_grade": null,
"lexical_density": null,
"compression_ratio": null
}
}
]
},
{
"heading": "Findings",
"level": 2,
"span": {
"byte_start": 199,
"byte_end": 210
},
"paragraphs": [
{
"span": {
"byte_start": 212,
"byte_end": 563
},
"text": "Three of the five markers were intact and matched their recorded positi…",
"kind": "paragraph",
"sentences": [ /* 4 items */ ],
"metrics": {
"readability_grade": 6.8750000000000036,
"lexical_density": 0.484375,
"compression_ratio": 0.5897435897435898
}
}
]
}
],
"metrics": {
"total_words": 91,
"total_sentences": 6,
"paragraph_count": 3,
"mean_sentence_length": 15.166666666666666,
"sentence_length_std": 3.3714487489307423,
"vocabulary_ttr": 0.6923076923076923,
"nominalization_ratio": 0.01098901098901099,
"passive_ratio": 0.5
},
"skipped": []
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
Programming model steps this same document through each stage of the pipeline, to show what each stage adds.
What to notice
sectionsfollow the headings: each hasheading, the heading's text,level, the number of#marks, andspan, where the heading lies in the file,#marks included.- The quote is kept as a paragraph whose
kindisquote, and it is never parsed: itssentenceslist is empty and all three of its measures arenull. Itsspancovers the>markers its text leaves out. - The Markdown reader is chosen by the
.mdextension in all three calls. Read as plain text, the same file is one section with no heading, and the heading lines and the quote are parsed as sentences,#marks and all.
Cluster sentences by meaning
Group the sentences of a document that say the same thing in different words, for example to audit LLM output for restatement, which lexical overlap cannot see. In Rust this needs the model2vec feature for the reference model; the Python package has it. What the clustering threshold does explains what a cluster means, and Semantic clusters lists the value, the model and the bounds.
What you get
A Clusters value: the groups of sentences whose pairwise cosine similarity cleared the threshold you pass, the pairs that cleared it, and a provenance recording the threshold and the identity of the model that scored them.
Start around 0.85 with the reference model on sentences, and calibrate on your own corpus: the threshold does not travel between models, domains or text lengths. Comparing whole documents shows how to read the raw scores before you choose one.
The model
Take the reference model, potion-base-8M. The first call downloads it (30.9 MB) and verifies it against a digest compiled into matra; every later call loads it from disk:
use matra::config::Config;
use matra::embed::model2vec::Model2Vec;
let model = Model2Vec::from_config(&Config::resolve()?)?;
from matra import Model2Vec
model = Model2Vec.potion_base_8m()
It goes into a directory named for the configured embedding model (models.embedding, potion-base-8M as shipped) inside the configured model directory. To keep it somewhere else, name a models directory: Model2Vec::potion_base_8m(dir) in Rust, Model2Vec.potion_base_8m(dir) in Python. dir is the directory that holds every model, the one MATRA_MODEL_DIR names, not the model's own: the three files land in dir/potion-base-8M/, beside the parsing model in dir, so passing the directory matra config show reports as model_dir reuses a model that is already there. The call refuses a directory that already holds another model's files and names it; remove them, or name another directory, or point MATRA_MODEL_DIR or models.embedding somewhere else.
To use files you already have, or a different model2vec artifact, put the three files in one directory and load it with Model2Vec::from_dir(dir), which never reaches the network:
$ mkdir -p ~/models/potion-base-8M && cd ~/models/potion-base-8M
$ for f in model.safetensors tokenizer.json config.json; do
curl -sSfLO "https://huggingface.co/minishlab/potion-base-8M/resolve/main/$f"
done
Nothing verifies files you load this way. To check that you have the reference model, compare model.model_hash with the digest in the reference.
Rust
use matra::config::Config;
use matra::embed::model2vec::Model2Vec;
use matra::{Engine, Input};
let cfg = Config::resolve()?;
let engine = Engine::from_config(&cfg)?;
let model = Model2Vec::from_config(&cfg)?;
let doc = engine.analyze(Input::text(text, matra::domain::Format::PlainText))?;
let clusters = matra::clusters(&doc, &model, 0.85)?;
for c in &clusters.clusters {
println!("restated {} times: sentences {:?}", c.members.len(), c.members);
}
With your own Embedder implementation in place of Model2Vec, clusters needs no feature at all; an embedder that returns vectors you already hold is how they get in.
Python
import matra
model = matra.Model2Vec.potion_base_8m()
engine = matra.Engine()
doc = engine.analyze(text, format="markdown") # or plain_text; a str alone reads as plain text
result = matra.clusters(doc, model, threshold=0.85)
for cluster in result["clusters"]:
print("restated:", cluster["members"])
Name the format the text is in. Read as plain text, a markdown document's heading markers (#, ##) become sentences of their own, and identical short sentences embed identically, so they come back as one cluster of "restatements". Read as markdown, a heading is a heading and never a sentence. engine.analyze(file=path) detects the format from the file.
Already hold embeddings? Pass an object whose embed returns them, one per sentence, and whose identity names the model that made them, as in Rust. And model.embed(texts) returns the raw vectors when you want to do something else with them.
The threshold comes back in result["provenance"]["parameters"]["threshold"] exactly as you passed it, beside the embedder's identity. An edge's score is computed in the vectors' f32 and widened, so compare scores at a few decimal places rather than for equality.
The command line
matra clusters notes.md --threshold 0.85 --json
matra clusters builds the reference embedder from the configuration, the model models.embedding names in the model directory, and takes its threshold from --threshold, else clusters.threshold in the config file, 0.85 as shipped. --json writes the same Clusters value inside the envelope; the table lists each cluster's sentence indices and text. It needs a build with model2vec, which the Python package has; a cargo install --features cli build refuses the command and names the feature. The CLI guide has the flags.
Comparing whole documents
clusters works over the sentences of one document, in every language. There is no cross-document primitive. Build one out of what you have: embed each document as a single text with model.embed, then compare the vectors yourself.
One bound decides what the answer means. Model2Vec caps every text at 512 tokens, pre-truncating on bytes and then truncating the token ids, so "embed each document as a single text" embeds roughly the first 512 tokens of it and nothing after. Tokens are not words, and how many words 512 tokens buys depends on what the file holds. Say which basis a figure is on. Swept over the 26 pages of this book long enough to reach it, with the call below, which embeds the raw file text, the cap ran out between 141 and 378 words. Diagrams, code fences and tables pull the low end down; the recipe below reads files off disk, so the low end is the one that applies to it. Two long texts that agree for as few as their first 141 words can already embed to byte-identical vectors. That cuts both ways: documents sharing a boilerplate opening score as near-duplicates on the opening alone, and two real paraphrases that diverge inside their first few hundred words never get compared on the part that matters.
If the tail carries the content, split each document into chunks and compare the chunks, and size the chunks from a token count rather than a word count. A word count is not a safe proxy on raw markup, and the gap is not small: sliding a window across the same pages, a window of 54 words landed inside an inline SVG block and had already filled the cap. Any word figure has to be measured against your own files, because getting it wrong does not raise. It returns a score computed on less text than you handed it.
Read every pairwise score before you choose a cutoff. That is how you calibrate:
import math
from itertools import combinations
from pathlib import Path
from matra import Model2Vec
model = Model2Vec.potion_base_8m()
docs = [
Path("site/content/guides/cli.md"),
Path("site/content/guides/rust.md"),
Path("site/content/roadmap.md"),
]
vectors = model.embed([p.read_text() for p in docs])
def cosine(a: list[float], b: list[float]) -> float:
norm = math.hypot(*a) * math.hypot(*b)
return sum(x * y for x, y in zip(a, b)) / norm if norm else 0.0
for a, b in combinations(range(len(docs)), 2):
print(f"{cosine(vectors[a], vectors[b]):.4f} {docs[a].name} <-> {docs[b].name}")
Those three are pages of this book, run from a checkout of the repository, so the numbers below are yours to reproduce. Two of them cover the same ground for different surfaces, and the third is unrelated. It prints:
0.8281 cli.md <-> rust.md
0.5800 cli.md <-> roadmap.md
0.6121 rust.md <-> roadmap.md
The separation is decisive, and the sentence-level starting point does not carry over: against a cutoff of 0.85 no pair clears, so the near-duplicate pair would have been reported as unrelated. This is what "the threshold does not travel" costs when it is taken on faith. Calibrate on scores you have read.
Two things carry over from clusters, and on this route they are yours to keep. Attribution: the vectors do not carry the model identity, so record model.model_hash beside every score you keep, as clusters records it in parameters["embedder"]["sha256"]. And a zero-magnitude vector, which is what an empty document embeds to, makes no claim: the cosine above returns 0.0 for it rather than dividing by zero.
Hand matra to an agent
Give an agent matra's own instructions, printed by the version it will run, rather than pointing it at these pages.
Print the skill
If the agent has a shell and nothing installed, give it this line:
uvx 'matra>=0.3' --skill
If matra is already installed, matra --skill prints the same text. It says what matra is for, when to reach for it, every command with its JSON shape, how to read each number and what it does not license the agent to conclude, the limits, and the errors.
Keep the version floor in the uvx line. The skill describes the command line of the version that prints it, and 0.2's spells its commands and its JSON differently from the 0.3 these pages describe. 0.1.0's has no --skill at all.
Give it the deeper references
matra --skill -r # the references, one per line: name, then summary
matra --skill -r json # one reference
The text is compiled into the binary and the wheel, so it matches the version that is running, and the test suite executes every command in it against the command line. The CLI reference has the flag's exit codes and its --json shapes.
Load it as a plugin
The same files install as a Claude Code plugin. Run claude --plugin-dir <checkout> over a clone of the repository, and plugin discovery picks up skills/matra/.
Point a reader of this site at llms.txt
An agent reading the documentation instead of running matra starts at llms.txt, served at the site's root: every page with its one-line summary.
Install the model behind a TLS-intercepting proxy
Place the parsing model by hand when a proxy that re-signs TLS stops matra from downloading it.
Recognise the failure
matra verifies TLS against root certificates compiled into the binary and never reads the system trust store. That is why it needs no ca-certificates package, on any platform, and it is also why a proxy that re-signs TLS cannot be trusted by installing its CA anywhere on the machine. The first run that needs the model fails with:
matra: io error: download https://lindat.mff.cuni.cz/...: the TLS certificate offered for
lindat.mff.cuni.cz was rejected. matra verifies TLS against root certificates compiled into
it and never reads the system trust store, so a proxy that re-signs TLS cannot be trusted by
installing its CA. Fetch english-ewt-ud-2.5-191206.udpipe by hand and put it in the model
directory instead. Underlying failure: io: invalid peer certificate: ...
In Python the same failure is a matra.IoError, an OSError, from matra.Engine(), with the URL in details["url"] and details["retryable"] false, since retrying meets the same proxy. Errors lists every way a download can fail.
Place the model by hand
The artifact is pinned by name, size and SHA-256, so a hand-placed file is exactly as trustworthy as a fetched one: it goes through the same verification on load. A file that is not the pinned model is never used and never deleted. matra tries to download the pinned model in its place, and where that download cannot get through, which is the reason to place the file by hand, the call fails and the file stays exactly as you left it.
- Find the model directory this machine resolves, and create it.
matra config showprints it asmodel_dir. - Download the model with a client that trusts the proxy, such as
curlwith the system store. - Check its SHA-256 before moving it in, so a truncated or proxy-rewritten download fails here rather than on the next run. The digest must read
784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957. - Move it into the model directory.
mkdir -p "$(matra config show | awk -F\" '/^model_dir/ {print $2}')"
curl -L -o english-ewt-ud-2.5-191206.udpipe \
"https://lindat.mff.cuni.cz/repository/server/api/core/bitstreams/handle/11234/1-3131/english-ewt-ud-2.5-191206.udpipe?sequence=17&isAllowed=y"
shasum -a 256 english-ewt-ud-2.5-191206.udpipe
# 784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957
mv english-ewt-ud-2.5-191206.udpipe "<the model_dir above>/"
MATRA_MODEL_DIR points at a directory of your own if the resolved one is not writable.
Check it worked
Run any command that parses. The command line writes matra: downloading ... to standard error before a fetch and nothing when the model is already in place, so a run that prints no such line has loaded the file you placed.
The embedding model
The reference embedding model has the same route: three artifacts into a directory loaded with Model2Vec::from_dir, described in semantic clusters.
matra
matra reports the structure of text and measurements over it. It parses a document into a typed tree of sections, paragraphs, sentences, and tokens, measures that tree, and ranks what is in it. Every value it returns is structure or a number over structure. What any of it means for your purpose is your code's decision, which is what makes the output reusable across purposes.
Four tiers of output
- Structure. Sections, paragraphs, sentences, and full CoNLL-U tokens, plus the structural primitives read off each dependency tree: negations, modals, clausal complements, root adverbials, Hearst pairs, and the root clause's mood.
- Measures. Readability, lexical density, and compression ratio per paragraph; counts, sentence length, vocabulary TTR, nominalization ratio, and passive ratio per document. Every document also records where it came from: the parser and its model, and a digest of the analysed text.
- Extraction. Two extractive summarizers (TF-IDF, TextRank) and two keyphrase extractors (RAKE, YAKE).
- Semantic clusters. Sentence embeddings grouped by cosine similarity, behind the
model2vecfeature, carrying the identity of the model that produced them.
End to end
import matra
engine = matra.Engine()
text = (
"The committee approved the proposal without debate. "
"Three amendments were submitted by the working group."
)
doc = engine.analyze(text)
print(doc["metrics"]["passive_ratio"])
print(doc["sections"][0]["paragraphs"][0]["metrics"]["readability_grade"])
The same analysis from Rust, and from the command line:
use matra::domain::Format;
use matra::{Engine, Input};
let engine = Engine::new()?;
let text = "The committee approved the proposal without debate. \
Three amendments were submitted by the working group.";
let doc = engine.analyze(Input::text(text, Format::PlainText))?;
println!("{:?}", doc.metrics.passive_ratio);
$ matra analyze essay.md
essay.md
sentences 35
words 332
mean sentence len 9.5
sentence len sd 7.4
passive ratio 0.057
All three return the same shape: a struct in Rust, a read-only Document that reads as a dict in Python, a table or --json at the command line. The second sentence in the example is passive, and amendments carries the relation nsubj:pass. matra records that and stops there.
Where to go next
Understand the model. Concepts covers the document tree, the four tiers, and what a structural primitive is. Situation model covers what the output is for and what it deliberately withholds.
Write against it. Programming model is the surface: Input, Engine, ports, bounds, and what Python exposes. Pragmatics answers the choices you have to make.
Get it running. Installation, then Rust, Python, or CLI.
Look something up. What matra gives you lists every field and function the pipeline returns.
Concepts
The document tree
Analysis returns one nested value. Each level owns the level below it and carries its own data.
| Level | Type | Carries |
|---|---|---|
| document | Document | the section list, plus vocabulary_ttr, nominalization_ratio, passive_ratio |
| section | Section | heading, level, and its paragraphs in document order |
| paragraph | Paragraph | its span, text and kind, its sentences, plus readability_grade, lexical_density, compression_ratio |
| sentence | Sentence | its text, id-sorted tokens, and the structural primitive fields |
| token | Token | the ten CoNLL-U columns (text, lemma, pos, xpos, features, head, dep, deps, misc, plus id) and the derived is_punct |
Paragraph.text is the text the parser reads, a markdown paragraph's inline markup removed. Sentence.text is rebuilt from the tokens, so its spacing can differ from the source. Both are normalized, and each carries a span, in UTF-8 bytes, saying where in the analysed text it came from. Each paragraph that is not a quote is parsed on its own, so a sentence always belongs to exactly one paragraph, rather than being matched back by text.
Inside a sentence, exactly one token has head equal to 0: the root. Every other token names its governor in head and the relation in dep. Those two columns are what turn a flat token list into a tree.
Words keep their reading order left to right; height is depth in the tree. built is the root, everything else hangs off it directly or through one more hop, so its tree_depth is 2.
The same two columns, read from matra's output for five short sentences. The table is what a Sentence holds; the arcs below it draw each head and dep. The first sentence carries a negation, the second a modal, the third a clausal complement, the fourth a root adverbial, and the fifth two Hearst pairs: the structural primitives described further down.
Dependency parse
1The committee did not approve the budget.
- The det of committee
- committee nsubj of approve
- did aux of approve
- not advmod of approve
- approve root
- the det of budget
- budget obj of approve
- . punct of approve
show the data
| # | Word | Lemma | POS | Head | Relation |
|---|---|---|---|---|---|
| 1 | The | the | DET | 2committee | det |
| 2 | committee | committee | NOUN | 5approve | nsubj |
| 3 | did | do | AUX | 5approve | aux |
| 4 | not | not | PART | 5approve | advmod |
| 5 | approve | approve | VERB | 0root | root |
| 6 | the | the | DET | 7budget | det |
| 7 | budget | budget | NOUN | 5approve | obj |
| 8 | . | . | PUNCT | 5approve | punct |
- Core argument
- Modifier
- Function word
- Other
Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.
Input: Five sentences, one structural primitive each: a negation, a modal, a reporting, a root adverbial, a Hearst pair, matra contributors. MIT, as part of the matra repository.
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: primitives/parse.json.
The four output tiers
The pipeline produces them in this order, and each is a different kind of result.
- Structure, read and parsed by
Engine::analyze. Sections, paragraphs, sentences, tokens, and the structural primitive fields. Derived from the parse, checkable against the source bytes. - Measures, filled by
Engine::analyzeas its last step. SixOption<f64>slots filled by the metric suite: three per paragraph, two per document, plus the materializedpassive_ratio.Nonemeans not computed, which is distinct from a computed zero. - Outcomes, from free functions over the
Document:summarizeandkeyphrases, with_withforms that name the method. They returnSummaryandKeyphrasesvalues, each carrying the document's provenance with the method and its parameters, never fields on the tree. - Semantic, with the reference embedder behind the
model2vecfeature.clustersreturns aClustersvalue whose provenance carries the embedder's identity and the threshold you supplied. This tier depends on the model, so by design it never becomes a field onDocumentorSentence.
Structural primitives
A structural primitive is a construction read off the dependency graph and reported as a field. Six of them are computed once and travel with the sentence into every language binding: five at Sentence construction, and hearst_pairs as the engine parses each paragraph, because its detector lives outside the domain (a Sentence built by hand has an empty hearst_pairs). They name an arc shape. They never name a judgment.
| Field | The arc shape | Grounded example |
|---|---|---|
negations | not, never, no, neither, nor on an advmod, det, or cc arc | It was never reviewed. gives one Negation: cue never, head reviewed |
modals | one of ten modal lemmas on an aux arc or with the AUX part of speech | You must complete the form by Friday. gives one Modal: must on complete |
clausal_complements | a verb governing a clausal complement (ccomp), with its subject when the parse has one | Smith reported that the effect vanished. gives one ClausalComplement: verb report, complement head vanished, subject Smith |
root_adverbials | every advmod arc into the root | Reportedly, the deal closed. gives one RootAdverbial: reportedly |
hearst_pairs | the six Hearst (1992) lexico-syntactic patterns, matched as dependency arcs | Animals such as dogs and cats need daily care. gives two HearstPair values, both tagged such_as, hypernym animal, hyponyms dog and cat |
root_mood | the Mood feature of the root clause, on the root or on its cop, aux, or aux:pass child, with the root's token id | The committee approved it. gives Ind on approved; It might have been done. gives none, because neither the participle root nor its auxiliaries carry a mood |
Two consequences of reporting arcs rather than readings:
root_mood consults neither negation nor modals. The detector reads the root and its finiteness carriers. Nothing else. The sky is not blue reports Ind and also carries a negation; an indicative root that no modal governs is Ind with no modal whose head_id is the root's id. Combining the facts is your code's job.
Overlap is intentional. never in the example above is both a Negation and a RootAdverbial, because it satisfies both arc shapes. Neither field is assigning it a meaning, so neither has to win.
Passive voice is the seventh construction, a flag rather than a list of arcs: Sentence::passive is true when a token carries nsubj:pass, nsubjpass, or aux:pass, and its document-level share is the passive_ratio field of DocumentMetrics.
Reporting verbs and root adverbials are open word classes, so matra ships no lexicon for either. Sentence::clausal_complements_in and Sentence::root_adverbials_in filter by a list you supply.
Where to go next
Situation model covers what this output is for. Programming model covers the surface that produces it.
Situation model
What matra is for
matra reports the structure of text and measurements over it. Text goes in; a typed tree, a set of measures, and a set of ranked extractions come out. Everything it returns is a value your code can read, store, and reason over.
It is a library, not an application. The analysis on top of it is yours to write.
What it reports and what it leaves to you
matra reports passive_ratio = 0.4. Whether 0.4 is too passive for the document at hand is your call, and the answer differs between a statute, a lab report, and a landing page.
That split runs through every tier:
| matra reports | Your code decides |
|---|---|
a sentence carries nsubj:pass | whether the missing agent matters here |
clausal_complements names a verb governing a ccomp | whether that verb is evidential and whether the source is credible |
hearst_pairs names a construction that conventionally signals hypernymy | whether the hypernymy relation actually holds |
modals names must on an aux arc | whether that must is obligation or inference |
a Cluster groups sentences whose vectors cleared your threshold | whether those sentences say the same thing |
The reason for the split is reuse. A judgment is fitted to one purpose and is dead weight to the next caller; the arc it was computed from is not.
Where it sits in a pipeline
Upstream of interpretation, downstream of ingestion.
files or strings -> matra -> rule evaluation
scoring and dashboards
retrieval and chunking
LLM prompt construction
Two properties make it usable that far downstream. Every value is serde-serializable, so the parse crosses into Python as a plain dict and can be stored as JSON. And every value has a place in the tree: a token id is scoped to its sentence, a sentence belongs to the paragraph whose text was parsed to produce it, and every heading, paragraph, sentence and token carries a span into the analysed text, so a value can be traced back to the bytes it came from.
What it needs from the environment
Nothing has to be set up first. Engine::new() in Rust, matra.Engine() in Python, and any matra command resolve where models live and fetch what is missing. The directory comes from an explicit argument if you pass one, else MATRA_MODEL_DIR, else the models subdirectory of the data root (MATRA_DATA_DIR, else $XDG_DATA_HOME/matra, else ~/.local/share/matra), with a non-empty ~/.matra/models from an older install used when the new location does not exist yet. The config file does not move the model directory, and it does not choose the parsing model, which is pinned in the source. Programming model has the resolution order.
Models are fetched, not bundled, and only against a pinned digest. The UDPipe English model arrives on the first call to Engine::new or Engine::from_config, in Rust or in Python; the reference embedding model arrives on the first call to Model2Vec::potion_base_8m or Model2Vec::from_config. Both fetch from URLs written in the source and load nothing whose SHA-256 does not equal a constant written beside them. A download that fails verification is fetched once more, in memory, and a second failed download raises without anything having been written. Exactly one artifact set can arrive this way for each.
Bringing your own embedding model is a first-class path. Model2Vec::from_dir loads what you supply and touches no network, whatever the directory holds. The parsing model is a pin. For those, the digest is identity rather than proof: it says which artifacts produced a value, not that they are the ones you meant to fetch.
Verification and load share one read. Bytes are hashed in memory and the model is loaded from those same bytes. Nothing re-reads the disk between the two.
Those pinned model fetches are the only network access anywhere in the library.
The split, on five sentences
The primitive fields from the table above, as matra fills them for five short sentences written to carry one each. Every mark is a construction in the tree; whether it matters for a text stays with the reader.
Structural primitives
- 1
The committee did not approve the budget.
root mood Ind - 2
Members must submit revised estimates by Friday.
- 3
The chair reported that the vote was postponed.
root mood Ind - 4
Reportedly, the new figures could arrive next week.
- 5
Agencies such as the treasury and the audit office publish quarterly reports.
root mood Ind
show the data
| # | Primitive | Read from | Attaches to | Field and detail |
|---|---|---|---|---|
| 1 | negation | 4not | 5approve | negations |
| 1 | root adverbial | 4not | none | root_adverbials |
| 1 | root mood | 5approve | none | root_mood, mood Ind |
| 2 | modal | 2must | 3submit | modals |
| 3 | clausal complement | 3reported | 8postponed | clausal_complements, subject 2 chair |
| 3 | root mood | 3reported | none | root_mood, mood Ind |
| 4 | modal | 6could | 7arrive | modals |
| 4 | root adverbial | 1Reportedly | none | root_adverbials |
| 5 | Hearst pair | 5treasury | 1Agencies | hearst_pairs, such_as, hyponym span 4 to 5 |
| 5 | Hearst pair | 9office | 1Agencies | hearst_pairs, such_as, hyponym span 7 to 9 |
| 5 | root mood | 10publish | none | root_mood, mood Ind |
- negation
- modal
- clausal complement
- root adverbial
- Hearst pair
- root mood
Each primitive is a field matra fills on Sentence from its dependency tree: the tinted
word is the one it is read from, the underlined word the one it attaches to. The root mood is read
off the whole root clause: the Mood feature on the root or its auxiliary. What any of
it means for a text is the reader's call; matra reports the construction.
Input: Five sentences, one structural primitive each: a negation, a modal, a reporting, a root adverbial, a Hearst pair, matra contributors. MIT, as part of the matra repository.
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: primitives/primitives.json.
Programming model
Two values: Input into Engine
Input is one document. Engine is what happens to it.
| Value | Constructors | Meaning |
|---|---|---|
Input | text(string, format), bytes(data, format), file(path), file_as(path, format), blocks(blocks) | one document: text in a named format, bytes from anywhere, a file read as its name says or as a format you name, or headings and paragraphs you supply |
Inputs | dir(path) | the files in a directory, an iterator of Input::file |
Engine | new(), from_config(cfg), from_config_with_notice(cfg, notice) | the assembled pipeline: UDPipe with its pinned model, and the readers this build ships |
Source variation lives in the data, not in the function namespace, which is why Engine::analyze is one function rather than one per source and format.
Two ways to run it:
| Call | Does |
|---|---|
analyze(input) | one document, end to end, returning a Document |
analyze_all(inputs) | a lazy iterator, one result per input, in order; partitioning successes from failures is the caller's |
And what to ask of the result, with free functions over the Document that need no engine and no parser:
| Call | Returns |
|---|---|
summarize(&doc, n), summarize_with(&doc, n, SummaryMethod) | Summary: the n sentences that best represent the document, in document order |
keyphrases(&doc, n), keyphrases_with(&doc, n, KeyphraseMethod) | Keyphrases: at most n phrases, highest score first, each with every place it occurs |
clusters(&doc, &embedder, threshold) | Clusters: sentences grouped by the similarity of their embeddings, with the embedder you pass |
The method is optional and never in a function's name; every result records the method that ran and its parameters in its provenance, because which method ran decides what a score means.
Reads are lazy. An Input reads nothing until the engine reaches it, and Inputs::dir lists entries up front, a listing failure being its Err, but reads no file until the iterator is pulled. A per-file failure (unreadable, oversized, a symlink, bytes that are not text) arrives as an Err item carrying the path, so one bad file cannot abort a directory walk.
Behind analyze, the input side is cut by decision, one crate-private module each: fetching the bytes (with the symlink and size guards), detecting the format (a PDF or DOCX signature, then the format you named, then the extension), reading one format into blocks with their spans, building the section tree from blocks, and parsing. A new format is a reader and a registration line; no published signature moves.
Inside the engine, analysis is two stages: reading and parsing (annotate), then measuring (compose). Both are crate-private, since a public parse-only call would publish a second shape of Document with every metric empty, and the library's tests pin that analysis is exactly the one followed by the other. Measuring is total: it reads what is attached and skips what is not, so it has no failure path.
One short Markdown document at each stage, as matra produces them:
One document through the pipeline
1 Input Input::text adds the text and its format.
# Field notes The survey team reached the northern ridge on the third day. Snow had closed the lower pass, so the route ran along the eastern face instead. > Radio contact was lost for six hours. ## Findings Three of the five markers were intact and matched their recorded positions. The fourth had been moved about ten metres downhill by a rockfall, and its base plate was cracked. The fifth could not be found, although the team searched the slope below it for most of an afternoon. A replacement marker will be set on the next visit, once the pass is open.
2 Read and parse Engine::analyze adds sections, paragraphs, sentences, tokens.
- section # Field notes
- ¶1 The survey team reached the northern ridge on… 2 sentences 29 tokens measures not computed yet
- ¶2 Radio contact was lost for six hours.… quote, not parsed
- section ## Findings
- ¶3 Three of the five markers were intact and… 4 sentences 72 tokens measures not computed yet
3 Measure Engine::analyze adds the measures.
- section # Field notes
- ¶1 The survey team reached the northern ridge on… 2 sentences 29 tokens grade 4.91 density 0.65 compression none
- ¶2 Radio contact was lost for six hours.… quote, not parsed grade none density none compression none
- section ## Findings
- ¶3 Three of the five markers were intact and… 4 sentences 72 tokens grade 6.88 density 0.48 compression 0.59
Document: vocabulary TTR 0.69, nominalization ratio 0.01, passive ratio 0.50
show the data
| ¶ | Quote | Sentences | Tokens | Grade | Density | Compression |
|---|---|---|---|---|---|---|
| 1 | 2 | 29 | 4.91 | 0.65 | none | |
| 2 | yes | 0 | 0 | none | none | none |
| 3 | 4 | 72 | 6.88 | 0.48 | 0.59 |
Each stage is what matra holds at that point, run by matra on the text shown: one call, Engine::analyze, reads and parses, then measures, and returns the last. The quote
is kept as a paragraph but never parsed, so it has no sentences and no measures. A measure whose
applicability conditions a paragraph does not meet stays None after measuring, shown as
none.
Input: Field notes: a short Markdown document with a heading, a quote and two sections, matra contributors. MIT, as part of the matra repository.
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: field-notes/pipeline.json.
Configuration
The engine needs to know where the UDPipe model lives. Config answers that question once so no caller has to invent a path.
let engine = Engine::new()?;
let engine = Engine::from_config(&Config::resolve()?)?;
Engine::new() resolves the locations from the environment alone and reads no configuration file; Engine::from_config(&Config::resolve()?) reads the file as well. An explicit directory, Config::with_model_dir, wins over everything below.
Config resolves locations and defaults, never behavior. It carries the model directory, the embedding model's directory name, and the default counts, methods and cluster threshold the command line uses, which are all things a caller could pass as arguments instead. It does not carry which metrics run or how output is shaped. A key 0.3.0 renamed or removed ([summarize], algorithm, [semantic], models.udpipe) is refused with Error::InvalidInput naming what replaced it, so a config file from 0.2 fails to load rather than half-load.
Resolution order, per key: an explicit argument, then the environment, then the config file, then the defaults compiled into the crate from config/default.toml. The argument rung has one producer, Config::with_model_dir, which returns the same configuration with the model directory replaced and that one key's source changed. It is the rung a command line's --model-dir lands on, so the flag reaches the adapter through Config rather than past it.
| Variable | Overrides |
|---|---|
MATRA_CONFIG_FILE | the config file path |
MATRA_DATA_DIR | the data root |
MATRA_MODEL_DIR | the model directory |
Paths follow the XDG conventions on Linux and on macOS.
| What | Default | Env override |
|---|---|---|
| config file | $XDG_CONFIG_HOME/matra/config.toml, else ~/.config/matra/config.toml | MATRA_CONFIG_FILE |
| data root | $XDG_DATA_HOME/matra, else ~/.local/share/matra | MATRA_DATA_DIR |
| models | the data root's models subdirectory | MATRA_MODEL_DIR |
One exception to the model directory rule: if the data root has no models subdirectory yet, and ~/.matra/models exists and is not empty, that older location is used. matra never creates ~/.matra. When an existing, non-empty legacy cache is selected it is used as the model directory, downloads and re-downloads included, because the resolved directory is the one handed to the UDPipe adapter and that is where it writes. Create the new location, or set MATRA_MODEL_DIR, to move off it.
An empty ~/.matra/models is not selected. There is no cache in it to keep working, and picking it would send every later download into a directory matra would otherwise never create. A fresh install therefore lands in the new place.
A missing config file is not an error; the built-in defaults stand. A malformed one is Error::InvalidInput naming the file and the line, and an algorithm name the build does not know is rejected the same way at resolve time rather than at call time. Every resolved value records which rung it came from, readable through Config::sources as a ValueSource.
annotate is the only route to the parser
The engine's reading stage, annotate, runs the size check on every Input, then hands each paragraph that is not a quote to the parser (NlpProvider::parse, a crate-private trait), and maps the token and sentence spans the parser reports against the paragraph's text into the analysed text. Nothing else in the library calls the parser. That is what makes the 8 MiB cap a property of the pipeline rather than of each entry point: no text over MAX_INPUT_BYTES can reach the parser, whichever call or constructor you started from.
It is also why the outcomes are cheap. Analyze once, and hand the same document to every outcome function: none of them parses anything. Nothing is parsed twice.
Ports and adapters
The pipeline depends on port traits, and adapters implement them. The domain depends on neither: the ports import the domain, never the reverse.
| Port | Trait | Adapters this build ships | Feature |
|---|---|---|---|
source/ | Source (crate-private) | FileSource; DirectorySource lists a directory for it and reads nothing | none |
decompose/ | Decomposer (crate-private) | the markdown, plain-text and block-input readers | none |
nlp/ | NlpProvider (crate-private) | Udpipe | udpipe (default) |
embed/ | Embedder | Model2Vec | model2vec |
Features are additive. Enabling udpipe adds UDPipe; disabling it removes UDPipe and nothing else. cargo check --no-default-features compiles. The other two features are cli (the command line, which implies udpipe) and python (the PyO3 bindings, which implies cli so that a wheel carries the command).
Every port is open to your own adapter. clusters takes any &dyn Embedder, and with your own implementation it needs no feature flag at all. Engine::with_parser takes a parser of your own (NlpProvider), in any build; with_reader registers a reader (Decomposer) for a format, a Format::Other you name included; and with_source reads the paths inputs name from somewhere else (Source). The engine checks what a parser or a reader returns before it reaches a Document, so a part of yours cannot hand the outcomes a tree with a cycle or a span outside its text. Without implementing anything, bytes you fetch yourself go in through Input::bytes, and a format matra has no reader for goes in as headings and paragraphs through Input::blocks.
Determinism and bounds
Every gate returns a typed variant of Error. There is no Result<T, String> anywhere in the library, and library code does not panic: UDPipe panics at the C boundary are caught in nlp/udpipe.rs and converted.
| Gate | Cap | Variant |
|---|---|---|
reading, every Input | 8 MiB (MAX_INPUT_BYTES) | InputTooLarge, what is "input" |
| file read | 8 MiB | InputTooLarge, what is "file_read" |
summarize, summarize_with | 2,000 sentences | InputTooLarge, what is "tfidf" or "textrank" |
clusters | 2,000 sentences | InputTooLarge, what is "clusters" |
keyphrases, keyphrases_with | 200,000 tokens | InputTooLarge, what is "rake" or "yake" |
| the pinned UDPipe model being downloaded | 64 MiB | InputTooLarge, what is "parser_model_download" |
| one pinned embedding artifact being downloaded | 64 MiB | InputTooLarge, what is "embedding_model_download" |
| the config file, checked against its metadata | 64 KiB | InputTooLarge, what is "config_file_read" |
The what discriminator names which gate fired, so a caller can route a document-too-big differently from a corpus-too-big.
Cycle safety is a value, not a ceiling: Sentence::tree_depth is None on a malformed parse containing a cycle rather than silently truncated.
Fields cross languages, methods do not
Values reach Python through serde and pythonize. A field has a serde representation and crosses with its name unchanged. A method has none, so there is nothing for it to cross with.
Rust-only, therefore: the Document iterators (paragraphs, sentences, tokens); the Sentence tree walks and filters (root_token, head_of, children_of, subtree, content_tokens, clausal_complements_in, root_adverbials_in); Paragraph::word_count and sentence_count; and Token::feat.
When an aggregate needs to be visible cross-language it is materialized as a field. The document's counts, sentence-length statistics and ratios are the fields of DocumentMetrics, which the metric suite fills, and a sentence's word_count, tree_depth and passive are fields set when it is built.
What Python exposes
The same words Rust has, in Python's syntax: Engine, the read-only Document, the outcome functions, Model2Vec, and the entry point the matra console script launches. A Rust Input constructor is a Python keyword, and a Rust _with form is Python's method=.
| Python | Rust equivalent |
|---|---|
Engine(), Engine.from_config(config_file=None) | Engine::new(), and Engine::from_config over Config::from_file(path), or over Config::resolve() with no file named |
engine.analyze(text, format=) | analyze(Input::text(text, format)), plain text when no format is named |
engine.analyze(file=), bytes=, blocks= | analyze on Input::file (Input::file_as with format=), Input::bytes or block input |
engine.analyze_all(dir=), texts= | analyze_all over Inputs::dir, or over Input::text per text, lazily |
doc["metrics"], doc.to_dict() | the fields of Document, through serde |
summarize(doc, n=3, method=), keyphrases(doc, n=10, method=) | summarize or summarize_with, keyphrases or keyphrases_with |
clusters(doc, embedder, threshold=0.85) | clusters |
span_text(text, span) | slicing the analysed text by a Span |
Model2Vec.from_dir, .model_hash, .dimensions, .embed, .identity | Model2Vec behind the Embedder port |
Model2Vec.potion_base_8m | Model2Vec::potion_base_8m; with no argument, over the model directory the environment names, reading no config file, as Engine() does |
Model2Vec.from_config | Model2Vec::from_config over Config::from_file or Config::resolve, as Engine.from_config is Engine::from_config |
any object with embed and identity | an Embedder implementor, wrapped by an adapter in the binding |
_core.cli_main(argv) | cli::run, which is what the matra console script launches |
A stream crosses item by item: analyze_all yields a Document or a DocumentError dict per input, so a failure on one file is one item rather than an exception that ends the walk. And clusters takes any object with embed and identity: the Embedder port was always the extension point, and the binding now reaches it. Model2Vec carries both methods too, so it satisfies the same protocol a caller's own object does rather than sitting beside it as a second accepted type. In the extension module that arm needs no embedding adapter compiled in. The matra package itself still requires an extension built with model2vec: its __init__ imports Model2Vec and raises ImportError naming the feature when it is absent, and every published wheel includes it.
An outcome takes a document you already analyzed, in both languages, so a text is parsed once however many outcomes are asked of it. Python's outcome functions take only a Document, never a dict: a dict could carry provenance no matra run produced. Parsing is the one thing no caller substitutes, in either language, because the size cap and the panic boundary live behind it. The Python binding lives in src/python/ and uses only the public surface, which boundary rule 7 holds it to.
Engine is unsendable in Python, because the UDPipe model holds C-side state that is not thread safe. Multi-process Python is fine; sharing one engine across threads is not.
One core, three install routes
matra is one Rust core published three ways: a library on crates.io, a command-line binary built from the same crate, and a Python package on PyPI. The routes differ in what they compile and what they print about themselves, and not in what an analysis returns. This page explains why the install commands look as they do. Install matra has the steps and Platforms and models the facts.
One program behind every matra command
The matra command the Python package installs is not a Python program. From 0.2.0 it is the Rust command line, compiled into the library and reached through the extension module, so pip install and cargo install give you the same program with two launchers. 0.1.0 had a second command line written in Python, with a --json shape of its own and a model directory hardcoded to the pre-0.2.0 location. Two implementations of one interface drifted apart, which is why the command line now lives in the library: a binary target cannot be reached from Python, and a library module can.
Why every install line names a version
An unpinned install line takes whatever the registry hands out. Today that is 0.3.x on both registries, and it will not stay that way. 0.1.0 is also still published, and its matra is the separate Python implementation above. So is 0.2, whose surface 0.3.0 renamed throughout. The failure an unpinned line risks is not an error: the install succeeds and gives you a program these pages do not describe, so its version banner and its output do not match theirs.
The Python lines take a floor, 'matra>=0.3', because what they claim holds for 0.3.0 and every release after it, and an exact pin would still name 0.3.0 after a later release ships. The cargo lines take a caret requirement, matra@0.3 and --version '^0.3', which is how cargo states a version. Cargo reads a change in the first non-zero component as breaking, so the caret accepts 0.3.0 and later 0.3 releases and stops short of 0.4.0, which a floor would not.
Same program, different builds
The two banners differ, and both are correct. The CLI binary from cargo install is built with the features the command line needs, udpipe and cli. The wheel is built with python and model2vec on top, because the Python package also carries the library, embedding included. matra --version prints the version and then the features of the build that is running, so a report quoting it says which build produced a result. Every analysis command behaves the same in both.
Free-threaded CPython is the one interpreter the wheels do not reach, and the reason is a tag, not a platform. The wheels are built once per platform against the CPython stable ABI and tagged abi3, which every GIL-enabled CPython from 3.12 accepts. A free-threaded interpreter accepts abi3t instead, so it falls back to building from source even where a wheel exists.
Why the first run is quiet in the library
The first call that needs the parsing model downloads it, about 16 MB from a university server, and that can take from a few seconds to over half a minute. The command line says so on stderr before it starts, because a blank terminal for that long is indistinguishable from a hung process. The library says nothing, because a program embedding matra chooses its own reporting: the constructors that take a notice callback hand it the same facts, and the program decides what to do with them.
What the clustering threshold does
matra can group the sentences of a document that say the same thing in different words. It embeds each sentence as a vector, scores every pair by cosine similarity, and links each pair whose score clears a threshold you choose. The clusters are the connected groups those links make. Cluster sentences by meaning has the calls.
Everything else matra returns is structure read off the text, checkable against its bytes. Clusters are not: they depend on a model's representation of meaning. So they arrive as their own value from a separate call, never as a field on Document, and their provenance carries the identity of the model that produced them and the threshold you chose.
Three things the shape means
- Co-membership is transitive, not pairwise. Clusters are connected components, so sentence A and sentence C can share a cluster because both resemble B, without resembling each other. The edges travel in the result so you can see which pairs actually cleared the bar. A missing edge is no claim, not a low score.
- Singletons are always excluded. A sentence with no pair above the threshold appears in no cluster, so "not in any cluster" is a meaningful count.
- The threshold is yours, and it does not travel. Published cutoffs for paraphrase detection span 0.67 to 0.9 with no consensus; the working value depends on the model, the domain and the text length. A cutoff calibrated on sentences is not the cutoff for whole documents, because a document vector is the mean over far more tokens.
Watch the threshold move
Ten short sentences, with the reference model. Before you move the threshold, predict: sentence 8 is about a different decision by the same committee. As the threshold falls from 0.85, at which value do you expect it to join sentences 1, 2 and 3, and through which of them? Then drag the threshold and find out.
Semantic clusters as the threshold moves
- 1The committee approved the budget on Tuesday.
- 2On Tuesday the committee passed the budget.
- 3The budget was approved by the committee.
- 4Heavy rain closed the mountain road overnight.
- 5The mountain road was shut overnight by heavy rain.
- 6The new release fixes a crash when saving large files.
- 7Saving large files no longer crashes the program in this release.
- 8The committee approved a new park plan on Monday.
- 9Bread rises faster in a warm kitchen.
- 10The museum opens at nine on weekdays.
At 0.85, matra's default: 3 clusters, 3 of 10 sentences in none.
show the data
| Threshold | Clusters | Edges (cosine) |
|---|---|---|
| 0.50 | {1, 2, 3, 8} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 2-8 (0.5646), 3-8 (0.6405), 4-5 (0.9592), 6-7 (0.8590) |
| 0.55 | {1, 2, 3, 8} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 2-8 (0.5646), 3-8 (0.6405), 4-5 (0.9592), 6-7 (0.8590) |
| 0.60 | {1, 2, 3, 8} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 3-8 (0.6405), 4-5 (0.9592), 6-7 (0.8590) |
| 0.65 | {1, 2, 3, 8} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 4-5 (0.9592), 6-7 (0.8590) |
| 0.70 | {1, 2, 3, 8} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 4-5 (0.9592), 6-7 (0.8590) |
| 0.75 | {1, 2, 3} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 4-5 (0.9592), 6-7 (0.8590) |
| 0.80 | {1, 2, 3} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 4-5 (0.9592), 6-7 (0.8590) |
| 0.85 | {1, 2, 3} {4, 5} {6, 7} | 1-2 (0.8521), 1-3 (0.8812), 4-5 (0.9592), 6-7 (0.8590) |
| 0.90 | {4, 5} | 4-5 (0.9592) |
| 0.95 | {4, 5} | 4-5 (0.9592) |
- in a cluster, named by its letter
- in no cluster
- what you point at or tap, with its cluster
Each arc is a pair of sentences whose cosine similarity cleared the threshold, labelled with the score. A cluster is a connected group, so two sentences can share one through a third without their own pair clearing the bar. The embeddings are fixed; only the threshold moves, over the values matra was run at, and between two of them the figure blends the two runs rather than compute one of its own. matra ships 0.85 as a starting point, not a verdict: which threshold is right depends on the text and the question.
Input: Three things said two or three ways each, and three said once, matra contributors. MIT, as part of the matra repository.
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3) and
the embedding model potion-base-8M (SHA-256 81c359215087). Data: paraphrases/clusters.json.
What sentence 8 did
It joins at 0.70, through sentence 1 alone: their pair scores 0.7239 and clears the bar, while its pairs with sentences 2 and 3 do not, yet it shares their cluster. A cluster can grow through a chain of pairs, on shared words ("committee approved") rather than shared meaning, and the clusters dissolve as the bar rises past 0.85.
Why a static model
The reference model is a static embedding model, a lookup table rather than a transformer: inference is a row gather, a mean, and a normalize. That costs roughly ten percent of a small transformer's benchmark quality, and buys bit-identical vectors on every platform and in every language binding, which is what lets the conformance suite pin exact vectors rather than tolerances.
Pragmatics
Which call to reach for, and how to read what it returns: the choices behind the summarizers, the keyphrase extractors, the document measures, the clustering threshold and the models, and why each answer is what it is. The caps each call enforces are listed under InputTooLarge; handing matra to an agent is its own guide.
Choosing a summarizer
summarize returns the top N sentences in document order, as a Summary whose sentences carry the score and their index in the document, and whose provenance names the method. summarize_with chooses the method.
TF-IDF (SummaryMethod::TfIdf, the default) scores each sentence by the mean TF-IDF of its lemmatized terms, treating every sentence as its own document for the IDF computation. It runs in linear time, and it favours sentences carrying vocabulary that is rare in the rest of the document.
TextRank (SummaryMethod::TextRank) builds a similarity graph over the sentences (shared content lemmas divided by the log of the two lengths, so long sentences are not favoured) and runs iterative PageRank scoring over it. It favours sentences many others resemble. The dense similarity matrix is quadratic in sentence count, which is why the 2,000-sentence cap exists.
Rare-in-this-document against central-to-this-document is the whole difference. On a document with one repeated theme they tend to agree; where they diverge, TF-IDF has found the unusual sentence and TextRank has found the typical one.
Choosing a keyphrase extractor
keyphrases returns Keyphrase values, highest score first, each with every place it occurs, and the provenance names the method. keyphrases_with chooses it.
RAKE (KeyphraseMethod::Rake, the default) splits at stop words and punctuation, keeps runs of NOUN, ADJ, and PROPN tokens as candidates, and scores each by the co-occurrence degree over frequency ratio. Multi-word phrases are what it is built to find, and it needs no corpus.
YAKE (KeyphraseMethod::Yake) scores individual terms on position, frequency, and context diversity, then assembles 1-word to 3-word candidates. YAKE's own score is lower-is-better, and matra inverts it so higher is more relevant in the returned values.
Their caps differ from the summarizers': both are bounded on total tokens rather than sentence count, because a corpus of many one-token sentences costs them nothing.
Reading a Document
| Question | Read |
|---|---|
| Is it passive-heavy? | DocumentMetrics.passive_ratio, the share of sentences carrying a passive construction |
| Is it dense? | ParagraphMetrics.lexical_density for the content-word share, alongside readability_grade |
| Is it repetitive in wording? | ParagraphMetrics.compression_ratio, where a lower value means the text compressed further and so repeats more on the surface |
| How varied is the vocabulary? | DocumentMetrics.vocabulary_ttr, but see the length caveat below |
| Is it repetitive in paraphrase? | none of the above: lexical measures cannot see restatement in different words. That is what Clusters is for |
| How nominal is the style? | DocumentMetrics.nominalization_ratio |
Two cautions. vocabulary_ttr is a raw type-token ratio and falls as text grows, so two documents of different lengths are not comparable on it without normalizing first. And any of these slots can be None, which means the metric was not computed (too short, a quote, or over a per-metric byte ceiling) and is distinct from a computed zero.
The semantic threshold
clusters takes the cosine threshold from you rather than choosing one, and records it in the result's provenance. Published cutoffs for paraphrase detection span 0.67 to 0.9 with no consensus, and the working value depends on the model, the domain, and sentence length.
Raising the threshold admits fewer edges: clusters get smaller and more of them, and more sentences end up in none. Lowering it merges components through chains, because clusters are connected components rather than cliques. That transitivity is why the clearing edges travel beside the clusters: two sentences can share a cluster without a direct edge, and the edge list is where you check.
Start around 0.85 with the reference model and calibrate on your own corpus.
Provisioning models
Both models work the same way, and the constructor you reach for is what decides whether anything is fetched.
Ask for nothing and you get the pinned model in the resolved place. Engine::new(), Engine::from_config(cfg) and Model2Vec::from_config(cfg) resolve the directory through Config and fetch the pinned model when it is absent. In Python that is matra.Engine() and Model2Vec.potion_base_8m() with no argument, which read the environment and no configuration file, and Engine.from_config() and Model2Vec.from_config(), which read one. This is the path to take unless you have a reason not to.
Naming the directory changes where, not what. Engine::from_config(&cfg.with_model_dir(dir)), Python's matra.Engine() under MATRA_MODEL_DIR, and Model2Vec::potion_base_8m(dir) fetch the same pinned artifacts into the models directory you name, each model in its own place there. Take them when the model belongs somewhere specific.
Downloading is pinned, or it does not happen. Either form fetches from URLs written in the source and loads nothing whose digest does not equal a constant written in the source next to them. Verification happens before the bytes are parsed, and the bytes that were verified are the bytes that get parsed, with no second read of the disk in between. A download that fails the digest is fetched once more into memory, and a second failed download raises; nothing that failed the digest is written at any point, so a failed attempt leaves nothing behind for a later call to pick up. Exactly one artifact set can arrive this way, which is what makes "matra downloads a model" a bounded claim rather than an open one.
A provisioner never deletes what it did not write. The embedding artifacts are named for their format (model.safetensors, tokenizer.json, config.json) rather than for one model, so a directory can already hold somebody else's. Model2Vec::potion_base_8m(dir) therefore downloads only into a directory holding none of the three. Finding all three with a digest that does not match, or finding only some of them, it raises and names the directory, having downloaded nothing and removed nothing. The UDPipe model file is named for its own release, so its provisioner has no such collision to guard against.
Bringing your own. Model2Vec::from_dir(dir) loads what you supply and never reaches the network, whatever the directory holds. That is the path for a model2vec artifact this build does not pin, from either language. The parsing model is a pin on every surface: no surface loads another UDPipe model file. Nothing verifies it, so the model_hash on a result is identity rather than proof: it tells you which artifacts produced a score, not that they are the ones you meant to fetch.
The pinned model is part of the contract in every case. A different model produces a different parse or a different vector space, and the conformance fixtures will fail, correctly. The semantic clusters reference records the embedding digest the conformance suite pins.
The expensive thing is the model, not the call: load an Engine once and reuse it.
How matra runs
engine.analyze(Input::file("essay.md")) yields a Document or an Error.
One call, end to end
Take a 40 KB markdown file. Analysing it runs five things in order, each a module hiding one decision, and only the last two are expensive.
Fetch. Engine::annotate reads the file through the engine's source, FileSource unless the caller supplied another with Engine::with_source. FileSource calls symlink_metadata, which does not traverse. A symlink is refused, and a directory is refused naming Inputs::dir. Anything else that is not a regular file is refused. A file whose metadata size exceeds MAX_INPUT_BYTES, 8 MiB, is refused before a byte is read, so a 1 GB file costs one stat call rather than a GB of resident memory. Then all 40 KB are read in as bytes.
Detect. The bytes are checked against the cap again (the same limit under the "input" label, because bytes and text handed in directly never passed the first gate), then given a format: a PDF or DOCX signature first, then a format the caller named, then the extension, so .md is Format::Markdown. A format with no reader in the engine's table is Error::UnsupportedFormat. Otherwise the reader registered for the format gets the bytes; a text reader decodes them as UTF-8, and bytes that are not are Error::UnsupportedFormat too. Unless the reader names an analysed text of its own, as a PDF reader would, the bytes become the analysed text, with no copy.
Read. The markdown reader runs a CommonMark parser over the text once, and turns its events into headings and paragraphs, each with its span in the text and its own text with the inline markup removed: a second copy of the prose. Front matter, code, tables and HTML are recorded in skipped with their spans rather than dropped. The engine then checks what the reader returned against the reader's contract: every span inside the text and on character boundaries, blocks in document order, no two spans overlapping, every heading at level 1 or more, and every paragraph's text lined up with the text its span covers. The structure step then makes the blocks a section tree, a heading opening each section.
Parse and measure. annotate walks every paragraph in document order, skips the quotes, and calls the parser on each remaining paragraph's text. Sixty body paragraphs means sixty calls into UDPipe. Each call returns sentences of tokens, and the engine checks them before it uses them: ids 1..=n, every head in range, one root, no cycle, and every span a span of the paragraph, in order, over its token's form. Then it maps the spans from the paragraph's text into the file's bytes and builds each Sentence itself, its text and the fields derived from its tree included, so no derived field depends on which parser ran. Then compose runs four metric functions over the attached tree; the document-level pair aggregates over the same attachment, so there is no second sentence set to keep in agreement.
At peak, three representations of the same document are resident at once: the original String, the paragraph copies, and the parsed tokens hanging off the paragraphs.
For a 40 KB input none of that matters. For a caller sizing a batch job against a memory ceiling it does, because every Token carries nine owned Strings.
A directory changes the shape by streaming. Inputs::dir lists the entries up front but reads nothing until pulled, and each pull of analyze_all runs one document through the whole pipeline before the next file is read. One document's representations are resident at a time; only what you retain accumulates, and collecting the stream retains everything.
The model is the expensive thing
Everything above assumes nlp already exists. Constructing it is the one genuinely slow operation matra performs.
Udpipe::english(dir) looks for the English UD-EWT model in the directory you name. If it is absent, the bytes are verified in memory and then written to a temporary subdirectory named for this call, and moved into place with a single rename, so two processes pointing at the same directory cannot leave each other a half-written file. The name carries more than the process id, because in a container every process is pid 1 and two cold starts would otherwise collide.
Udpipe::from_config(&cfg) is the same call with the directory resolved rather than named, and it is what the public constructors reach: Engine::new() with a Config resolved from the environment alone, Engine::from_config(&cfg) with the one you pass. Config is read once, at construction; nothing in the call path above consults it, which is why configuration cannot change what a call computes. Model2Vec::potion_base_8m(dir) and Model2Vec::from_config(&cfg) do the same for the reference embedding model, against a three-file digest rather than a single one.
The cached file is then checked against a size constant, 16,309,608 bytes, and a SHA-256 constant pinned in nlp/udpipe.rs. A size mismatch fails before the file is hashed. A mismatch downloads the model, and a download that also fails the digest is retried once, so a call makes at most two downloads. The file is replaced only if downloaded bytes verify; if both downloads fail, the call returns Error::ModelInvalid and leaves the old file untouched. An unverified model is never loaded.
The verify step returns the bytes it hashed, and those exact bytes go to the loader. Nothing re-reads the disk in between. That ordering is the whole point: a hash-then-reopen sequence leaves a window in which an attacker with write access to the model directory can swap the file after the check and before the read.
No surface loads a model file the caller names. The pinned model is the only one a parse can come from, so the identity on every result is proof as well as a name.
Load once, reuse. After construction, the model is a live handle to a C++ object. Parsing does not reload it. The Engine owns the provider, so a service builds one engine and keeps it for its lifetime.
One engine per thread. The underlying Model is Send and deliberately not Sync: its parse path mutates internal workspace caches. The Engine that owns it is Send and not Sync for the same reason: an engine moves to another thread, but sharing one between two is a compile error rather than a race. To parallelize you build an engine on each thread, paying another 16 MB load for each, or serialize calls behind a mutex.
Python gets the same constraint through a different mechanism. Its Engine is declared unsendable, so PyO3 records the creating thread and refuses access from any other at runtime. A ProcessPoolExecutor is unaffected, because each process loads its own model.
The unit of work is the paragraph
Parsing per paragraph looks wasteful. Sixty FFI calls where one would do, and a parser that never sees a paragraph's neighbours as context. The reason it is worth that is a defect class the alternative could not avoid.
The earlier implementation joined every paragraph into one string, parsed once, then wired each returned sentence back to a paragraph by matching text prefixes. Three regression tests in src/lib.rs hold the current behaviour in place, one per way that matching failed:
- Two paragraphs sharing their first thirty characters. A sentence lands in whichever one matched first, and the other paragraph silently loses it.
- One paragraph containing another paragraph's opening words mid-text. The greedy match steals the sentence from its rightful owner.
- An empty paragraph with trailing whitespace, which confused the match entirely.
Follow that down and the root is not the matching algorithm. It is that the relationship between a sentence and its paragraph was being reconstructed from text, and text does not carry identity. Two paragraphs can be character-for-character similar and still be different paragraphs.
So any reconstruction from that data has to guess, and a guess in this position fails silently. The Document that comes out is still well-formed, still serializes, still passes every shape check. It is simply wrong about which paragraph said what, and nothing in the output says so.
The two identical sentences are the ones from the regression fixture in src/lib.rs. On the left nothing in the text distinguishes them, so the assignment is a coin toss and one sentence is left over. On the right the arrows run the other way: the call produced the sentences, so the owner is known before any text is compared.
Per-paragraph parsing deletes the wiring step instead of improving it. A paragraph's sentences are the return value of the call made on that paragraph's text. The relationship comes from the call graph, so there is nothing left to get wrong.
Blockquote paragraphs are skipped at this stage, which is why they reach the end with no sentences and all three metric slots at None. And a parse failure in any single paragraph aborts that document with Error::ParseFailed; the partial document is dropped, not returned, and in a stream the failure travels as that document's DocumentError while the next document proceeds.
Measure fills slots, extract is a separate call
The metric suite is four functions with one signature, Box<dyn Fn(&mut Document)>, run in sequence by compose. Readability, lexical density, and compression write per-paragraph slots. The document pass writes vocabulary_ttr, nominalization_ratio and passive_ratio. None of the four reads another's output, so the suite is a list, not a chain.
Each carries its own applicability condition, and this is where unexplained None values come from:
| Slot | Written when |
|---|---|
readability_grade | more than 10 words, not a quote |
lexical_density | at least one word, not a quote |
compression_ratio | more than 50 words, not a quote, paragraph text at most 256 KiB |
vocabulary_ttr, nominalization_ratio | at least one word in the document |
passive_ratio | at least one sentence in the document |
A word here is a non-punctuation token, the count Paragraph::word_count returns.
None is not zero. A three-word paragraph has no meaningful Flesch-Kincaid grade, and the slot says so rather than reporting a number nobody should use. The compression cap is a CPU bound: the brotli window is 2^18 bytes, and a paragraph larger than one window is skipped instead of pegging a core on adversarial input.
Summarization and keyphrase extraction are not part of this. The pipeline never calls them. They are free functions over the analysed Document, invoked by the caller, and they need no engine:
let doc = engine.analyze(Input::file("essay.md"))?;
let summary = matra::summarize(&doc, 3)?;
let phrases = matra::keyphrases(&doc, 10)?;
Keeping them out of the pipeline is a cost decision. TextRank builds a dense sentence-similarity matrix that is quadratic in sentence count, roughly 32 MB of f64 at its 2,000-sentence ceiling. Folding it into the analysis pass would charge that to every caller who only wanted a readability grade. Inputs above a cap return Error::InputTooLarge rather than allocating.
What can fail, and where it surfaces
Every guard sits at the earliest point where the check is still cheap.
Three separate lanes, because these are three separate call paths. Only the middle one runs during a pipeline call. The model check fires once when you construct the engine, and the outcome caps fire only when you call an outcome function, which is why neither shows up in a pipeline stack trace.
The catch_unwind is the one that matters. A panic crossing an FFI boundary does not unwind into your code, it aborts the process: interpreter death in Python, a trap in WASM, from a library that promised typed errors.
The boundary that converts such a panic into an Error lives in nlp/udpipe.rs, and that file is the only one permitted to import the UDPipe crate. A second import path would be a second process-abort surface with no boundary in front of it. That boundary covers parsing only; model loading translates the loader's own error and is not wrapped.
Every cap a gate checks, and every minimum a metric needs, is a constant in domain::limits, defined once and read where it applies. It sits inside the domain, the one module every row may import, so the gates in the adapters, the outcome functions and the metrics all read the same number without boundary rule 5 bending for it.
The parser and the readers are ports anyone can implement, so the engine trusts none of them, matra's own included: what a reader returns is checked once per document, and what the parser returns once per paragraph, before either reaches a Document. A break is Error::ParseFailed with a reason naming the check and a cause naming the token or block (Errors lists the checks). Each check is a pass over what it reads. Measured over the docsite's inputs and its own reference and guide pages, nearly two thousand paragraphs, the parse checks cost about 0.014% of UDPipe's parse time, roughly 3 microseconds a paragraph, and the reader checks about 7% of the readers' own time, under a tenth of a millisecond for a 140 KB page: together 0.015% of reading and parsing. Every document an engine returns therefore holds the invariants Sentence documents, whichever parts built it.
Two guards live in the domain types rather than at an edge, because a document can arrive from outside the engine, read back from JSON or built by hand, and its shape is not guaranteed. Sentence::tree_depth is computed by walking head references with a visited set, and is None when they form a cycle. Sentence::subtree carries the same visited set so it terminates. The None is deliberate: an earlier version capped depth at a magic 20, which silently truncated malformed parses and legitimate deep ones alike, and a sentinel number would read as a very deep tree to anyone who did not check for it.
Directory reads fail differently. analyze_all yields successes and per-document failures side by side, so one unreadable file does not cost you the other ninety-nine; Inputs::dir is Err only when the listing itself fails. Full detail on every gate, every variant, and the Python exception each maps to is in Errors.
What you can replace
Every arrow points down, and that is the rule: a module imports from the row below it, never the row above. One arrow runs sideways: src/bin/matra.rs reaches into src/cli/, which is the whole of that file: the command line lives in the library so both launchers run one program, and the binary only collects arguments, locks the streams, and returns the code. One arrow from src/cli/ reaches past lib.rs, and it still points down: matra clusters builds Model2Vec from the configuration, because the library builds no embedder inside an outcome call and choosing the default one is the application's decision. Boundary rule 7 admits embed for the command line as it does for the Python binding; the other three ports are public too, and neither needs them, so the rule keeps them out of both.
src/config.rs sits in the composition-root row beside lib.rs, and it is the one box an adapter is allowed to import from the row above. Udpipe::from_config and Model2Vec::from_config take a &Config, which is the traffic RFC-0011 intends and boundary rule 7 records: Config imports no port and no adapter, so nothing travels back down with it, and an adapter that reads it is reading a resolved path rather than reaching for the environment itself.
Config is not one of the extension points below, and the distinction is worth keeping. An extension point substitutes an implementation; Config only supplies values a caller could have passed as arguments. That is why it carries locations and defaults and never behavior: the moment a file selects which metrics run, configuration stops being adaptability and becomes dispatch that nothing type-checks.
Every port is open to outside implementers, and each is one or two methods over domain types. NlpProvider has two, parse and identity: a paragraph in, sentences of tokens out, and who parsed them, which the engine records in every document's provenance. Decomposer has one, decompose: a format's bytes in, blocks and skipped constructs out, built with Read::new and ReadBlock::heading and ReadBlock::paragraph. Source has one, read: a path in, bytes out. Embedder is the same shape one tier over: clusters takes it as &dyn Embedder, and its identity travels into every result's provenance so the scores stay attributable to the model that produced them. That is what makes UDPipe an implementation detail rather than the library.
The engine holds a parser, a table mapping each Format to a boxed reader, and a source, and a caller replaces any of them without forking matra. Engine::with_parser(parser) builds an engine around a parser of your own, in any build, with the shipped readers and the file source; .with_reader(format, reader) registers a reader for a format, replacing a shipped one, supplying one for Format::Pdf or Format::Docx, or reading a Format::Other you name; .with_source(source) reads the paths inputs name from somewhere else. Both methods compose with the standard constructors, so Engine::new()?.with_reader(Format::Pdf, MyPdf) is UDPipe with your PDF reader. It is one constructor and two consuming methods rather than a builder type: every part but the parser has a default, so there is nothing for a builder to hold that the engine does not already hold. Inputs::dir still lists the local filesystem, and Input::bytes and Input::blocks reach no source and no reader, so a caller with bytes or blocks from anywhere needs neither. Bring your own parser or reader walks through one of each.
The traits are frozen once someone implements them, since a new method would break every implementation, so they ask for only what the engine cannot derive. A parser returns tokens, not Sentence values: the sentence's text, span and every field derived from its tree are the engine's, computed the same way whichever parser ran. A reader returns opaque blocks, so the block vocabulary can grow without breaking one. And neither trait promises well-formed output in its signature; the engine checks it instead, at the one place each kind of output enters.
The metric suite is data as well, inside the crate: a list of functions over &mut Document that compose runs. It is private, so a new measure is a function and a line in the list. A caller who wants another measure writes a function over the Document it was handed, which is all a metric is.
What you gain from the arrangement is concrete and mostly shows up in test suites. Nothing under metrics/ or extraction/ imports a port, which is the one column above that skips the port row entirely, so those functions run with no model loaded. A new metric is testable against hand-built Sentence values, with no 16 MB download and no C++ toolchain. The rules that keep it that way, and how weakly some of them are enforced, are listed in Boundary rules.
Crossing into Python copies everything
The Rust side runs the same pipeline, then hands the Document to pythonize, which walks the serde representation and builds a Python dict. That is a full deep copy. Every token becomes a dict of eleven keys. For a document with seven thousand tokens, that is seven thousand dicts allocated on the Python heap.
Fields cross. Methods do not. A Rust method has no serde representation, so it has nothing to cross with, which is why every aggregate a reader needs is stored: compose writes the counts, sentence-length statistics and ratios into DocumentMetrics and ParagraphMetrics, and Sentence::new sets each sentence's word_count, tree_depth and passive (RFC-0008). A Python caller reads them rather than recomputing them. Domain types draws that boundary member by member.
Errors cross by type and as data. Each variant is raised as a class of its own, named as the variant is and deriving from matra.MatraError, and the exception carries kind and details, the values Error::kind and Error::details compute, so a method's answer crosses as a field. The choice of class is a match with no wildcard arm, so adding a variant to domain::Error fails to compile until someone decides which class it becomes. A wildcard would let new failure modes fall through to another variant's class unnoticed.
Every Python method that takes text routes through the same pipeline, so the 8 MiB text cap fires uniformly. The outcome methods add their per-method caps on top.
Where to look next
What matra gives you is the output side: every value the pipeline produces and what each one measures.
Domain types is the type graph: what each type owns, which values are stored and which are computed on demand.
Boundary rules is the enforcement side: the eight rules, why each exists, and which of them have a mechanical gate.
How matra is maintained
matra is maintained by its owner and by Claude, Anthropic's AI, working through a set of agents, skills and checks that live in the repository beside the code. This page explains that arrangement for a reader outside it: someone deciding whether to depend on matra, someone thinking of contributing, or someone who wants to borrow the pattern. The mechanics of opening a pull request are in CONTRIBUTING.md; this page says why the project is run the way it is.
Two roles
The owner decides, and Claude does most of the engineering. The owner brings what is outside the codebase: what is worth building, when to ship it, and the judgment calls that depend on context no file holds. Claude brings the convergent work inside a frame: drafting, implementing, reviewing and keeping a large codebase consistent with its own rules. Neither substitutes for the other, and the working rule is to think with, not for: a claim that does not hold is challenged, from either side, with the evidence for the challenge.
| Who | What they do | What only they do |
|---|---|---|
| The owner | Sets direction, reviews what matters to them, comments on records in progress | Accepts an enhancement proposal by merging the change of its status to accepted, accepts an API change proposal, approves each release |
| Claude | Writes code, docs and proposals; reviews; merges; prepares and dispatches releases | Nothing the owner cannot also do. Claude merges every pull request except one that accepts a proposal, each with a comment giving its reason |
The agents
The work is split across seven agents, each a Claude Code subagent defined by one file in .claude/agents/. They are not separate people or separate models: each is Claude with a different set of instructions, and what makes one worth having is that it reasons in a way the others do not. Every agent file states the method it owns and what it refuses to do, and no two agents own the same method.
| Agent | Its method | What it leaves to others |
|---|---|---|
| maintainer | Orchestration: holds the whole codebase in view, makes the call on a change, authors and argues proposals, and hands each part of the work to the agent whose method fits | Accepting its own proposals, which only the owner does |
| reviewer | Falsification: steel-mans a change, then demands evidence for it, against the review gates | Resilience findings, and writing the fix |
| portsmith | Minimal-contract design for the four port traits, and the test for moving one into its own crate | Adapter internals that leave a trait unchanged |
| ffi-keeper | Keeping the Rust and Python surfaces in agreement: what crosses, how errors map, how the packages are built and pinned | Rust-only surface that never crosses |
| resilience | Failure-mode analysis: size caps at the entry, panic boundaries at the C++ parser, atomic writes, verified bytes used once, cycle-safe tree walks | General review. It is the one agent that blocks a merge on these grounds |
| archivist | Lockstep: the CHANGELOG, the index of records, statuses and tracking issues move in the same pull request as the code they describe | Authoring proposals |
| newcomer | Naive literal execution: arrives knowing nothing, follows the pages exactly, records what was expected against what happened, and fixes nothing | Reading the source to explain a failure |
The newcomer is the agent most unlike the rest. Its first two passes, before the 0.2.0 release, found six blocking defects that every automated check had passed, the worst of them on the boundary between what a page promised and what a platform did.
The skills
The judgment the agents apply lives in .claude/skills/, one directory per practice: what good looks like in this codebase, and why. Two are non-negotiable and are read together on every structural change. ACES (adaptable, composable, extensible) is the design test: does the change leave the system easier or harder to change, to recombine and to extend. The resilience floor is the operational one: does it survive hostile input, a crashing dependency and two processes racing. The others cover the hexagonal architecture, Rust design choices, the Python binding, test strategy, keeping records in step, the review gates, and checking that a built artifact installs for a real user.
They are practices rather than tool instructions, so they survive a change of tooling, and they are versioned with the code, so a change to one is reviewed like a change to the code.
Routing
Each agent's and each skill's description, at the top of its file, says what it is for and what it is not for. Claude Code reads those descriptions and picks by them, and they are the only routing table: there is no second list to drift out of step with the first. A pull request review is where several are convened, and only by what the diff touches: the reviewer always, the portsmith when a port module changes, the ffi-keeper when the Python binding or its packaging changes, resilience when a change reads files, writes them or takes outside input, and the archivist when the public surface or the records change. That rule is written once, in the /review command.
The gate before a merge
Two layers stand between a pull request and main, and they are not equal.
The deterministic checks are the verification. just check runs them locally. On GitHub, branch protection on main requires twenty status checks to pass, and holds administrators to them: the Rust build and tests on Linux and macOS under each feature set, the minimum supported Rust version, a WebAssembly build check, the API docs, the dependency audit, the semver check, the Python wheels and type check, CodeQL, the boundary rules as tested semgrep rules, and the docsite floor, which builds the site and checks its links, its published URLs, its figures and its worked examples. The practice for a new check is to plant the violation it exists for and watch it fail before trusting it.
The model review is a cold re-read. Every pull request is also read by Claude against the review gates in the pr-review skill. That reader is a model of the same family as the author, so it is a cold re-read, not independent verification: it shares the author's blind spots, and a separate reader of the same family has not been shown to catch more than the author's own second pass would. What it adds is a reading of intent, the part of a change no check can see. Its verdict is not a required status check. Claude does not merge over a "blockers" verdict, and fixes it and asks for a re-read instead; that holds by discipline, and the comment thread on each pull request shows whether it held.
Who is accountable
Two kinds of act are the owner's alone, whatever trust has built up.
- Decisions. An enhancement proposal is accepted only when the owner merges the change of its status line to
accepted, and an API change proposal only when the owner says so on its issue. Claude writes and argues both, and may merge a proposal while its status isproposed, so it can be read on this site; that decides nothing, and Claude never merges a pull request that accepts one. - Releases. Claude prepares a release and dispatches the workflow, which verifies, tags, builds and attests it, then stops at two deployment environments, one per registry, each requiring the owner's approval. That approval is the release decision, because a published version cannot be withdrawn. An agent never approves a deployment.
Everything else merges on Claude's standing authority: once CI is green and the review raises no blockers, Claude merges and leaves a comment giving its reasoning, which is the audit trail. Every commit Claude makes carries a Co-Authored-By trailer naming the model, and every release the release workflow builds carries an attestation that SECURITY.md shows how to verify.
One limit is worth stating plainly. Today every merge, comment and approval is recorded under the owner's GitHub login, so the record alone cannot show which of them the owner made by hand, and no branch rule yet makes an acceptance wait for the owner. The rules above hold by practice. The Identity section of CONTRIBUTING.md sets out how Claude is to get a GitHub identity of its own, and how a required check then holds an acceptance for the owner's review.
Which record a change takes
The process follows the Rust project's RFC process, and the size of a change sets the weight of its record.
| Change | Record |
|---|---|
| A substantial change users notice: the public surface in Rust, Python, the command line or the JSON output; behaviour, including the default model; removing a substantial feature; an architecture boundary | An enhancement proposal (EPR-NNNN), proposed as a pull request. It may merge as proposed, so it renders for review, and is accepted when the owner merges the change of its status to accepted |
| A minor addition to the public surface, or a minor removal | An API change proposal: a short issue labelled acp |
| Tooling, docs, CI, refactors, measured performance work, bug fixes | The pull request itself. A rule it leaves behind lives in the file that governs it |
Each accepted proposal gets a tracking issue, labelled tracking, that holds its milestone checklist and links the pull request for each milestone; it closes when the work ships. A plan that several agents carry out in parallel gets an enhancement plan (EPL-NNNN) beside the proposal. An accepted proposal is never rewritten: a change of mind is a new proposal that supersedes it, so the reasoning behind every decision stays readable. The full process and the index of records are in blueprints/README.md.
Proposals and plans are numbered from 0001, from a baseline on 2026-10-04. The records written before it, cited as RFC-NNNN and EP-NNNN, are kept unchanged in blueprints/legacy/: they are the reasons behind the code that exists, and the comments and pages that cite them still mean them.
Where a record converges
The proposals and plans are rendered on this site, in its Blueprints area, each with its status; it is the one area of the site that shows a status, and the documentation keeps to what ships. Before a record is settled, the owner reads it there in the local development server, where any passage on any page can be selected and commented on. The threads are saved as files in the repository, so the next Claude session reads them, answers in them and changes the record, and the owner reads the answer in the same place. The discussion that shaped a record stays beside it rather than in a chat log that is gone the next day.
How a lesson becomes a check
When something goes wrong twice, the fix is a check rather than a note, because a note is read only by whoever already knows to look for it. The pattern is the same each time: write the check, plant the failure it exists for and watch it fail, then run it from just check and in CI.
- The boundary rules were partly a text search and partly review alone, and the search missed whole forms of import. They are now semgrep rules, each tested against a fixture that holds a violation.
- A check twice passed silently on a machine where the tool it needed was missing. Every check now fails when its tool is absent.
- When the design records moved to their present process, every citation of a record in the tree changed at once. A check now fails when a cited record has no file of its own kind: a proposal, a plan, or a legacy RFC or EP.
- The agent and skill files drifted from the code twice: an agent cited a class at a line it had long left, and another counted files that had been removed. A check now fails when a file and line cited under
.claude/points at a missing file or past its end.
What a check cannot see, such as whether a design is sound or whether a paragraph still describes the code it names, stays with review. The repository's own check scripts say which is which: each ends by naming what it examined, so a pass is not mistaken for a reading.
Where the parts live
| Part | Where |
|---|---|
| The agents | .claude/agents/ |
| The skills | .claude/skills/ |
| The review command | .claude/commands/review.md |
| The rules Claude works under | CLAUDE.md and AGENTS.md |
| Who decides, and how a release runs | CONTRIBUTING.md |
| The records and the process | blueprints/ |
| The boundary rules | Boundary rules |
| The local checks | justfile and scripts/ |
What matra gives you
Every value the pipeline returns, by tier. The tables say which language surfaces carry each value: fields cross to Python, methods do not.
1. Structure
Produced by Engine::analyze, as the document is read and parsed.
Token
The ten CoNLL-U columns plus one derived flag. All cross to Python.
| Field | Holds |
|---|---|
id | 1-based position in the sentence |
text | the word as written |
lemma | dictionary form (approved becomes approve) |
pos | universal part of speech (NOUN, VERB, ADJ) |
xpos | treebank-specific tag, finer grained than pos |
features | morphology (tense, number, person), as a map from feature to value |
head | id of the token this one depends on, 0 for the root |
dep | the dependency relation to that head |
deps | secondary dependencies, always _ from the UDPipe adapter, whose binding does not surface them |
misc | annotation |
is_punct | punctuation flag |
Token::feat(key) reads one morphological feature. Rust only.
Sentence
| Field | Holds |
|---|---|
text | sentence text, rebuilt by the provider from its tokens |
tokens | Vec<Token>, id-sorted |
word_count | non-punctuation tokens |
tree_depth | nesting depth, None on a malformed cycle |
passive | whether the sentence carries nsubj:pass, nsubjpass, or aux:pass |
negations | Vec<Negation>: cue id, cue lemma, head id |
modals | Vec<Modal>: auxiliary id, lemma, head id |
clausal_complements | Vec<ClausalComplement>: verb id and lemma, ccomp head id, subject when present |
root_adverbials | Vec<RootAdverbial>: adverbial id and lemma |
hearst_pairs | Vec<HearstPair>: pattern tag plus hypernym and hyponym HearstSpan values |
root_mood | Option<RootMood>: the root clause's Mood feature and the root's id |
All eleven cross to Python. The methods below are Rust only.
| Method | Returns |
|---|---|
root_token() | the token everything else hangs off |
head_of(id) | the governor of a token |
children_of(id) | its direct dependents |
subtree(id) | the whole clause under it |
content_tokens() | non-punctuation tokens |
clausal_complements_in(lexicon) | the clausal complements whose verb lemma is in a list you supply |
root_adverbials_in(lexicon) | the root adverbials whose lemma is in a list you supply |
Paragraph, Section, Document
| Type | Fields (cross to Python) | Methods (Rust only) |
|---|---|---|
Paragraph | span, text, kind, sentences, metrics | word_count(), sentence_count() |
Section | heading, level, paragraphs | none |
Document | provenance, sections, metrics | paragraphs(), sentences(), tokens() |
provenance records the matra version, the schema version ("2.0"), the parser with its model, digest and licence, the format, and the input's path, and the length and SHA-256 of the analysed text.
2. Metrics
Produced by Engine::analyze, as its last step, into ParagraphMetrics on each paragraph and DocumentMetrics on the document. Every ratio is Option<f64>, and None means not computed, which is distinct from a computed zero. Methodology gives each formula and the exact condition for None.
| Field | On | Measures |
|---|---|---|
readability_grade | ParagraphMetrics | Flesch-Kincaid grade level, from sentence and syllable length |
lexical_density | ParagraphMetrics | content words as a share of all words |
compression_ratio | ParagraphMetrics | brotli compressed size over raw size, a repetition proxy |
total_words, total_sentences, paragraph_count | DocumentMetrics | the counts, as integers |
mean_sentence_length, sentence_length_std | DocumentMetrics | words per sentence, and its sample standard deviation |
vocabulary_ttr | DocumentMetrics | distinct lemmas over total words, and it falls as text grows, so documents of different lengths are not comparable on it |
nominalization_ratio | DocumentMetrics | nouns ending in a nominalizing suffix (-tion, -ment, -ness, -ity, -ence, -ance) over all words, a surface proxy for nouns formed from verbs (decide becoming decision) |
passive_ratio | DocumentMetrics | share of sentences carrying a passive construction |
Every measure is per document. Aggregating across documents is the caller's decision: the counts sum, and vocabulary_ttr does not.
3. Summarization
Free functions over a Document, needing no engine and no parser. Both return a Summary: the document's provenance with the method and n recorded, and sentences, each a SummarySentence (index, span, text, score), in document order, capped at 2,000 sentences.
| Function | Ranks by |
|---|---|
summarize(doc, n), summarize_with(doc, n, SummaryMethod::TfIdf) | mean TF-IDF of the sentence's lemmatized terms, each sentence its own document for IDF (tfidf, the default) |
summarize_with(doc, n, SummaryMethod::TextRank) | PageRank over a similarity graph of shared content lemmas normalized by log length (textrank) |
Python spells them matra.summarize(doc, n=3) and matra.summarize(doc, n=3, method="textrank"), over the Document an engine returned.
TextRank's summary of The Federalist No. 10 at the default length, and every sentence's score it was chosen from:
TextRank: the summary and the scores it came from
- 6Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority.
- 47When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens.
- 48To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed.
show the data
| # | ¶ | Score | Summary | Sentence |
|---|---|---|---|---|
| 1 | 1 | 0.0086 | AMONG the numerous advantages promised by a well constructed Union, none deserves to be more accurately developed than its tendency to break and control the violence of faction. | |
| 2 | 1 | 0.0097 | The friend of popular governments never finds himself so much alarmed for their character and fate, as when he contemplates their propensity to this dangerous vice. | |
| 3 | 1 | 0.0058 | He will not fail, therefore, to set a due value on any plan which, without violating the principles to which he is attached, provides a proper cure for it. | |
| 4 | 1 | 0.0112 | The instability, injustice, and confusion introduced into the public councils, have, in truth, been the mortal diseases under which popular governments have everywhere perished; as they continue to be the favorite and fruitful topics from which the adversaries to liberty derive their most specious declamations. | |
| 5 | 1 | 0.0064 | The valuable improvements made by the American constitutions on the popular models, both ancient and modern, cannot certainly be too much admired; but it would be an unwarrantable partiality, to contend that they have as effectually obviated the danger on this side, as was wished and expected. | |
| 6 | 1 | 0.0221 | yes | Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority. |
| 7 | 1 | 0.0050 | However anxiously we may wish that these complaints had no foundation, the evidence, of known facts will not permit us to deny that they are in some degree true. | |
| 8 | 1 | 0.0163 | It will be found, indeed, on a candid review of our situation, that some of the distresses under which we labor have been erroneously charged on the operation of our governments; but it will be found, at the same time, that other causes will not alone account for many of our heaviest misfortunes; and, particularly, for that prevailing and increasing distrust of public engagements, and alarm for private rights, which are echoed from one end of the continent to the other. | |
| 9 | 1 | 0.0099 | These must be chiefly, if not wholly, effects of the unsteadiness and injustice with which a factious spirit has tainted our public administrations. | |
| 10 | 2 | 0.0211 | By a faction, I understand a number of citizens, whether amounting to a majority or a minority of the whole, who are united and actuated by some common impulse of passion, or of interest, adversed to the rights of other citizens, or to the permanent and aggregate interests of the community. | |
| 11 | 3 | 0.0145 | There are two methods of curing the mischiefs of faction: the one, by removing its causes; the other, by controlling its effects. | |
| 12 | 4 | 0.0191 | There are again two methods of removing the causes of faction: the one, by destroying the liberty which is essential to its existence; the other, by giving to every citizen the same opinions, the same passions, and the same interests. | |
| 13 | 5 | 0.0050 | It could never be more truly said than of the first remedy, that it was worse than the disease. | |
| 14 | 5 | 0.0068 | Liberty is to faction what air is to fire, an aliment without which it instantly expires. | |
| 15 | 5 | 0.0096 | But it could not be less folly to abolish liberty, which is essential to political life, because it nourishes faction, than it would be to wish the annihilation of air, which is essential to animal life, because it imparts to fire its destructive agency. | |
| 16 | 6 | 0.0036 | The second expedient is as impracticable as the first would be unwise. | |
| 17 | 6 | 0.0095 | As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed. | |
| 18 | 6 | 0.0090 | As long as the connection subsists between his reason and his self-love, his opinions and his passions will have a reciprocal influence on each other; and the former will be objects to which the latter will attach themselves. | |
| 19 | 6 | 0.0127 | The diversity in the faculties of men, from which the rights of property originate, is not less an insuperable obstacle to a uniformity of interests. | |
| 20 | 6 | 0.0100 | The protection of these faculties is the first object of government. | |
| 21 | 6 | 0.0182 | From the protection of different and unequal faculties of acquiring property, the possession of different degrees and kinds of property immediately results; and from the influence of these on the sentiments and views of the respective proprietors, ensues a division of the society into different interests and parties. | |
| 22 | 7 | 0.0143 | The latent causes of faction are thus sown in the nature of man; and we see them everywhere brought into different degrees of activity, according to the different circumstances of civil society. | |
| 23 | 7 | 0.0210 | A zeal for different opinions concerning religion, concerning government, and many other points, as well of speculation as of practice; an attachment to different leaders ambitiously contending for pre-eminence and power; or to persons of other descriptions whose fortunes have been interesting to the human passions, have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to co-operate for their common good. | |
| 24 | 7 | 0.0062 | So strong is this propensity of mankind to fall into mutual animosities, that where no substantial occasion presents itself, the most frivolous and fanciful distinctions have been sufficient to kindle their unfriendly passions and excite their most violent conflicts. | |
| 25 | 7 | 0.0051 | But the most common and durable source of factions has been the various and unequal distribution of property. | |
| 26 | 7 | 0.0093 | Those who hold and those who are without property have ever formed distinct interests in society. | |
| 27 | 7 | 0.0028 | Those who are creditors, and those who are debtors, fall under a like discrimination. | |
| 28 | 7 | 0.0112 | A landed interest, a manufacturing interest, a mercantile interest, a moneyed interest, with many lesser interests, grow up of necessity in civilized nations, and divide them into different classes, actuated by different sentiments and views. | |
| 29 | 7 | 0.0193 | The regulation of these various and interfering interests forms the principal task of modern legislation, and involves the spirit of party and faction in the necessary and ordinary operations of the government. | |
| 30 | 8 | 0.0089 | No man is allowed to be a judge in his own cause, because his interest would certainly bias his judgment, and, not improbably, corrupt his integrity. | |
| 31 | 8 | 0.0174 | With equal, nay with greater reason, a body of men are unfit to be both judges and parties at the same time; yet what are many of the most important acts of legislation, but so many judicial determinations, not indeed concerning the rights of single persons, but concerning the rights of large bodies of citizens? | |
| 32 | 8 | 0.0085 | And what are the different classes of legislators but advocates and parties to the causes which they determine? | |
| 33 | 8 | 0.0031 | Is a law proposed concerning private debts? | |
| 34 | 8 | 0.0101 | It is a question to which the creditors are parties on one side and the debtors on the other. | |
| 35 | 8 | 0.0025 | Justice ought to hold the balance between them. | |
| 36 | 8 | 0.0134 | Yet the parties are, and must be, themselves the judges; and the most numerous party, or, in other words, the most powerful faction must be expected to prevail. | |
| 37 | 8 | 0.0033 | Shall domestic manufactures be encouraged, and in what degree, by restrictions on foreign manufactures? | |
| 38 | 8 | 0.0082 | are questions which would be differently decided by the landed and the manufacturing classes, and probably by neither with a sole regard to justice and the public good. | |
| 39 | 8 | 0.0108 | The apportionment of taxes on the various descriptions of property is an act which seems to require the most exact impartiality; yet there is, perhaps, no legislative act in which greater opportunity and temptation are given to a predominant party to trample on the rules of justice. | |
| 40 | 8 | 0.0050 | Every shilling with which they overburden the inferior number, is a shilling saved to their own pockets. | |
| 41 | 9 | 0.0121 | It is in vain to say that enlightened statesmen will be able to adjust these clashing interests, and render them all subservient to the public good. | |
| 42 | 9 | 0.0024 | Enlightened statesmen will not always be at the helm. | |
| 43 | 9 | 0.0202 | Nor, in many cases, can such an adjustment be made at all without taking into view indirect and remote considerations, which will rarely prevail over the immediate interest which one party may find in disregarding the rights of another or the good of the whole. | |
| 44 | 10 | 0.0110 | The inference to which we are brought is, that the CAUSES of faction cannot be removed, and that relief is only to be sought in the means of controlling its EFFECTS. | |
| 45 | 11 | 0.0119 | If a faction consists of less than a majority, relief is supplied by the republican principle, which enables the majority to defeat its sinister views by regular vote. | |
| 46 | 11 | 0.0070 | It may clog the administration, it may convulse the society; but it will be unable to execute and mask its violence under the forms of the Constitution. | |
| 47 | 11 | 0.0274 | yes | When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens. |
| 48 | 11 | 0.0224 | yes | To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed. |
| 49 | 11 | 0.0093 | Let me add that it is the great desideratum by which this form of government can be rescued from the opprobrium under which it has so long labored, and be recommended to the esteem and adoption of mankind. | |
| 50 | 12 | 0.0040 | By what means is this object attainable? | |
| 51 | 12 | 0.0052 | Evidently by one of two only. | |
| 52 | 12 | 0.0201 | Either the existence of the same passion or interest in a majority at the same time must be prevented, or the majority, having such coexistent passion or interest, must be rendered, by their number and local situation, unable to concert and carry into effect schemes of oppression. | |
| 53 | 12 | 0.0045 | If the impulse and the opportunity be suffered to coincide, we well know that neither moral nor religious motives can be relied on as an adequate control. | |
| 54 | 12 | 0.0087 | They are not found to be such on the injustice and violence of individuals, and lose their efficacy in proportion to the number combined together, that is, in proportion as their efficacy becomes needful. | |
| 55 | 13 | 0.0198 | From this view of the subject it may be concluded that a pure democracy, by which I mean a society consisting of a small number of citizens, who assemble and administer the government in person, can admit of no cure for the mischiefs of faction. | |
| 56 | 13 | 0.0219 | A common passion or interest will, in almost every case, be felt by a majority of the whole; a communication and concert result from the form of government itself; and there is nothing to check the inducements to sacrifice the weaker party or an obnoxious individual. | |
| 57 | 13 | 0.0087 | Hence it is that such democracies have ever been spectacles of turbulence and contention; have ever been found incompatible with personal security or the rights of property; and have in general been as short in their lives as they have been violent in their deaths. | |
| 58 | 13 | 0.0118 | Theoretic politicians, who have patronized this species of government, have erroneously supposed that by reducing mankind to a perfect equality in their political rights, they would, at the same time, be perfectly equalized and assimilated in their possessions, their opinions, and their passions. | |
| 59 | 14 | 0.0122 | A republic, by which I mean a government in which the scheme of representation takes place, opens a different prospect, and promises the cure for which we are seeking. | |
| 60 | 14 | 0.0085 | Let us examine the points in which it varies from pure democracy, and we shall comprehend both the nature of the cure and the efficacy which it must derive from the Union. | |
| 61 | 15 | 0.0210 | The two great points of difference between a democracy and a republic are: first, the delegation of the government, in the latter, to a small number of citizens elected by the rest; secondly, the greater number of citizens, and greater sphere of country, over which the latter may be extended. | |
| 62 | 16 | 0.0192 | The effect of the first difference is, on the one hand, to refine and enlarge the public views, by passing them through the medium of a chosen body of citizens, whose wisdom may best discern the true interest of their country, and whose patriotism and love of justice will be least likely to sacrifice it to temporary or partial considerations. | |
| 63 | 16 | 0.0088 | Under such a regulation, it may well happen that the public voice, pronounced by the representatives of the people, will be more consonant to the public good than if pronounced by the people themselves, convened for the purpose. | |
| 64 | 16 | 0.0041 | On the other hand, the effect may be inverted. | |
| 65 | 16 | 0.0112 | Men of factious tempers, of local prejudices, or of sinister designs, may, by intrigue, by corruption, or by other means, first obtain the suffrages, and then betray the interests, of the people. | |
| 66 | 16 | 0.0108 | The question resulting is, whether small or extensive republics are more favorable to the election of proper guardians of the public weal; and it is clearly decided in favor of the latter by two obvious considerations: | |
| 67 | 17 | 0.0143 | In the first place, it is to be remarked that, however small the republic may be, the representatives must be raised to a certain number, in order to guard against the cabals of a few; and that, however large it may be, they must be limited to a certain number, in order to guard against the confusion of a multitude. | |
| 68 | 17 | 0.0165 | Hence, the number of representatives in the two cases not being in proportion to that of the two constituents, and being proportionally greater in the small republic, it follows that, if the proportion of fit characters be not less in the large than in the small republic, the former will present a greater option, and consequently a greater probability of a fit choice. | |
| 69 | 18 | 0.0163 | In the next place, as each representative will be chosen by a greater number of citizens in the large than in the small republic, it will be more difficult for unworthy candidates to practice with success the vicious arts by which elections are too often carried; and the suffrages of the people being more free, will be more likely to centre in men who possess the most attractive merit and the most diffusive and established characters. | |
| 70 | 19 | 0.0071 | It must be confessed that in this, as in most other cases, there is a mean, on both sides of which inconveniences will be found to lie. | |
| 71 | 19 | 0.0154 | By enlarging too much the number of electors, you render the representatives too little acquainted with all their local circumstances and lesser interests; as by reducing it too much, you render him unduly attached to these, and too little fit to comprehend and pursue great and national objects. | |
| 72 | 19 | 0.0110 | The federal Constitution forms a happy combination in this respect; the great and aggregate interests being referred to the national, the local and particular to the State legislatures. | |
| 73 | 20 | 0.0188 | The other point of difference is, the greater number of citizens and extent of territory which may be brought within the compass of republican than of democratic government; and it is this circumstance principally which renders factious combinations less to be dreaded in the former than in the latter. | |
| 74 | 20 | 0.0198 | The smaller the society, the fewer probably will be the distinct parties and interests composing it; the fewer the distinct parties and interests, the more frequently will a majority be found of the same party; and the smaller the number of individuals composing a majority, and the smaller the compass within which they are placed, the more easily will they concert and execute their plans of oppression. | |
| 75 | 20 | 0.0207 | Extend the sphere, and you take in a greater variety of parties and interests; you make it less probable that a majority of the whole will have a common motive to invade the rights of other citizens; or if such a common motive exists, it will be more difficult for all who feel it to discover their own strength, and to act in unison with each other. | |
| 76 | 20 | 0.0080 | Besides other impediments, it may be remarked that, where there is a consciousness of unjust or dishonorable purposes, communication is always checked by distrust in proportion to the number whose concurrence is necessary. | |
| 77 | 21 | 0.0156 | Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic,--is enjoyed by the Union over the States composing it. | |
| 78 | 21 | 0.0105 | Does the advantage consist in the substitution of representatives whose enlightened views and virtuous sentiments render them superior to local prejudices and schemes of injustice? | |
| 79 | 21 | 0.0051 | It will not be denied that the representation of the Union will be most likely to possess these requisite endowments. | |
| 80 | 21 | 0.0132 | Does it consist in the greater security afforded by a greater variety of parties, against the event of any one party being able to outnumber and oppress the rest? | |
| 81 | 21 | 0.0114 | In an equal degree does the increased variety of parties comprised within the Union, increase this security. | |
| 82 | 21 | 0.0080 | Does it, in fine, consist in the greater obstacles opposed to the concert and accomplishment of the secret wishes of an unjust and interested majority? | |
| 83 | 21 | 0.0055 | Here, again, the extent of the Union gives it the most palpable advantage. | |
| 84 | 22 | 0.0055 | The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States. | |
| 85 | 22 | 0.0109 | A religious sect may degenerate into a political faction in a part of the Confederacy; but the variety of sects dispersed over the entire face of it must secure the national councils against any danger from that source. | |
| 86 | 22 | 0.0110 | A rage for paper money, for an abolition of debts, for an equal division of property, or for any other improper or wicked project, will be less apt to pervade the whole body of the Union than a particular member of it; in the same proportion as such a malady is more likely to taint a particular county or district, than an entire State. | |
| 87 | 23 | 0.0100 | In the extent and proper structure of the Union, therefore, we behold a republican remedy for the diseases most incident to republican government. | |
| 88 | 23 | 0.0067 | And according to the degree of pleasure and pride we feel in being republicans, ought to be our zeal in cherishing the spirit and supporting the character of Federalists. |
- in the summary (3 sentences, matra's default)
- not picked
TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.
Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: federalist-10/textrank.json.
4. Keyphrases
Free functions over a Document. Both return a Keyphrases: the document's provenance with the method and n recorded, and phrases, each a Keyphrase (phrase, score, occurrences, the span of every place it was found), highest score first, capped at 200,000 tokens.
| Function | Ranks by |
|---|---|
keyphrases(doc, n), keyphrases_with(doc, n, KeyphraseMethod::Rake) | co-occurrence degree over frequency, on NOUN, ADJ, and PROPN runs between stop words (rake, the default) |
keyphrases_with(doc, n, KeyphraseMethod::Yake) | per-term position, frequency, and context diversity, assembled into 1-word to 3-word candidates, score inverted so higher is more relevant (yake) |
Python spells them matra.keyphrases(doc, n=10) and matra.keyphrases(doc, n=10, method="yake").
The two rank by different properties of the text, and on The Federalist No. 10 their top twelve lists share no phrase:
RAKE and YAKE on the same text
show the data
| Phrase | RAKE rank | RAKE score | YAKE rank | YAKE score |
|---|---|---|---|---|
| fate contemplate propensity | not ranked | 1 | 32553.6 | |
| substantial occasion present | 1 | 9.000 | 2694 | 0.451 |
| many judicial determination | 2 | 7.833 | 2642 | 0.468 |
| tendency break | not ranked | 2 | 22722.0 | |
| develop tendency break | not ranked | 3 | 22609.3 | |
| federal constitution form | 3 | 6.429 | 2983 | 0.335 |
| many lesser interest | 4 | 5.875 | 2273 | 0.576 |
| tendency break control | not ranked | 4 | 16940.2 | |
| fate contemplate | not ranked | 5 | 4523.1 | |
| lesser interest | 5 | 4.042 | 1722 | 0.727 |
| adequate control | 6 | 4.000 | 2426 | 0.520 |
| alone account | 6 | 4.000 | 1486 | 0.823 |
| american constitution | 6 | 4.000 | 1793 | 0.703 |
| attractive merit | 6 | 4.000 | 2849 | 0.404 |
| candid review | 6 | 4.000 | 1465 | 0.840 |
| character fate contemplate | not ranked | 6 | 3038.8 | |
| civilized nation | 6 | 4.000 | 2136 | 0.615 |
| dangerous vice | 6 | 4.000 | 1294 | 0.967 |
| complaint foundation evidence | not ranked | 7 | 1730.8 | |
| contemplate propensity | not ranked | 8 | 463.1 | |
| contemplate propensity dangerous | not ranked | 9 | 455.5 | |
| break control violence | not ranked | 10 | 362.2 | |
| annihilation air essential | not ranked | 11 | 218.2 | |
| tendency | 272 | 1.000 | 12 | 158.4 |
- in RAKE's top 12
- in YAKE's top 12
- only one method produces the phrase
Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 466 RAKE and 3111 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.
Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: federalist-10/keyphrases.json.
5. Semantic clusters
Not a field on any type above. The port and clusters are always compiled; the Model2Vec adapter is behind the model2vec feature.
| Item | Is |
|---|---|
Embedder | the port: embed(&[&str]) returning one Embedding per text, plus identity() |
Model2Vec | the adapter, reading model.safetensors, tokenizer.json, and config.json from a directory. from_dir(dir) loads what is there; potion_base_8m(dir), with dir a models directory, and from_config(cfg) fetch the pinned reference model into its potion-base-8M subdirectory when it is absent |
clusters(doc, embedder, threshold) | embeds a document's sentences with the embedder you pass, then clusters them; vectors you already hold arrive through an Embedder that returns them |
Clusters | the document's provenance with the method (cosine), the threshold and the embedder's identity recorded, clusters, and edges |
Cluster, SimilarityEdge | a cluster's members (sentence indices), and a pair a, b whose score cleared the threshold |
Capped at 2,000 sentences. Python spells them matra.clusters(doc, embedder, threshold=0.85) and Model2Vec (including Model2Vec.potion_base_8m); matra.clusters takes a Model2Vec or any object with embed and identity.
The pipeline surface
| Value | Constructors | Carries |
|---|---|---|
Input | text(string, format), bytes(data, format), file(path), blocks(blocks) | one document: text, bytes, a file, or headings and paragraphs you supply |
Inputs | dir(path) | the files in a directory, as an iterator of Input |
Engine | new(), from_config(cfg), from_config_with_notice(cfg, notice) | analyze one input, analyze_all of many |
Config | resolve(), from_file(path), from_sources(...), with_model_dir(dir) | where models live and what the defaults are, plus the ValueSource each value came from |
Engine::new() is the no-setup constructor: locate the model directory from the environment, then build the standard pipeline, fetching the pinned English model if it is not on disk. It reads no configuration file; Engine::from_config(&Config::resolve()?) reads one.
engine.analyze_all(inputs) returns a lazy iterator of per-document results, one per input in order, and one bad file does not abort the stream. Python's Engine.analyze_all(dir=path) is the same lazy walk, yielding a Document or a DocumentError dict per file.
Traceability
Every heading, paragraph, sentence and token carries a span into the analysed text, in UTF-8 bytes, and skipped lists what the reader saw and did not analyse (a table, a code block, front matter), each with its span. Every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span, so nothing is dropped without a trace. The text fields are normalized: Paragraph.text is what the parser reads, a markdown paragraph's inline markup removed, and Sentence.text is rebuilt by the provider, the UDPipe adapter joining the token forms with a space unless a token carries SpaceAfter=No. The span, not the text, says where a value came from. Each paragraph is parsed on its own and its sentences are attached to it, so the chain from a token up through its sentence, paragraph, and section holds by construction rather than by text matching.
Where to go next
Domain model is the full type graph. Methodology is every formula. Errors is every failure.
Domain types
matra::domain holds every type the library hands back. The module depends on serde, thiserror, and the standard library, and on nothing else, so a caller can depend on these types without inheriting a C++ toolchain.
It is not a copy of the generated item-level documentation. For that, run cargo doc --no-deps --all-features.
The type graph
Five types nest, each owning the next. Nothing is owned twice.
The left column is what the pipeline writes into the struct. The right column is recomputed from the tree every time you ask, and an empty right column means the type only holds. So there is no second list of paragraphs anywhere in Document: the flat iterators walk the section tree. Every number a reader needs from a document is stored, in DocumentMetrics and ParagraphMetrics and in a sentence's word_count, tree_depth and passive, because a field crosses the language boundary and a method does not.
The derived structural types (Negation, Modal, ClausalComplement, RootAdverbial, RootMood, HearstPair with its HearstSpan and HearstPattern) sit inside the nest, owned by Sentence, and the metric records DocumentMetrics and ParagraphMetrics are owned by Document and Paragraph. Provenance, with its Identity and ProvenanceSource, is owned by Document, and so is skipped, a list of Skipped records, each a SkippedKind and a Span. Every heading, paragraph, sentence and token holds a Span, and a paragraph its ParagraphKind. The other types sit outside it:
| Type | Role |
|---|---|
Block | a heading or paragraph a caller supplies, the input to Input::blocks; never returned |
Summary, SummarySentence | output of summarize and summarize_with |
Keyphrases, Keyphrase | output of keyphrases and keyphrases_with |
Clusters, Cluster, SimilarityEdge | output of clusters, described in Semantic clusters |
Parameters | what shaped an outcome, recorded in its provenance |
Format | which reader a document needs |
Error | every failure the library can return |
ErrorDetails, DetailValue | the values a caller acts on in an error, by name: what Error::details returns |
DocumentError | one document's failure, an Err item of analyze_all's stream |
Embedding | one vector from an Embedder, the input to clustering |
ProvisionNotice | what a model fetch is about to download, handed to the _with_notice constructors' callback, described in Errors |
Where each of those enters and leaves the pipeline is drawn in Architecture.
Attributes shared across the module
| Type | Derives | #[non_exhaustive] |
|---|---|---|
Token | Debug, Clone, Serialize, Deserialize | yes |
TokenBuilder | none | no |
Sentence | Debug, Clone, Serialize, Deserialize | yes |
Paragraph | Debug, Clone, Serialize, Deserialize | yes |
Section | Debug, Clone, Serialize, Deserialize | yes |
Document | Debug, Clone, Serialize, Deserialize | yes |
DocumentMetrics, ParagraphMetrics | Debug, Clone, Default, PartialEq, Serialize, Deserialize | yes |
Provenance | Debug, Clone, PartialEq, Serialize, Deserialize | yes |
Parameters | Debug, Clone, Default, PartialEq, Serialize, Deserialize | yes |
Identity, ProvenanceSource | Debug, Clone, PartialEq, Eq, Serialize, Deserialize | yes |
Summary, SummarySentence, Keyphrases, Keyphrase | Debug, Clone, PartialEq, Serialize, Deserialize | yes |
Clusters, Cluster, SimilarityEdge | Debug, Clone, PartialEq, Serialize, Deserialize | yes |
Format | Debug, Clone, PartialEq, Eq, Serialize, Deserialize | yes |
Span | Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize | yes |
Skipped, SkippedKind, ParagraphKind | Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize | yes |
Block | Debug, Clone, PartialEq, Eq | no |
Error | Debug, thiserror::Error; Serialize by hand | yes |
ErrorDetails | Debug, Clone, Default, PartialEq, Eq; Serialize by hand | no |
DetailValue | Debug, Clone, PartialEq, Eq, Serialize | yes |
Negation, Modal, ClausalComplement, RootAdverbial | Debug, Clone, Serialize, Deserialize | yes |
RootMood | Debug, Clone, PartialEq, Eq, Serialize, Deserialize | yes |
HearstPair, HearstSpan | Debug, Clone, PartialEq, Eq, Serialize, Deserialize | yes |
HearstPattern | Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize | yes |
DocumentError | Debug; Serialize by hand | yes |
Embedding | Debug, Clone, PartialEq, Serialize, Deserialize | no |
ProvisionNotice | Debug, Clone, PartialEq, Eq | yes |
Block is the one input type. It has no serde derives and no public fields, so it needs no #[non_exhaustive]: a caller builds one with Block::heading or Block::paragraph and only matra reads it. Its JSON form is the blocks format, whose schema is spec/schema/v2/blocks.schema.json.
No type in the module derives Hash, and the parse tree derives no equality: two Token, Sentence, Paragraph, Section or Document values cannot be compared with ==. Format derives PartialEq and Eq, which is what the reader table keys on, and the metric records, the provenance types, Span, Skipped, the kind enums, Block, RootMood, the Hearst types, the outcome types, Embedding, ProvisionNotice, ErrorDetails and DetailValue derive PartialEq. Error has no Clone, no equality and no serde derives; its Serialize is written by hand, as DocumentError's and ErrorDetails' are. ErrorDetails has no public fields, like Block, so it needs no #[non_exhaustive]. Embedding is the one public type with public fields and without #[non_exhaustive]: on a tuple struct the attribute would make the constructor private, and an Embedder implemented outside the crate has to build one.
#[non_exhaustive] has two effects outside the crate. Struct literal syntax is unavailable, so you build values through the associated constructors listed below. The outcome types have no public constructor: matra produces them, and serde reads them back. Matches on Format, ParagraphKind, SkippedKind and Error need a catch-all arm, because variants can be added in a minor release.
Every field on these types is pub and readable. TokenBuilder and Block are the exceptions: their fields are private, and you reach them through the builder's setters and the block's constructors.
Token
One parsed token carrying the full CoNLL-U annotation set plus one derived flag.
| Field | Type | CoNLL-U column | Contents |
|---|---|---|---|
id | usize | 1 | Position within the sentence, 1-based |
span | Span | none | Where the surface form lies in the analysed text, in UTF-8 bytes |
text | String | 2 | Surface form |
lemma | String | 3 | Dictionary form |
pos | String | 4 | Universal POS tag, for example NOUN, VERB, ADJ |
xpos | String | 5 | Language-specific POS tag |
features | BTreeMap<String, String> | 6 | Morphological features as a map from feature name to value |
head | usize | 7 | Id of the governing token; 0 marks the root |
dep | String | 8 | Dependency relation to the head, for example nsubj, obj, aux:pass |
deps | String | 9 | Enhanced dependency graph |
misc | String | 10 | Miscellaneous annotations, for example SpaceAfter=No |
is_punct | bool | derived | Whether the token is punctuation |
Provider-specific notes for the UDPipe adapter, the one adapter that ships:
is_punctis set when the UPOS tag isPUNCT.depsis always the string_, the CoNLL-U empty marker. The underlyingudpipe-rsbinding does not surface column 9.xposandmisccarry whatever the model emits, and can be_or empty.featuresis converted from the column:Mood=Ind|Tense=Presbecomes two entries, a multi-valued feature (Case=Nom,Acc) keeps its value unsplit, and_becomes an empty map.
A token id or head that does not fit in usize is rejected at the adapter with Error::ParseFailed rather than being silently coerced.
span comes from aligning each form with the paragraph text the parser was given, then mapping it into the analysed text. A form the adapter cannot find where the text says it should be is Error::ParseFailed naming the token, never a guessed span, and the engine checks every parser's spans by the same rule, so a span from any parser that does not line up with its form is Error::ParseFailed with reason token_form_mismatch. In a markdown paragraph whose markup the reader removed, a token's span can cover markup inside it (foo**bar** is one token from the f to the last r), and a token read from an escape or an entity (&) covers all of it.
Construction
let token = matra::domain::Token::builder(
1, // id
"committee".to_string(), // text
"committee".to_string(), // lemma
"NOUN".to_string(), // pos
3, // head
"nsubj".to_string(), // dep
)
.xpos("NN".to_string())
.feats("Number=Sing".to_string())
.is_punct(false)
.build();
Token::builder(id, text, lemma, pos, head, dep) -> TokenBuilder takes the six CoNLL-U essentials. TokenBuilder carries one setter per optional field (xpos, deps, misc, each taking String; feats, taking the CoNLL-U column 6 string and converting it into features; is_punct taking bool; span taking a Span), each consuming and returning the builder. build() returns the Token. Unset string fields default to empty, features to an empty map, is_punct to false, and span to the empty span at 0.
Methods
| Signature | Returns | Behavior |
|---|---|---|
feat(&self, key: &str) | Option<&str> | The value of one feature in features, by exact name. None when the feature is absent |
feat("Mood") returns Some("Ind") when the token carries Mood=Ind. The value is what the provider emitted, so a multi-valued feature (Case=Nom,Acc) comes back unsplit: matra does not normalise it. A token whose column 6 was _ or empty has no features, so every lookup on it returns None.
feat is Rust-only by design. features crosses FFI as a map, so a lookup over it adds no information to the wire; a Python or TypeScript caller indexes the same map (RFC-0009).
Sentence
| Field | Type | Contents |
|---|---|---|
span | Span | Where the sentence lies in the analysed text: from its first token's start to its last token's end, derived at construction |
text | String | Sentence text rebuilt from the token forms, normalized, not a slice of the input |
tokens | Vec<Token> | Tokens in ascending id order |
word_count | usize | Non-punctuation tokens, derived at construction |
tree_depth | Option<usize> | Longest path from any token to the root, the root at depth 0, derived at construction. None when the head references form a cycle |
passive | bool | Whether any token's dep is nsubj:pass, nsubjpass, or aux:pass, derived at construction |
negations | Vec<Negation> | Negation cues, derived at construction |
modals | Vec<Modal> | Modal auxiliaries, derived at construction |
clausal_complements | Vec<ClausalComplement> | Verbs governing a clausal complement (ccomp), each with its subject when the parse has one, derived at construction |
root_adverbials | Vec<RootAdverbial> | Adverbial modifiers attached to the root, derived at construction |
hearst_pairs | Vec<HearstPair> | Candidate hypernymy pairs from the six Hearst (1992) patterns, filled by the pipeline at the annotate stage |
root_mood | Option<RootMood> | The root clause's Mood feature with the root's token id, derived at construction. None when the root clause carries no mood |
Invariants that downstream code relies on: tokens are id-sorted; exactly one token has head == 0; every other head names a token in the same sentence; following head never loops. Every sentence the engine builds holds them, because it checks what the parser returns before it builds one and fails the document with Error::ParseFailed naming the check otherwise. A sentence built by hand with Sentence::new, which trusts its caller, or read back from JSON can break any of them, and the tree walks stay safe when the last one breaks (see tree_depth below).
tree_depth is computed in time linear in token count with a visited set, and is None when the head references form a cycle, which reports a malformed parse as a value a reader must handle instead of truncating it or returning a number that reads as a very deep tree. An empty sentence has depth Some(0). word_count counts tokens the NLP provider identified, minus punctuation. It is not a whitespace split, and the two counts differ on the same text.
The engine builds text by joining the token surface forms a parser returned, inserting a space unless the preceding token carries SpaceAfter=No in misc. The string is a reconstruction, not a slice of the input, so whitespace can differ from the source text; span is where the sentence came from.
Derived structural fields
The fields after tokens are computed once from the dependency graph and serialized with the sentence, so every language binding and the CLI's JSON read one Rust detection as data (RFC-0008). Most are computed at Sentence::new; hearst_pairs is the exception, filled by the pipeline at the annotate stage because its detector lives in the hearst module outside the domain, so a hand-built Sentence carries an empty vector until the caller runs hearst::hypernymy_pairs. Each field reports structure only; the reading is the caller's.
Negation reports one cue by token id: cue_id, cue_lemma (not, never, no, neither, nor), and head_id, the token the cue attaches to. A cue fires only on its carrying relation (advmod, det, cc), so nothing as a subject does not fire and tokenizer-split cannot fires exactly once, on the not token. Whether the negation reverses the sentence's claim is not matra's call.
Modal reports one modal auxiliary by token id: aux_id, aux_lemma, and head_id, the token the auxiliary attaches to (0 when the modal is itself the root). The closed class is the ten lemmas the UD English treebank tags MD: can, could, may, might, must, ought, shall, should, will, would, matched by lemma on the aux relation or the AUX part of speech. The part-of-speech arm catches the shapes the model does not park on aux: VP ellipsis promotes the modal to root ("Yes, you should.") and coordinated auxiliaries parse as root plus aux or conj ("He can and will succeed."). Precision holds because the model's only other AUX lemmas (be, have, do, get) are outside the class. Contracted forms fire on their lemma (wo in won't lemmatizes to will), the noun will never fires because it never carries aux or AUX, and a multi-auxiliary chain (might have been done) reports its modal with head_id on the content verb. English modals are ambiguous across readings, and resolving must into obligation or inference takes context matra does not model, so no epistemic, deontic, or dynamic category is assigned: the auxiliary and its arc are reported, the reading is the caller's.
RootMood reports the grammatical mood of the root clause: root_id, the root's token id, and mood, the Mood feature as the model emitted it (Ind, Imp, and an open set beyond). The feature is read off the root token or, when the root carries none, off its first cop, aux, or aux:pass child that does, because the model parks finiteness on the auxiliary in copular ("The sky is blue."), do-support ("He did leave.") and passive clauses. A modal carries VerbForm=Fin and no mood, so a clause whose root a modal governs ("You must complete the form.") usually has no root mood; whether a modal governs the root is the separate fact in modals. It reads the root clause only, so a mood in a subordinate clause does not reach it. The bare assertoric form, an indicative root no modal governs, is mood == "Ind" with no modal whose head_id is root_id.
ClausalComplement reports one verb governing a clausal complement by token id: verb_id and verb_lemma for the verb, ccomp_id for the head of the complement, and subject_id plus subject_lemma for the verb's subject (nsubj, nsubj:pass) when the parse has one in the same sentence. The construction fires for every verb that fills it ("Smith reported that the effect vanished.", "These results suggest that the mechanism is shared.", "We show that the method works." all match), because verbs taking a clausal complement are an open class and any list matra shipped would be incomplete while looking authoritative. Which verb lemmas count as reporting or evidential is the caller's lexicon, supplied as a parameter to clausal_complements_in or applied to the field directly in any language. A ccomp under a non-verb head ("I am sure that it works." parses the complement under the adjective) is outside the construction by decision. The subject is optional by observed fact: UDPipe splits "Smith et al. reported that ..." into two sentences at the period in "et al.", stranding the attribution in the previous sentence while the verb keeps its ccomp. That upstream segmentation defect is recorded here and in the test suite rather than papered over.
RootAdverbial reports one adverbial modifier attached to the root by token id: adv_id and adv_lemma. The root-attached advmod arc is where sentence-scope adverbs land ("Reportedly, the deal closed."), but the dependency graph does not distinguish sentence scope from manner ("The team quickly shipped." puts quickly on the same arc), so every root-attached adverbial is reported and the caller's lexicon selects the evidential ones, via root_adverbials_in or a filter over the field. Negation particles on the same arc ("not" in "The plan is not ready.") appear in both negations and root_adverbials; both report arcs, neither assigns a reading, and the overlap is intentional.
HearstPair reports one candidate hypernymy pair from the six lexico-syntactic patterns of Hearst (1992), detected as dependency-arc patterns rather than regex over surface text: NP such as NP, such NP as NP, NP, including NP, NP, especially NP, NP and other NP, NP or other NP. The pattern field carries a HearstPattern tag naming the construction that matched (such_as, such_np_as, including, especially, and_other, or_other on the wire), and hypernym and hyponym each carry a HearstSpan: head_id and head_lemma for the noun phrase head, plus first_id and last_id for the contiguous token range covering the head and its adjacent nominal modifiers, with the pattern's own marker words (such, other) excluded, so traceability holds against tokens. Precision is favoured over recall: each detector requires the full verified arc shape, so a clausal coordination ("She invited John, and other guests brought gifts."), a definite contrastive coordination ("The teacher and the other students laughed."), or an emphasized plain coordination ("France and especially Spain") does not fire, and a sentence the model misparses is missed rather than misreported. The pair is a candidate by design: matra reports that the sentence used a construction which conventionally signals hypernymy, and does not build a taxonomy or assert the relation is true.
Methods
Most methods below are walks over the head and dep columns; the two _in filters are views over the derived fields against a caller-supplied lexicon. Concepts draws one parsed sentence as the tree these walk.
| Signature | Returns | Behavior |
|---|---|---|
new(text: String, tokens: Vec<Token>) | Sentence | Constructor. Derives span and the structural fields from tokens. Does not validate the invariants |
content_tokens(&self) | Vec<&Token> | Tokens where is_punct is false |
root_token(&self) | Option<&Token> | The token with head == 0, if there is one |
children_of(&self, id: usize) | Vec<&Token> | Tokens whose head equals id. Empty for an unknown id |
head_of(&self, id: usize) | Option<&Token> | The governing token. None for the root and for an unknown id |
subtree(&self, id: usize) | Vec<&Token> | The token and all its descendants, sorted by id. Empty for an unknown id |
clausal_complements_in(&self, lexicon: &[&str]) | Vec<&ClausalComplement> | Clausal complements whose verb_lemma is in the caller-supplied lexicon |
root_adverbials_in(&self, lexicon: &[&str]) | Vec<&RootAdverbial> | Root adverbials whose adv_lemma is in the caller-supplied lexicon |
subtree carries a visited set, so it terminates on cyclic input.
Paragraph
| Field | Type | Contents |
|---|---|---|
span | Span | Where the paragraph lies in the analysed text, trailing whitespace excluded. For markdown it covers the quote and list markers on its lines |
text | String | The text the parser reads, normalized: for markdown the inline text with the markup removed (emphasis, link and image syntax, code span backticks, quote and list markers), soft line breaks as \n, escapes and entities decoded; for plain text and block input a verbatim slice |
kind | ParagraphKind | Paragraph, Quote (inside a block quote at any depth) or ListItem; paragraph, quote, list_item on the wire |
sentences | Vec<Sentence> | Sentences from parsing this paragraph |
metrics | ParagraphMetrics | The paragraph's three measures |
ParagraphMetrics holds the three:
| Field | Type | Contents |
|---|---|---|
readability_grade | Option<f64> | Flesch-Kincaid grade level, when the paragraph holds more than ten words |
lexical_density | Option<f64> | Content-word ratio, 0.0 to 1.0, when the paragraph holds a word |
compression_ratio | Option<f64> | Brotli compressed size over original size, when the paragraph holds more than fifty words |
Each slot holds None until Engine::compose fills it, and stays None when the paragraph does not meet the metric's applicability condition; no outer Option wraps the record, because each slot gates on its own.
Engine::annotate skips quotes when it parses, so a paragraph whose kind is ParagraphKind::Quote reaches compose with no sentences and keeps all three slots at None.
Methods
| Signature | Returns | Behavior |
|---|---|---|
new(text: String, kind: ParagraphKind, span: Span) | Paragraph | Constructor. Empty sentences, every metric None |
word_count(&self) | usize | Non-punctuation tokens summed across this paragraph's sentences |
sentence_count(&self) | usize | Number of sentences |
Section
| Field | Type | Contents |
|---|---|---|
heading | Option<String> | Heading text, its markup removed |
level | usize | Heading depth |
span | Option<Span> | Where the heading lies in the analysed text, markers included; None with no heading |
paragraphs | Vec<Paragraph> | Paragraphs in document order |
The markdown reader sets level to the heading's level (the number of # characters, or 1 and 2 for a setext heading) and heading to its text. Content before the first heading becomes a section with heading: None and level: 0. Plain text produces at most one section, always with heading: None and level: 0. Block input opens a section at each Block::heading, at the level it names.
Section::new(heading: Option<String>, level: usize, span: Option<Span>, paragraphs: Vec<Paragraph>) -> Section is the constructor. There are no other methods.
Document
The output of the pipeline.
| Field | Type | Contents |
|---|---|---|
provenance | Provenance | Where the document came from and what produced it, recorded by Engine::annotate |
sections | Vec<Section> | The section tree, which owns every paragraph, sentence, and token |
metrics | DocumentMetrics | The document's aggregates, filled by Engine::compose |
skipped | Vec<Skipped> | What the reader saw and did not analyse, in document order |
DocumentMetrics holds what the command line's table prints and what a binding would otherwise rebuild:
| Field | Type | Contents |
|---|---|---|
total_words | usize | Non-punctuation tokens across every sentence |
total_sentences | usize | Sentences across every paragraph |
paragraph_count | usize | Paragraphs, quotes included |
mean_sentence_length | Option<f64> | total_words over total_sentences. None with no sentences |
sentence_length_std | Option<f64> | Sample standard deviation of sentence length in words. None with fewer than two sentences, where it is undefined |
vocabulary_ttr | Option<f64> | Type-token ratio over lemmas. Falls as text grows, so documents of different lengths are not comparable on it |
nominalization_ratio | Option<f64> | Share of nominalizing nouns |
passive_ratio | Option<f64> | Passive sentences over total sentences. None with no sentences |
Before the document is measured every count is 0 and every ratio None; None after it means the document did not meet the metric's condition, never a computed zero. Engine::analyze always measures, so this holds for a document you build by hand with Document::new.
Provenance holds matra, the version that produced the document; schema, the schema version as "major.minor" (SCHEMA_VERSION, "2.0"), which appears nowhere else; parser, the parser's Identity; format, the Format the input was read as; and source, a ProvenanceSource with the input's path (None for anything but a file), and the length in bytes and the sha256 of the analysed text every span points into, computed once at annotation. For markdown and plain text that is the input; for block input, the blocks joined. On a document, method and parameters are None and absent from the JSON; an outcome derived from the document carries a copy of its provenance with both set (see Summary). There is no timestamp, so two analyses of the same input by the same build are identical. Identity names a model-backed component: name (udpipe), model (english-ewt-ud-2.5-191206), sha256 of the model artifact, and license. The UDPipe adapter derives it from the bytes it loaded, so a model matra does not pin reports its digest with model and license set to None; an embedder reports one in the same shape (model2vec for Model2Vec), recorded in a cluster result's parameters.embedder. Provenance::new(parser, format, source) fills matra and schema from the build.
Methods
| Signature | Returns | Behavior |
|---|---|---|
new(provenance: Provenance, sections: Vec<Section>, skipped: Vec<Skipped>) | Document | Constructor. An empty metric record |
paragraphs(&self) | impl Iterator<Item = &Paragraph> | Every paragraph, in document order |
paragraphs_mut(&mut self) | impl Iterator<Item = &mut Paragraph> | The mutable form, which the engine uses as it parses and measures |
sentences(&self) | impl Iterator<Item = &Sentence> | Every sentence across every paragraph |
tokens(&self) | impl Iterator<Item = &Token> | Every token across every sentence |
Summary
What summarize and summarize_with return. The selector is matra::SummaryMethod, at the crate root (TfIdf, the default, and TextRank, #[non_exhaustive]).
| Field | Type | Contents |
|---|---|---|
provenance | Provenance | The document's, with method (tfidf or textrank, an open set) and parameters (n) |
sentences | Vec<SummarySentence> | At most n sentences, in document order |
SummarySentence is one sentence a summary keeps:
| Field | Type | Contents |
|---|---|---|
index | usize | The sentence's position among the document's sentences (Document::sentences), counting from 0 |
span | Span | Where it lies in the analysed text, as on the Sentence |
text | String | Its text, as on the Sentence |
score | f64 | The method's score, higher is more representative |
Both summarizers select the top n by score and then return them in ascending index, so the result reads in document order while score carries the ranking. A score compares only with another from the same method.
Parameters records what shaped an outcome, each field present only when the method takes it: n (Option<usize>) for a summary and keyphrases, threshold (Option<f64>) and embedder (Option<Identity>, the identity the embedder reports, the same shape as the parser's) for clusters. In JSON an absent parameter is an absent key, so a summary's is {"n": 3}.
Keyphrases
What keyphrases and keyphrases_with return. The selector is matra::KeyphraseMethod (Rake, the default, and Yake, #[non_exhaustive]).
| Field | Type | Contents |
|---|---|---|
provenance | Provenance | The document's, with method (rake or yake, an open set) and parameters (n) |
phrases | Vec<Keyphrase> | At most n phrases, highest score first; equal scores in the order the phrases are first met, the same on every run |
Keyphrase is one ranked phrase:
| Field | Type | Contents |
|---|---|---|
phrase | String | The phrase, built from lowercased lemmas joined by single spaces |
score | f64 | Relevance score, higher is more relevant |
occurrences | Vec<Span> | Every stretch of the analysed text it was found in, in document order, from its first word's start to its last word's end |
Clusters
What clusters returns: provenance (the document's, with method cosine and parameters threshold and embedder), clusters (Vec<Cluster>, each members, sentence indices) and edges (Vec<SimilarityEdge>, each a, b and score: f64). Semantic clusters describes each field and what co-membership does and does not mean.
Format
Which reader a document needs.
| Variant | Wire name | Assigned to |
|---|---|---|
Markdown | markdown | Extension .md or .markdown |
PlainText | plain_text | Every other extension, and a file with no extension |
Blocks | blocks | No extension: a caller names it, and Input::blocks takes it |
Pdf | pdf | Extension .pdf, or bytes opening with %PDF |
Docx | docx | Extension .docx, or a zip archive holding [Content_Types].xml |
Other(String) | the name it holds | Never detected: a caller names it, built with Format::named |
Format::from_path(path: impl AsRef<Path>) -> Format is the extension column, and it is the only copy of it. Reading a file with Input::file applies it, and the command line reads it for the name given to --stdin-filename when --format names none, so a name means the same thing on both routes. When the bytes are read, a PDF or DOCX signature outranks both the extension and a format the caller names.
Pdf, Docx and an Other have no reader that ships, so analyzing such a document returns Error::UnsupportedFormat with that format, unless the engine was given a reader for it with Engine::with_reader. Other names a format matra has no variant for, the one a caller's own reader reads: Format::named("html"). Format::named gives the variant matra has for a name it knows, so Format::named("markdown") is Markdown, and a name read back from JSON does the same, an unknown one reading as Other rather than failing. Input::text and Input::file_as take the format as an argument; Input::bytes takes one optionally; Input::file detects one. Format derives PartialEq and Eq, which is what the reader table keys on.
Block
A heading or a paragraph a caller supplies, for a document matra has no reader for. Input only: built by Block::heading(level: u8, text: impl Into<String>) and Block::paragraph(text: impl Into<String>), handed to Input::blocks, and never returned. It has no public fields.
The analysed text of block input is the blocks' texts joined by a blank line ("\n\n"), in order, and every span in the result points into it. A heading opens a section at its level; a level of 0 is refused with Error::InvalidInput when the input is analysed. The same blocks as JSON are the blocks format: an array of {"kind": "heading", "level": 1, "text": "..."} and {"kind": "paragraph", "text": "..."} objects, with an unknown kind or field refused by name.
Span
| Field | Type | Contents |
|---|---|---|
byte_start | usize | The first byte |
byte_end | usize | One past the last byte |
A stretch of the analysed text in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is the stretch. Bytes rather than characters: a caller in a language that indexes strings by character converts. The unit is in the field names because a byte span used to slice a character-indexed string, a Python str, is right on ASCII and wrong, with no error, after the first character outside it. Span::new(byte_start, byte_end) is the constructor.
Skipped
| Field | Type | Contents |
|---|---|---|
kind | SkippedKind | What the construct is |
span | Span | Where it lies in the analysed text, trailing whitespace excluded |
One construct the reader saw and did not analyse. SkippedKind names it: Table, Code, FrontMatter, Html, ThematicBreak, LinkDefinition (table, code, front_matter, html, thematic_break, link_definition on the wire), a set a reader treats as open. Skipping is specified behaviour beside a correct result, not an error. Skipped::new(kind, span) is the constructor.
Together with the heading and paragraph spans, skipped accounts for the input: every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span.
DocumentError
| Field | Type | Contents |
|---|---|---|
path | Option<PathBuf> | Where the failure occurred, None for input that is not a file |
error | Error | What went wrong |
One per-document failure: the Err item Engine::analyze_all yields in the failed input's place, so documents plus errors equals inputs consumed. Display prints path: error when a path exists and the bare error otherwise; source() returns the wrapped Error. DocumentError::new(path, error) is the constructor. Not Clone, because Error wraps std::io::Error.
It serializes as its error's form plus the path, {"path", "kind", "message", "details"}, with path null for input that is not a file and written lossily where it is not UTF-8. The JSON Schema is spec/schema/v2/error.schema.json.
matra collects nothing across documents: a stream is the collection, and how to combine documents is the caller's decision (a count sums; vocabulary_ttr does not).
Constants, aliases, and the error type
| Item | Definition | Role |
|---|---|---|
limits | module of usize constants | Every cap and metric minimum, below |
SCHEMA_VERSION | &str, "2.0" | The document schema version this build emits, as "major.minor", written into Provenance::schema. The JSON Schema for each major is in spec/schema/ |
Result<T> | std::result::Result<T, Error> | Return type of every fallible library function |
Error | enum, #[non_exhaustive] | Every failure the library returns. See Errors |
ErrorDetails | ordered names and values; get(key), iter(), len(), is_empty() | What Error::details returns: the values a caller acts on, by name. Serializes as an object. See kinds and details |
DetailValue | enum, #[non_exhaustive]: Text(String), Integer(u64), Bool(bool) | One value in an ErrorDetails. Serializes as the bare value |
Limits
domain::limits holds every bound matra holds its work to, each defined once and read where it applies. They are published so you can size work before a call instead of learning a bound from a refusal: split an input under MAX_INPUT_BYTES, or chunk a document's sentences under the cap of the method you select, each named for that method as SummaryMethod and KeyphraseMethod name it. They are constants, not settings: no configuration changes them. They are this release's values and may change in any 0.x release, so read them from domain::limits rather than copying the numbers. A cap that is exceeded returns InputTooLarge, whose limit detail is the constant and whose what detail names the gate; a paragraph outside a metric's bounds keeps that metric None. Each constant's rustdoc carries the cost model behind its value.
| Constant | Value | Bounds |
|---|---|---|
MAX_INPUT_BYTES | 8,388,608 (8 MiB) | Text one input may hold; the "input" and "file_read" gates |
MAX_CONFIG_FILE_BYTES | 65,536 (64 KiB) | The configuration file; the "config_file_read" gate |
MAX_DOWNLOAD_BYTES | 67,108,864 (64 MiB) | Any one model download; the "parser_model_download" and "embedding_model_download" gates |
MAX_TFIDF_SENTENCES | 2,000 | Sentences a TF-IDF summary ranks; "tfidf" |
MAX_TEXTRANK_SENTENCES | 2,000 | Sentences a TextRank summary ranks; "textrank" |
MAX_CLUSTER_SENTENCES | MAX_TEXTRANK_SENTENCES | Sentences or vectors clustered; "clusters". Defined as TextRank's, since both build the same quadratic graph |
MAX_RAKE_TOKENS | 200,000 | Tokens RAKE reads, punctuation included; "rake" |
MAX_YAKE_TOKENS | 200,000 | Tokens YAKE reads, punctuation included; "yake" |
MAX_COMPRESSION_PARAGRAPH_BYTES | 262,144 (256 KiB) | The largest paragraph given a compression_ratio |
MIN_READABILITY_WORDS | 11 | The fewest words for a readability_grade (more than 10) |
MIN_COMPRESSION_WORDS | 51 | The fewest words for a compression_ratio (more than 50) |
Two caps with the same value stay two constants where their costs differ, so tuning one does not move the other: TF-IDF's 2,000 is not TextRank's, and YAKE's 200,000 is not RAKE's.
What crosses the language boundary
The Python surface serializes values with pythonize. Fields have a serde representation and cross with their names unchanged. Methods have none, so there is nothing for them to cross with.
Token::feat, Document::sentences, Sentence::subtree, and every other method in the tables above are available to Rust callers only. The two methods on Error, kind and details, are different: their values are computed into the error's serde form and onto the Python exception, so they cross as fields. Nothing a reader needs is method-only: every aggregate is a field of DocumentMetrics or ParagraphMetrics, and a sentence carries its word_count, tree_depth and passive, so each crosses with the record that holds it.
| Rust type | Python shape |
|---|---|
Token | Token |
Sentence | Sentence |
Paragraph | Paragraph |
Section | Section |
Document | matra.Document, read-only, whose to_dict() is a DocumentDict |
DocumentMetrics, ParagraphMetrics | DocumentMetrics, ParagraphMetrics |
Provenance, Identity, ProvenanceSource | Provenance, Identity, ProvenanceSource |
ClausalComplement, RootMood | ClausalComplement, RootMood |
Span, Skipped | Span, Skipped |
Parameters | Parameters |
Summary, SummarySentence | Summary, SummarySentence |
Keyphrases, Keyphrase | Keyphrases, Keyphrase |
Clusters, Cluster, SimilarityEdge | Clusters, Cluster, SimilarityEdge |
DocumentError | DocumentError |
Error | matra.ErrorInfo, and the exception classes under matra.MatraError |
ErrorDetails | ErrorDetails |
Block | BlockDict, a HeadingBlock or a ParagraphBlock |
Format | its name, a FormatName |
Document and DocumentError are the two items Python's Engine.analyze_all yields. A Document stays a Rust value behind a read-only handle, and every read of it crosses through serde like every shape above it, with one field replaced: its provenance.source.path is decoded with the platform's filesystem decoding (os.fsdecode on Unix), so a name that is not valid UTF-8 crosses exactly instead of as serde's lossy string. A DocumentError crosses through serde too, as {"path", "kind", "message", "details"}, with its path decoded the same way. kind is Error::kind on the domain type, pinned across bindings by spec/tests/corpus/items.json and exported to Python as matra.ERROR_KINDS; message is the Rust error's own Display text; details is Error::details. An Error raised rather than returned crosses as a Python exception carrying the same three, kind, str(exc) and details: a method is computed into a value before it crosses, as an aggregate is. A future binding takes the same serde form.
The Python shapes are TypedDict declarations available at runtime. They are declared in matra.types and re-exported from the package root, so either import works:
from matra.types import DocumentDict, Keyphrases, Summary
from matra import DocumentDict, Keyphrases, Summary
Field names and nesting match the Rust types exactly. Option<f64> becomes float | None; Option<String> becomes str | None; Vec<T> becomes list[T]; usize becomes int.
Platforms and models
What each of matra's install routes needs and installs, and the models matra downloads. Install matra has the steps; One core, three install routes explains the choices behind them.
Install routes
| Route | Command | Builds from source | Features in the build |
|---|---|---|---|
| Rust library | cargo add matra@0.3 | yes | udpipe (the default) |
| CLI binary | cargo install matra --version '^0.3' --features cli | yes | udpipe cli |
| Python package | pip install 'matra>=0.3' | only where no wheel fits | udpipe model2vec python cli |
The Python package installs the library and the matra command, which is the Rust CLI reached through the extension module, and has no runtime dependencies of its own.
Toolchains
| Needed for | Requirement |
|---|---|
| Every route that builds from source | Rust 1.88 or later, the minimum supported Rust version |
| Every route that builds from source | A C++ compiler: UDPipe is a C++ library, which udpipe-rs builds during the cargo build. A C compiler alone is not enough. |
| The Python package | CPython 3.12 or later |
Python wheels
| Platform | Wheel |
|---|---|
| Linux x86_64, Linux aarch64 | manylinux2014: glibc 2.17 or newer, which covers Debian 11, Ubuntu 20.04, RHEL 8 and Amazon Linux 2 |
| macOS x86_64, macOS arm64 | yes |
| Windows | none; the UDPipe build under MSVC is unverified, so the sdist route there is untested |
The wheels are built against the CPython stable ABI and tagged abi3, so one wheel per platform serves 3.12 and every later 3.x on GIL-enabled CPython. A free-threaded interpreter (python3.14t, for one) accepts a different tag, abi3t, so it gets no wheel on any platform and builds the sdist; pyo3 gains the free-threaded stable ABI at 3.15. Wherever no wheel fits, pip builds the sdist, which needs the toolchains above.
Every release file from 0.2.1 on carries a build provenance attestation; Verifying a release has the commands.
Version requirements
| Requirement | Used by | Accepts |
|---|---|---|
matra@0.3, --version '^0.3' | cargo | 0.3.0 or a later release semver-compatible with it, so not 0.4.0 |
'matra>=0.3' | pip, uv, uvx | 0.3.0 and every later release |
An install line with no version takes the newest release on the registry. 0.1.0 is still published, and its matra command is a separate Python implementation with its own --json shape and a model directory fixed at ~/.matra/models. 0.2 is published too, with the surface Migrate from 0.2 to 0.3 moves you off.
Build features
matra --version prints the version, then the features the build was compiled with:
matra 0.3.0
features: udpipe cli
That is the CLI binary from cargo install. The Python package prints features: udpipe model2vec python cli. Every analysis command behaves the same in both builds except matra clusters, which needs model2vec and refuses, naming it, in a build without; cargo install matra --features cli,model2vec builds one with it.
The model directory
The library, the CLI and the Python package resolve the model directory the same way, first match wins:
- A directory passed explicitly:
--model-diron the command line,Config::with_model_dirin Rust, or the directory argument of the constructors that take one, such asModel2Vec::potion_base_8m. MATRA_MODEL_DIR.- The
modelsdirectory of the data root, which isMATRA_DATA_DIR, else$XDG_DATA_HOME/matra, else~/.local/share/matra. - A pre-existing, non-empty
~/.matra/modelsfrom an older install, when the location in 3 does not exist yet. matra never creates~/.matra; a selected legacy cache is used as the model directory, downloads included.
Where matra keeps things has the config file and data root beside it, and matra config show prints which rung each value came from.
The model download
| Parsing model | Embedding model | |
|---|---|---|
| Fetched by | Engine::new and Engine::from_config, in Rust and in Python, and the command line, on first use | Model2Vec::potion_base_8m, Model2Vec::from_config, their Python forms and matra clusters, on first use |
| Size | 16.3 MB | 30.9 MB, three files |
| Verified against | a SHA-256 pinned in the source | a SHA-256 over the three files, pinned in the source |
| Download cap | 64 MiB | 64 MiB per file |
The download is held in memory, checked against the pinned hash, and written only if it matches, so nothing unverified reaches the model directory and an interrupted transfer leaves nothing behind. A mismatch downloads once more; a second mismatch is an error, and nothing is loaded. A cached parsing model that fails the check is replaced only when new bytes verify, so a re-download that cannot reach the network leaves the file that was there. The embedding model is never written over files already in its directory; Provisioning has what it does instead.
The library prints nothing while it downloads. The command line prints two lines on stderr before either model's download, naming the model, its size, where it is going and its license; --quiet silences them. Errors has what each failure returns.
Model licenses
matra's code is MIT. The models it downloads are separate files, published by other people under licenses of their own.
| Model | Used for | Source | License |
|---|---|---|---|
english-ewt-ud-2.5-191206.udpipe | every parse | LINDAT record 11234/1-3131, published by ÚFAL, Charles University | CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International) |
| potion-base-8M | semantic clusters only | minishlab/potion-base-8M on Hugging Face | MIT, per its model card |
The parsing model's license does not permit commercial use. The UDPipe models page states that the UDPipe 1 models it publishes for Universal Dependencies are distributed under CC BY-NC-SA. This page reports what the licenses say and does not interpret them; read the license text for the terms.
The command line and the config file always use this parsing model: --model-dir, MATRA_MODEL_DIR and the other settings change where it is stored, not which model it is. models.udpipe, which earlier releases wrote into the config file, is gone, and a file that still sets it fails to load, saying so. No surface loads a different UDPipe model file: the model is a pin.
Semantic clusters
The value semantic clustering returns, the model it embeds with, and its bounds. Cluster sentences by meaning has the calls; What the clustering threshold does explains what a cluster means.
The value
clusters(&doc, &embedder, threshold) and Python's matra.clusters(doc, embedder, threshold) return a Clusters, a standalone value and never a field on Document. Its JSON Schema is spec/schema/v2/clusters.schema.json.
Clusters
provenance the document's, plus method "cosine" and parameters:
threshold the cutoff you supplied
embedder the embedder's identity: name, model, sha256, license
clusters each: member sentence indices
edges every pair that cleared the threshold
| Type | Field | Holds |
|---|---|---|
Clusters | provenance | The document's Provenance, with method and parameters. parameters.embedder is the Identity the embedder reports, the shape provenance.parser has: for Model2Vec, name model2vec, sha256 the digest over its three artifacts (the value its model_hash returns), and model potion-base-8M and license MIT when that digest is the pinned model's, None otherwise. parameters.threshold is the cosine cutoff the call was given |
clusters | The connected components, ordered by smallest member index | |
edges | Every pair whose cosine cleared the threshold, ordered by a, then b; both ends lie in one cluster | |
Cluster | members | Sentence indices in document order, sorted, at least two |
SimilarityEdge | a, b | The two sentence indices, a < b |
score | Their cosine similarity, an f64 |
A sentence with no pair above the threshold is in no cluster. Two members of one cluster need not share an edge: they can be joined through a chain of edges.
The threshold is recorded exactly as it was passed. A score is computed in the vectors' own f32 and widened to f64, so it carries more digits than the model resolved: compare scores with a tolerance, not for equality with a number you typed.
Vectors you already hold go through clusters with an Embedder that returns them, in Rust and in Python alike: the clustering always runs over a document's sentences, so its result always carries a provenance.
The model
The Model2Vec adapter, behind the model2vec feature, loads static embedding models in the model2vec artifact format: an embedding matrix (model.safetensors), a tokenizer.json, and a config.json in one directory. clusters and the Embedder port are always compiled; with your own Embedder implementation, clustering needs no feature at all.
| Constructor | Rust | Python | Reaches the network |
|---|---|---|---|
| The configured directory | Model2Vec::from_config(&cfg) | Model2Vec.from_config(), or Model2Vec.from_config(config_file) | when the pinned model is absent |
| The directory the environment names, or the default one; no config file read | Model2Vec::potion_base_8m(dir) with the directory you resolve | Model2Vec.potion_base_8m() | when the pinned model is absent |
A models directory you name, the model in its potion-base-8M subdirectory | Model2Vec::potion_base_8m(dir) | Model2Vec.potion_base_8m(dir) | when the pinned model is absent |
| Files you supply | Model2Vec::from_dir(dir) | Model2Vec.from_dir(dir) | never |
The configured directory is models.embedding (potion-base-8M as shipped) inside the configured model directory.
The reference model
potion-base-8M, 30.9 MB in three files, MIT per its model card. The SHA-256 over its three files, concatenated in the order above, is a constant compiled into the library:
81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4
It is the digest spec/tests/semantic/reference-model.json pins and the value model_hash reports once the model is loaded. For files loaded with from_dir, model_hash is identity rather than verification: it names the artifacts that produced a score, and says nothing about whether they are the ones you meant to fetch.
Provisioning
- The three artifacts are verified as a set, in memory, before anything is parsed. A mismatch downloads once more without writing anything; a second mismatch raises with the directory as the call found it.
- Downloading happens only into a directory holding none of the three file names. If all three are there and the digest does not match, or only some of them are, the call raises and names the directory, having downloaded nothing and deleted nothing.
- Each artifact download is capped at 64 MiB.
Inference
A static model is a lookup table: inference is a row gather, a mean, and a normalize, and the vectors are bit-identical on every platform and in every language binding. Every text is capped at 512 tokens, pre-truncated on bytes and then truncated on token ids, so a longer text is embedded from its opening only. An empty text embeds to the zero vector.
Bounds and failure
| Condition | Rust | Python |
|---|---|---|
| More than 2,000 sentences, checked before the embedding pass runs | InputTooLarge, what is "clusters" | matra.InputTooLargeError, a ValueError |
| Vectors that disagree on dimension, a non-finite value, a non-finite threshold | InvalidInput | matra.InvalidInputError, a ValueError |
| An embedder returning the wrong number of vectors | InvalidInput | matra.InvalidInputError, a ValueError |
The 2,000 cap counts the document's sentences. The contract violations mean the call site or the embedder is wrong, never the text. A zero-magnitude vector has no defined cosine with anything, so it gets no edge at any threshold and no cluster. Errors has the full table, the provisioning failures included.
Use matra from Rust
You already have matra in your Cargo.toml.
Every snippet below assumes it runs inside a function returning matra::domain::Result<()>, which is why ? works.
The no-setup path
One line, no arguments, no model on disk yet:
use matra::Engine;
let engine = Engine::new()?;
Engine::new locates the model directory from the environment, downloads the English UDPipe model into it if it is not already there, and wires the readers this build ships. The directory is MATRA_MODEL_DIR if you set it, otherwise the models subdirectory of $XDG_DATA_HOME/matra, which defaults to ~/.local/share/matra. It reads no configuration file. Programming model has the full resolution order.
When you want your configuration file read, or the resolved values but not all of them, hold the Config yourself:
use matra::config::Config;
let cfg = Config::resolve()?.with_model_dir("/opt/matra/models");
let engine = Engine::from_config(&cfg)?;
with_model_dir puts the directory on the Argument rung, which outranks the environment and the defaults, and cfg.sources() reports where every other value came from. Config is also what Model2Vec::from_config reads. Engine::from_config_with_notice takes a closure called once before each model download, with what is about to arrive and its licence, so a caller that renders can say so.
Where the model lives
The parser is UDPipe with one pinned English model, english-ewt-ud-2.5-191206, and the engine builds it for you, so there is no parser to construct unless you bring your own (below). Name a directory you own: provisioning trusts what it finds there, and a shared one such as /tmp is writable by anybody on the machine. The engine creates the directory if it is missing, downloads the model (about 16 MB) on first use, verifies it against a pinned SHA-256 hash, and loads from the cached file on every call after that. The verified bytes are written to a temporary subdirectory whose name is unique to the call (the process id, the clock in nanoseconds, and a counter) and moved into place with a single rename, so two processes pointed at the same directory cannot corrupt each other's file, even in containers where every process is pid 1.
The path is passed to std::fs::create_dir_all unchanged, and ~ in it is taken literally: neither Rust nor matra expands it, so with_model_dir("~/models") creates a directory literally named ~ under your current working directory. The same holds for MATRA_MODEL_DIR and the other variables when the shell has not already expanded them. Pass an absolute path, or expand the home directory yourself.
Construction is the expensive step: it loads and validates the model once. Build one engine and reuse it for every document. An Engine is Send but not Sync: it moves to another thread, but it is not shared between two, and a stream borrowed from it stays on the thread that holds it. That is a compile error, not a runtime surprise. To parallelize, build one engine on each worker thread, or serialize calls behind a mutex.
The pipeline
matra's public Rust surface is one pipeline assembled from two values. Input says what one document is; Engine says what happens to it.
use matra::domain::Format;
use matra::{Engine, Input, Inputs};
let engine = Engine::new()?;
// One document.
let doc = engine.analyze(Input::text("Plain text in memory.", Format::PlainText))?;
println!("{} sentences", doc.metrics.total_sentences);
// Many documents, lazily and in order. A failure is one item, and the
// walk goes on past it.
for item in engine.analyze_all(Inputs::dir("./corpus")?) {
match item {
Ok(doc) => println!("{:?}", doc.provenance.source.path),
Err(e) => eprintln!("skipped: {e}"),
}
}
Input has five constructors. None of them reads anything or fails: the read, the format and the size cap are settled when the engine reaches the input.
| Constructor | The document is | Format from |
|---|---|---|
Input::text(string, format) | the text itself | the argument |
Input::bytes(data, format) | bytes from anywhere | a PDF or DOCX signature, else the argument, else plain text |
Input::file(path) | the file, read when analysed | a PDF or DOCX signature, else the extension |
Input::file_as(path, format) | the file, read when analysed | a PDF or DOCX signature, else the argument |
Input::blocks(blocks) | headings and paragraphs you supply (Block::heading, Block::paragraph) | blocks |
.md and .markdown read as markdown; any other extension, or none, reads as plain text when the bytes are UTF-8. Format::Blocks names the same headings and paragraphs as a JSON array, for Input::text, Input::bytes or Input::file_as, since no extension implies it, with the schema in spec/schema/v2/blocks.schema.json. Inputs::dir(path) lists one level of a directory, up front, and is an iterator of Input::file; its Err is the listing failing. Input::file on a directory fails with Error::InvalidInput naming Inputs::dir.
Engine exposes the pipeline at two grains:
| Method | Takes | Returns | Use it when |
|---|---|---|---|
analyze | one Input | Result<Document> | one document, end to end |
analyze_all | anything yielding Input | a lazy stream of Result<Document, DocumentError> | any number of documents; a failure is one item carrying the path, and the stream goes on |
The grains always agree: analyze_all is analyze mapped over the inputs, for every Input constructor. Inside, analyze reads and parses the document, then measures it; those two stages are the engine's own, and the library's test suite pins that analysis is exactly the one followed by the other, so the grains cannot drift apart silently.
The readers this build ships are markdown, plain text and block input. Format::Pdf and Format::Docx are recognised, by extension and by signature, and have no reader: analysing one returns Error::UnsupportedFormat with that format, and so do bytes that are not UTF-8 and carry no signature matra recognises (with format: None). The signature outranks the extension and the format you name, so a PDF renamed notes.txt is still refused as a PDF rather than read as text.
What the size gates reject
Seven caps guard the pipeline and the functions over its output. Each one returns Error::InputTooLarge, and the what field names which gate fired so you can route on the label instead of guessing from context.
what | Limit | Fires in |
|---|---|---|
"input" | 8 MiB (domain::limits::MAX_INPUT_BYTES) | the engine's reading stage, which is the only route from text to the parser, on the bytes handed in and on the analysed text, so every Input constructor and every pipeline call inherits it |
"file_read" | 8 MiB on disk | Input::file, checked against file metadata before any read |
"tfidf" | 2000 sentences | summarize, and summarize_with with SummaryMethod::TfIdf |
"textrank" | 2000 sentences | summarize_with with SummaryMethod::TextRank |
"rake" | 200000 tokens summed across all sentences | keyphrases, and keyphrases_with with KeyphraseMethod::Rake |
"yake" | 200000 tokens summed across all sentences | keyphrases_with with KeyphraseMethod::Yake |
"clusters" | 2000 sentences | clusters, before the embedding pass |
Three more labels guard what matra reads that is not your text: "parser_model_download" and "embedding_model_download" cap a model download at 64 MiB, and "config_file_read" caps the config file at 64 KiB. Errors lists all ten.
A document from disk passes the gates in sequence: the metadata check when the file is read, then the byte and text checks before it is parsed. Block input is checked on its analysed text, the blocks joined, so blocks each under the cap that join to more than it are refused. A file at exactly the cap is accepted; one byte over is rejected.
The outcome caps are counted over the parsed document, so a document under 8 MiB can still exceed the TF-IDF sentence cap. Check doc.metrics.total_sentences before calling if you need to decide between summarizing and chunking.
Parse once, use many
The parse inside analyze is the expensive step: it runs the parser once per paragraph. The outcome functions are free functions over the Document it returns, and never parse anything themselves, so they need no engine and run in a build without UDPipe. Analyze once, and hand the same document to as many outcomes as you need:
use matra::{Input, KeyphraseMethod, SummaryMethod};
use matra::domain::Format;
let doc = engine.analyze(Input::text(text, Format::Markdown))?;
let summary = matra::summarize(&doc, 3)?;
let ranked = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Yake)?;
Each outcome reads the same document; nothing re-parses. The sentences it reads are the ones the reader kept, so markdown headings, code, tables and quotes never reach a summary as if they were prose.
The method is optional. summarize runs TF-IDF and keyphrases runs RAKE, the defaults config/default.toml ships; summarize_with and keyphrases_with take a SummaryMethod or KeyphraseMethod, both #[non_exhaustive]. Every result is a record carrying the document's provenance plus the method that ran (tfidf, textrank, rake, yake) and the parameters it ran with (n), because the method decides what a score means: TF-IDF and TextRank, or RAKE and YAKE, score on unrelated scales.
| Function | Returns | Holds |
|---|---|---|
summarize, summarize_with | Summary | sentences, at most n, in document order: each a SummarySentence with its index among the document's sentences, its span, text and score |
keyphrases, keyphrases_with | Keyphrases | phrases, at most n, highest score first: each a Keyphrase with its phrase (lowercased lemmas), score and occurrences, the span of every place it was found |
clusters | Clusters | clusters of sentence indices and every edges pair that cleared the threshold; see Cluster sentences by meaning |
Keyphrases with equal scores come back in the same order on every run, the order the phrases are first met, so a tie at the requested count is decided the same way each time.
Why a metric slot is None
Every ratio in ParagraphMetrics and DocumentMetrics is an Option, and each metric has its own threshold. A None means the metric declined to run, not that it computed nothing. The counts in DocumentMetrics (total_words, total_sentences, paragraph_count) are plain integers.
| Slot | Filled when |
|---|---|
ParagraphMetrics::readability_grade | the paragraph has more than 10 words and is not a quote |
ParagraphMetrics::lexical_density | the paragraph has at least 1 word and is not a quote |
ParagraphMetrics::compression_ratio | the paragraph has more than 50 words, is not a quote, and its text is at most 256 KiB |
DocumentMetrics::vocabulary_ttr | the document's attached sentences hold at least one non-punctuation token |
DocumentMetrics::nominalization_ratio | the same condition as vocabulary_ttr |
DocumentMetrics::passive_ratio, DocumentMetrics::mean_sentence_length | the document holds at least one sentence |
DocumentMetrics::sentence_length_std | the document holds at least two sentences |
"Words" here means Paragraph::word_count(), the non-punctuation token count summed over the paragraph's attached sentences. A paragraph whose sentences vector is empty has a word count of zero and therefore no per-paragraph metrics at all.
Quotes are never parsed. A paragraph inside a block quote, at any depth, has kind ParagraphKind::Quote; the parser never sees it, so its sentences stays empty, its metric slots stay None, and it contributes nothing to total_words or passive_ratio, though it counts in paragraph_count. It stays in the section tree with its text, and its span covers the > markers.
What the markdown reader skips
A markdown document (by format argument or by .md extension) is read as CommonMark, with tables, strikethrough, task lists and front matter. Headings, paragraphs, list items and quotes become the section tree; a # with no space after it, as in #hashtag, is text, and ## References is a heading like any other. Each paragraph's text is what the parser reads, with the inline markup removed (emphasis, link and image syntax, the backticks of a code span) and escapes and entities decoded.
What it does not analyse it records in Document::skipped, each entry a SkippedKind and a span, never dropping it silently:
FrontMatter: YAML between---lines, or TOML between+++lines, at the very startCode: fenced and indented code blocksTable: a table, header and rowsHtml: an HTML blockThematicBreak:---,***or___on a line of its ownLinkDefinition: a link reference definition,[label]: https://example.com
Every byte of the input that is not whitespace lies in exactly one heading span, paragraph span or skipped span. The test suite checks that over every conformance fixture and thousands of generated documents.
Spans
Every heading (Section::span), paragraph, skipped construct, sentence and token carries a Span { byte_start, byte_end } into the analysed text, in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is what it came from. For markdown and plain text the analysed text is the input; for block input it is the blocks' texts joined by a blank line. The text fields are normalized (a sentence's text is rebuilt from its tokens, a paragraph's has its markup removed); the span is the authority on where a value came from. provenance.source records the analysed text's length and SHA-256.
Handle domain::Error
analyze and the outcome functions return domain::Result<T>, an alias for Result<T, domain::Error>. The stream wraps the same error with its path: analyze_all yields DocumentError, whose path field is Some for a file input and whose error field is the domain::Error to match on:
use matra::domain::Error;
for outcome in engine.analyze_all(matra::Inputs::dir("notes/")?) {
match outcome {
Ok(doc) => { /* doc.metrics */ }
Err(doc_err) => match doc_err.error {
Error::InputTooLarge { limit, actual, what } => {
eprintln!("{what} gate: {actual} bytes over the {limit} cap");
}
Error::UnsupportedFormat { format, .. } => {
eprintln!("no reader for {format:?}");
}
Error::Io(e) => eprintln!("could not read the file: {e}"),
Error::ParseFailed(msg) => eprintln!("the parser failed: {msg}"),
e => eprintln!("analysis failed: {e}"),
},
}
}
Error::ParseFailed also carries panics that crossed the UDPipe FFI boundary. The adapter wraps the C-side call in catch_unwind and converts a panic into ParseFailed with the payload message, so a bug inside UDPipe surfaces as a matchable error instead of aborting your process. It is also what the parser returns when it cannot place a token in the text it was given, rather than reporting a span it guessed.
To pass an error on to another program, use what every channel carries rather than the message. e.kind() is a stable string such as "input_too_large". e.details() holds the values to act on by name (every dynamic value in the message, and on invalid_input the reason naming the check that failed), so e.details().get("limit") reads a cap without matching the variant. serde_json::to_value(&e) is {"kind", "message", "details"}, and a DocumentError serializes the same way with its path beside them. Errors lists the keys for every kind.
domain::Error is #[non_exhaustive], so a trailing _ => (or a named catch-all binding, as above) is required even once you have named every variant that exists today. A future variant becomes a compile requirement for code that matches without a wildcard, and a routed catch-all for code that has one.
The domain structs carry #[non_exhaustive] for the same reason. From outside the crate you cannot write Token { .. } or Sentence { .. } as struct literals. Use the constructors, which matters most when you are building fixtures for tests:
Token::builder(id: usize, text: String, lemma: String, pos: String, head: usize, dep: String) -> TokenBuilder
Sentence::new(text: String, tokens: Vec<Token>) -> Sentence
Every string argument is an owned String, not a &str, so a literal needs .to_string() on the way in. TokenBuilder takes the remaining CoNLL-U columns through xpos, feats, deps and misc, each also a String, plus is_punct for the derived flag and span for where the token lies in its text, and build() hands back the Token. Sentence::new derives the sentence's span from its first and last token. feats takes column 6 as CoNLL-U writes it and converts it into the token's features map:
use matra::domain::Token;
let token = Token::builder(
1,
"approved".to_string(),
"approve".to_string(),
"VERB".to_string(),
0,
"root".to_string(),
)
.feats("Mood=Ind|Tense=Past".to_string())
.build();
assert_eq!(token.feat("Tense"), Some("Past"));
assert_eq!(token.features.get("Mood").map(String::as_str), Some("Ind"));
Analyze a directory
Inputs::dir lists one level of a directory, and analyze_all streams it, one result per file:
let (documents, errors): (Vec<_>, Vec<_>) = engine
.analyze_all(matra::Inputs::dir("./corpus")?)
.partition(Result::is_ok);
let words: usize = documents
.iter()
.flatten()
.map(|doc| doc.metrics.total_words)
.sum();
println!("{} documents, {words} words", documents.len());
for err in errors.iter().filter_map(|r| r.as_ref().err()) {
eprintln!("skipped: {err}");
}
Inputs::dir is Err only when the listing itself fails, such as the directory not existing. Everything else is an Err item carrying its path: an unreadable file as Io, a file over the cap as InputTooLarge with what = "file_read", a non-UTF-8 file, a PDF or a DOCX (by signature or by name) as UnsupportedFormat, and a parser failure as ParseFailed. Documents plus errors always equals inputs consumed, and each document names its file in provenance.source.path.
Two kinds of entry are skipped without an error entry, so they will not appear in either list: subdirectories, because the walk is one level deep, and symlinks, which are filtered out with symlink_metadata so a link cannot redirect the read elsewhere. Files are attempted in sorted path order. How to aggregate across documents is yours to decide: a count such as total_words sums, and a ratio such as vocabulary_ttr does not, since it falls as a document grows.
You do not have to collect. The stream from analyze_all is lazy, so a loop that renders each document as it completes holds one document in memory at a time, not the corpus.
Walk a Document
Document holds a sections: Vec<Section> tree; sections hold paragraphs: Vec<Paragraph>; paragraphs hold sentences: Vec<Sentence>. Flat iterators cut across the tree when you do not need the structure:
for sentence in doc.sentences() {
println!("{}", sentence.text);
}
doc.paragraphs() and doc.tokens() flatten at the other two levels.
A Sentence carries its dependency tree in tokens: Vec<Token>, where each token's head field points at another token's id in the same sentence and head == 0 marks the root. Navigate it with the methods Sentence provides rather than walking tokens by hand:
for sentence in doc.sentences() {
let Some(root) = sentence.root_token() else { continue };
println!("root: {} ({})", root.text, root.pos);
for child in sentence.children_of(root.id) {
println!(" {} --{}--> {}", child.text, child.dep, root.text);
}
}
Token fields carry the CoNLL-U columns under Rust names: text is the surface form, pos is the universal POS tag, xpos is the language-specific tag, and dep is the dependency relation. features is the FEATS column as a map from feature name to value (token.feat("Mood") reads one), deps and misc hold their raw column strings, and is_punct is set by the parser.
subtree(id) collects every token reachable from a given token, sorted back into document order, which is the tool for pulling a phrase out from under a dependency relation rather than a single token. Pulling the direct object out of every sentence that has one:
for sentence in doc.sentences() {
for token in &sentence.tokens {
if token.dep == "obj" {
let phrase: Vec<&str> = sentence
.subtree(token.id)
.iter()
.map(|t| t.text.as_str())
.collect();
println!("object phrase: {}", phrase.join(" "));
}
}
}
matra reports the dependency label UDPipe assigned ("obj" here). It does not interpret what the phrase means. That interpretation is your application's job.
The remaining navigation methods: head_of(id) returns the head token, or None for the root and for an id that is not in the sentence; content_tokens() gives you the non-punctuation view. Three counts are fields, computed when the sentence is built: word_count, the non-punctuation token count; passive, whether any token carries nsubj:pass, nsubjpass, or aux:pass; and tree_depth.
tree_depth is the longest root-to-leaf path in the sentence, an Option<usize>. On a malformed parse where tokens form a cycle it is None rather than a depth truncated at some arbitrary ceiling, so a broken parse is a value the compiler makes you handle rather than a number that reads as a deep tree. subtree and children_of stay safe on the same malformed input; both use a visited set.
Bring your own parser or reader
The engine is assembled from three parts, and each is a trait you can implement: the parser (matra::nlp::NlpProvider), a reader for one format (matra::decompose::Decomposer), and the source that reads the paths inputs name (matra::source::Source). Plug one in and the rest of the pipeline, every metric and every outcome function, runs on what it produces, with no fork of matra.
A parser takes one paragraph's text and returns its sentences, each a Vec<Token>. It does not build Sentence values: the engine builds them from your tokens, text, span and every derived field included, so passive, negations and mood mean the same thing whichever parser ran. Engine::with_parser takes it, in any build, udpipe or not:
use matra::domain::{Format, Identity, Result, Span, Token};
use matra::nlp::NlpProvider;
use matra::{Engine, Input};
struct Words;
impl NlpProvider for Words {
fn parse(&self, text: &str) -> Result<Vec<Vec<Token>>> {
let mut tokens = Vec::new();
let mut at = 0;
for word in text.split_whitespace() {
let start = at + text[at..].find(word).unwrap_or(0);
at = start + word.len();
let id = tokens.len() + 1;
let head = if id == 1 { 0 } else { 1 };
tokens.push(
Token::builder(id, word.into(), word.to_lowercase(), "X".into(), head, "dep".into())
.span(Span::new(start, at))
.build(),
);
}
Ok(if tokens.is_empty() { Vec::new() } else { vec![tokens] })
}
fn identity(&self) -> Identity {
Identity::new("words", None, None, None)
}
}
let doc = Engine::with_parser(Words).analyze(Input::text("The river rose.", Format::PlainText))?;
The contract is on the trait. In short: ids run 1..=n in each sentence; every head is 0 or a token of the sentence, and exactly one is 0; the heads form no cycle; and every span is where the token's form lies in the paragraph text you were handed, in UTF-8 bytes relative to that text, in token order. Set is_punct for punctuation and SpaceAfter=No in misc where no space follows, since the sentence text is rebuilt from the forms. identity names your parser and its model, and lands in every document's provenance.parser. examples/custom_parser.rs is a runnable version that splits sentences too: cargo run --example custom_parser --no-default-features.
A reader takes a format's bytes and returns a Read: its blocks, built with ReadBlock::heading(level, text, span) and ReadBlock::paragraph(kind, text, span), and the constructs it skipped. A paragraph's text is the analysed text at its span. For a format that is not text, extract the text and name it with Read::with_text, and every span points into that. Register a reader for a format with with_reader, which composes with the standard constructors:
let engine = Engine::new()?
.with_reader(Format::Pdf, MyPdfReader) // matra recognises PDF and ships no reader
.with_reader(Format::named("html"), MyHtmlReader); // a format matra has no name for
let doc = engine.analyze(Input::file_as("page.html", Format::named("html")))?;
A PDF or DOCX is detected by its bytes, so a registered reader for it runs on Input::file with no format named. A Format::Other is never detected: name it on the input, as above. Replacing a shipped reader, say Format::Markdown, works the same way.
A source takes a path and returns the bytes there. with_source swaps it in for the local filesystem, so Input::file("bucket/notes.md") reads through it, the extension still decides the format, and the path is still what provenance.source.path records. Inputs::dir still lists the local filesystem, and Input::bytes and Input::blocks need no source at all.
The engine trusts none of the three. What a reader returns is checked before the section tree is built, and what the parser returns is checked for each paragraph before a sentence is built from it. A break fails that document with Error::ParseFailed, whose reason names the check (head_out_of_range, token_span_order, block_span_invalid, and the rest Errors lists) and whose cause names the paragraph, sentence and token, or the block. That is also how to test an implementation: run your own texts through Engine::analyze, and it conforms when no document fails with one of those reasons. tests/providers.rs in the repository holds a broken parser and a broken reader for each check. A failure of your own is ParseFailed with reason None, or the kind the trait lists for it, and never a panic: a parser wrapping native code catches a panic at its own boundary, as the UDPipe adapter does.
These traits are Rust only in 0.3.0. Python takes an embedder of your own, and not yet a parser, a reader or a source.
Use matra from Python
You installed matra with uv add 'matra>=0.3' or pip install 'matra>=0.3'. That gave you two things: the library described on this page, and the matra command, which is the Rust CLI reached through the extension module rather than a Python program of its own. Everything the command does is in the CLI guide; the Python package adds no behavior to it.
The Python surface spells what Rust spells. You ask an Engine to analyze a document, then ask for an outcome of the result: summarize, keyphrases or clusters.
import matra
engine = matra.Engine()
doc = engine.analyze(file="README.md")
summary = matra.summarize(doc, n=3)
phrases = matra.keyphrases(doc, n=10, method="yake")
print(doc["metrics"]["total_words"])
The no-setup path
No argument, no model on disk yet:
import matra
engine = matra.Engine()
Engine() downloads the English UDPipe model on first use, verifies it against a pinned SHA-256 hash, and loads from the cache after that. Before the download it writes two lines to sys.stderr, the ones the command line prints for the same wait:
matra: downloading english-ewt-ud-2.5-191206.udpipe (16.3 MB) into /home/you/.local/share/matra/models
matra: the model is licensed CC BY-NC-SA 4.0 (non-commercial), separately from matra: https://creativecommons.org/licenses/by-nc-sa/4.0/
Once the model is on disk nothing is written. Engine.from_config(), Model2Vec.potion_base_8m() and Model2Vec.from_config() announce a download the same way. Redirect sys.stderr if a notice must not reach your terminal. It finds the model directory from the environment alone: MATRA_MODEL_DIR if you set it, otherwise the models subdirectory of $XDG_DATA_HOME/matra, which defaults to ~/.local/share/matra. A pre-existing ~/.matra/models is used instead when the new location does not exist yet and that one is not empty, so a cache from 0.1.0 keeps working. Programming model has the full resolution order.
To put the model somewhere specific, set MATRA_MODEL_DIR before you build the engine. Name a directory you own: provisioning trusts what it finds there, and a shared one such as /tmp is writable by anybody on the machine. The value is used as a path unchanged, so ~ is taken literally.
Engine.from_config() is the same engine with a configuration file read and checked as well: the one the environment names, or the one you pass, Engine.from_config("matra.toml"), which must exist. The file does not move the model directory, so most callers want Engine().
The two ways provisioning can fail raise different exceptions, and the split is the one the errors reference describes. A download that never arrived, or a directory that could not be written, raises matra.IoError, an OSError. Bytes that arrived and then failed the hash raise matra.ModelInvalidError, a RuntimeError. A bootstrap that wants to survive both catches matra.MatraError. A download that did not arrive carries details["retryable"], which says whether calling again may succeed.
The parser model is pinned. There is no way to load another model file from Python, as there is none from Rust.
Construction is the expensive step. Build one Engine and reuse it across calls.
Threads and processes
The underlying model holds C-side state that is not thread-safe, so the Rust binding marks the class unsendable. Touching one Engine from any thread other than the one that created it raises pyo3_runtime.PanicException. That exception derives from BaseException, not Exception, so a bare except Exception will not catch it and a thread pool will not report it the way you expect. Do not share an engine across threads. A Document is not tied to a thread.
Separate processes are fine, because each process gets its own model state. Build the engine once per worker with an initializer rather than once per task, so the model is loaded as many times as you have workers and not as many times as you have chunks:
from concurrent.futures import ProcessPoolExecutor
import matra
_engine: matra.Engine | None = None
def _init() -> None:
global _engine
_engine = matra.Engine()
def analyze_chunk(text: str) -> matra.DocumentDict:
assert _engine is not None
return _engine.analyze(text).to_dict()
if __name__ == "__main__":
chunks = ["The report was filed.", "Nobody read it.", "The committee met again on Friday."]
with ProcessPoolExecutor(initializer=_init) as pool:
results = list(pool.map(analyze_chunk, chunks))
print(len(results), [r["metrics"]["total_words"] for r in results])
The if __name__ == "__main__": guard is required. On macOS and Windows a worker process starts by importing your script afresh (the spawn start method), so a pool created at module level would start again inside every worker, and Python stops that with a RuntimeError about the bootstrapping phase. Under the guard, only the parent builds the pool. A Document does not pickle, so a worker returns to_dict(). Concurrent first runs against the same model directory are safe. Each process downloads into its own temporary subdirectory and moves the file into place with a single rename.
Analyze one document
Engine.analyze takes exactly one input, by keyword, and returns a Document:
| Call | Reads |
|---|---|
analyze(text) | a str, as plain text, or as format when you name one |
analyze(file=path) | a file, its format detected from its name and bytes, or as format when you name one |
analyze(bytes=data) | bytes, as format, or detected when you name none |
analyze(blocks=[...]) | headings and paragraphs you supply |
doc = engine.analyze("The committee approved the proposal without debate.")
doc = engine.analyze(text, format="markdown")
doc = engine.analyze(file="notes.md")
A str is always text, never a path: engine.analyze("notes.md") analyzes the eight characters notes.md. Name a file with file=. Passing no input, or two, raises TypeError at the call. format is one of markdown, plain_text and blocks, and it applies to text, bytes and file. With no format, a file is read by what it is: .md as markdown, anything else that is UTF-8 as plain text, and a PDF, a DOCX or bytes that are not text as matra.UnsupportedFormatError. A JSON file of blocks is engine.analyze(file="notes.json", format="blocks"), since no extension implies it; a PDF or DOCX signature still outranks the format you name.
A file's path travels into the result as doc["provenance"]["source"]["path"]. On Unix it is decoded the way Python decodes any filesystem path, so os.fsencode on it hands back the bytes the name came from, even when the name is not valid UTF-8. Every path argument takes a str or a pathlib.Path alike.
Text with its own section breaks
Text that comes from somewhere matra does not read (a PDF you extracted, a CMS, a transcript) can arrive as headings and paragraphs, with no markdown in between:
doc = engine.analyze(blocks=[
{"kind": "heading", "level": 1, "text": "Field notes"},
{"kind": "paragraph", "text": "The river rose overnight."},
])
Each block is a BlockDict, a heading (kind, level, text) or a paragraph (kind, text). It is the same blocks input format the JSON schema in spec/schema/v2/blocks.schema.json publishes, read by the same reader, so a kind or a key matra does not know raises matra.InvalidInputError naming it. The analysed text is the blocks' texts joined by a blank line, and every span points into it.
Analyze many documents
Engine.analyze_all takes a directory or an iterable of texts and returns an iterator, one item per document, in input order:
for item in engine.analyze_all(dir="docs/"):
if isinstance(item, matra.Document):
print(item["provenance"]["source"]["path"], item["metrics"]["vocabulary_ttr"])
else:
print(f"{item['path']}: {item['kind']}: {item['message']}")
Nothing is read or parsed until you pull the next item, so a large directory costs nothing up front. One unreadable file costs one item, not the walk. A document that analyzed arrives as a Document. One that did not arrives as a DocumentError dict (path, kind, message, details), holding the position the document would have had. Test isinstance(item, matra.Document) to tell them apart; a type checker narrows the union on that test.
analyze_all(texts=[...]) does the same for texts you hold, read as plain text or as format. A str alone raises TypeError, since it would otherwise be read one character at a time.
The walk is not recursive, and symlinks and subdirectories are skipped rather than followed. Only a failure listing the directory itself raises: a missing directory is matra.IoNotFoundError, a FileNotFoundError, because there is no per-document result for it to travel in.
Read a document
A Document holds the parse matra made, and you read it as the JSON reads:
doc["metrics"]["total_words"]
doc["sections"][0]["heading"]
doc["provenance"]["schema"] # "2.0"
whole = doc.to_dict() # the whole document, as a dict
The keys are provenance, sections, metrics and skipped. list(doc), len(doc) and "metrics" in doc work as they do on a dict. Each read builds a new value from the parse, so changing what a read returned changes nothing in the document. Hold a field in a variable when you read it in a loop.
A Document is read-only. There is nothing to assign to, and the outcome functions take a Document and never a dict, so a summary cannot carry provenance no matra run produced. A Document does not pickle; to_dict() is what you store or send.
Summaries, keyphrases and clusters
The outcome functions run on a Document you already have, so a text is parsed once however many outcomes you ask for:
| Function | Returns |
|---|---|
matra.summarize(doc, n=3, *, method=None) | Summary |
matra.keyphrases(doc, n=10, *, method=None) | Keyphrases |
matra.clusters(doc, embedder, threshold=0.85) | Clusters |
summary = matra.summarize(doc, n=3)
summary = matra.summarize(doc, n=3, method="textrank")
phrases = matra.keyphrases(doc, n=10, method="yake")
method names how the ranking is done: tfidf or textrank for a summary, rake or yake for keyphrases. With none, matra's default runs, which is tfidf and rake today and the same on the command line. An unknown name raises matra.InvalidInputError. The defaults for n are the command line's.
Each result is a dict with a provenance, the document's own with the method that ran and the parameters it ran with ({"n": 3}), beside the result itself. A summary returns its sentences in document order, not score order: read index to see where a sentence sits among the document's sentences, span for where it lies in the text, and score to see how it ranked. Keyphrases return their phrases in score order, highest first, each with occurrences, the span of every place the phrase was found.
Keyphrases come back as lowercased lemmas joined with spaces, not as the surface text. A document about "Dependency Parses" yields the phrase dependency parse; its occurrences' spans point at the surface text. Phrases with identical scores come back in the same order on every run, so a tie straddling the n cutoff is decided the same way each time.
The score on a Keyphrase is an ordering key and nothing more. RAKE's is a sum of per-word ratios that are each at least 1, so a phrase of k words scores at least k, which is a floor per length and not an order across lengths; YAKE's is the reciprocal of a product of per-word scores, so it is unbounded above, carries no unit, and does not track phrase length in either direction. The two scales are unrelated to each other, and neither is comparable across documents or with another tool's. Do not surface either number to a reader as a strength without saying which method produced it. The CLI guide works through what the ranking does and does not tell you, and Methodology carries the formulas.
Bring your own embeddings
matra.clusters takes the embedder as an argument, always. Model2Vec is one, and so is any object with two methods:
class MyEmbedder:
def embed(self, texts: list[str]) -> list[list[float]]:
return my_service.encode(texts) # one vector per text, in order
def identity(self) -> matra.types.Identity:
# Names the geometry, not the run, in the shape the parser uses.
return {"name": "my-service", "model": "v3", "sha256": None, "license": None}
found = matra.clusters(doc, MyEmbedder(), threshold=0.85)
assert found["provenance"]["parameters"]["embedder"]["model"] == "v3"
Embedder in matra.types is the protocol those two methods satisfy, and it is what clusters is annotated with. Importing it is optional and buys you a type checker's opinion, not a runtime check.
The object is asked for its identity before anything is embedded, so an embedder that cannot name its geometry is refused for the price of one method call.
The contract is the one the Rust port carries: exactly one vector per sentence, in order, every vector the same length. Break it and the call raises matra.InvalidInputError, a ValueError, with the same message a Rust implementor gets, because the check is in the library and not in the binding. An exception raised inside your embed arrives as matra.InvalidInputError too, with your exception's own text inside the message.
identity returns the shape provenance["parser"] has, matra.types.Identity: a name, and the model, its sha256 and its license, each None when you do not know it. It is read once, when the object is handed over, and travels into the result's provenance as parameters["embedder"], so the model's licence travels with the scores as the parsing model's does with the parse. A bare string is refused. Two embedders that can disagree must not report the same identity, or scores end up attributed to a geometry that did not produce them.
Size limits
Every input over 8 MiB is rejected with matra.InputTooLargeError, a ValueError, whose details carry what, limit and actual. Text, bytes and blocks are checked as input; a file is checked before it is read, as file_read. The gate lives in the one step every input passes through to reach the parser, so there is no input that skips it. In analyze_all, an oversized input is one item whose kind is input_too_large, and the stream goes on.
The outcome functions add caps of their own: 2000 sentences for a summary and for clusters, and 200000 tokens for keyphrases, each raising matra.InputTooLargeError with the method in details["what"].
Every cap, and the word counts a paragraph needs before it gets a readability grade or a compression ratio, is a constant in matra.limits, under the name Rust's domain::limits gives it, read from the extension so it is the value the library enforces:
from pathlib import Path
import matra
data = Path("transcript.txt").read_bytes()
if len(data) > matra.limits.MAX_INPUT_BYTES:
... # split it before analysing it
The eleven are MAX_INPUT_BYTES, MAX_CONFIG_FILE_BYTES, MAX_DOWNLOAD_BYTES, MAX_TFIDF_SENTENCES, MAX_TEXTRANK_SENTENCES, MAX_CLUSTER_SENTENCES, MAX_RAKE_TOKENS, MAX_YAKE_TOKENS, MAX_COMPRESSION_PARAGRAPH_BYTES, MIN_READABILITY_WORDS and MIN_COMPRESSION_WORDS.
The shapes of the data
A Document reads as a dict, and every outcome returns one. DocumentDict, Section, Paragraph, Sentence, Token, Summary, Keyphrases and Clusters are TypedDict definitions in matra.types that describe the shapes at both runtime and type-check time:
from matra import Clusters, DocumentDict, Keyphrases, Paragraph, Section, Sentence, Summary, Token
A document nests down to the token level:
for section in doc["sections"]: # heading, level, span, paragraphs
for para in section["paragraphs"]: # span, text, kind, sentences,
# metrics
print(para["metrics"]["readability_grade"]) # float or None
for sentence in para["sentences"]: # span, text, tokens, word_count,
# tree_depth, passive, negations,
# modals, clausal_complements,
# root_adverbials, hearst_pairs,
# root_mood
for token in sentence["tokens"]:
print(token["lemma"], token["pos"], token["dep"])
print(doc["metrics"]["vocabulary_ttr"]) # float or None
print(doc["metrics"]["nominalization_ratio"]) # float or None
print(doc["skipped"]) # [{"kind": "table", "span": {...}}, ...]
Every heading, paragraph, skipped construct, sentence and token has a span, {"byte_start": int, "byte_end": int}, into the analysed text. It counts UTF-8 bytes, not characters, as the field names say: text[span["byte_start"]:span["byte_end"]] on a str is right on ASCII and wrong, with no error, after the first character outside it. Slice with matra.span_text(text, span), which encodes the text and decodes the stretch for you. The text fields are normalized (a sentence's text is rebuilt from its tokens, a markdown paragraph's has its markup removed); the span is where the value came from.
A Token carries the ten CoNLL-U columns under these keys: id, text, lemma, pos, xpos, features, head, dep, deps, misc, plus its span and the derived is_punct. features is a dict from feature name to value, {"Number": "Plur"}. head is the id of the governing token in the same sentence, and 0 marks the root.
Summary is provenance and sentences, each a SummarySentence of index, span, text and score. Keyphrases is provenance and phrases, each a Keyphrase of phrase, score and occurrences. Clusters is provenance, clusters and edges.
Why a metric field is None
Each metric declines to run below its own threshold, and a None records that decision rather than a failed computation. The first three are in each paragraph's metrics, the rest in the document's.
| Field | Filled when |
|---|---|
readability_grade | the paragraph has more than 10 words and its kind is not quote |
lexical_density | the paragraph has at least 1 word and its kind is not quote |
compression_ratio | the paragraph has more than 50 words, its kind is not quote, and the paragraph text is at most 256 KiB |
vocabulary_ttr | the document has at least one non-punctuation token |
nominalization_ratio | the same condition as vocabulary_ttr |
passive_ratio, mean_sentence_length | the document has at least one sentence |
sentence_length_std | the document has at least two sentences |
"Words" means the count of tokens with is_punct false across the paragraph's sentences. Quotes (a kind of quote) are never parsed at all: their sentences list is empty, their three paragraph metrics are None, and they contribute nothing to any document total.
Markdown is read as CommonMark, and what matra does not analyse it lists in skipped rather than dropping it: front matter, fenced and indented code, tables, HTML blocks, thematic breaks and link reference definitions, each with its kind and span. List items are paragraphs of kind list_item. ## References is a heading like any other, and a #hashtag is text.
The document-level aggregates
Every document-level aggregate is a field of metrics. Read them rather than recomputing them:
doc = engine.analyze("The report was filed. The committee met.")
print(list(doc))
# ['provenance', 'sections', 'metrics', 'skipped']
print(doc["metrics"])
# {'total_words': 7, 'total_sentences': 2, 'paragraph_count': 1,
# 'mean_sentence_length': 3.5, 'sentence_length_std': 0.7071067811865476,
# 'vocabulary_ttr': 0.8571428571428571, 'nominalization_ratio': 0.0,
# 'passive_ratio': 0.5}
The counts are always integers. Each ratio is None when its condition was not met: mean_sentence_length and passive_ratio need a sentence, sentence_length_std (the sample standard deviation, as statistics.stdev computes it) needs two.
Methods do not cross the FFI boundary, only fields do, which is why these are fields: the metric suite materializes them so the aggregates cross as data instead of being re-derived once per language binding (RFC-0008). The same holds per sentence: word_count, tree_depth and passive are keys on each sentence dict.
passive = [
s["text"]
for section in doc["sections"]
for para in section["paragraphs"]
for s in para["sentences"]
if s["passive"]
]
doc["provenance"] says where the document came from: matra (the version), schema ("2.0", the schema version), parser (name, model, sha256, license), format (markdown, plain_text or blocks), and source (path, bytes, sha256 of the analysed text). Read schema before anything else when a result was stored by another process: ignore keys you do not recognize, and reject a major version you do not know.
Exceptions
Every domain::Error variant crosses the FFI boundary as a class of its own, named for the variant with PEP 8's Error suffix; the name it shares with Rust and the JSON is its kind. Each derives from matra.MatraError and from a builtin class, the one matra raised before 0.3.0, so an except ValueError written then still catches what it caught:
| Situation | Python class | Also a |
|---|---|---|
A model artifact is missing from the directory Model2Vec.from_dir was given | ModelNotFoundError | FileNotFoundError |
| Input exceeds the 8 MiB cap, or an outcome's cap | InputTooLargeError | ValueError |
| Input format has no reader, or the bytes are not UTF-8 text | UnsupportedFormatError | ValueError |
| A caller broke a documented contract, such as an unknown method or an embedder returning the wrong number of vectors | InvalidInputError | ValueError |
| File I/O error, or a download that did not arrive | IoError | OSError |
| A path that does not exist | IoNotFoundError | FileNotFoundError |
| A path matra may not read or write | IoPermissionError | PermissionError |
| Model file corrupt or wrong format, or bytes that arrived and failed the pinned hash | ModelInvalidError | RuntimeError |
| NLP parsing failed, including a panic caught at the UDPipe boundary | ParseFailedError | RuntimeError |
IoNotFoundError and IoPermissionError derive from IoError, so except matra.IoError catches every I/O failure. matra.IoError is not the builtin IOError, which is another name for OSError. Every exception carries kind, the stable string from matra.ERROR_KINDS, and details, a dict of the values to act on: what, limit and actual on an InputTooLargeError, path on a missing file, url and retryable on a download that did not arrive. The errors reference lists the keys for every kind.
A call with the wrong arguments (no input, two inputs, format with blocks) raises the builtin TypeError, as any Python function does. It is a mistake in the call, not a failure of the analysis.
Loading the model and analyzing text fail in different ways, so catch them separately:
import matra
try:
engine = matra.Engine()
except matra.MatraError as e:
raise SystemExit(f"cannot load the model: {e.kind}: {e}")
try:
doc = engine.analyze(file=path)
except FileNotFoundError as e:
print(f"no such file: {e.details['path']}")
except matra.InputTooLargeError as e:
print(f"{e.details['what']} gate: {e.details['actual']} > {e.details['limit']}")
except matra.MatraError as e:
print(f"{e.kind}: {e}")
The message on each exception is the Rust error's own Display output, for a person to read. Branch on the class or on kind, and read values from details. An InputTooLargeError names its gate in details["what"]: input and file_read for an input, tfidf, textrank, rake, yake and clusters for an outcome, and config_file_read, parser_model_download and embedding_model_download for the constructors that read a configuration or download a model. An InvalidInputError names the check that failed in details["reason"], a stable name such as method_unknown, embedder_raised or config_malformed, so a caller tells a bad argument from a broken embedder or a config file to fix without reading the message.
The mapping is exhaustive and enforced at compile time on the Rust side. The match that picks the class has no wildcard arm over the variants, so adding a new domain::Error variant without also giving it a class fails to build. A new variant cannot fall through to another variant's class by accident.
The one failure that does not follow this table is the cross-thread access described above, which arrives as pyo3_runtime.PanicException and bypasses except Exception entirely.
Use the matra CLI
matra reads a file, a directory or stdin and writes to stdout. matra --help describes the tool in the line built into the command: "matra parses text into CoNLL-U structure and measures it. It reports what is there; interpretation is yours."
Install
Nothing installed, nothing configured:
uvx 'matra>=0.3' analyze essay.md
The version floor is load-bearing. A bare uvx matra takes whatever release is newest, which is 0.3.x today and will not stay that way, and 0.1.0 and 0.2 are still on PyPI for anything pinned to them. 0.2's command line spells its JSON and its config file differently; Migrate from 0.2 to 0.3 lists each change. 0.1.0's analyze is a separate Python implementation rather than this one missing a flag: --json prints a bare document with no envelope, the human table is a different renderer, and the model cache lands in ~/.matra/models instead of the resolved model directory. It does not fail, it succeeds and gives you something else, so pin it.
With the floor in place the command ships two ways beyond that, and all three run the same program. The Python package's matra command is the Rust CLI reached through the extension module, not a second implementation, so the flags, the output, and the exit codes are the same whichever route you took. One thing does differ: which features were compiled in. cargo install matra --version '^0.3' --features cli reports features: udpipe cli, while the published wheel reports features: udpipe model2vec python cli, so the embedding adapter is present on the Python route and absent from a --features cli build, whose matra clusters refuses and names the feature. cargo install matra --features cli,model2vec builds one with it. matra --version prints the list of the binary in front of you on its second line.
# Rust binary, no Python involved
cargo install matra --version '^0.3' --features cli
# or through the Python package
uv add 'matra>=0.3' # then: matra --help
Every route caches the model at the resolved model directory (about 16 MB) and downloads it the first time any command needs it. No flag and no environment variable is required to make that happen. If the cached file fails its SHA-256 check, matra never deletes it: a download that verifies replaces it through an atomic rename, and a download that fails leaves the file you had exactly as it was.
Where matra keeps things
| What | Path | Override |
|---|---|---|
| config file | $XDG_CONFIG_HOME/matra/config.toml, else ~/.config/matra/config.toml | MATRA_CONFIG_FILE |
| data root | $XDG_DATA_HOME/matra, else ~/.local/share/matra | MATRA_DATA_DIR |
| models | the data root's models directory | MATRA_MODEL_DIR, or --model-dir on the command line |
A pre-existing ~/.matra/models from an older install is still used when the new location does not exist yet. matra never creates ~/.matra; when a non-empty legacy cache is selected it is the resolved model directory, downloads and re-downloads included. Create the new location, or set MATRA_MODEL_DIR, to move off it.
Per key, --model-dir outranks every environment variable, which outranks the config file, which outranks the defaults compiled into the crate. The config file has no key for the model directory, so that value comes from the flag, the environment, or the defaults. matra config show prints which rung each value actually came from.
Commands
analyze, summarize, keyphrases and clusters each read a file, or - for stdin. analyze also reads a directory, one result per file. The other three report on one document, so they refuse a directory and name analyze, which reads one; picking one file out of it would answer a question nobody asked.
Every subcommand routes through the library pipeline, which rejects symlinks, rejects files over 8 MiB before reading them, and picks a reader from the file's bytes and extension. --format markdown|plain_text|blocks names the format instead, which is how a JSON file of blocks is read, since no extension implies one. A PDF or DOCX signature in the bytes still outranks it.
matra analyze notes.json --format blocks --json
matra analyze
Analyzes a file and reports its metrics.
matra analyze essay.md
matra analyze essay.md --sections
matra analyze essay.md --json
--sections adds a per-section breakdown under the metric table: one row per section with its heading level, heading, and the paragraph, sentence, and word counts underneath it.
Given a directory, analyze reads every regular file in it, one level deep and in name order, and reports each one: a table per file, or under --json one envelope per line. A file that fails is reported where it happens, on stderr and under --json as its own line with its error, and the walk goes on to the next file. The exit code is 2 if any file failed. Symlinks and subdirectories are skipped, as the library's Inputs::dir skips them. --format is refused with a directory, because each of its files is read as its own name says.
matra analyze docs/ --json --depth document | jq -c '{input, words: .result.metrics.total_words}'
--depth decides how much of the parse --json writes. document writes the provenance, the metrics and skipped, and no sections; paragraph adds the sections and their paragraphs, with no sentences; sentence adds the sentences, with no tokens; and token, the default, writes everything. A level below the depth is absent as a key, never an empty array, because an empty sentences would say a paragraph has none, and what remains keeps the whole document's key order, so the two diff line for line. Tokens are most of a document's JSON, so a caller that reads only the measures pays for no more than it reads. --depth without --json is a usage error.
On a markdown file, front matter, code blocks, tables, HTML blocks, thematic breaks and link reference definitions are not analyzed. They are listed in the JSON's skipped, each with its kind and span, rather than dropped. ## References is a heading like any other.
--json emits the full Document (provenance, sections, and metrics, which carries the counts and ratios the table prints) inside the envelope described below:
matra analyze essay.md --json | jq '.result.sections[0].paragraphs[0].metrics.readability_grade'
That query returns null more often than you might expect. Per-paragraph metrics have thresholds: readability_grade needs more than 10 words, compression_ratio needs more than 50, and quotes are skipped entirely. Filter for non-null before aggregating:
matra analyze essay.md --json \
| jq '[.result.sections[].paragraphs[].metrics.readability_grade | select(. != null)] | add / length'
matra summarize
Extracts the top-N sentences as an extractive summary.
matra summarize essay.md
matra summarize essay.md -n 5 --method textrank
| Option | Default | Values |
|---|---|---|
-n | summary.n from the config, 3 as shipped | any positive integer |
--method | summary.method from the config, tfidf as shipped | tfidf, textrank |
TF-IDF scores each sentence by term frequency against the document. TextRank scores by a graph-coherence measure over sentence similarity. TF-IDF is the default because it is cheap and needs no graph, not because it picks better sentences. It favours sentences dense in the document's rarer terms, and on narrative prose that can mean short sentences that make little sense on their own. Reach for TextRank when the summary should carry the document's main thread, when sentences restate and build on each other, or when the TF-IDF picks read as fragments. Read both on your own text before you rely on either. Both reject input over 2000 sentences.
Both methods select their top-N sentences by score internally, then return that selection re-sorted into document order, not score order. The score field tells you how each sentence ranked; the list order tells you where each sentence sits in the source text.
The human table prints each score at three decimals. Two sentences whose scores differ in the fourth therefore print the same number, and a summary of three sentences can arrive looking as though the ranker scored them identically when it did not. --json carries the full float, so reach for it whenever the ordering has to be defended rather than read.
matra summarize essay.md --json | jq '.result.sentences[0]'
# {"index": 4, "span": {"byte_start": 812, "byte_end": 951}, "text": "...", "score": 0.312}
matra summarize essay.md --json | jq '.result.provenance.method'
# "tfidf"
Under --json the result is a summary: the document's provenance, with the method that ran and its parameters ({"n": 3}), and the sentences, each with its index among the document's sentences and its span in the input.
matra keyphrases
Extracts ranked keyphrases.
matra keyphrases essay.md
matra keyphrases essay.md -n 20 --method yake
| Option | Default | Values |
|---|---|---|
-n | keyphrases.n from the config, 10 as shipped | any positive integer |
--method | keyphrases.method from the config, rake as shipped | rake, yake |
RAKE splits on stop words and scores the remaining word runs by a co-occurrence degree-to-frequency ratio. It is rule-based and fast. YAKE scores individual terms by position, frequency, and how varied their surrounding context is, then builds one-word to three-word candidates from those scores. It tends to surface longer, more specific phrases and to notice terms that first appear later in a long document, at more computation than RAKE. Both reject input over 200000 tokens.
Unlike summarize, keyphrases output stays in score order, highest first. A phrase is not tied to one location the way a sentence is, so under --json each carries occurrences, the span of every place in the input it was found, beside the result's provenance with its method and parameters.
Phrases are printed as lowercased lemmas, not as the surface text of the document. "Dependency Parses" appears as dependency parse, and its occurrences point at the surface text. The lemma is the parser's, read in each sentence's context, and the parser does not always lemmatize one word the same way twice. "Reading room" can come out as reading room in one sentence and read room in another, two phrases for one surface form. Merge such pairs yourself if they matter. Phrases with equal scores come back in the same order on every run, the order they are first met, so a tie at the -n boundary is decided the same way each time.
Reading the scores
Neither number is a probability, a percentage, or comparable with the other. RAKE sums a per-word degree-over-frequency ratio across the words of the phrase, and that ratio is at least 1 for every word, so a phrase of k words scores at least k. That is a floor per length, not an order across lengths: on this book's methodology page the two-word lexical density scores 5.667 and the three-word model file name scores 5.200. YAKE multiplies its words' term scores and returns the reciprocal, so it is unbounded above and carries no interpretable unit, and phrase length does not predict where a YAKE score lands. Use either score to order phrases within one document under one method. Do not read it as a magnitude, and do not compare a RAKE number with a YAKE number.
Expect the top phrase not to be the document's topic. On this book's errors page, RAKE puts error implement st ::error, read off the code span std::error::Error and the words around it, first at 12.135 and scores the bare error at 1.468, while YAKE's first is move copy at 2257.447. What is low there is the score, not the position: 1.468 sits not far above 1.000, the floor any single word scores, and 229 of the 589 phrases sit flat on that floor underneath it, which puts error around the middle of the ranking, 306th. Both methods are behaving as specified: they rank properties of word co-occurrence, not aboutness. Methodology carries both formulas and every departure from the published versions.
matra keyphrases essay.md --json | jq '.result.phrases[0:3]'
matra clusters
Groups a document's sentences by the similarity of their embeddings.
matra clusters essay.md
matra clusters essay.md --threshold 0.8 --json
On prose that does not repeat itself, expect the first line to find nothing. 0.85 joins near-restatements, and an essay whose sentences each say something new has none, so the table is empty, stderr says so, and the exit code is 1:
matra: found no clusters in essay.md at --threshold 0.85; a lower threshold joins less similar sentences
That is a result, not a failure. summarize and keyphrases say the same when they find nothing, and under --json or --quiet none of the three writes the line.
| Option | Default | Values |
|---|---|---|
--threshold | clusters.threshold from the config, 0.85 as shipped | a cosine similarity |
Each sentence is embedded with potion-base-8M, the embedding model models.embedding names, downloaded into the model directory on first use (30.9 MB), with the same two lines on stderr the parsing model's first download prints. Two sentences whose cosine similarity is at least the threshold are joined, and a cluster is a connected component of that graph, so co-membership is transitive and is not pairwise similarity. A sentence with no partner is in no cluster. The table prints each cluster with its members' sentence indices and text; --json writes the Clusters value, with edges, every pair that cleared the threshold and its score, and the embedder's identity in provenance.parameters. Finding no cluster is exit 1.
The threshold does not travel between models, domains or granularities, and 0.85 is a starting point for sentences only. Semantic clusters says how to choose one.
The command needs a build with the model2vec feature, which the Python wheel has. A build without it refuses before it reads anything, with invalid_input naming the feature.
matra config
matra config show prints every resolved value with the rung it came from, one key per line, in the shape cargo config get --show-origin uses:
matra config show
data_dir = "/home/you/.local/share/matra" # default
model_dir = "/home/you/.local/share/matra/models" # default
models.embedding = "potion-base-8M" # default
summary.n = 5 # /home/you/.config/matra/config.toml
summary.method = "tfidf" # default
keyphrases.n = 10 # default
keyphrases.method = "rake" # default
clusters.threshold = 0.85 # default
models.embedding is the name of the embedding model's directory under model_dir, which matra clusters, Model2Vec::from_config and Python's Model2Vec.from_config() provision into and load from. The file's sections are [models], [summary] and [keyphrases], each of the last two with n and method, and [clusters] with threshold. They configure the command line only: the library and Python take every count, method and threshold as an argument.
A config file from 0.2 does not load. 0.3.0 renamed [summarize] to [summary], algorithm to method, and semantic.threshold to clusters.threshold, and dropped models.udpipe, since the parsing model is pinned in the source beside its digest. Every command that reads such a file fails with invalid_input, naming the file and each key's replacement, so the file can be fixed from the message. matra config init --force writes a fresh one.
The origin is default for a value compiled into the crate, a path for one read from your config file, environment variable ... for one an environment variable set, and command line for one a flag set. With --json, each key carries its value, the rung, and what the rung pointed at.
What that listing does not carry is the config file path. Every key whose origin is default names no file, so a run with no config file on disk prints no path at all, and "where is my config" is not a question the human table answers. Two commands do answer it, once there is a file. matra config show --json puts the config file's path in the envelope's input field when that file exists, and null when it does not: input names what the run read, and a run with no config file read none. matra config init prints the path it wrote to, which is also where config show looks.
matra config show --json | jq -r .input
matra config init writes the shipped defaults (models.embedding, and the summary, keyphrase and cluster keys) to the resolved config path and prints where it wrote them. It creates the parent directories, writes through a temporary file in the same directory so a reader never sees a half-written config, and refuses to overwrite an existing file unless you pass --force.
matra config init
matra config init --force
matra completions
Prints a completion script for bash, zsh, or fish.
mkdir -p ~/.zfunc && matra completions zsh > ~/.zfunc/_matra # ~/.zfunc must be on your fpath
matra completions bash > /etc/bash_completion.d/matra
For an agent
matra --skill prints the agent skill: what matra is for, when to reach for it, every command with its JSON shape, how to read each number and what it does not mean, the limits, and the errors. It is the whole hand-off. If you are pointing an agent at matra and it has nothing installed, uvx 'matra>=0.3' --skill is the line to give it. Pin the version: --skill arrived in 0.2.0, and a bare uvx matra takes whatever release is newest, which before 0.2.0 was a different command line that does not have the flag.
matra --skill # the skill itself
matra --skill -r # the references, one per line: name, then summary
matra --skill -r json # one reference
The text is compiled into the program with include_str! from skills/matra/SKILL.md and skills/matra/references/*.md, so what you read is what the installed version does, not a copy of the docs that may have moved on. The same files are what the repository ships as a plugin. Every command in them is executed against the command line by the test suite, so an incantation that no longer runs fails CI.
--skill outranks a subcommand: matra analyze essay.md --skill prints the skill and never looks at the file. -r without --skill is an error that says so, and an unknown reference name exits 2 naming the ones that exist. Everything else exits 0. --quiet does not apply, as it does not to completions: the text is the command's output, not a rendering of a result.
Under --json the skill uses the same envelope every other command uses, with input null because the command reads no document:
{
"format_version": 2,
"command": "skill",
"input": null,
"result": { "name": "SKILL", "body": "---\nname: matra\n..." },
"error": null
}
result.name is SKILL for the top level and the reference's own name for matra --skill -r <name> --json. The list replaces result with one key:
{
"format_version": 2,
"command": "skill",
"input": null,
"result": { "references": [{ "name": "json", "summary": "The JSON envelope every command but completions emits, ..." }] },
"error": null
}
The list is every file under references/, in file-name order, and each summary is read from that file's own frontmatter rather than from a second list that could drift from it.
The same files install as a plugin: claude --plugin-dir <checkout> over a clone of the repository picks up skills/matra/, which is where plugin discovery looks. And this documentation site serves llms.txt at its root, the map of every page with its one-line summary, for an agent reading the human door instead.
Reading from stdin
- in place of the path reads stdin, as bytes, and the format is detected the way it is for a file. --stdin-filename gives that input a name, which is the label in the output and the envelope, and whose extension stands in for a file's. --format names the format outright. With neither, the input is read as plain text and labelled <stdin>. Bytes that are not UTF-8 text fail as unsupported_format, as a file's do.
pandoc notes.docx -t markdown | matra analyze - --stdin-filename notes.md
pandoc notes.docx -t markdown | matra analyze - --format markdown
cat essay.txt | matra keyphrases -
The 8 MiB cap applies to stdin as it does to a file, and it is enforced while reading rather than after, so an unbounded stream is refused instead of buffered. Stdin is read before the model loads, so an oversized stream costs no download.
Color, quiet, and version
--color takes auto (the default), always, or never. Under auto, matra colors an interactive terminal and nothing else, and honors NO_COLOR: if that variable is present and not empty, color is off whatever its value. --color always and --color never are explicit requests and outrank NO_COLOR in both directions.
--quiet suppresses the human-readable output while leaving the exit code alone, so a script can branch on "found something" without collecting the table. It has no effect on --json. It also silences the one line the command prints to standard error before it downloads a model, which names the artifact, its size and where it is going. That line is the only output a first run produces during a wait that can reach half a minute, so silence it deliberately rather than by habit.
--version prints the version and then the features this build was compiled with:
matra 0.3.0
features: udpipe model2vec python cli
The JSON envelope
--json emits one object for every command but completions, success or failure:
{
"format_version": 2,
"command": "analyze",
"input": "essay.md",
"result": { "provenance": {}, "sections": [], "metrics": {}, "skipped": [] },
"error": null
}
command is analyze, summarize, keyphrases, clusters, config, skill or schema. input is the path that was read, or the name --stdin-filename gave it. For config it is the config file the run read, and null when there is none, and for skill and schema, which read no document, it is always null. result is the serde form of the domain value the command produced: a Document for analyze, a Summary for summarize, a Keyphrases for keyphrases and a Clusters for clusters, each of the last three carrying the document's provenance with the method that ran and its parameters. For config it is an object keyed by setting, each carrying value, source and origin, and for skill the shapes shown above.
All five keys are always present, and exactly one of result and error is null. A failure writes the envelope with result null and error holding the failure's kind, message and details, the same three Rust serializes and Python raises:
{
"format_version": 2,
"command": "analyze",
"input": "missing.md",
"result": null,
"error": {
"kind": "io",
"message": "io error: cannot read missing.md: No such file or directory (os error 2)",
"details": { "io_kind": "not_found", "path": "missing.md", "cause": "cannot read missing.md: No such file or directory (os error 2)" }
}
}
The message still goes to stderr and the exit code is still 2, so a script that reads only the exit code is unchanged. Branch on error.kind and read values from error.details, never from the message. The command line's own refusals carry the kind whose remedy matches, which for every one of them is invalid_input, a call to change, with error.details.reason naming which refusal it is: a directory given to a command that reads one document, --format with a directory, -r without --skill, an unknown reference or schema name, config init over an existing file, and clusters in a build without an embedder. A usage error under --json, such as an unknown flag, is an envelope too, with command null when the arguments name none. Errors lists every kind.
analyze DIR --json writes one envelope per line, one line per file, each compact rather than indented, so the output is a stream of JSON values that jq reads one at a time.
Stability: format_version increments on any change to the envelope, and whenever the document schema's major version moves; the result value is the serde form of the documented domain types. That is the same promise cargo makes for cargo metadata --format-version. Pin your consumer to a format_version you have tested against.
A Document also names its own schema version, in result.provenance.schema ("2.0"), and that is the only place it appears. A minor version adds optional fields only; anything else is a major. So ignore fields you do not recognize, treat every string set you receive as open (format, pos, dep, mood, pattern, the feature names), and reject a major you do not know.
The JSON Schemas for this major are compiled into the program. matra --schema lists them with their titles, and matra --schema NAME prints one: document, summary, keyphrases, clusters and error for the outputs, and blocks for block input. Like --skill it outranks a subcommand and reads no document, and under --json it is the envelope with command schema. The same files are in the repository under spec/schema/v2/.
matra --schema
matra --schema document > document.schema.json
Exit codes
0 on success when the command found something, 1 on success when it found nothing (an empty summary, no keyphrases, no clusters, a file with zero sentences), and 2 when an error occurred. Model load failure, a missing or unreadable file, a directory where one document is read, input over the size cap, a bad argument, and a parse failure all land on 2, with the message on stderr prefixed matra: and, under --json, the envelope with its error on stdout. For a directory the code is 2 if any file failed, 0 if none failed and any had a sentence, and 1 otherwise. A broken pipe, which is what you get piping into head, is treated as success and exits 0.
matra keyphrases notes.txt > /dev/null
case $? in
0) echo "phrases found" ;;
1) echo "no phrases" ;;
2) echo "failed" ;;
esacMethodology
What each number matra reports is computed from, how to cite the work it comes from, and what it does not license you to conclude. Every formula on this page is transcribed from the implementation, not from the publication it descends from; where matra departs from the published method, the departure is stated.
Related pages: Domain types for the fields these values live in, Errors for the caps each algorithm enforces.
The parse layer
Model identity
The English model is pinned in src/nlp/udpipe.rs.
| Property | Value |
|---|---|
| File | english-ewt-ud-2.5-191206.udpipe |
| Size | 16,309,608 bytes |
| SHA-256 | 784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957 |
| Treebank | English Web Treebank, Universal Dependencies 2.5, release 191206 |
| Distributor | LINDAT/CLARIAH-CZ repository, handle 11234/1-3131 |
The engine's constructors, Engine::new and Engine::from_config, download the file into the model directory when it is absent, checks the size, computes the SHA-256, and loads the same bytes it hashed. There is no second read from disk between verification and load. A cached file that fails verification is not loaded: the model is downloaded, and a download that also fails the digest is retried once, so a call makes at most two downloads. The file is replaced only by bytes that verify. If both downloads fail the digest the call returns Error::ModelInvalid, nothing is loaded, and the file that was there is still there.
That path is the only way a parsing model arrives: no surface loads another model file, so every result names the pinned model it came from.
Changing the model means editing the pinned constants, which makes the change visible in version control and in any caller that pins a matra version.
What the parse produces
One Token per token with all ten CoNLL-U columns, grouped into sentences. Two properties of the shipped adapter matter when you report results:
deps(CoNLL-U column 9) is always_. Theudpipe-rsbinding does not surface enhanced dependencies.Sentence::textis built from token surface forms, with a space inserted unless the preceding token carriesSpaceAfter=Noinmisc. It is a reconstruction, not a substring of your input.
Which text is parsed
The route through the library changes what gets segmented into sentences.
Every pipeline route parses each paragraph that is not a quote separately. Quotes are not parsed, so they contribute nothing to any metric. Calling a provider's parse directly parses the whole string in one call with no decomposition, and its segmentation can differ from the pipeline's at paragraph boundaries.
Input::file picks the format from the file's bytes and then its extension, so the same text segments differently under .md and under .txt; Input::text takes the format as an argument.
Paragraph-at-a-time parsing means a sentence can never be attributed to the wrong paragraph. It also means sentence segmentation cannot run across a paragraph boundary. Report which format your numbers were produced under.
Citation
Straka, M., & Straková, J. (2017). Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe. Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, 88-99.
Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., & Manning, C.D. (2014). A Gold Standard Dependency Corpus for English. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014).
Zeman, D., et al. (2019). Universal Dependencies 2.5. LINDAT/CLARIAH-CZ digital library, Institute of Formal and Applied Linguistics, Charles University.
Limitations
UDPipe is a statistical syntactic parser. It performs no named entity recognition, no coreference resolution, and no semantic role labeling. Tags and relations are predictions, not ground truth, and error rates on text unlike the training corpus are higher than on text like it. The shipped configuration is English only.
Paragraph metrics
Two different counts appear below, and confusing them changes the answer. Paragraph::word_count counts non-punctuation tokens from the parse, and every applicability gate is stated in those terms. The formulas themselves split the paragraph text on whitespace. A gate stated in word_count terms is not a gate on whitespace words.
Flesch-Kincaid grade level
grade = 0.39 * (words / sentences) + 11.8 * (syllables / words) - 15.59
| Input | As computed |
|---|---|
words | Count of whitespace-separated pieces of the paragraph text |
sentences | Count of ., !, and ? characters in the paragraph text, floored at 1 |
syllables | Sum over words of the syllable estimate below. A word with no alphabetic characters contributes 0 |
The syllable estimate for one word: keep only alphabetic characters, lowercase them; if the result is three bytes or shorter, return 1; otherwise count runs of the letters a, e, i, o, u, y, where adjacent vowels count once; subtract 1 if the word ends in e and the count is above 1; return at least 1.
Applied to paragraphs that are not quotes and whose word_count is above 10. Stored in ParagraphMetrics::readability_grade, the paragraph's metrics.readability_grade. The value is unbounded and can be negative on short, simple text. The formula is computed inside the metric suite only; matra publishes no function that computes it on a string.
Citation. Kincaid, J.P., Fishburne, R.P., Rogers, R.L., & Chisholm, B.S. (1975). Derivation of New Readability Formulas for Navy Enlisted Personnel. Research Branch Report 8-75, Naval Technical Training Command.
Limitations. The formula was fitted to Navy training material in 1975 and predicts nothing about conceptual difficulty, argument quality, or required background knowledge. Two paragraphs with the same syllable and length profile score the same whatever they assert. The original publication specifies no syllable-counting algorithm, so grade values from different tools are not comparable; the heuristic above miscounts many words, and non-ASCII letters register as consonants. Counting sentence-ending characters treats abbreviations, decimals, and ellipses as sentence boundaries.
Lexical density
lexical_density = content_words / total_words
| Input | As computed |
|---|---|
total_words | Count of whitespace-separated pieces of the paragraph text |
content_words | Those pieces whose alphabetic characters, lowercased, form a non-empty string that is not in the stop word list |
A piece made only of punctuation counts in the denominator and never in the numerator.
Applied to paragraphs that are not quotes, whose word_count is not zero, and whose text yields at least one whitespace-separated piece. Stored in ParagraphMetrics::lexical_density. Range 0.0 to 1.0.
Citation. Ure, J. (1971). Lexical density and register differentiation. In G. Perren & J.L.M. Trim (Eds.), Applications of Linguistics. Cambridge University Press.
Limitations. The value depends entirely on the stop word list, which is matra's own list of 105 English words, not a standard one. Ure's definition counts lexical items by part of speech; matra approximates that with a stop word list over whitespace pieces. Density says nothing about whether the content words are used well, or at all correctly.
Compression ratio
compression_ratio = compressed_bytes / original_bytes
| Input | As computed |
|---|---|
original_bytes | UTF-8 byte length of the paragraph text |
compressed_bytes | Byte length of the brotli-compressed text at quality 6 with a window parameter of 18, which is a 256 KiB window |
Applied to paragraphs that are not quotes, whose word_count is above 50, and whose text is at most 262,144 bytes. Paragraphs above that byte ceiling are skipped and keep None, which bounds worst-case compression time. The slot also stays None if the encoder fails to consume the text. Stored in ParagraphMetrics::compression_ratio. A lower ratio means the text compressed further, which means more surface repetition.
Citation. Alakuijala, J., & Szabadka, Z. (2016). Brotli Compressed Data Format. RFC 7932, Internet Engineering Task Force.
Limitations. This is a redundancy proxy over bytes, not a measure of meaning, quality, or novelty. Precise technical prose that reuses terminology compresses much like repetitive filler. The number is a property of one encoder at two specific parameter settings: a different compressor, or a different version of this one, gives different values.
The three paragraph measures, computed for every paragraph of one longer text, The Federalist No. 10:
Paragraph measures across one document
- Mean readability
- 18.20
- Vocabulary TTR
- 0.25
- Nominalization ratio
- 0.04
- Passive ratio
- 0.42
show the data
| # | Opens with | Words | Grade | Density | Compression |
|---|---|---|---|---|---|
| 1 | AMONG the numerous advantages promised by a well… | 384 | 20.72 | 0.47 | 0.46 |
| 2 | By a faction, I understand a number of… | 51 | 24.89 | 0.43 | 0.56 |
| 3 | There are two methods of curing the mischiefs… | 22 | 10.69 | 0.45 | none |
| 4 | There are again two methods of removing the… | 40 | 18.89 | 0.40 | none |
| 5 | It could never be more truly said than… | 79 | 12.90 | 0.46 | 0.52 |
| 6 | The second expedient is as impracticable as the… | 157 | 14.59 | 0.44 | 0.45 |
| 7 | The latent causes of faction are thus sown… | 263 | 18.68 | 0.53 | 0.42 |
| 8 | No man is allowed to be a judge… | 267 | 12.92 | 0.47 | 0.46 |
| 9 | It is in vain to say that enlightened… | 80 | 13.54 | 0.47 | 0.55 |
| 10 | The inference to which we are brought is,… | 32 | 12.87 | 0.35 | none |
| 11 | If a faction consists of less than a… | 172 | 16.10 | 0.43 | 0.45 |
| 12 | By what means is this object attainable? Evidently… | 121 | 13.74 | 0.47 | 0.50 |
| 13 | From this view of the subject it may… | 179 | 21.57 | 0.45 | 0.48 |
| 14 | A republic, by which I mean a government… | 61 | 13.91 | 0.44 | 0.58 |
| 15 | The two great points of difference between a… | 50 | 23.03 | 0.48 | none |
| 16 | The effect of the first difference is, on… | 175 | 20.22 | 0.49 | 0.48 |
| 17 | In the first place, it is to be… | 123 | 26.53 | 0.45 | 0.42 |
| 18 | In the next place, as each representative will… | 76 | 32.06 | 0.42 | 0.55 |
| 19 | It must be confessed that in this, as… | 103 | 17.28 | 0.46 | 0.52 |
| 20 | The other point of difference is, the greater… | 216 | 24.70 | 0.44 | 0.45 |
| 21 | Hence, it clearly appears, that the same advantage… | 169 | 14.14 | 0.49 | 0.48 |
| 22 | The influence of factious leaders may kindle a… | 128 | 19.86 | 0.48 | 0.51 |
| 23 | In the extent and proper structure of the… | 52 | 14.75 | 0.48 | 0.54 |
One panel per paragraph measure, paragraphs in document order along the bottom. A cross marks a
paragraph matra declined to measure: the field is None because the paragraph falls
outside the measure's applicability conditions above, and the line breaks there rather than
inventing a value. The dashed line is the document's mean readability: the mean of the grades
of the paragraphs matra measured. matra computes no document-level lexical density or
compression ratio, so those panels have no line.
Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: federalist-10/metrics.json.
Document metrics
Every field of DocumentMetrics is computed over the sentences attached to the document's paragraphs, which excludes quotes. There is exactly one sentence set, and it is derivable from the tree in one way, so the numbers cannot disagree with the structure they describe.
Vocabulary type-token ratio
vocabulary_ttr = distinct_lemmas / total_lemmas
| Input | As computed |
|---|---|
total_lemmas | Count of non-punctuation tokens across the sentence slice |
distinct_lemmas | Count of distinct lemma strings in that same set, compared exactly, without case folding |
Applied to the whole slice. Stored in DocumentMetrics::vocabulary_ttr, which stays None when the slice contains no non-punctuation token.
Citation. Johnson, W. (1944). Studies in language behavior: A program of research. Psychological Monographs, 56(2), 1-15.
Limitations. Type-token ratio falls as documents get longer, because repetition accumulates. Values from documents of different lengths are not comparable without a length correction, and matra applies none. Because lemmas are compared exactly, a provider that emits case-varying lemmas inflates the distinct count.
Nominalization ratio
nominalization_ratio = nominalizing_nouns / total_lemmas
| Input | As computed |
|---|---|
total_lemmas | The same denominator as vocabulary type-token ratio: non-punctuation tokens across the slice |
nominalizing_nouns | Tokens whose pos is NOUN and whose text, lowercased, ends in tion, ment, ness, ity, ence, or ance |
The suffix test reads the surface form, not the lemma, so a plural nominalization such as conditions or measurements is missed: the plural -s falls after the suffix, so "conditions".ends_with("tion") is false even though the singular condition matches.
Applied to the whole slice. Stored in DocumentMetrics::nominalization_ratio, which stays None when the slice contains no non-punctuation token.
Limitations. Six suffixes are a heuristic, not a morphological analysis. Words that end in those letters without being nominalizations count as false positives, and nominalizations formed otherwise, such as growth or failure, are missed. The test depends on the POS tagger having assigned NOUN, so tagging errors propagate. The denominator counts every running word rather than every noun, so the value is a share of the text, not a share of the nouns in it.
Passive ratio
passive_ratio = passive_sentences / total_sentences
A sentence counts as passive when any of its tokens carries a dep of nsubj:pass, nsubjpass, or aux:pass. The first two are the Universal Dependencies and older Stanford spellings of a passive subject; the third is a passive auxiliary.
Stored in DocumentMetrics::passive_ratio when the document is measured, and per sentence in Sentence::passive, which Sentence::new sets; both cross to Python as fields. The field is None when there are no sentences.
Citation. de Marneffe, M.-C., Manning, C.D., Nivre, J., & Zeman, D. (2021). Universal Dependencies. Computational Linguistics, 47(2), 255-308.
Limitations. This counts sentences that contain a passive construction, not passive clauses, so a sentence with three passives counts once. Detection is only as good as the parser's relation labels. Nothing here judges whether a passive is appropriate.
Sentence length statistics
DocumentMetrics::mean_sentence_length is total_words / total_sentences, where total_words counts non-punctuation tokens. DocumentMetrics::sentence_length_std is the sample standard deviation of per-sentence non-punctuation token counts, with denominator n minus 1. Both are None when undefined: no sentences for the mean, fewer than two for the deviation. Measuring the document stores both, with the counts total_words, total_sentences and paragraph_count, so all of them cross to Python.
Corpus aggregates
matra computes none. Every measure above is of one document, and how to combine documents is a decision about what the combination is for, so it is the caller's. The counts sum across documents. A ratio pools by summing its numerator and denominator rather than averaging the ratios, so the passive ratio of a set is its passive sentences over its sentences. vocabulary_ttr does not pool at all: it falls as text grows, so documents of different lengths are not comparable on it, alone or summed. A mean of paragraph readability grades averages paragraphs, not documents, so a long document contributes more terms than a short one; leave out the paragraphs whose slot is None.
Summarization
Both summarizers take a Document and a count, score every sentence, keep the highest N, and return them in ascending index, the sentence's position among the document's sentences. Neither generates text: the output sentences are the input sentences.
Shared inputs
A term is a token's lemma with punctuation tokens removed and stop words removed. The stop word test lowercases the lemma before looking it up; the term itself is the lemma as the provider produced it, without case folding. The stop word list holds 105 English words and is shared with the paragraph lexical density metric.
TF-IDF
Each sentence is treated as a document when computing document frequency.
score(s) = ( sum over distinct terms t in s of tf(t, s) * idf(t) ) / distinct_terms(s)
tf(t, s) = occurrences of t in s / total terms in s
idf(t) = ln( total_sentences / df(t) )
df(t) = number of sentences containing t
A sentence with no terms scores 0.0. The mean divides by the number of distinct terms, not by the number of term occurrences. A term present in every sentence has an idf of 0 and contributes nothing.
Cap. 2,000 sentences. Above it, Error::InputTooLarge with what set to "tfidf".
Citation. Luhn, H.P. (1958). The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development, 2(2), 159-165. Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11-21. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24(5), 513-523.
Limitations. Treating sentences as documents is an adaptation of a corpus-level weighting scheme, and the resulting weights are not the weights you would get from a real document collection. Scores measure term rarity within one text, not informativeness. Selected sentences are not guaranteed to read coherently together, and nothing checks for redundancy between them.
TextRank
similarity(a, b) = shared(a, b) / ( ln(1 + |a|) + ln(1 + |b|) )
shared(a, b) = sum over terms t in both of min(count_a(t), count_b(t))
|a| = number of distinct terms in a
Similarity is 0 when either sentence has no terms or they share none. The matrix is symmetric and its diagonal stays 0, so a sentence never reinforces itself.
score(i) at step k+1 = (1 - d) / N
+ d * sum over j not equal to i of
( similarity(j, i) / out_sum(j) ) * score(j) at step k
d = 0.85
N = number of sentences
out_sum(j) = sum of row j of the similarity matrix
All scores start at 1 / N. Iteration stops after 50 passes, or earlier when the largest single-sentence change in a pass falls below 1e-6. Sentences whose out_sum is 0 send no weight.
Cap. 2,000 sentences, which bounds the dense similarity matrix at roughly 32 MB of f64. Above it, Error::InputTooLarge with what set to "textrank".
Citation. Mihalcea, R., & Tarau, P. (2004). TextRank: Bringing Order into Text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, 404-411. Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank Citation Ranking: Bringing Order to the Web. Stanford InfoLab Technical Report 1999-66.
Limitations. Similarity here is lemma overlap. Two sentences expressing one idea in different words have similarity 0. The normalization by the logarithm of distinct-term counts is matra's, not the one in the 2004 paper, so scores are not comparable with other TextRank implementations. Reaching the iteration ceiling without converging is possible and is not reported.
Keyphrase extraction
Both extractors take &[Sentence] and a maximum count, and return Keyphrase values in descending score order. Both cap input at 200,000 tokens counted across the slice with punctuation included.
RAKE
Candidates. Within each sentence, a candidate is a maximal run of tokens whose pos is NOUN, ADJ, or PROPN and whose lowercased lemma is not a stop word. Punctuation, stop words, and any other part of speech end the run. Candidate words are lowercased lemmas.
for each candidate phrase p of length k, for each word w in p:
freq(w) += 1
degree(w) += k
word_score(w) = degree(w) / freq(w)
phrase_score(p) = sum over w in p of word_score(w)
When the same phrase string occurs more than once, the highest score is kept. Phrases are then sorted by score descending and truncated to the requested count.
Cap. 200,000 tokens. Above it, Error::InputTooLarge with what set to "rake".
Citation. Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010). Automatic Keyword Extraction from Individual Documents. In M.W. Berry & J. Kogan (Eds.), Text Mining: Applications and Theory. John Wiley and Sons.
Limitations. Published RAKE delimits candidates with stop words and punctuation over surface words. matra adds a part-of-speech filter and works over lemmas, so scores and candidate sets differ from a reference implementation. A high degree over frequency ratio means a word appears in long phrases relative to how often it appears at all; that is a structural property, not evidence of topical centrality.
YAKE
Terms. Non-punctuation tokens whose lowercased lemma is not a stop word and is longer than one character. Position is a running index over every non-punctuation token, so stop words and single-character tokens occupy positions without becoming terms.
pos_score(t) = ln( 1 + mean_position(t) / total_positions )
freq_score(t) = occurrences(t) / total_positions
context_diversity(t) = distinct_neighbors(t) / neighbor_observations(t)
term_score(t) = ( pos_score(t) + context_diversity(t) ) / ( freq_score(t) + 1.0 )
total_positions is the count of all non-punctuation tokens across the slice, floored at 1. A neighbor is the adjacent non-punctuation token on either side within the same sentence, recorded only when it is not a stop word; a term with no recorded neighbor gets a context diversity of 0.0. Lower term_score means more relevant at this stage.
ngram_score(p) = product over words w in p of term_score(w)
output_score(p) = 1.0 / ngram_score(p)
Candidates are sliding windows of 1, 2, and 3 words over each sentence's term list, with stop words and single-character tokens already removed, so an n-gram can join words that were not adjacent in the source. A sentence with fewer than three terms yields correspondingly shorter windows. When a phrase string occurs more than once, the lowest ngram_score is kept.
Candidates whose score is not strictly positive, or is not finite, are dropped before inversion. A term reaches a score of exactly 0 when it occurs only at position 0 and has no eligible neighbor, and any n-gram containing it is dropped with it. The remainder are sorted by output_score descending, so higher is more relevant in the returned Keyphrase.
Cap. 200,000 tokens. Above it, Error::InputTooLarge with what set to "yake".
Citation. Campos, R., Mangaravite, V., Pasquali, A., Jorge, A., Nunes, C., & Jatowt, A. (2020). YAKE! Keyword extraction from single documents using multiple local features. Information Sciences, 509, 257-289.
Limitations. Published YAKE combines five features, including casing and sentence spread, and applies deduplication over similar candidates. matra implements three features, no deduplication, and a different combination formula. Scores are not comparable with the reference implementation, and the two will not agree on rankings. The inversion means output scores are unbounded above and have no interpretable unit; use them for ordering, not as magnitudes.
Determinism and reproducibility
What is fixed
matra draws no random numbers, sets no seeds, samples nothing, and runs the analysis path on one thread. The model is pinned by SHA-256, and the bytes that pass verification are the bytes that get loaded. Given the same model file and the same input, the parse is the same.
Given the same parse, these are reproducible bit for bit on one machine: every paragraph metric, every document metric, every derived method on Document, TextRank scores and selection, RAKE word and phrase scores, YAKE term and n-gram scores, and the order of keyphrases with equal scores. TextRank in particular is order-stable because its shared-term counts are integer sums and its iteration walks sentence indices in order.
What is not bit-stable
One place carries hash map iteration order into the result.
-
TF-IDF sentence scores. The per-sentence score sums over a hash map whose iteration order is not fixed between runs. Floating-point addition is not associative, so the low bits of a score can differ between runs.
Selection changes only when two sentences differ by less than that rounding difference.
RAKE and YAKE collect their candidates in the order each phrase is first met and sort them by score with a stable sort, so phrases with exactly equal scores come out in the same order on every run, and a tie straddling the truncation point is decided the same way each time.
Two further sources of variation are worth recording in a methods section.
- Natural logarithms come from the platform's math library. Values can differ by an ulp across operating systems and toolchains, so compare cross-platform results with a tolerance rather than for equality.
- The compression ratio depends on the brotli encoder. A change in the encoder version can change the value for unchanged text.
What to record
To let someone reproduce a result, report: the matra version, the model file name and its SHA-256, which format the document was analyzed under, the method, and the parameter values it ran with, such as the requested sentence count or maximum phrase count. A summary, keyphrases or clusters carry every one of these in their provenance: matra, parser, format, method and parameters.
matra <crate version> (https://github.com/mox-labs/matra)
parse: UDPipe, english-ewt-ud-2.5-191206.udpipe
SHA-256 784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957
format: markdown
method: tfidf
parameters: n = 5
Cite the algorithm publications listed above for the methods themselves, and the UDPipe and Universal Dependencies references for the parse layer.
Cite matra separately from the methods. The CITATION.cff file at the repository root carries the software entry, and it identifies the implementation and its version. The publications identify the method. Because matra departs from several published methods in the ways recorded on this page, a reader who has only the method citation does not know which values you produced, and a reader who has only the software citation does not know what the values are meant to measure. Both are needed.
General limitations
These hold for every number on this page.
- matra reports structure and stops. It measures; your application decides what a measurement means.
- No output is a judgment of quality, correctness, originality, or authorship. Nothing here detects machine-generated text.
- No output carries meaning. Passive ratio counts a syntactic pattern, compression ratio counts bytes, keyphrase scores count co-occurrence. None of them read.
- Metrics that share a name with published measures are matra's implementations of them, with the departures listed per metric. Do not compare matra's values with another tool's values for a like-named metric.
- Every value downstream of the parse inherits the parser's errors. On text unlike the English Web Treebank, that inheritance is larger.
Errors
Every fallible function in matra returns domain::Result<T>, which is std::result::Result<T, domain::Error>. No function in the library returns Result<T, String>. matra signals failure by returning Error, not by unwinding: a panic raised inside the UDPipe C++ boundary is caught at the adapter and converted into Error::ParseFailed.
Error is #[non_exhaustive]. Variants can be added in a minor release, so a match on it from another crate needs a catch-all arm. Error does not derive Clone; it is moved, not copied. The stream surface wraps it in DocumentError, which adds the path the failure occurred at, and yields it as that input's Err item.
Every error reads the same way on every channel. It has a kind, a stable string to branch on. It has a message, for a person. And it has details: the values a caller acts on, as named data, so nothing has to parse the message. For every kind but invalid_input that is every dynamic value in the message; an invalid_input names the check that failed, and its message adds the values that failed it for a person. In Rust these are Error::kind, Display and Error::details. The serde form is the three together, and Python exceptions carry the same three. See kinds and details.
The variants
| Variant | Payload | Returned when |
|---|---|---|
ModelNotFound | PathBuf | A model file does not exist at the given path |
ModelInvalid | String | Bytes that arrived could not be verified against the pin, or could not be loaded |
ParseFailed | { reason: Option<&'static str>, cause: String } | The parser or a reader failed on the input, or returned what its contract forbids. reason names the engine's check that failed, when one did |
InputTooLarge | { limit: usize, actual: usize, what: &'static str } | A size gate rejected the input. what names the gate |
UnsupportedFormat | { format: Option<Format>, feature: Option<&'static str> } | The input is in a format no reader in this build reads, or is not UTF-8 text |
InvalidInput | { reason: &'static str, message: String } | The call cannot be honored as made. reason names the check that failed |
Io | std::io::Error | A filesystem operation failed, or a download did not arrive |
Error implements std::error::Error and Display through thiserror, and From<std::io::Error>, so ? converts I/O failures automatically.
ModelNotFound
Returned by Model2Vec::from_dir when any of the three artifacts is absent. The payload is the path of the first one missing. Neither the UDPipe provisioner, behind Engine::new and Engine::from_config in Rust and Python, nor Model2Vec::potion_base_8m returns this variant: both download the model when the file is absent.
ModelInvalid
Returned by:
- The UDPipe provisioner when the bytes it downloaded still fail SHA-256 verification after one refetch. The payload is
SHA-256 mismatch after re-download from <url>. Nothing that failed verification is written, so the model directory is left as the call found it. Model2Vec::from_dirwhen an artifact does not parse, uses an embedding dtype other than f32, or panics the loader.Model2Vec::potion_base_8mwhen the directory already holds all three artifacts and their digest is not the pinned one, or holds only some of them. The payload names the directory and the ways out; nothing there is downloaded over or removed.Model2Vec::potion_base_8mwhen the artifacts it downloaded still fail the digest after one re-download. Both attempts verify in memory, so neither writes anything, and the directory is left exactly as the call found it. The payload names the directory and the expected digest.
A file whose hash does not match the pinned constant is treated as untrusted and is never loaded.
ModelInvalid is about bytes that arrived. A download that never arrived, a server that answered about the request rather than with a model, and a filesystem that refused the write are Io, not ModelInvalid (RFC-0015). Before 0.2.0 the UDPipe download funnelled all three here, so a DNS failure reported model_invalid; a consumer that branched on that reads the provisioning failures section below.
ParseFailed
The parser or a reader failed on the input, or returned what its contract forbids. The remedy is the component, not the call: the same input through the same parser fails the same way. cause is free text for a person, and reason is present when the failure is one of the engine's checks.
A parser or a reader reports its own failure with reason None. The UDPipe adapter's parse does so in four situations:
- The underlying parser returns an error. The cause is that error's message.
- The underlying parser panics. The cause is
udpipe panicked: <message>. Thecatch_unwindboundary that produces this lives innlp/udpipe.rs, and it exists because an unhandled panic crossing the C++ boundary aborts the host process instead of unwinding. - A token id or head value cannot be represented as
usize. The cause names the token and its sentence. - A token's form cannot be found in the paragraph text the parser was given, at or after the previous token, so the token has no span. The cause names the token, its sentence and the byte the search began at. The adapter reports this rather than a guessed span.
The engine trusts no parser and no reader, matra's own included. Before anything a reader returns becomes a section, and before anything the parser returns for a paragraph becomes a sentence, the engine checks it against the contract its trait states (matra::decompose::Decomposer, matra::nlp::NlpProvider), and a break fails the document with reason naming the check and cause naming the block, or the paragraph, sentence and token:
reason | Check that failed | Of |
|---|---|---|
token_id_sequence | A sentence's token ids do not run 1..=n in order | the parser |
head_out_of_range | A head is neither 0 nor the id of a token in the sentence | the parser |
root_count | A sentence has no token whose head is 0, or more than one; a sentence with no tokens has none | the parser |
head_cycle | Following heads from a token never reaches the root | the parser |
token_span_invalid | A token's span is reversed, runs past the paragraph, or splits a character | the parser |
token_span_order | A token's span starts before the previous token's ends, within a sentence or across two | the parser |
token_form_mismatch | The paragraph text at a token's span is not its form (a run of whitespace in the form matches any run of whitespace) | the parser |
block_span_invalid | A block's or a skipped construct's span is reversed, runs past the analysed text, or splits a character | a reader |
block_span_order | The blocks are out of document order, or two spans among the blocks and skipped constructs overlap | a reader |
heading_level | A heading is at level 0 | a reader |
paragraph_text_mismatch | A paragraph's text is not the analysed text its span covers | a reader |
A conforming parser or reader is one whose output passes these checks on every input it accepts. The checks are a pass over what they read: on the docsite's own pages they cost about 0.014% of UDPipe's parse time, about 3 microseconds a paragraph.
InputTooLarge
Ten gate labels produce this variant. The what field carries the label so a caller can route each gate differently.
what | Gate | Limit | Measured over |
|---|---|---|---|
"input" | The engine's reading stage, the only route from text to the parser, for every Input constructor, and the CLI's stdin read, which applies the same cap first | MAX_INPUT_BYTES, 8 MiB (8,388,608) | Byte length of the input handed in, and of the analysed text the reader produces (for block input, the blocks joined); the CLI counts the bytes as they arrive, before decoding them |
"file_read" | Reading the file an Input::file names | MAX_INPUT_BYTES, 8,388,608 bytes | File size reported by the filesystem, checked before any read |
"tfidf" | summarize, and summarize_with with SummaryMethod::TfIdf | MAX_TFIDF_SENTENCES, 2,000 | Number of sentences in the document |
"textrank" | summarize_with with SummaryMethod::TextRank | MAX_TEXTRANK_SENTENCES, 2,000 | Number of sentences in the document |
"rake" | keyphrases, and keyphrases_with with KeyphraseMethod::Rake | MAX_RAKE_TOKENS, 200,000 | Total tokens across the document's sentences, punctuation included |
"yake" | keyphrases_with with KeyphraseMethod::Yake | MAX_YAKE_TOKENS, 200,000 | Total tokens across the document's sentences, punctuation included |
"clusters" | clusters, before anything is embedded | MAX_CLUSTER_SENTENCES, 2,000 | Number of sentences in the document |
"parser_model_download" | The UDPipe provisioner | MAX_DOWNLOAD_BYTES, 64 MiB (67,108,864) | Bytes read from the response, which stops one past the cap |
"embedding_model_download" | Model2Vec::potion_base_8m, per artifact | MAX_DOWNLOAD_BYTES, 64 MiB (67,108,864) | Bytes read from the response, which stops one past the cap |
"config_file_read" | Config::resolve and Config::from_file, reading a config file | MAX_CONFIG_FILE_BYTES, 64 KiB (65,536) | File size reported by the filesystem, checked before any read |
limit carries the cap and actual carries the measured size, so an error message can be built without hardcoding the constants. Each cap is a published constant in domain::limits, named in the table; the module's documentation carries the cost model behind each value. No configuration changes any of them.
The caps bound worst-case memory and time. TextRank builds a dense similarity matrix that reaches roughly 32 MB of f64 at 2,000 sentences. RAKE and YAKE build phrase-keyed maps whose size follows token count rather than sentence count, which is why their caps are stated in tokens. The two download caps are the gates whose input is not the caller's: they bound what a redirected or misbehaving server can make the process hold, and the read stops at the bound rather than continuing, so actual reports the bound that was breached rather than the response's full length. Both are 64 MiB per file, against the 16.3 MB parsing model and the embedding model's largest file, 30.2 MB of its 30.9 MB set, which is headroom for a later version and still finite.
A document from disk crosses both gates in sequence: "file_read" when the file is read, "input" before it is parsed. In practice "file_read" fires first, since both carry the same 8 MiB limit and the file size is checked before the read.
Every route from text to a parser, a parser of your own included, runs through the engine, so no text bypasses the "input" gate on its way to one.
The label has a second producer. The CLI reads stdin through read_capped in src/cli/mod.rs, which reads at most one byte past MAX_INPUT_BYTES and returns InputTooLarge with what set to "input" before the bytes ever reach the engine. Same limit, same label, so a caller sees one gate however the text arrived. The CLI never decodes what it reads: the bytes go to the engine as Input::bytes, so an oversized pipe is reported as too large, and a pipe that is not UTF-8 is UnsupportedFormat, as a file's bytes would be.
The four ranking methods check their caps after their empty-result checks. summarize_with(&doc, 0, ..) and keyphrases_with(&doc, 0, ..) return an empty result without evaluating the cap, whatever the size of the document. The same holds for a document with no sentences. clusters has no count parameter; its cap is checked first, before anything is embedded, then its contract, and a document with no sentences returns empty clusters.
A metric has no gate of its own. The compression ratio skips any paragraph over 262,144 bytes (MAX_COMPRESSION_PARAGRAPH_BYTES) and leaves its metric slot at None rather than returning an error, and a paragraph under a metric's minimum word count (MIN_READABILITY_WORDS, MIN_COMPRESSION_WORDS) does the same.
InvalidInput
The call cannot be honored as made, and making it again as it is will fail the same way. The remedy is to change something the caller controls: the arguments, the configuration, the data handed in, or the embedder passed to clusters. Nothing about analysed text produces it.
One kind covers every such check, because they share that remedy, and reason says which check failed: a stable snake_case name, in details["reason"] on every channel, to branch on where the remedy differs. message names the values that failed it, for a person. A reader treats the set of reasons as open, since a later version can add a check.
reason | Check that failed | Raised by |
|---|---|---|
threshold_not_finite | The threshold is NaN or infinite | clusters |
embedder_vector_count | The embedder returned a different number of vectors than there are sentences | clusters |
embedder_dimensions_differ | Two of the embedder's vectors have different lengths | clusters |
embedder_value_not_finite | A vector holds a NaN or an infinity | clusters |
not_a_directory | The path is not a directory | Inputs::dir |
is_a_directory | The path is a directory where one file is read | Engine::analyze of an Input::file or Input::file_as |
blocks_malformed | Block input is not an array of heading and paragraph blocks | Engine::analyze of input read as Format::Blocks; Python's Engine.analyze(blocks=...) |
heading_level_zero | A heading block has level 0 | Engine::analyze of block input, from Input::blocks, JSON or Python |
config_malformed | The config file does not parse, holds an unknown key, or a value of the wrong type | Config::resolve, Config::from_file |
config_key_renamed | The config file sets a key 0.3.0 renamed or removed ([summarize], algorithm, [semantic], models.udpipe); the message names each replacement | Config::resolve, Config::from_file |
config_method_unknown | summary.method or keyphrases.method names a method this build does not know | Config::resolve, Config::from_file |
config_threshold_not_finite | clusters.threshold is NaN or infinite | Config::resolve, Config::from_file |
config_not_path_component | models.embedding is not a single directory name | Config::resolve, Config::from_file |
config_not_utf8 | The config file is not valid UTF-8 | Config::resolve, Config::from_file |
config_default_missing | The built-in defaults lack a key: a defect in this build, to report | Config::resolve, Config::from_file |
no_data_directory | None of MATRA_DATA_DIR, XDG_DATA_HOME or HOME is set | Config::resolve, Engine::new |
format_unknown | A format name matra does not know | Python's Engine.analyze |
method_unknown | A method name matra does not know | Python's summarize and keyphrases |
embedder_identity_failed | The embedder has no identity(), or it raised | Python's clusters |
embedder_identity_shape | identity() returned something other than an Identity dict | Python's clusters |
embedder_raised | The embedder's embed() raised | Python's clusters |
embedder_return_shape | embed() returned something other than a sequence of number sequences | Python's clusters |
usage | The arguments do not parse: an unknown flag, a missing value | the command line, under --json |
no_command | No subcommand and neither --skill nor --schema | the command line |
reference_without_skill | -r without --skill | the command line |
reference_unknown | --skill -r names no reference | the command line |
schema_unknown | --schema names no schema | the command line |
format_with_directory | --format given with a directory | matra analyze |
directory_for_one_document | A directory given to a command that reads one document | matra summarize, keyphrases, clusters |
method_unavailable | The configured method is one this build cannot run | matra summarize, keyphrases |
embedder_unavailable | The build has no embedder (no model2vec feature) | matra clusters |
config_file_exists | The config file exists and --force was not given | matra config init |
no_config_file_location | None of MATRA_CONFIG_FILE, XDG_CONFIG_HOME or HOME is set | matra config init |
config_key_unrendered | A resolved key has no renderer: a defect in this build, to report | matra config show |
embedded_file_malformed | A schema or skill reference compiled into the binary does not read: a defect in this build, to report | matra --schema, matra --skill |
A config file that is simply absent is not an error for Config::resolve, and the built-in defaults stand; Config::from_file refuses one with Io, because the caller named it.
UnsupportedFormat
Returned by Engine::analyze in two cases:
- The engine has no reader for the input's format:
PdforDocx, whether named by the caller, read off the extension, or recognised from the bytes (%PDF, or a zip archive holding[Content_Types].xml, which outranks the extension), or aFormat::Otherthe caller named and registered no reader for.formatis that format, andfeaturenames the Cargo feature that would read it; no feature of this version reads any of them, sofeatureisNone. A reader registered withEngine::with_readerreads any of them.Markdown,PlainTextandBlocksalways have a reader. - The bytes are not UTF-8 text and carry no signature matra recognises.
formatandfeatureare bothNone. This used to beIowith kindInvalidData; nothing went wrong reading the bytes, so it is now a kind with a remedy.
A UTF-8 file with an extension matra does not know is not this error: it reads as plain text.
Io
Wraps std::io::Error, produced by:
- Reading a file rejecting a symlink, with
ErrorKind::Unsupportedand the messagerefusing to read symlink: <path>;Inputs::dirrefuses a symlinked directory the same way. The path is alsodetails.path. - Reading a file rejecting a path that is neither a regular file nor a directory, with
ErrorKind::InvalidInputand the messagenot a regular file: <path>, and the path indetails.path. A directory isInvalidInputinstead, namingInputs::dir. - Any read, directory listing, directory creation, write, or rename that fails.
- The UDPipe provisioner when a download fails at the transport or answers with a non-2xx status. Same shape as the embedding path below: the message names the URL, and the kind is
TimedOutpast the 300-second fetch budget or the 30-second connect budget,NotConnectedfor an unreachable host, and whatever the socket reported otherwise. The URL is alsodetails.url, anddetails.retryableistrue, except for a rejected certificate. - The UDPipe provisioner when the model directory cannot be created, or when the verified bytes cannot be written or renamed into place. The message names the operation and the path, so a full disk reads
cannot write the model to <path>: No space left on device (os error 28)rather thanPermission denied (os error 13)with nothing to act on. The path is alsodetails.path. Model2Vec::potion_base_8mwhen a download fails at the transport or answers with a non-2xx status. The message names the URL, and the kind isTimedOutpast the 300-second fetch budget or the 30-second connect budget,NotConnectedfor an unreachable host, and whatever the socket reported otherwise. The URL andretryableare details, as on the UDPipe path. Bytes that arrived and then failed the digest areModelInvalidinstead; this variant is for the ones that never arrived.Model2Vec::potion_base_8mwhen the model directory cannot be created, when an artifact cannot be read, or when a verified artifact cannot be written or renamed into place. The message names the operation and the path, exactly as the UDPipe path's does, and the path isdetails.path.Model2Vec::from_dirreports an unreadable artifact the same way.Model2Vec::potion_base_8mwhen the temporary file an artifact lands through cannot be created, with the kind the open reported (AlreadyExistswhen something is already sitting at that path). The temporary is opened exclusively, so a path already there, symlink or not, fails the open rather than being written through.Engine::analyzewhen reading a file yields no document, withErrorKind::InvalidDataand the messagesource returned no documents.
Provisioning failures
The UDPipe provisioner (behind Engine::new and Engine::from_config, in Rust and Python) and Model2Vec::potion_base_8m obtain a pinned artifact over the network. Both fetch through matra's own client, under the same bounds, and both classify a failure the same way.
| Condition | Variant | Kind | What the message carries | Details |
|---|---|---|---|---|
| DNS failure, unreachable host, refused connection | Io, ErrorKind::NotConnected | io | The URL | url, retryable: true |
| Connect budget exceeded (30 seconds) | Io, ErrorKind::TimedOut | io | The URL | url, retryable: true |
| Fetch budget exceeded (300 seconds, lookup through last byte) | Io, ErrorKind::TimedOut | io | The URL | url, retryable: true |
| TLS certificate rejected | Io | io | The host, why matra cannot be made to trust it, the way out, then the underlying failure | url, retryable: false |
| Non-2xx status | Io | io | The URL and the status | url, retryable: true |
| Response past 64 MiB | InputTooLarge | input_too_large | what is "parser_model_download" or "embedding_model_download" | what, limit, actual |
| Directory, write or rename failed | Io | io | The operation and the path | path |
| Bytes arrived and failed the pinned digest, twice | ModelInvalid | model_invalid | The URL, or the directory and the expected digest | cause |
| Bytes arrived, passed the digest, and did not load | ModelInvalid | model_invalid | The loader's message | cause |
The rule behind the table: model_invalid is about bytes that arrived. Anything that stopped a fetch from arriving, or a filesystem from accepting it, is io (RFC-0015).
retryable is true when the same call can be made again as it is and may succeed. A failed fetch writes nothing, so a retry starts from where the first call did. A rejected certificate is false: a retry meets the same proxy, and the message names the way out instead.
The rule has a consequence in Python that is easy to miss, because it changes which except clause fires rather than only which string a message carries. IoError is an OSError and ModelInvalidError is a RuntimeError, so from 0.2.0 a DNS failure, a rejected certificate, a timeout or a non-2xx status raises an OSError from the call that builds the engine, where it used to raise RuntimeError. A caller that wrapped a bootstrap in except RuntimeError catches nothing now and the OSError propagates past it. Catch matra.MatraError, which both derive from, and branch on kind.
Nothing that failed the digest is ever written. Both provisioners fetch into memory, verify there, and write only what verified, so a download that fails adds nothing to the model directory and a run killed mid-transfer leaves nothing at all. A failed run removes nothing either. A cached file that is already there and fails verification is not loaded and not deleted: the UDPipe provisioner downloads again, at most twice, and replaces the file only with bytes that verified, through the same atomic rename every install lands through. A run whose downloads fail, at the network or at the digest, leaves that file byte-identical. Model2Vec::potion_base_8m replaces nothing and refuses a directory whose artifacts fail verification, because its three filenames belong to the artifact format rather than to this one model and so may be a caller's own (RFC-0015). A temporary left by a killed process is reclaimed by the next download that finds it older than ten minutes, which is twice the fetch budget and therefore older than any transfer that could still be running. The UDPipe provisioner leaves a temporary directory and Model2Vec::potion_base_8m three temporary files, and each reclaims its own by the same rule.
Behind a TLS-intercepting proxy
matra verifies TLS against root certificates compiled into the binary and never reads the system trust store, so a proxy that re-signs TLS cannot be made trusted. The fetch fails as Io, kind io, and the message names the host, says why its certificate cannot be trusted, names the file to fetch by hand, and ends with the underlying failure:
matra: io error: download https://lindat.mff.cuni.cz/...: the TLS certificate offered for
lindat.mff.cuni.cz was rejected. matra verifies TLS against root certificates compiled into
it and never reads the system trust store, so a proxy that re-signs TLS cannot be trusted by
installing its CA. Fetch english-ewt-ud-2.5-191206.udpipe by hand and put it in the model
directory instead. Underlying failure: io: invalid peer certificate: ...
Install the model behind a TLS-intercepting proxy has the steps.
The first run says so
A cold run has to fetch 16.3 MB before it can answer, and how long that takes is the link's business rather than matra's. The command line writes two lines to standard error before the transfer and nothing at all when the model is already there:
matra: downloading english-ewt-ud-2.5-191206.udpipe (16.3 MB) into /home/u/.local/share/matra/models
matra: the model is licensed CC BY-NC-SA 4.0 (non-commercial), separately from matra: https://creativecommons.org/licenses/by-nc-sa/4.0/
The second line is there because the model is a separate file with its own license, which is not matra's MIT; model licenses has both models and their sources.
Standard error, so --json output stays a single object on standard output. --quiet silences it. A library caller gets the same facts as a domain::ProvisionNotice from Engine::from_config_with_notice, and decides the wording itself.
The first matra clusters says the same for the embedding model, whose three files are fetched as one set of 30.9 MB:
matra: downloading potion-base-8M (30.9 MB) into /home/u/.local/share/matra/models/potion-base-8M
matra: the model is licensed MIT, separately from matra: https://opensource.org/license/mit
Python writes the same two lines to sys.stderr before Engine(), Engine.from_config(), Model2Vec.potion_base_8m() or Model2Vec.from_config() downloads; redirect sys.stderr to silence them. A Rust caller gets the facts from Engine::from_config_with_notice and Model2Vec::from_config_with_notice and chooses the wording, since the library writes to no terminal. The other Rust constructors, Engine::new, Engine::from_config, Model2Vec::potion_base_8m and Model2Vec::from_config, take no notice and say nothing for the length of their download.
Per-document failures: DocumentError
The stream surface never lets one bad document abort the rest. Engine::analyze_all yields Result<Document, DocumentError> per input, where DocumentError pairs the Error with the path it occurred at (None for an input that is not a file, so a path-less document is distinguishable from one at an empty path). Its Display prints path: error when a path exists and the bare error otherwise, and its source() is the wrapped Error.
The stream yields one item per input, in order: the Document, or the DocumentError in its place, so documents plus errors equals inputs consumed, and partitioning them is the caller's. Inputs::dir is Err only when the listing itself fails; both read failures and analysis failures travel as stream items.
Two kinds of entry never appear on either side. The directory listing skips symlinks, so a symlink produces no document and no error. It also skips anything that is not a regular file, and the walk is one level deep, so a subdirectory is skipped the same way.
DocumentError serializes as its error's form plus the path: {"path", "kind", "message", "details"}, with path null for input that never came from disk. Python's Engine.analyze_all yields that form as a dict for each failed document. kind is the stable string Error::kind names for the variant, and the table below carries all seven. Error::kind is a match with no wildcard arm, so a new Error variant fails to compile until someone names it, and spec/tests/corpus/items.json pins the vocabulary with a runner per crust, so every crust spells a failure the same way. Both item shapes come from serde. Either's path (provenance.source.path on a document, path on an error) is decoded with os.fsdecode, so os.fsencode on it names the file the walk read even when that name is not valid UTF-8. The Python call yields the items in order and the caller partitions them, testing isinstance(item, matra.Document).
Kinds and details
Error::details returns an ErrorDetails: the values a caller acts on, by name, which for every kind but invalid_input is every dynamic value in the message. Each value is a DetailValue, which is text, a whole number or a boolean. A caller reads a value instead of parsing the message.
| Kind | Details |
|---|---|
model_not_found | path |
model_invalid | cause |
parse_failed | cause; reason when what the parser or a reader returned failed one of the engine's checks, from the table above |
input_too_large | what, limit, actual |
unsupported_format | format when one was recognised; feature when a Cargo feature would read it |
invalid_input | reason, the check that failed, from the table above |
io | io_kind and cause; path when the failure concerns a path matra named; url and retryable when a download did not arrive |
- One word, one meaning.
reasonis always the name of a check that failed, stable and snake_case, to branch on: oninvalid_inputalways, and onparse_failedwhen the engine's checks refused what a parser or a reader returned.causeis always free text for a person, like the message: the text the variant holds, or the operating system's account of an I/O failure. io_kindis the I/O failure's class in snake case:not_found,permission_denied,already_exists,unsupported,invalid_input,invalid_data,timed_out,not_connected,connection_refused,connection_reset,unexpected_eof,interrupted,not_a_directory,is_a_directoryorstorage_full. Any other class isother. Treat the set as open.retryableappears only on a download that did not arrive. Its absence makes no claim either way.
A detail that does not apply is absent, never empty: an unsupported_format for bytes that are not text carries no details at all. A key a kind carries keeps appearing in later versions. A later version may add keys, so read the ones you know and ignore the rest.
The serde form is {"kind", "message", "details"}:
{ "kind": "input_too_large", "message": "input too large: 9000000 > limit 8388608",
"details": { "what": "input", "limit": 8388608, "actual": 9000000 } }
spec/schema/v2/error.schema.json is the published JSON Schema for this form and for DocumentError's, with the keys each kind carries.
Display strings and kinds
These are the strings Display produces, which are also the strings Python's str(exc) returns, alongside the kind Error::kind reports for each variant. A message is for a human to read; a kind is what code branches on.
| Variant | Format | Kind |
|---|---|---|
ModelNotFound(path) | model not found: {path} | model_not_found |
ModelInvalid(s) | invalid model: {s} | model_invalid |
ParseFailed { reason, cause } | parse failed: {cause} | parse_failed |
InputTooLarge { limit, actual, what } | {what} input too large: {actual} > limit {limit}, or input too large: {actual} > limit {limit} when what is "input" | input_too_large |
UnsupportedFormat { format, feature } | unsupported format: {format}; no feature of this matra version reads it, unsupported format: {format}; no reader is registered for it for a Format::Other, unsupported format: {format}; enable the `{feature}` feature to read it, or unsupported format: the input is not UTF-8 text and carries no signature matra recognises when format is None | unsupported_format |
InvalidInput { reason, message } | invalid input: {message} | invalid_input |
Io(e) | io error: {e} | io |
UnsupportedFormat renders the format's wire name, so the string reads unsupported format: pdf; no feature of this matra version reads it.
Matching in Rust
use matra::domain::{Error, Format};
use matra::{Engine, Input};
fn report(text: &str, engine: &Engine) {
match engine.analyze(Input::text(text, Format::PlainText)) {
Ok(doc) => println!("{} sentences", doc.metrics.total_sentences),
Err(Error::InputTooLarge {
what,
actual,
limit,
}) => eprintln!("{what} gate: {actual} over limit {limit}"),
Err(Error::UnsupportedFormat { format, .. }) => eprintln!("no reader for {format:?}"),
Err(other) => eprintln!("{}: {}", other.kind(), other),
}
}
The details carry the same values without a match on the variant, which is what a caller forwarding an error to another program wants:
use matra::domain::{DetailValue, Error};
fn limit_of(err: &Error) -> Option<u64> {
match err.details().get("limit") {
Some(DetailValue::Integer(limit)) => Some(*limit),
_ => None,
}
}
Python exception mapping
The PyO3 binding raises each Error as a class of its own, named for the variant with PEP 8's Error suffix. The name a failure shares across languages is its kind, not its class. Every class derives from matra.MatraError and from the builtin class matra raised before 0.3.0, so except ValueError still catches an InputTooLargeError. The routing is a match with no wildcard arm over the variants, so adding a variant to Error without assigning it a class fails to compile.
| Variant | Python class | Also a |
|---|---|---|
ModelNotFound | ModelNotFoundError | FileNotFoundError |
ModelInvalid | ModelInvalidError | RuntimeError |
ParseFailed | ParseFailedError | RuntimeError |
InputTooLarge | InputTooLargeError | ValueError |
UnsupportedFormat | UnsupportedFormatError | ValueError |
InvalidInput | InvalidInputError | ValueError |
Io, any other kind | IoError | OSError |
Io, kind NotFound | IoNotFoundError | FileNotFoundError |
Io, kind PermissionDenied | IoPermissionError | PermissionError |
Io routes on the I/O kind it wraps. A missing file is an IoNotFoundError, which is a FileNotFoundError; a refused permission is an IoPermissionError, which is a PermissionError. Both derive from IoError, so except matra.IoError catches all three. matra.IoError is matra's class, not the builtin IOError, which is another name for OSError.
Every exception carries the same three things as the serde form: kind, the string Error::kind names; details, a dict of the details; and the Display string as its message, which str(exc) returns. Branch on the class or on kind, and read values from details:
import matra
try:
engine.analyze(huge)
except matra.InputTooLargeError as exc: # also a ValueError
print(exc.kind, exc.details["what"], exc.details["limit"], exc.details["actual"])
IoNotFoundError is a FileNotFoundError, so the conventional Python idiom works:
try:
doc = engine.analyze(file="notes.md")
except FileNotFoundError as exc:
print(exc.details["path"]) # notes.md
The classes name matra as their module, so a traceback reads matra.InputTooLargeError, and an exception survives pickle with its kind and details.
Every Python input reaches the parser through the engine's reading stage, so the 8 MiB "input" gate applies uniformly, to text, bytes and blocks alike, and a file meets "file_read" first. The outcome functions take a Document, never text, and their own caps apply to it.
One further failure has no Error variant behind it. Python's Engine is #[pyclass(unsendable)], because the loaded model holds C-side state that is not thread-safe. Accessing one engine from a thread other than the one that created it fails at runtime. Multi-process use is unaffected.
At the command line
One program is installed under the name matra, by either of two routes. The Rust binary comes from cargo install matra --features cli; the Python console script comes from the wheel and calls the same matra::cli::run through the extension module. The exit-code contract below is the program's, so it holds on both.
| Exit code | Meaning |
|---|---|
| 0 | The command succeeded, and where applicable something was found |
| 1 | The command succeeded and found nothing |
| 2 | An error occurred |
On exit code 2 the command writes matra: followed by the error's Display string to standard error, prefixed with the path when the failure concerns a file. Under --json it also writes the envelope to standard output, with result null and error holding the serde form above, {kind, message, details}: the same three a Rust caller serializes and a Python caller reads off the exception. The envelope's input names the file, so the error object carries no path of its own.
{ "format_version": 2, "command": "analyze", "input": "scan.pdf", "result": null,
"error": { "kind": "unsupported_format", "message": "unsupported format: pdf; no feature of this matra version reads it", "details": { "format": "pdf" } } }
spec/tests/parity/parity.json pins that equality: the same files through Rust, Python and the command line give the same kind, message and details, and a runner per surface fails on a drift.
A broken pipe, which is what happens when the reading end of matra analyze file.md | head goes away, exits 0 and prints nothing.
matra analyze DIR reads every file in the directory and keeps going past a failure. Each failed file is a line on standard error and, under --json, its own envelope line with its error; the run exits 2 if any file failed, after the walk has finished. This is the one partial success the command line has: per-item failure in a bulk operation.
The table is the whole contract, and no command narrows it. The command line's own refusals carry the kind whose remedy matches, which for each of them is InvalidInput, a call to change: a directory given to summarize, keyphrases or clusters, --format with a directory, matra --skill -r <name> or matra --schema <name> with a name nothing answers to (both name the ones that exist), -r without --skill, config init over an existing file without --force, clusters in a build without the model2vec feature, and a usage error such as an unknown flag. A missing path is not one of them: it is the library's own Io failure with not_found, raised before the engine is built so it costs no model load. matra --skill and matra --schema in any of their accepted forms exit 0; they read no document, so they have nothing to find and never exit 1.
The exception classes above belong to the library surface, not to the command line. A domain::Error reaching the command line is rendered and turned into an exit code and, under --json, an envelope; it never surfaces as a Python exception, whichever launcher started it.
Boundary rules
Eight rules hold matra's hexagonal architecture in place.
CLAUDE.md carries a summary list and points here for the reasoning.
What enforcement means here
Rust offers no directional import control between modules inside a single crate, and matra is a single crate by RFC-0004, so the compiler enforces none of these rules except rule 6. Seven of the eight are checked by semgrep rules in .semgrep/, which read the import and path forms each rule forbids. Review remains the gate for what an import cannot show.
| Rule | Checked by | What the check catches | What review still reads for |
|---|---|---|---|
| 1. Domain dependency set | boundary-rule1-* | any use or inline path in src/domain.rs rooted outside std, core, alloc, serde, thiserror | a #[macro_use] macro used unqualified |
| 2. Ports import only domain | boundary-rule2-* | crate:: or super:: into anything but domain, external crates, the port's own adapters | a trait whose shape leaks an adapter |
| 3. No cross-port import | boundary-rule3-* | a port named by use, brace group, inline path or trait bound | macro-generated paths |
4. Single udpipe_rs importer | boundary-rule4-* | udpipe_rs named anywhere else, and anything pub in the adapter that names it | a private alias made public under another name |
| 5. Metrics and extraction purity | boundary-rule5-* | any crate module other than domain and stopwords | a function taking text instead of parsed structure |
| 6. No-default-features build | CI | the whole rule, by compiling | |
| 7. Composition root knows the whole | boundary-rule7-* | src/cli/ reaching past Engine, Input, Inputs, Block, the outcome functions and their selectors, config and domain; src/python/ reaching past the same, embed and cli | wiring outside src/lib.rs in any other file |
8. No tracing in domain or ports | boundary-rule8-* | the identifier in code in the five files |
Each finding names its rule, says why the boundary exists, and links the section of this page to read. The full table, with every form each check covers and misses and the reasons the checks are built the way they are, is EP-0014. Code inside #[cfg(test)] modules is exempt from rules 1, 2, 3, 5 and 7, and read by rules 4 and 8.
scripts/check-boundaries.sh runs the checks: first each rule file against its fixture of violating and allowed code, then the scan of src/, then a check that the scan read every Rust file there. It runs from just check, from the pre-commit hook that scripts/install-hooks.sh installs, and from the Boundary check job in ci.yml. The hook is opt-in, runs on every commit regardless of which files are staged, and skips the check with a warning when semgrep is not installed; CI does not skip it.
Rule 6 is the only rule CI verifies by compiling. The rust job runs cargo check, cargo clippy, and cargo test with and without default features, on Linux and macOS. The msrv job runs cargo check on Rust 1.88, once with all features and once with no default features. CI fires on pushes to main and alpha and on pull requests targeting them, so work on a feature branch is ungated until the pull request opens.
Rule 1: the domain dependency set
The rule. src/domain.rs depends on serde, thiserror, and std, and on nothing else.
Scope. One file, plus the non-optional entries in [dependencies].
Why it is drawn there. Domain types are what every language surface serializes. A dependency added here enters the closure of every caller on every target. Changing the set takes an RFC. thiserror was admitted that way: it emits no public API and replaced roughly 35 lines of hand-written Display and Error implementations.
Enforcement. boundary-rule1-domain-imports fails on any use or extern crate in src/domain.rs rooted outside std, core, alloc, serde and thiserror, including crate::, super:: and self::: the domain imports nothing from the rest of matra. boundary-rule1-domain-inline-paths fails on an inline path with any other lowercase root, because a crate listed in Cargo.toml can be named inline with no use line; for that reason the domain writes std paths in full (std::fmt::Display). A dependency added to [dependencies] and used in domain.rs is therefore caught wherever domain.rs names it. Review reads for the one route left, a macro or derive brought into scope crate-wide by #[macro_use] in src/lib.rs and invoked unqualified.
Rule 2: ports import only from domain
The rule. Each port module imports from crate::domain and std, and from no other module or crate.
Scope. src/source/mod.rs, src/decompose/mod.rs, src/nlp/mod.rs, src/embed/mod.rs.
Why it is drawn there. A port is a contract. Whatever the contract imports becomes a requirement on everyone who implements it, so a domain-only port stays implementable by someone who has never read matra's adapters.
Enforcement. boundary-rule2-port-imports and boundary-rule2-port-inline-paths fail on any import or inline path in those four files that reaches past crate::domain and std: another crate module, a crate-root item such as crate::Engine, a glob, an external crate, or the port's own adapter through self::. In a port's mod.rs, super:: is the crate root and is read as such. Review reads the trait method signatures for what an import cannot show: a contract shaped around one adapter's needs.
Rule 3: no port module imports another port module
The rule. The four ports are peers and name each other nowhere.
Scope. The same four files.
Why it is drawn there. Stage order belongs to the composition root, not to the contracts. If Decomposer knew about NlpProvider, the pipeline's shape would be encoded in the traits and a stage could no longer be replaced on its own.
Enforcement. boundary-rule3-port-names-port fails when one of the four files names a port through crate:: or super::: in a use line, a brace group (a comment inside a multi-line group does not hide it), a fully qualified inline path, or a trait bound. A type alias cannot launder the path, because the only module a port may import is the domain, and the domain may import nothing from the crate (rule 1). A crate-root re-export of another port's trait is caught by rule 2.
Rule 4: one file imports udpipe_rs
The rule. src/nlp/udpipe.rs is the only file in the crate that imports udpipe_rs.
Scope. All of src/.
Why it is drawn there. This is a resilience rule. UDPipe is C++ across an FFI boundary holding state that is not Send, and a panic on the C side aborts the host process rather than unwinding, which means interpreter death in Python. The catch_unwind boundary in nlp/udpipe.rs converts that into a domain::Error. Confining the import is what makes that boundary the only entrance rather than one entrance among several. A second NLP backend gets its own adapter file with its own panic boundary, which is the pattern working rather than an exception to it.
Enforcement. boundary-rule4-udpipe-rs-outside-adapter fails on udpipe_rs anywhere in code under src/ outside src/nlp/udpipe.rs, test modules included. boundary-rule4-udpipe-rs-exposed fails inside the adapter on anything pub, at any restriction, that names udpipe_rs: a re-export, a type alias, a signature or a field, any of which would let another file hold the C-backed type while the first check stays green. Review reads for the laundering route left: a private alias (use udpipe_rs::Model as M;) made public under another name.
The analog. src/embed/model2vec.rs is the only file that imports safetensors and tokenizers, enforced by the same pair of checks (boundary-rule4-model2vec-crates-*) with the same one route left to review. Those crates are pure Rust, so the confinement is not a C panic boundary; it is what keeps the adapter swappable and the model-format vocabulary out of every other file's reach. A second embedding backend gets its own adapter file with its own confinement line.
Rule 5: metrics, extraction, and structure readers stay pure
The rule. No file under src/metrics/ or src/extraction/, and no structure-reading module (src/hearst.rs), imports from any crate module other than crate::domain and crate::stopwords.
Scope. Intra-crate imports only. External crates that the computation itself needs are unaffected: the compression metric uses brotli, and several extraction files use std::collections.
Why it is drawn there. These are pure functions over already-parsed structure. Purity is what lets them run with no model loaded, be unit-tested without fixtures, and be called by someone who parsed elsewhere. It is also what makes the parse-once-use-many contract real: a metric that reached for an NlpProvider would re-parse internally, and a caller who had already parsed would pay twice.
Enforcement. boundary-rule5-pure-imports fails on any crate:: path in those trees, in a use line, a brace group or inline, naming something beyond domain and stopwords, and on super::super::, which leaves the tree. boundary-rule5-root-super reads super:: the same way in src/metrics/mod.rs, src/extraction/mod.rs and src/hearst.rs, where it is the crate root. Sibling imports within a tree (super::textrank) and external crates stay allowed. Review reads for what purity means rather than what it imports: any function there taking raw text instead of a Document or its sentences.
Rule 6: the no-default-features build compiles
The rule. cargo check --no-default-features succeeds.
Scope. The whole crate.
Why it is drawn there. This is the mechanical proxy for features being additive and the core standing alone. It proves that the domain and the ports compile with no UDPipe, which is the configuration a type-only caller needs.
Enforcement. CI, as described above. Code that only compiles with udpipe enabled belongs behind #[cfg(feature = "udpipe")].
Rule 7: the composition root knows the whole
The rule. src/lib.rs is the only place that knows all adapters and all ports.
Scope. Every file except src/lib.rs. Test modules are exempt.
Why it is drawn there. Knowledge of the full assembly is a cost paid once. Concentrating it in one file means a reader learns how matra is wired by reading one file, and adding an adapter is a one-file change. Two files that both know the wiring drift, and the place that was missed becomes the bug.
Enforcement. For src/cli/ and src/python/, the checks described below. Everywhere else, review: read for any file other than lib.rs importing from two or more adapter modules, and for any helper outside the composition root that matches on Format to pick a reader.
src/config.rs sits in this tier alongside lib.rs. It imports domain, std, serde and toml, and it imports no port and no adapter. The traffic runs the other way: an adapter may import Config to offer a from_config constructor (RFC-0011), which is why Udpipe::from_config lives in src/nlp/udpipe.rs and Model2Vec::from_config in src/embed/model2vec.rs, not in the composition root. That import gives the adapter a default, not a second opinion about the wiring, so rule 7 still holds: lib.rs remains the only file that knows every adapter and every port.
src/cli/ sits above that tier: it is the application, compiled into the library so both launchers run one program. From the crate it uses the public surface lib.rs exports (Engine, Input, Inputs, Block, the outcome functions summarize, summarize_with, keyphrases, keyphrases_with and clusters, and their selectors SummaryMethod and KeyphraseMethod), config, domain and embed, and never another port module, a reader or parser adapter, or a crate-private module such as extraction. It reaches the pipeline through Engine::from_config and the outcomes through the functions at the crate root, so rule 7 holds there too. All four port modules are public since the provider traits opened, but only embed is on the command line's list: it implements no parser, reader or source and plugs none in, so nlp, decompose or source there would be wiring the composition root does, by another name, and the check keeps them out. Model2Vec is public behind its feature. The library builds no embedder inside an outcome call, because building the default one downloads a model, so matra clusters builds it from the configuration with Model2Vec::from_config and hands it to clusters. Choosing that default is the application's decision, and the command line is the application.
boundary-rule7-cli-imports reads both spellings of a violation, because the inline one is the one it would most likely take. The declared form is a use crate::... line in src/cli/; the inline form is a qualified path in a function body. The CLI already reaches Engine, Input and the outcome functions by inline path rather than by use, so the inline form is the ordinary idiom here and the one a violation would blend into. The check is an allowlist: any crate:: path in src/cli/ must start with Engine, Input, Inputs, Block, summarize, summarize_with, keyphrases, keyphrases_with, clusters, SummaryMethod, KeyphraseMethod, config, domain, embed or cli, so another port, a reader or parser adapter, a crate-private module, or a crate-root re-export of an adapter fails it. boundary-rule7-cli-root-super reads super:: in src/cli/mod.rs, where it is the crate root, the same way. Either spelling would put adapter selection in the command line, which is the composition root's job.
src/python/ is the Python binding, a driving adapter that translates and decides nothing, held to the same rule by boundary-rule7-python-imports and boundary-rule7-python-root-super in .semgrep/rule7-python.yml. Its allowlist is the command line's plus cli, because the console script runs cli::run. It reaches embed for its own reason: a Python caller hands clusters an embedder (Model2Vec, or any object the binding wraps as an Embedder). It reaches no other port: Python cannot implement a parser, a reader or a source in 0.3.0, so the binding has nothing to wrap as one, and when it can, the wrapper is the reason to add the port to its list, as embed was added. Its own tree is crate::python. Before 0.3.0 the binding lived inside src/lib.rs, out of this check's reach, and it decided things the command line decided differently: it read markdown as plain text before a summary. Held to the public surface, it cannot.
Rule 8: no tracing in the domain or the ports
The rule. tracing is forbidden in src/domain.rs and in the four port modules.
Scope. Those five files.
Why it is drawn there. Observability is an adapter and composition-root concern. A domain type that emits spans holds an opinion about the host's runtime and subscriber configuration, and a port that traces forces that opinion onto every implementor. In domain.rs it is also rule 1 by another route, since it would be a fourth dependency.
Enforcement. boundary-rule8-no-tracing fails on the identifier tracing anywhere in code in those five files, test modules included, whether in a use line, a brace group, a path or an attribute. The rule is preemptive: tracing is not a dependency of matra at all. The line was drawn before the first import could land.
Which rules apply to which files
| File touched | Rules in scope |
|---|---|
src/domain.rs | 1, 8 |
src/source/mod.rs, src/decompose/mod.rs, src/nlp/mod.rs, src/embed/mod.rs | 2, 3, 8 |
src/nlp/udpipe.rs | 4, 6 |
| Other adapters | 6, 7 |
src/config.rs | 6, 7 |
src/cli/ | 7 |
src/python/ | 7 |
src/metrics/, src/extraction/, src/hearst.rs | 5, 6 |
src/lib.rs | 6, 7 |
Cargo.toml | 1, 6 |
Running the checks
pip install --require-hashes -r .github/requirements/semgrep.txt # once
just boundary # scripts/check-boundaries.sh on its own
just check # every local gate, including the boundary check
just install-hooks # install the pre-commit hook that runs it
The check needs semgrep, at the version .github/requirements/semgrep.txt pins, and must run from the root of a git work tree: semgrep resolves each rule's paths against the work tree, and outside one a path-scoped rule matches nothing. Each finding prints the file, the line and the rule's message. On a clean tree the script ends with check-boundaries: N of N Rust files in src/ read, 0 finding(s), and it exits non-zero on any finding, any failed rule test, any Rust file under src/ the scan did not read, or any rule scoped to a path git no longer has. A rule scoped to a moved file would scan nothing and pass, so when a file moves, the rules that name it move with it.
A violation is a merge blocker. The remedy is a change to the structure, or an RFC that changes the rule deliberately. It is never a change to the check. A check that flags correct code is a bug in the check: fix the rule and add the case to its fixture, as EP-0014 describes.
Roadmap
What matra does not do yet, why, and the condition that would change that.
Every other page of this documentation describes what ships today. This page is the only one that describes what does not, which is what lets you trust the rest without auditing each claim.
matra follows a discipline borrowed from road-building: dirt road first, then cobblestone when traffic justifies it, then tarmac when the route is established. Each capability below is deferred not because it is unimportant, but because adding it before the trigger condition fires would add complexity that nobody needs yet, or that requires design decisions that need real usage patterns to make correctly.
The discipline is Chesterton's fence applied to features: before you add a capability, be able to state what problem it solves for which caller today. If the answer is "we might need this eventually," the fence stays up.
Every entry below names the trigger condition. When the condition is met, write an enhancement proposal and proceed.
The scoping principle: lowest verifiable tier
A survey of the computational landscape (internal, 2026-08-21) sorts NLP affordances into three tiers: deterministic tooling whose output is checkable against the source (tokenizers, morphology, dependency parsers, document-structure extraction), specialist models (coreference, discourse relations), and general model judgment, which has the widest coverage and the lowest verifiability. The engineering rule it supports: route each operation to the lowest tier that meets the accuracy and verifiability requirement, and escalate only when the cheaper tier provably fails.
matra is the first tier, deliberately. Everything it ships is deterministic and grounds back to the bytes it came from, which is what makes its output usable as data rather than as interpretation. Two consequences bound the roadmap:
Capabilities with no reliable affordance stay out, and the gap is recorded rather than simulated. Pragmatic enrichment (implicature, purport, illocutionary force) and claim or fallacy certification have no deterministic path and no reliable model path either; the honest output there is a structural gap marker, not simulated confidence. matra's idiom already encodes this (Option slots that mean "declined to compute", a tree_depth of None on a cycle, UnsupportedFormat, DocumentError, the skipped list), and new capabilities inherit it.
Capabilities that need a specialist model are adapter work, not extensions of the parse. Coreference is the clearest case: it sits a tier above UDPipe, the research synthesis places it above matra beside SRL and NLI, and pretending otherwise would put unverifiable output behind a surface that promises verifiability. If a specialist coreference adapter ever lands, it arrives as a new parser adapter behind its own feature flag, with its tier stated plainly.
After 0.3.0
0.3.0 settled the public surface: one set of outcome names in Rust, Python and the command line, input through Input, a span and provenance on every result, schema 2.0 with a published JSON Schema, and an error kind on every channel. Migrate from 0.2 to 0.3 lists every change. Four things are planned for 0.4.0, each an addition to that surface:
- A feed for text arriving over time, in Rust, Python and the command line at once. See Streaming source.
- Caller extensions: predicates and the findings they produce, over a
Documentand its schema, in their own proposal. See Rule evaluation over parsed structure. - TypeScript: types generated from the published schema, and a web component that renders a parse from matra's JSON. The spike on compiling UDPipe to WebAssembly returned a go, and EPR-0007 proposes a package that parses in the browser and in Node.js; it is proposed, not accepted. See WASM binding for TypeScript/browser.
- A permissively licensed parsing model, as a second pinned model chosen by id. See A permissively licensed parsing model.
Rule evaluation over parsed structure
What it is. Declarative rules expressed as predicates over Document. A Rule names a structural pattern (a dependency arc, a token sequence, a metric threshold) and fires when the pattern is present. The output is a Finding with a SourceSpan that points back to the bytes in the original text where the rule matched.
This is the capability that bridges the record tier (what matra exposes today) and the abstract tier: relation extraction, modality detection, speech act classification, voice signature analysis. The record tier gives you the tokens and arcs. Rule evaluation gives you the named patterns over them.
Where it lands. Inside matra, in a new src/rules/ module, occupying the abstract extension point RFC-0007 reserves between structure and purpose-fitted output. The vocabulary is locked by RFC-0006: Rule, Predicate, Finding, SourceSpan. Rule evaluation is not a separate crate; callers compose against one surface. The Finding type's shape (trait vs enum) is deferred to Phase 2; the name is locked. Since 0.3.0 every heading, paragraph, sentence and token carries a Span into the analysed text, so a finding can point at bytes without a new addressing scheme. Caller extensions are planned for 0.4.0, in their own proposal, on schema 2. Rules are deterministic predicates over structure, so their findings stay at the verifiable tier: a Finding names the pattern and points at the bytes, and whether the pattern matters remains the caller's judgment.
Trigger condition. FIRED, 2026-05-23. The condition was at least one concrete caller pattern that direct Document field access and application-side logic cannot adequately serve.
An internal research synthesis dated 2026-05-23 names five such patterns and states the case directly: "these five primitives are the consumer sites. The deferral has fired." They are negation, modal classification, evidentiality, Hearst patterns, and typed morphological features, each grounded in a distinct literature (FactBank polarity, CoNLL-2010 hedges, Aikhenvald evidentiality, Hearst 1992).
A second, sharper piece of evidence sat inside this repository. Sentence::is_passive was a method, and methods do not cross FFI, so python/matra/cli.py re-implemented passive detection over raw tokens: matra's own language binding duplicated its own primitive. RFC-0008 settled the channel (derived structural facts cross FFI as fields, computed once in Rust at a single point in the pipeline) and the Python re-implementation is deleted.
The five primitives shipped, 2026-08-21, in blueprints/legacy/eps/0007-structural-primitives.md, which carries the milestones, their rubrics, and the reasoning trail. Rule and Predicate are designed after them, from the shape the five actually took, which is what "pulled from real use, not anticipated" asked for.
What the five revealed about that shape. Written from the landed code, not from anticipation. A Predicate must be able to name:
- A single dependency arc, selected by relation and lemma. Negation is one
advmodarc carrying a closed-class cue; a modal is oneauxarc whose lemma sits in the ten-item UD closed class, widened by theAUXpart of speech to catch modals the parser promotes under ellipsis and coordination. Both landed as structs of token ids and lemmas (Negation,Modal). Arc-plus-lemma is the smallest pattern unit, and two of the five primitives are exactly one of them. - A morphological feature at a tree position. The root-mood detector reads
MoodthroughToken::featon the root or on acoporauxchild of it. A predicate vocabulary with arcs but without feature lookups cannot express the test that separates asserted from modalized clauses. - A multi-arc construction with optional participants. The reporting construction is a verb governing a
ccomp, plus itsnsubjwhen the sentence has one (ClausalComplement, named for what the detector finds). The subject slot is optional because sentence segmentation strands attribution across boundaries; a construction predicate must match with an absent argument rather than fail. - A lexicon as a parameter, for open classes. Reporting verbs and evidential adverbs have no closed list, so the detectors report every structural match and
Sentence::clausal_complements_inandSentence::root_adverbials_infilter by a caller-supplied lexicon. ARuleover an open class carries its lexicon as data; embedding one would ship an incomplete list that looks authoritative. - Span pairs with token-id traceability, as output. A Hearst match returns a pattern tag plus two spans, each a head token and a token-id range (
HearstPair,HearstSpan).SourceSpantherefore has to address a range, not a single token, and aFindinghas to be able to carry more than one of them.
Two mechanism facts carry over. Predicates read materialized fields, not the raw parse: RFC-0008 computes each derivation once at a single point in the pipeline and serializes it, so rule evaluation is field access over Document before it is graph walking. And that line held under real pressure: every discriminant that landed names a construction (HearstPattern variants, modal lemmas, the reporting shape), never a semantic judgment, which is exactly the Frame-3 contract RFC-0006 imposes on Finding.
WASM binding for TypeScript/browser
What it is. A wasm-bindgen surface that exposes the same Document types to TypeScript running in a browser or Node.js environment. The same names cross; the same methods-don't-cross constraint applies. This makes matra usable in frontend applications, browser-based document editors, and Node.js pipelines without requiring a Python or Rust runtime.
What is blocking it. A decision, not the parser. This entry used to say that C FFI cannot run in a WASM sandbox, and that a parser in the browser would need a different parser or a server. A spike run between 2026-11-14 and 2026-11-18 found otherwise. UDPipe's C++, compiled with Emscripten, gave output byte-identical to the native build over matra's twelve example texts, in Node.js and in the Chromium, Firefox and WebKit builds that Playwright ships, at about half the native speed. EPR-0007 records every measurement and what was not measured, which includes Safari itself, any phone, and a model fetched over a real network.
The NlpProvider trait required no change. In the spike matra's core compiled for WebAssembly without the udpipe feature, with the compiled parser handed in through Engine::with_parser, and the outcome functions over a Document need no parser at all.
What remains open is the design, which EPR-0007 proposes and the owner has not accepted: where the binding lives, what a caller sees when something fails, which browsers the parser build admits, what the package is called on npm, what a second reader for one format does, and whether text is detected as bytes are. One question it lays out without a recommendation: who serves the parsing model to a browser. The model is licensed CC BY-NC-SA 4.0, so the package cannot carry it, and its publisher's host refuses requests from other sites, so a page cannot fetch it from there.
Planned, in steps, each usable alone. EPR-0006 placed three in 0.4.0: TypeScript types generated from the published JSON Schema in spec/schema/v2/ with a web component that renders a parse from matra's JSON, the pure-Rust core in WASM, and the spike, which has now returned a go. EPR-0007 orders what follows as eight milestones, in the order they are built: the types, for TypeScript and Python from the same schema; the ports and the names, so that a reader or parser written outside matra can do what matra's own do, every document names the reader that read it, and each method's name is written once; one copy in Rust of the mapping from the parser's output to tokens, the core in WebAssembly with the outcome functions, the parser, the release to npm, the component that renders a parse, and the playground that shows its result with it. The component needs only the types, so it may be built any time after them. None starts before the proposal is accepted, and nothing reaches npm without the owner's approval of that release.
Trigger condition. The owner accepts EPR-0007. The condition this entry used to carry, an NlpProvider implementation that compiles to WASM with sufficient parse quality, is met as far as the spike measured: it is the same parser giving the same output.
PDF and DOCX decomposers
What they are. Readers (crate-private Decomposer implementations) for PDF and DOCX documents. The variants already exist in domain::Format, and detection already recognises both, by extension and by their bytes (%PDF, or a zip archive holding [Content_Types].xml). Analysing one returns Error::UnsupportedFormat today, whose feature is None because no feature adds a reader; there is no reader to hand the bytes to.
What is blocking them. The Rust PDF extraction ecosystem. PDF text extraction requires handling layout ordering (columns, reading order), encoding diversity (Type1, CFF, CID fonts), and embedded-font subsetting. No current Rust crate handles all of these consistently and without correctness holes on real-world documents. DOCX extraction is more tractable but has not been required by any caller yet.
Trigger condition. A documented caller requirement for PDF or DOCX analysis, paired with a Rust extraction crate that is stable (post-1.0, maintained, no known correctness holes on the document shapes the caller needs). When a reader lands, no published signature changes: it is a reader file and a registration line in the composition root, behind a feature that UnsupportedFormat then names, emitting the same headings and paragraphs with spans as the markdown reader.
Recursive directory walk
What it is. Inputs::dir descending into subdirectories rather than stopping at depth one. This enables processing nested document collections (a repository of markdown files organized by directory, for example) without the caller having to walk the tree manually.
What is blocking it. No caller has required it. Recursive walks have non-obvious behavior: symlink cycles, permission errors mid-walk, and large directory trees with heterogeneous content all require explicit policy decisions. What is the depth limit? How are symlinks handled at nested levels? How do per-file errors propagate: do they abort the entire walk or continue? Building these policies into the API without a caller to validate them risks building the wrong policies.
Trigger condition. A caller that documents a requirement to process nested directory trees, with enough specificity to make the symlink, error-tolerance, and depth-limit policy decisions correctly.
Streaming source
What it is. A feed: text that arrives over time, such as a transcript or a message queue, analysed as it comes rather than handed over whole. It is planned for 0.4.0, in Rust, Python and the command line at once, so no surface gets it before the others.
What is in place. Engine::analyze_all is already lazy: it takes any iterator of Input, yields one result per input in order, and Inputs::dir reads each file only when the stream reaches it. So a caller can already stream documents that are each complete. What the feed adds is text that is not yet complete when analysis starts. The readers and the parser are private to the crate, so the feed arrives as an addition to the public surface, not a change to a signature that ships.
What is blocking it. The design: where a paragraph or a sentence ends when more text may still arrive, and what a partial result promises. The async reactor decision (deferred per RFC-0004) bears on the most useful form, async processing; a synchronous feed is possible without it.
Trigger condition. FIRED. EPR-0006 places the feed in 0.4.0 (decision 2), as an addition in all three syntaxes.
Separate matra-nlp-api port crate
What it is. Extracting the NlpProvider trait, public since 0.3.0, into a separately published minimal crate, so third-party NLP provider implementors can depend on the port contract without depending on all of matra. This follows the Pattern 6 criterion: separately publish a minimal port crate if and only if an external implementor ecosystem exists that needs to pin the contract independently of the main crate's version churn.
What is blocking it. No third-party NLP provider implementor crate exists today. The criterion is ecosystem size, not architectural elegance. Extracting matra-nlp-api before any external implementor exists would add a publishing burden (two crates to version, two changelogs, two sets of CI) for zero current benefit.
Trigger condition. A published Rust crate that depends on matra solely for NlpProvider (a matra-stanza, matra-spacy, or similar, shipped by a non-matra maintainer). At that point, extract matra-nlp-api as a minimal port crate, have matra depend on it, and keep matra as the caller-facing surface.
The pattern across all of these: capability waits for the caller that justifies it. Adding structure in advance of use is complexity that the next collaborator pays for. The trigger conditions are the guard against that. When one fires, write the enhancement proposal, explain why this trigger and not the next one, and proceed.
HTML report
What it is. A self-contained HTML rendering of a Document: the parse, the metrics, and the extracted phrases in a form suitable for visual inspection, notebook display, and paper supplementary material. The surface would be Document::to_html_report() in Rust, report(text, format="html") in Python, and matra report on the CLI.
What is blocking it. Nothing structural. It is unbuilt because no caller has needed it yet.
Trigger condition. A caller that needs to eyeball a parse rather than consume it programmatically, most likely a researcher working in a notebook.
A permissively licensed parsing model
What it is. A second pinned English parsing model whose licence permits commercial use, trained on UD English EWT, chosen by id beside the model that ships. The model that ships, english-ewt-ud-2.5-191206, is CC BY-NC-SA 4.0, which does not.
Where it lands. As a pin, like the first: a URL, a SHA-256 and a licence compiled into the adapter, and an id a caller names. Every document already records the parser and its model's digest in provenance.parser, so a reader sees which model produced a parse rather than discovering it as a shift in the numbers. Choosing by id is an addition; 0.3.0 removed loading a model file by path so that this would not be a break.
Trigger condition. FIRED. EPR-0006 places it in 0.4.0. What remains is training the model and measuring it against the one that ships.
Browser playground
What it is. A page where text can be pasted and its structure rendered live, with no install.
What is blocking it. It depends on the WASM binding above, and on a decision about who serves the parsing model to a browser. The model's licence is non-commercial and its publisher's host refuses requests from other sites, so a page needs the model at an address it may fetch. EPR-0007 lays out three options and what each costs: the docs site serves the current model with attribution, the playground waits for the permissively licensed model, or each visitor supplies the file. The choice is the owner's.
Trigger condition. The WASM binding ships and that choice is made.
Voice fingerprint metrics
What it is. Three additions that make matra's existing metrics usable as a stylometric signature rather than as per-document readouts: a length-normalized lexical diversity measure, burstiness as a first-class value, and a contraction count.
Why these three specifically. An internal note dated 2026-04-21 records a writing-assistant agent blocked on a calibration loop against a measured baseline: 26.6 word mean sentence length, 36.9 percent passive, 0 percent contractions, 7.7 percent nominalisation, 0.76 burstiness. Checked against what ships, three of those five are already Document fields. Burstiness is derivable from metrics.sentence_length_std and metrics.mean_sentence_length, both fields since 0.3.0, but is not itself exposed. Contractions are not counted at all. That note estimated the remaining work as a small configuration layer rather than a rewrite.
One of them is a defect, not an addition. vocabulary_ttr is a raw type-token ratio and TTR falls mechanically as text grows. Measured on this repository: README scores 0.690 against architecture/design.md at 0.227, but README has 35 sentences and the other has 266, so most of that gap is length rather than voice. As a cross-document feature it is unsound, which matters most when a directory is analyzed (Engine::analyze_all over Inputs::dir, or analyze_all(dir=) in Python), because the result invites the comparison. The limitation is documented on the field; what remains is a length-normalized measure beside it (MTLD, MATTR, or standardized TTR).
Why this is the strongest candidate of anything on this page. The four faces of voice all map onto shipping output with no hole: agentive onto nsubj and nsubj:pass, modal onto aux and feats, structural onto section hierarchy and sentence length, stylistic onto lexical density and compression. That is unlike claim atomization, which needs an LLM, and unlike deriving a text's situation model, for which no shipping output is a foundation at all.
Trigger condition. Met in substance: a named caller with a documented baseline exists and has been blocked for months. What is missing is confirmation that the caller still wants matra rather than the thin-wrapper alternative it also considered. Confirm that, then proceed.
Self-similarity and redundancy metrics
What it is. Structural detection of a document repeating itself: clusters of sentences restating the same content, with quantitative measures over them. The concrete caller pattern is auditing LLM-generated text, whose characteristic failure is high-lexical-overlap restatement.
The shape, all deterministic and traceable back to the source bytes:
| Output | Kind |
|---|---|
| Similarity clusters: sentence groups whose pairwise content-lemma overlap (Jaccard or TF-IDF cosine) exceeds a caller-supplied threshold, returned as span sets with the shared lemmas as evidence | qualitative |
| Redundancy ratio: share of sentences in clusters of size above one | quantitative |
| rep-n and distinct-n: n-gram repetition rates, the established NLG-literature measures | quantitative |
| Skeleton repetition: sentences sharing root-verb lemma plus subject and object lemmas | qualitative |
| Opener formulae: repeated sentence-initial lemma sequences | quantitative |
| Document-scope compression ratio, closing the per-paragraph measure's cross-paragraph blind spot | quantitative |
| POS-sequence compression ratio: the same compression measure over the tag sequence, catching structural repetition independent of vocabulary (Shaib et al. 2024) | quantitative |
| Syntactic template rate: recurring POS n-grams, the published operationalization of templated LLM output (Shaib et al., EMNLP 2024) | quantitative |
| Span recurrence: long token spans repeating verbatim, the loop-degeneration signature | qualitative |
matra reports the clusters and the numbers; whether they constitute fluff is the caller's reading. The word never appears in the output. That restraint is now empirical as well as principled: the 2026 slop-measurement literature found repetition and templatedness are weak predictors of human quality judgments on their own, so these are structural facts for a caller's argument, never a judgment. Synonym-level paraphrase (different vocabulary, same meaning) is out of reach of lexical overlap by design; catching it needs semantic similarity, which sits above the verifiable tier. That half shipped 2026-09-05 via EP-0009: the Embedder port, the model2vec static adapter, and the clustering that clusters runs (see the semantic clusters guide). The deterministic family above remains the open half of this entry. Published similarity thresholds for that task span 0.67 to 0.9 with no consensus, which is why every threshold here is caller-supplied.
Where it lands. Mostly paid for already: extraction/textrank.rs builds a pairwise sentence-similarity matrix and projects it down to centrality ranks; re-projecting the same matrix as clusters is the core of this capability. Thresholds are caller-supplied parameters, not constants matra pretends to know.
Trigger condition. FIRED, 2026-08-21. A concrete caller pattern was named that field access cannot serve: quantifying restatement across a document for LLM-output auditing. Design against EP-0007's primitives and the rule-vocabulary shape; an enhancement proposal settles whether this is a metric family, an extractor, or the first rule pack.
Information density
What it is. Deterministic measures of how much a text says per word, the complement of the redundancy family: redundancy catches the same thing said twice, density catches little said at length. The anchor is propositional idea density, a validated psycholinguistics measure (propositions per word) with two published computational forms. CPIDR counts propositions from Penn Treebank tags, which UDPipe already emits in Token::xpos; DEPID counts them from dependency relations, which is matra's core output, and agrees with human raters slightly better than CPIDR does. DEPID-R, the variant counting distinct relation-lemma triples rather than tokens of them, catches fluent-but-repetitive text and costs nothing extra once lemmas are in hand. Around the anchor sit smaller settled measures that are direct tree walks: mean dependency distance, modifiers per noun phrase, words before the main verb, content-to-function ratio.
Where it lands. Metric functions over Document, same family as the shipping metrics suite. The DEPID relation inventory is Stanford-style, so an explicit mapping to UD relations is part of the design; the CPIDR path additionally holds only for English models, since Token::xpos carries whatever tagset the provider's model emits and PTB tags are the English case. Three counting rules the literature leaves genuinely unsettled (direct objects, coordinating conjunctions, the denominator) get settled by an enhancement proposal rather than inherited silently, and the proposal also states the non-English story. The reference implementations are GPL, so the derivation is clean-room from the published rules, and exact numeric parity with them is a non-goal. No implementation of idea density exists for CoNLL-U input in any language; this would be the first.
What stays out, and why. Hedging and booster density need a lexicon, and no canonical machine-readable hedge list exists; per the open-class precedent (clausal_complements_in), any such lexicon is caller-supplied data, never shipped as if authoritative. Surprisal needs frequency norms or a language model, and sycophancy detection has no deterministic path at all; both gaps are recorded rather than simulated.
Trigger condition. FIRED, 2026-09-04. The same LLM-output-auditing caller pattern that fired the redundancy entry names low density as the second failure mode, and the 2026-09-04 landscape survey (internal, not in this repository) established implementability from matra's existing parse output. Design together with the redundancy family; the same proposal should place both.
Configuration-driven invocation
What it is. A configuration file selecting which metrics run, which extractors run, and how output is shaped, so both the library and the CLI are driven by declaration rather than by argument lists.
What is blocking it. The metric suite is private to the crate since 0.3.0, so choosing metrics from a file would first need a public way to name them. What is absent beyond that is the surface: a format, a resolution order, and a decision about whether configuration is a library concern or belongs only to the application tier.
That last question is the real one. matra's discipline is that the library returns typed data and the binary decides presentation. A config format that reaches into the library risks putting policy where the composition root belongs.
Shipped, 2026-09-05, via EP-0010: Config resolves locations and defaults per key from argument, environment, config file, then the defaults compiled into the crate; Engine::with_defaults() (since 0.3.0 Engine::new(), which reads no configuration file, and matra.Engine() in Python, which replaced Matra.english()) and the from_config constructors are the no-setup path on every surface; matra config show and matra config init are the command-line half. The question this entry asks is answered narrowly: the file carries locations and defaults, never which metrics run or how output is shaped. Selecting behavior from a file remains unbuilt, and reopening it needs its own enhancement proposal rather than a wider schema here.
Trigger condition. FIRED, 2026-09-05. The condition was a caller running matra repeatedly with the same non-default selection, with agent-driven use the likely first instance. It fired as owner direction: matra works with no setup on every surface, follows developer-tool conventions for config and paths, and keeps Rust as the core with Python and TypeScript as thin reach layers. RFC-0011 settles the library-or-application question the paragraph above asks (a resolver for locations and defaults in the library, behavior selection in the application), and blueprints/legacy/eps/0010-foundations.md carries the milestones.
Agent surface
What it is. A --skill flag on the CLI that prints a self-contained description of matra's semantics for an agent: what it is for, when to reach for it, the incantations with their JSON shapes, how to read the numbers, and the limits. Progressive disclosure follows the shape of a skill on disk: --skill prints the short top level, --skill -r <name> prints one deeper reference. --help stays the framework-generated reference for humans. The same file is what a plugin marketplace distributes.
Why a flag and not only a docs page. Human attention is the scarce input. The people who would read the whole docsite are few; the agents that will run matra on their behalf are many, and an agent that can print the semantics it needs, from the tool it is about to run, needs no link and no prior knowledge. The docs stay for human comprehension, with citations and readable benchmarks; the skill is the other door, derived from the same code and tested against it.
Precedent. A survey of seventeen tools (docs/surveys/2026-09-05-conventions.md) found one that prints its own agent-facing instructions with a second tier of detail, Vercel Labs' agent-browser, whose stated reason is that instructions served from the installed binary always match its version. The same survey records the surrounding conventions this entry adopts alongside the flag: an llms.txt on the docsite, an AGENTS.md in the repository, and a CITATION.cff so the research behind each measure is citable from the repository page.
Shipped, 2026-09-06, via EP-0011: matra --skill prints the top level, matra --skill -r lists the references and matra --skill -r <name> prints one. The text is embedded with include_str!, so an installed program's instructions are the ones that match it, and a test executes the commands the skill names, so an incantation that stops working fails a gate rather than a reader. --help is unchanged and stays the reference for humans. The surrounding conventions the precedent paragraph names landed alongside it: llms.txt on the docsite with a currency gate, AGENTS.md in the repository, a plugin manifest, and CITATION.cff.
Trigger condition. FIRED, 2026-09-05, by owner direction, sequenced after EP-0010: the skill documents the CLI contract, so the CLI had to be one implementation with a pinned JSON shape first. EP-0010 landed on 2026-09-06; RFC-0012 records the decisions and blueprints/legacy/eps/0011-agent-surface.md carries the milestones.
Terminal UI for the Rust CLI
What it is. An interactive terminal interface over the same cli module: browse a parsed document, its sections, sentences, and dependency trees, and the metrics beside them, without leaving the terminal.
Trigger condition. FIRED, 2026-09-06. EP-0010 and the agent surface have both met their acceptance gates. The TUI is a renderer over a contract that has to be stable first; building it before that means re-doing it.