Skip to content
matra
DocsBlueprintsLab
Search
apidiscussionsissues
  • Home

Tutorials

  • Your first analysis

How-to guides

  • Install matra
  • Migrate from 0.2 to 0.3
  • How-to guides
    • Parse a sentence into its dependency tree
    • Find negations and modals
    • Compare readability across paragraphs
    • Summarize a long document
    • Extract a document's keyphrases
    • Read a Markdown document's structure
    • Cluster sentences by meaning
  • Hand matra to an agent
  • Install the model behind a TLS-intercepting proxy

Explanation

  • Introduction
  • Concepts
  • Situation model
  • Programming model
  • One core, three install routes
  • What the clustering threshold does
  • Pragmatics
  • How matra runs
  • How matra is maintained

Reference

  • What matra gives you
  • Domain model
  • Platforms and models
  • Semantic clusters
  • Rust
  • Python
  • CLI
  • Methodology
  • Errors
  • Boundary rules

What is planned

  • Roadmap
apidiscussionsissues github

Every page of this documentation, in reading order.

matra

matra parses English text into sections, paragraphs, sentences and words, and returns measures, summaries and keyphrases over them, from Rust, Python or the command line.

Quick start#

Install matra, save one sentence, and run matra on it.

Rust

1Install

$ cargo new quickstart && cd quickstart
$ cargo add matra@0.3 serde_json

2Save the text

$ echo 'The committee approved the budget on Tuesday.' > note.txt

3Run: save this as src/main.rs, then run it

// cargo add matra@0.3 serde_json
use matra::{Engine, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads the English model, downloading it on first use.
    let engine = Engine::new()?;
    let doc = engine.analyze(Input::file("note.txt"))?;
    println!("{}", serde_json::to_string_pretty(&doc)?);
    Ok(())
}
$ cargo run

Python

1Install

$ pip install 'matra>=0.3'

2Save the text

$ echo 'The committee approved the budget on Tuesday.' > note.txt

3Run: save this as quickstart.py, then run it

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()  # downloads the English model on first use
doc = engine.analyze(file="note.txt")  # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False))
$ python quickstart.py

CLI

1Install

$ pip install 'matra>=0.3'   # or: cargo install matra --version '^0.3' --features cli

2Save the text

$ echo 'The committee approved the budget on Tuesday.' > note.txt

3Run

$ matra analyze --json note.txt

It prints JSON; here is the first word:

[
  {
    "id": 1,
    "span": {
      "byte_start": 0,
      "byte_end": 3
    },
    "text": "The",
    "lemma": "the",
    "pos": "DET",
    "xpos": "DT",
    "features": {
      "Definite": "Def",
      "PronType": "Art"
    },
    "head": 2,
    "dep": "det",
    "deps": "_",
    "misc": "",
    "is_punct": false
  }
  // … 7 more
]

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

Every word, drawn. Each arc runs from a word to the word that depends on it:

Dependency parse

1The committee approved the budget on Tuesday.

Dependency arcs for: The committee approved the budget on Tuesday.rootdetnsubjdetcaseobjnmodpunctTheDETcommitteeNOUNapprovedVERBtheDETbudgetNOUNonADPTuesdayPROPN.PUNCT
show the data
#WordLemmaPOSHeadRelation
1ThetheDET2committeedet
2committeecommitteeNOUN3approvednsubj
3approvedapproveVERB0rootroot
4thetheDET5budgetdet
5budgetbudgetNOUN3approvedobj
6ononADP7Tuesdaycase
7TuesdayTuesdayPROPN5budgetnmod
8..PUNCT3approvedpunct
  • Core argument
  • Modifier
  • Function word
  • Other

Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.

Input: The committee approved the budget on Tuesday., matra contributors. MIT, as part of the matra repository. Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: quickstart/parse.json.

Where to go next#

  • Tutorial: run matra on two short paragraphs and find each part of what it returns. About ten minutes.
  • How-to guides: one task each on real texts: summarize a document, extract keyphrases, find negations and modals, compare readability, cluster sentences by meaning.
  • Explanation: why the output has the shape it has, with figures you can step through.
  • Reference: every type and field, the Rust, Python and command-line surfaces, the methods behind each measure, and every error.

Your first analysis

grade12.7density0.57compressionnone

You will run matra on two short paragraphs of meeting minutes, then find four things in what it returns: the document's structure, the grammar it read off each sentence, the measures of each paragraph, and the measures of the whole document. You need matra installed; Installation covers every route.

Save the text#

Minutes of a committee meeting, two paragraphs, matra contributors. MIT, as part of the matra repository. 36 words.

The calls read it as minutes.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/first-analysis/minutes.txt

or download minutes.txt.

The committee did not approve the budget on Tuesday. The chair said that a second vote may follow next week.

Three amendments were submitted by the working group. Each one narrows the scope of the plan.

Run matra#

grade3.9density0.53compressionnone

Pick the language you will use. Each tab installs matra and runs the same analysis.

Rust

1Install

$ cargo new first-analysis && cd first-analysis
$ cargo add matra@0.3 serde_json

2Run: save this as src/main.rs, then run it

// cargo add matra@0.3 serde_json
use matra::{Engine, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads the English model, downloading it on first use.
    let engine = Engine::new()?;
    let doc = engine.analyze(Input::file("minutes.txt"))?;
    println!("{}", serde_json::to_string_pretty(&doc)?);
    Ok(())
}
$ cargo run

Python

1Install

$ pip install 'matra>=0.3'

2Run: save this as first_analysis.py, then run it

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()  # downloads the English model on first use
doc = engine.analyze(file="minutes.txt")  # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False))
$ python first_analysis.py

CLI

1Install

$ pip install 'matra>=0.3'   # or: cargo install matra --version '^0.3' --features cli

2Run

$ matra analyze --json minutes.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "analyze",
  "input": "minutes.txt",
  "result": /* the output below */,
  "error": null
}

Read the output#

grade2.6density0.55compressionnone

Every tab prints the same value. The words of each sentence are shown here as a count; the next section finds them.

Trimmed: tokens shown as a count, long strings shortened.

{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "plain_text",
    "source": {
      "path": "minutes.txt",
      "bytes": 204,
      "sha256": "b4f55f24335ecfae3b641caec0db70fb9566f76143bba54401a99869e29185d2"
    }
  },
  "sections": [
    {
      "heading": null,
      "level": 0,
      "span": null,
      "paragraphs": [
        {
          "span": {
            "byte_start": 0,
            "byte_end": 108
          },
          "text": "The committee did not approve the budget on Tuesday. The chair said tha…",
          "kind": "paragraph",
          "sentences": [
            {
              "span": {
                "byte_start": 0,
                "byte_end": 52
              },
              "text": "The committee did not approve the budget on Tuesday.",
              "tokens": [ /* 10 items */ ],
              "word_count": 9,
              "tree_depth": 2,
              "passive": false,
              "negations": [
                {
                  "cue_id": 4,
                  "cue_lemma": "not",
                  "head_id": 5
                }
              ],
              "modals": [],
              "clausal_complements": [],
              "root_adverbials": [
                {
                  "adv_id": 4,
                  "adv_lemma": "not"
                }
              ],
              "hearst_pairs": [],
              "root_mood": {
                "root_id": 5,
                "mood": "Ind"
              }
            },
            {
              "span": {
                "byte_start": 53,
                "byte_end": 108
              },
              "text": "The chair said that a second vote may follow next week.",
              "tokens": [ /* 12 items */ ],
              "word_count": 11,
              "tree_depth": 3,
              "passive": false,
              "negations": [],
              "modals": [
                {
                  "aux_id": 8,
                  "aux_lemma": "may",
                  "head_id": 9
                }
              ],
              "clausal_complements": [
                {
                  "verb_id": 3,
                  "verb_lemma": "say",
                  "ccomp_id": 9,
                  "subject_id": 2,
                  "subject_lemma": "chair"
                }
              ],
              "root_adverbials": [],
              "hearst_pairs": [],
              "root_mood": {
                "root_id": 3,
                "mood": "Ind"
              }
            }
          ],
          "metrics": {
            "readability_grade": 3.650000000000002,
            "lexical_density": 0.55,
            "compression_ratio": null
          }
        },
        {
          "span": {
            "byte_start": 110,
            "byte_end": 203
          },
          "text": "Three amendments were submitted by the working group. Each one narrows …",
          "kind": "paragraph",
          "sentences": [
            {
              "span": {
                "byte_start": 110,
                "byte_end": 163
              },
              "text": "Three amendments were submitted by the working group.",
              "tokens": [ /* 9 items */ ],
              "word_count": 8,
              "tree_depth": 2,
              "passive": true,
              "negations": [],
              "modals": [],
              "clausal_complements": [],
              "root_adverbials": [],
              "hearst_pairs": [],
              "root_mood": {
                "root_id": 4,
                "mood": "Ind"
              }
            },
            {
              "span": {
                "byte_start": 164,
                "byte_end": 203
              },
              "text": "Each one narrows the scope of the plan.",
              "tokens": [ /* 9 items */ ],
              "word_count": 8,
              "tree_depth": 3,
              "passive": false,
              "negations": [],
              "modals": [],
              "clausal_complements": [],
              "root_adverbials": [],
              "hearst_pairs": [],
              "root_mood": {
                "root_id": 3,
                "mood": "Ind"
              }
            }
          ],
          "metrics": {
            "readability_grade": 3.7550000000000026,
            "lexical_density": 0.5625,
            "compression_ratio": null
          }
        }
      ]
    }
  ],
  "metrics": {
    "total_words": 36,
    "total_sentences": 4,
    "paragraph_count": 2,
    "mean_sentence_length": 9,
    "sentence_length_std": 1.4142135623730951,
    "vocabulary_ttr": 0.8611111111111112,
    "nominalization_ratio": 0,
    "passive_ratio": 0.25
  },
  "skipped": []
}

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

Find the structure#

grade5.6density0.53compressionnone

The output is a tree. The document holds sections; this one has a single section, because the text has no headings. The section holds paragraphs, two of them, one for each block of text separated by a blank line. Each paragraph holds its sentences, and each sentence its tokens.

grade5.2density0.51compressionnone

The tokens are shown above only as a count. Each one carries its text, its lemma, its part of speech (pos), the id of the word it depends on (head) and the relation between them (dep). The quick start shows one sentence's tokens in full.

Find the grammar#

grade6.0density0.47compressionnone

matra reads a few facts off each sentence's parse and stores them on the sentence.

  1. In the first sentence, negations holds one entry. Its cue_lemma is not, and its head_id is 5: not negates the fifth word, approve.
  2. In the second sentence, modals holds may, over word 9, follow. clausal_complements holds say: the chair (subject_lemma) said something, and what was said is the clause headed by word 9.
  3. The third sentence is passive: the amendments were submitted, and its passive is true. Every sentence also carries its word_count, its tree_depth, and its root_mood: the grammatical mood of its main clause, Ind for a plain statement.

Find the paragraph measures#

grade9.0density0.48compression0.57

Each paragraph carries three measures in its metrics. readability_grade is a grade level, and lexical_density is the share of content words. compression_ratio is null for both paragraphs: matra computes it only for a paragraph of more than 50 words, and these are shorter. A null means the measure declined to run, never that it came out as zero.

Find the document measures#

grade7.8density0.51compression0.56

The document's own metrics describe the whole of it: the counts (total_words, total_sentences, paragraph_count), the mean and spread of sentence length, and three ratios. passive_ratio is 0.25: one sentence of the four is passive. vocabulary_ttr is the share of distinct lemmas among all of them. nominalization_ratio is the share of words that are nouns ending in a suffix such as -tion or -ment. It is 0 here: the one candidate, amendments, ends in an s, and the test reads the word as written (Methodology has the rule).

Where to go next#

  • How-to guides: the same calls on longer, real texts, one task each.
  • Concepts: why the output is a tree, and what each part of it records.
  • Domain model: every field named above, with its type.

Install matra

grade8.0density0.54compression0.60

Install matra as a Rust library, a command-line binary or a Python package, then check that it works. Each route is one command; take the one for the language you call matra from, or more than one. Platforms and models lists what each route needs and installs, and One core, three install routes explains why the commands look as they do.

grade11.2density0.53compressionnone

To check a release file before you install it, follow Verifying a release in the security policy.


Requirements#

grade7.0density0.53compression0.61

For the Rust library and the CLI, and for the Python package on a platform with no wheel, install Rust 1.88 or later (check with rustc --version) and a C++ compiler. A C compiler on its own is not enough: without C++ the build stops with error occurred in cc-rs: failed to find tool "c++".

PlatformPackage
Debian, Ubuntuapt install build-essential
Fedora, RHEL, Rocky, Almadnf install gcc-c++
Alpineapk add g++
Archpacman -S base-devel
macOSxcode-select --install
grade10.0density0.60compression0.57

For the Python package, use Python 3.12 or later (check with python3 --version, or python --version inside an activated virtual environment; stock macOS has only python3). Python wheels lists the platforms with a prebuilt wheel; on any other, and on a free-threaded interpreter, pip builds from source and needs the Rust toolchain and the C++ compiler above.


The Rust library#

cargo add matra@0.3
grade12.0density0.55compressionnone

The default features include udpipe, so matra::Engine::new() builds the pinned UDPipe parser without a feature flag. The UDPipe adapter itself is private to the crate: you reach it through Engine, and a parser of your own reaches it through Engine::with_parser.


The CLI binary#

cargo install matra --version '^0.3' --features cli
grade9.8density0.52compressionnone

--features cli is required: the binary is gated behind it. This compiles from source and places the binary in your cargo bin directory, which can take a minute or more. Confirm it landed:

matra --version
gradenonedensity1.00compressionnone

Expected output:

matra 0.3.0
features: udpipe cli
grade11.1density0.53compressionnone

The second line names the features this build was compiled with, so the Python package's banner below is longer.


The Python package#

pip install 'matra>=0.3'    # or: uv add 'matra>=0.3'
grade7.2density0.51compression0.59

This installs the library and the matra command together; the command is the same program as the CLI binary above, so uvx 'matra>=0.3' analyze essay.md and the installed binary do the same thing. Keep the version floor in the line: why the install lines name a version says what an unpinned one can give you.

grade5.9density0.50compressionnone

On a platform with a wheel this installs in seconds. Its version banner reads:

matra 0.3.0
features: udpipe model2vec python cli
grade9.8density0.53compressionnone

The verify step below doubles as the check: if import matra fails there, the package did not install correctly.


The English model#

grade9.0density0.56compressionnone

There is nothing to install. The library, the CLI and the Python package each download the English UDPipe model (about 16 MB) on first use, verify it against a pinned hash, and cache it in the model directory, by default ~/.local/share/matra/models.

grade11.1density0.63compressionnone

To keep it somewhere else, set MATRA_MODEL_DIR, pass --model-dir to the CLI, or pass the directory through Config::with_model_dir in Rust. Python reads MATRA_MODEL_DIR. The model directory gives the full resolution order, and the model download what happens when a download fails its check.


Model licenses#

grade8.3density0.55compression0.58

matra's code is MIT, and the two models it downloads are separate works with licenses of their own. The English parsing model comes from LINDAT under CC BY-NC-SA 4.0, which does not permit commercial use; read it before you use matra commercially. The embedding model, potion-base-8M, comes from Hugging Face under MIT per its model card, and is used for semantic clusters only. Model licenses has both in a table, with links to the license texts.

grade6.3density0.65compressionnone

The command line, the config file, Rust and Python always use this model. No surface loads a different UDPipe model file: the model is a pin.


Verify the install#

grade15.0density0.50compressionnone

Run this once. No arguments and no environment: it resolves the model directory, downloads and caches the model on this first run, then parses a sentence through it and prints two results:

import matra

engine = matra.Engine()

doc = engine.analyze("The committee approved the proposal without debate.")
print("sections:", len(doc["sections"]))
print("vocabulary_ttr:", doc["metrics"]["vocabulary_ttr"])
grade6.9density0.52compressionnone

To name the directory, set MATRA_MODEL_DIR before the call. Use a real path: the value is used unchanged, so ~ is not expanded.

gradenonedensity1.00compressionnone

Expected output:

sections: 1
vocabulary_ttr: 0.8571428571428571
grade8.6density0.50compression0.54

That first run fetches the model from a university server in Prague, and before it does, matra.Engine() writes two lines to standard error: what is downloading, its size and where to, then the model's licence. They are the lines the command line prints for the same download. Cold starts measured on a fast connection ranged from 3 to 35 seconds; on a slow network it takes longer, and the call is waiting on the network rather than working. Every run after that loads the cached file and touches no network, in about a second.

grade9.4density0.55compression0.53

If matra.Engine() raises OSError (a matra.IoError), either the download never arrived or the model directory could not be written, and the message says which by naming the URL or the path, as details["url"] or details["path"] do. Check your network connection first; then run matra config show to see which directory matra resolved and check the permissions on it. If it raises RuntimeError (a matra.ModelInvalidError), bytes did arrive and then failed the pinned hash check; run the snippet again.


What you have#

  • matra installed as a Rust crate, a CLI binary, a Python package, or some combination, all from the same 0.3.x core.
  • The English UDPipe model cached in the resolved model directory, verified against a pinned hash.
  • A confirmed working call from matra.Engine() through analyze() to a result.
grade9.0density0.50compressionnone

Next: Your first analysis walks through what that result holds, or go to Rust, Python or CLI for the surface you are calling from.

Migrate from 0.2 to 0.3

grade5.3density0.50compression0.53

0.3.0 changes Rust, Python, the command line, the JSON and the config file at once. Nothing keeps its old name. There are no deprecated aliases, no hidden flags, no fields written under both names and no setting that brings back the 0.2 output. A renamed or removed call fails at once in 0.3 rather than running with a different meaning: a compile error in Rust, an AttributeError or TypeError in Python, an unknown key in the config file, and a new format_version and schema major in the JSON. A few results do change for the same call, and the last section lists them.

grade-0.7density0.48compressionnone

This page lists every change a 0.2 caller meets, with the 0.2 form and the 0.3 form side by side. Work through the section for each surface you use.

Update the version#

grade5.7density0.63compressionnone

Move the version requirement first. The old requirement keeps you on 0.2, because cargo reads a change in the first non-zero component as breaking.

cargo add matra@0.3                                   # was matra@0.2
cargo install matra --version '^0.3' --features cli   # was '^0.2'
pip install 'matra>=0.3'                              # was 'matra>=0.2'
uvx 'matra>=0.3' --skill                              # the skill for 0.3

Every rename at a glance#

0.20.3Where
Engine::new(nlp, decomposers), Engine::with_defaults()Engine::new(), Engine::from_config(&Config::resolve()?)Rust
Ingest::text, Ingest::pathInput::text, Input::file, Inputs::dirRust
Engine::analyze(Ingest), Engine::analyze_oneEngine::analyze(Input), Engine::analyze_all(inputs)Rust
Engine::annotate, Engine::composenone: analyze is the callRust
Ingested, RawDocument, CorpusEntry, Corpus, CorpusResultnone: Document, its path in provenance.source.pathRust
extraction::tfidf_summarize, textrank_summarizesummarize, summarize_with(.., SummaryMethod::TextRank)Rust
extraction::rake_keyphrases, yake_keyphraseskeyphrases, keyphrases_with(.., KeyphraseMethod::Yake)Rust
embed_and_cluster, extraction::semantic_clustersclustersRust
ScoredSentence, SemanticClusters, SemanticCluster, SemanticEdgeSummarySentence in a Summary, Clusters, Cluster, SimilarityEdgeRust, Python types
domain::MAX_INPUT_BYTESdomain::limits::MAX_INPUT_BYTESRust
Error::InvalidInput(message)Error::InvalidInput { reason, message }Rust
Error::ParseFailed(message)Error::ParseFailed { reason, cause }Rust
Embedder::identity() -> &strEmbedder::identity() -> IdentityRust
Config::summarize_n, summarize_algorithm, keyphrases_algorithmConfig::summary_n, summary_method, keyphrases_methodRust
Config::udpipe_model, Config::semantic_thresholdnone; Config::clusters_threshold is newRust
Engine::new(Box::new(parser), decomposers)Engine::with_parser(parser), then .with_reader(format, reader) per readerRust
NlpProvider::parse -> Vec<Sentence>NlpProvider::parse -> Vec<Vec<Token>>, one list of tokens per sentenceRust
Decomposer::decompose(&str) -> Vec<Section>Decomposer::decompose(&[u8]) -> Result<Read>, built from ReadBlock::heading and ReadBlock::paragraphRust
Source::read -> Vec<RawDocument>, Source::acceptsSource::read -> Vec<u8>; accepts is goneRust
Udpipe, Decomposers, MarkdownDecomposer, PlainTextDecomposer, FileSource, DirectorySource, standard_decomposers, metrics, hearst, extractionprivate to the crateRust
Matra.english(), Matra.from_path(path)matra.Engine()Python
Matra.analyze(text), analyze_markdown(text)engine.analyze(text), engine.analyze(text, format="markdown")Python
Matra.analyze_path(path)engine.analyze(file=path), engine.analyze_all(dir=path)Python
Matra.tfidf_summarize(text, n), textrank_summarizematra.summarize(doc, n, method=...)Python
Matra.rake_keyphrases(text, max_phrases), yake_keyphrasesmatra.keyphrases(doc, n, method=...)Python
Matra.semantic_clusters(text, threshold, model)matra.clusters(doc, model, threshold)Python
matra.semantic_clusters(vectors, threshold, model_hash)none: a cosine over the vectors you holdPython
an embedder's identity() -> str, Model2Vec.identity()identity() returning an Identity dict, {name, model, sha256, license}Python
Model2Vec.potion_base_8m() reading the config fileModel2Vec.from_config()Python
Document, CorpusEntry, CorpusItem (types)matra.Document (a class), DocumentDict, and Document or DocumentError itemsPython
paragraph["readability_grade"] and its siblingsparagraph["metrics"]["readability_grade"]JSON, Python
doc["vocabulary_ttr"] and its siblingsdoc["metrics"]["vocabulary_ttr"]JSON, Python
Paragraph.in_blockquoteParagraph.kind ("quote")all
Sentence.reportings, Reporting, reportings_inSentence.clausal_complements, ClausalComplement, clausal_complements_inall
Sentence.bare_assertionSentence.root_moodall
Token.feats (a string)Token.features (a map)all
Sentence.tree_depth() returning usize::MAX on a cycleSentence.tree_depth, None on a cycleall
SemanticClusters.model_hashClusters.provenance.parameters.embedder.sha256all
InputTooLarge what: file_source, config_file, udpipe_download, embedding_downloadfile_read, config_file_read, parser_model_download, embedding_model_downloadall
invalid_input details.reason, the message textdetails.reason, the name of the check that failedall
model_invalid, parse_failed and io details.reason, the error's textdetails.causeall
Format serialized as "Markdown", "PlainText""markdown", "plain_text", "blocks"Rust serde, JSON
[summarize], algorithm, [semantic] threshold, [models] udpipe[summary], method, [clusters] threshold, noneconfig file
--json envelope format_version: 1format_version: 2, with an error keycommand line

Rust#

Build the engine#

grade8.0density0.57compressionnone

Engine::new takes no arguments and builds the pinned parser itself. It reads no configuration file. Engine::with_defaults is gone, because it read one under a name that did not say so.

// 0.2
let engine = Engine::with_defaults()?;
let engine = Engine::new(Box::new(Udpipe::english(dir)?), standard_decomposers());
// 0.3
let engine = Engine::new()?;
let engine = Engine::from_config(&Config::resolve()?)?;                     // reads the config file
let engine = Engine::from_config(&Config::resolve()?.with_model_dir(dir))?; // a model directory you name
grade6.8density0.48compressionnone

There is no provider to construct and no reader table to pass for the standard pipeline. No surface loads a UDPipe model file of your choosing: the model is a pin.

grade5.9density0.44compressionnone

A parser, reader or source of your own still plugs in, through the engine rather than a table:

// 0.2
let engine = Engine::new(Box::new(MyParser), standard_decomposers().with(Format::Markdown, Box::new(MyReader)));
// 0.3
let engine = Engine::with_parser(MyParser).with_reader(Format::Markdown, MyReader);
let engine = Engine::new()?.with_reader(Format::Pdf, MyPdfReader);   // UDPipe, plus a reader of yours
let engine = Engine::new()?.with_source(MyStore);                    // paths read from somewhere else
grade10.2density0.61compression0.51

The traits changed shape. NlpProvider::parse returns one Vec<Token> per sentence instead of Sentence values: the engine builds each sentence from the tokens, its text and every derived field included, so drop your Sentence::new call and return the tokens. Spans are required now, relative to the paragraph text parse was handed. Decomposer::decompose takes the format's bytes and returns a Result<Read> of ReadBlock values with their spans, where it returned sections; a text reader decodes the bytes itself. Source::read returns the document's bytes, and accepts is gone, since the engine never asked it. Whatever a parser or a reader returns is checked against its trait's contract, and a break is Error::ParseFailed naming the check (Errors); 0.2 passed it through unchecked. Bring your own parser or reader has the contracts.

Analyze one document#

grade8.4density0.52compressionnone

Input is one document. Engine::analyze returns its Document or the error. There is no stream to unwrap and no CorpusEntry around the document.

// 0.2
let analysis = engine
    .analyze(Ingest::path("essay.md")?)
    .next()
    .unwrap()
    .map_err(|e| e.error)?
    .analysis;
let doc = engine
    .analyze_one(RawDocument::new(text, None, Format::Markdown))
    .map_err(|e| e.error)?
    .analysis;
// 0.3
let doc = engine.analyze(Input::file("essay.md"))?;
let doc = engine.analyze(Input::text(text, Format::Markdown))?;
grade11.4density0.60compressionnone

Input has five constructors: text, bytes, file, file_as (a file read as the format you name) and blocks (headings and paragraphs you supply for a format matra has no reader for). Input::file on a directory fails with InvalidInput, naming Inputs::dir.

Analyze a directory#

grade7.9density0.59compressionnone

Ingest::path on a directory is Inputs::dir, and analyze_all streams one result per file. A failed file is an Err item carrying its path. The path of a document that analyzed is doc.provenance.source.path.

// 0.2
for item in engine.analyze(Ingest::path("docs/")?) {
    match item {
        Ok(entry) => println!("{:?}: {}", entry.path, entry.analysis.total_words()),
        Err(e) => eprintln!("{:?}: {}", e.path, e.error),
    }
}
// 0.3
for item in engine.analyze_all(Inputs::dir("docs/")?) {
    match item {
        Ok(doc) => println!("{:?}: {}", doc.provenance.source.path, doc.metrics.total_words),
        Err(e) => eprintln!("{e}"),
    }
}
grade10.2density0.56compressionnone

Corpus and CorpusResult are gone, with Corpus::total_words, passive_ratio and mean_readability. Collect the stream yourself. A count such as total_words sums across documents. A ratio such as vocabulary_ttr does not, because it falls as a document grows.

Read the measures#

grade9.2density0.50compressionnone

The aggregate methods are fields now, so they reach Python and the JSON too. The paragraph measures moved into Paragraph.metrics, and the document measures into Document.metrics.

// 0.2
doc.total_words();
doc.passive_ratio();            // 0.0 on a document with no sentences
doc.vocabulary_ttr;
paragraph.readability_grade;
sentence.word_count();
sentence.is_passive();
sentence.tree_depth();          // usize::MAX on a cycle
// 0.3
doc.metrics.total_words;
doc.metrics.passive_ratio;      // None on a document with no sentences
doc.metrics.vocabulary_ttr;
paragraph.metrics.readability_grade;
sentence.word_count;
sentence.passive;
sentence.tree_depth;            // None on a cycle
grade11.9density0.57compressionnone

DocumentMetrics also carries total_sentences, paragraph_count, mean_sentence_length and sentence_length_std. In 0.2 these were methods, so they never reached Python or the JSON.

Read the record facts#

grade6.7density0.45compressionnone

The facts on a sentence are named for what the detector finds. ClausalComplement is a verb governing a clausal complement. Which of those verbs report speech is your lexicon, as before.

// 0.2
sentence.reportings;
sentence.reportings_in(&["say", "claim"]);
sentence.bare_assertion;
token.feats;                    // "Mood=Ind|Tense=Past"
// 0.3
sentence.clausal_complements;
sentence.clausal_complements_in(&["say", "claim"]);
sentence.root_mood;             // Option<RootMood { root_id, mood }>
token.features;                 // BTreeMap: {"Mood": "Ind", "Tense": "Past"}
grade9.4density0.57compressionnone

The 0.2 bare_assertion flag is root_mood.mood == "Ind" with no modal whose head_id is root_mood.root_id. Token::feat("Mood") reads one feature, as before. Paragraph.in_blockquote is Paragraph.kind: in_blockquote: true is ParagraphKind::Quote, and a list item is now ParagraphKind::ListItem.

Summaries, keyphrases and clusters#

grade7.2density0.51compression0.59

The outcomes are free functions at the crate root. They take the analysed &Document, not a slice of its sentences, and need no engine, so they run in a build without UDPipe. The method is optional and never in the name. Each returns a record whose provenance names the method that ran and its parameters.

// 0.2
let sentences: Vec<Sentence> = doc.sentences().cloned().collect();
let top = extraction::tfidf_summarize(&sentences, 3)?;          // Vec<ScoredSentence>
let ranked = extraction::textrank_summarize(&sentences, 3)?;
let phrases = extraction::yake_keyphrases(&sentences, 10)?;     // Vec<Keyphrase>
let groups = embed_and_cluster(&doc, &model, 0.85)?;           // SemanticClusters
// 0.3
let top = matra::summarize(&doc, 3)?;                                    // Summary
let ranked = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Yake)?; // Keyphrases
let groups = matra::clusters(&doc, &model, 0.85)?;                      // Clusters
gradenonedensity0.60compressionnone

What moved inside the results:

  • ScoredSentence { text, score, position } is SummarySentence { index, span, text, score }, in Summary.sentences. position is index.
  • Keyphrase gains occurrences, every span the phrase was found in, and sits in Keyphrases.phrases.
  • SemanticClusters { model_hash, threshold, clusters } is Clusters { provenance, clusters, edges }. The embedder's identity is provenance.parameters.embedder, an Identity like the parser's (the old hash is its sha256), and the threshold provenance.parameters.threshold. The edges moved out of each cluster into one list, ordered by a and then b. Each Cluster holds only members.
  • The threshold and every score are f64, not f32.
  • Model2Vec::potion_base_8m(dir) takes the models directory, the one MATRA_MODEL_DIR names, and keeps the three files in dir/potion-base-8M/. In 0.2 dir was the model's own directory. Pass the parent: potion_base_8m(cfg.model_dir()) where 0.2 took potion_base_8m(cfg.model_dir().join("potion-base-8M")). Python's Model2Vec.potion_base_8m(dir) changed the same way.

Errors#

gradenonedensity0.78compressionnone

Error keeps its variants and kind(). Three shapes changed.

// 0.2
Error::UnsupportedFormat(format)
Error::InvalidInput(message)
Error::ParseFailed(message)
domain::MAX_INPUT_BYTES
// 0.3
Error::UnsupportedFormat { format, feature }   // both Option
Error::InvalidInput { reason, message }       // reason: "threshold_not_finite", ...
Error::ParseFailed { reason, cause }          // reason: None, or "head_out_of_range", ...
domain::limits::MAX_INPUT_BYTES
grade13.1density0.61compression0.50

InvalidInput's reason names the check that failed, a stable name the errors reference lists, and details["reason"] carries it, where 0.2 put the message text there. On model_invalid, parse_failed and io the error's own text moves from details["reason"] to details["cause"], so reason names a check wherever it appears. Four InputTooLarge labels name what you read rather than matra's modules: file_source is file_read, config_file is config_file_read, udpipe_download is parser_model_download and embedding_download is embedding_model_download. A match on the old strings no longer fires.

grade6.5density0.59compression0.54

Error::details() is new. It returns every value in the message by name, so no caller parses a message. Error and DocumentError serialize as {kind, message, details}, and DocumentError adds path. A failed read now names its path: io error: cannot read essay.md: No such file or directory (os error 2). A file whose bytes are not UTF-8 is UnsupportedFormat, not Io. Every cap is in domain::limits, with the same values as 0.2. Errors lists every kind and its details.

What left the public surface#

grade12.4density0.59compression0.51

These modules and items are private to the crate in 0.3: the Udpipe adapter, metrics with Metric, default_suite, run_suite and readability::flesch_kincaid_grade, hearst, extraction, the readers MarkdownDecomposer and PlainTextDecomposer with the Decomposers table, FileSource and DirectorySource, and standard_decomposers. The port modules nlp, decompose and source stay public with their traits, reshaped as above. The measures stay public as fields of the Document. For another measure, write a function over &Document. Embedder stays public and keeps its shape. Its identity returns a domain::Identity, the shape the parser reports, where it returned &str: Identity::new("my-service", Some("v3".into()), None, None) in place of "my-service/v3". Return ModelInvalid, Io or InputTooLarge for your own failures, as the trait documents.

Configuration#

gradenonedensity0.60compressionnone

Config reads the renamed sections, so its accessors follow them.

// 0.2
cfg.summarize_n();
cfg.summarize_algorithm();
cfg.keyphrases_algorithm();
cfg.semantic_threshold();   // f32
cfg.udpipe_model();
// 0.3
cfg.summary_n();
cfg.summary_method();
cfg.keyphrases_method();
cfg.clusters_threshold();   // f64
grade10.5density0.47compressionnone

Config::from_file(path) is new: the configuration with a file you name, where a missing file is an error.

Python#

Build the engine#

grade7.4density0.55compressionnone

Matra is Engine. A model directory is named by MATRA_MODEL_DIR, or by a config file you pass to Engine.from_config. Matra.from_path is gone: no surface loads a model file of your choosing.

# 0.2
from matra import Matra

m = Matra.english()
m = Matra.english("/opt/matra/models")
# 0.3
import matra

engine = matra.Engine()                                  # MATRA_MODEL_DIR, else the default
engine = matra.Engine.from_config("matra.toml")          # a config file you name
engine = matra.Engine.from_config()                      # the config file the environment names

Analyze#

grade5.2density0.63compressionnone

engine.analyze takes exactly one input. A str alone is text, never a path, and reads as plain text unless you name a format. It returns a matra.Document, not a dict.

# 0.2
doc = m.analyze(text)
doc = m.analyze_markdown(text)
items = m.analyze_path("essay.md")              # a list of one
items = m.analyze_path("docs/")                 # a list, built in full
# 0.3
doc = engine.analyze(text)
doc = engine.analyze(text, format="markdown")
doc = engine.analyze(file="essay.md")           # read as its extension and bytes say
for item in engine.analyze_all(dir="docs/"):    # lazy, one item per file
    ...
grade14.2density0.74compressionnone

analyze_path returned CorpusEntry dicts ({"path", "analysis"}) and DocumentError dicts ({"path", "error": {"kind", "message"}}). analyze_all yields a matra.Document or a DocumentError dict, {"path", "kind", "message", "details"}. Tell them apart with isinstance(item, matra.Document). The path of a document is item["provenance"]["source"]["path"].

# 0.2
for item in m.analyze_path("docs/"):
    if "error" in item:
        print(item["path"], item["error"]["kind"])
    else:
        print(item["path"], item["analysis"]["passive_ratio"])
# 0.3
for item in engine.analyze_all(dir="docs/"):
    if isinstance(item, matra.Document):
        print(item["provenance"]["source"]["path"], item["metrics"]["passive_ratio"])
    else:
        print(item["path"], item["kind"])

Read a document#

grade6.1density0.59compressionnone

A matra.Document reads like the dict it replaced: doc["metrics"]["total_words"], list(doc) and len(doc) work. It is read-only. Call doc.to_dict() where your code needs a dict, changes the result, or pickles it. The keys follow the schema 2.0 record, so the measures moved:

# 0.2
doc["passive_ratio"]
doc["sections"][0]["paragraphs"][0]["readability_grade"]
doc["sections"][0]["paragraphs"][0]["in_blockquote"]
# 0.3
doc["metrics"]["passive_ratio"]
doc["sections"][0]["paragraphs"][0]["metrics"]["readability_grade"]
doc["sections"][0]["paragraphs"][0]["kind"] == "quote"

Summaries, keyphrases and clusters#

grade8.8density0.56compression0.55

The four algorithm-named methods and semantic_clusters are module functions over a Document. A text is parsed once however many outcomes you ask of it, and in the format you analysed it in: 0.2's methods read every text as plain text, so a README's headings could land in its summary. Analyse a markdown file with file=, or a markdown string with format="markdown", and its headings stay headings. max_phrases is n. Each returns a dict whose provenance names the method and its parameters.

# 0.2
top = m.tfidf_summarize(text, 3)                   # [{"text", "score", "position"}]
ranked = m.textrank_summarize(text, 3)
phrases = m.yake_keyphrases(text, max_phrases=10)  # [{"phrase", "score"}]
groups = m.semantic_clusters(text, 0.85, model)
# 0.3
doc = engine.analyze(text)
top = matra.summarize(doc, 3)                       # {"provenance", "sentences": [{"index", "span", "text", "score"}]}
ranked = matra.summarize(doc, 3, method="textrank")
phrases = matra.keyphrases(doc, 10, method="yake")  # {"provenance", "phrases": [{"phrase", "score", "occurrences"}]}
groups = matra.clusters(doc, model, 0.85)
grade10.4density0.68compressionnone

The vectors-in matra.semantic_clusters(vectors, threshold, model_hash) is gone, because Rust never had it. To compare vectors you already hold, compute the cosine yourself, as the semantic clusters guide shows.

Exceptions#

grade7.2density0.59compressionnone

Every exception is a matra.MatraError subclass with kind and details. Each also derives from the builtin class 0.2 raised, so except ValueError and except OSError catch what they caught.

# 0.2
try:
    m.analyze(text)
except ValueError as e:          # which ValueError: read the message
    ...
# 0.3
try:
    engine.analyze(text)
except matra.InputTooLargeError as e:   # also a ValueError
    print(e.kind, e.details["limit"], e.details["actual"])
grade10.5density0.42compressionnone

A missing file was a bare OSError in 0.2. It is now matra.IoNotFoundError, which is a FileNotFoundError and an OSError, so except FileNotFoundError now catches it. A permission failure is matra.IoPermissionError, a PermissionError.

Types#

grade15.5density0.63compressionnone

matra.types follows the record. ScoredSentence is SummarySentence inside Summary; SemanticClusters, SemanticCluster and SemanticEdge are Clusters, Cluster and SimilarityEdge; Reporting is ClausalComplement; CorpusEntry and CorpusItem are gone. The TypedDict that was named Document is DocumentDict, the shape to_dict() returns, and matra.Document is the class.

Command line#

The config file#

grade4.6density0.61compressionnone

A config file from 0.2 does not load. Every command that reads one fails with invalid_input, naming the file and each key's replacement. Edit the file, or write a fresh one with matra config init --force.

# 0.2
[models]
udpipe = "english-ewt-ud-2.5-191206"
embedding = "potion-base-8M"

[semantic]
threshold = 0.85

[summarize]
n = 3
algorithm = "tfidf"

[keyphrases]
n = 10
algorithm = "rake"
# 0.3
[models]
embedding = "potion-base-8M"

[summary]
n = 3
method = "tfidf"

[keyphrases]
n = 10
method = "rake"

[clusters]
threshold = 0.85
grade8.5density0.65compressionnone

models.udpipe goes without a replacement, because the parsing model is pinned in the source beside its digest.

The JSON envelope#

grade6.1density0.53compressionnone

format_version is 2, and the envelope gains a fifth key, error. All five are always present, and exactly one of result and error is null. A consumer pinned to format_version: 1 should fail, and then move.

{ "format_version": 1, "command": "analyze", "input": "essay.md", "result": {} }
{ "format_version": 2, "command": "analyze", "input": "essay.md", "result": {}, "error": null }
gradenonedensity0.67compressionnone

result changed too:

  • analyze: a schema 2.0 Document. See The JSON record below.
  • summarize: a Summary record, {provenance, sentences}, not a bare list of {text, score, position}.
  • keyphrases: a Keyphrases record, {provenance, phrases}, not a bare list of {phrase, score}.
# 0.2
matra summarize essay.md --json | jq '.result[0].text'
matra keyphrases essay.md --json | jq '.result[].phrase'
# 0.3
matra summarize essay.md --json | jq '.result.sentences[0].text'
matra keyphrases essay.md --json | jq '.result.phrases[].phrase'

Failures under --json#

grade4.2density0.51compression0.56

In 0.2 a failure under --json wrote nothing to stdout. In 0.3 it writes the envelope there, with result null and error holding the kind, message and details. It still writes matra: <message> to stderr and still exits 2, so a script that reads only the exit code is unchanged. A usage error under --json is an envelope too.

# 0.3
matra analyze missing.md --json | jq -r '.error.kind'      # io
grade2.4density0.58compressionnone

The message for a missing file changed with it: io error: cannot read missing.md: ... where 0.2 printed no such file: missing.md. Branch on error.kind and read error.details, never the message.

Standard input#

grade6.3density0.47compression0.61

Standard input is read as bytes, and its format is detected as a file's is. --stdin-filename still names it, and its extension still picks the format. Bytes that are not UTF-8 fail as unsupported_format, as a file's do, where 0.2 failed with a read error that had no kind.

What is new#

grade6.7density0.61compressionnone

These do not break a 0.2 command, and are listed so you know they exist: matra analyze DIR, one result per file; matra clusters; --format on every document command; --depth on analyze --json; and matra --schema. The CLI guide has each.

The JSON record#

grade7.3density0.55compressionnone

The same record is the Rust Document, the Python matra.Document and analyze --json's result. It now names its own schema version in provenance.schema, "2.0". The schema is published: matra --schema document prints it.

0.20.3
no provenanceprovenance: the matra version, the schema, the parser and its model's SHA-256, the format, and the input's path, and the size and SHA-256 of the analysed text
vocabulary_ttr, nominalization_ratio, passive_ratiometrics.vocabulary_ttr, metrics.nominalization_ratio, metrics.passive_ratio
absentmetrics.total_words, total_sentences, paragraph_count, mean_sentence_length, sentence_length_std
absentskipped: each construct the reader saw and did not analyse, with its kind and span
sections[].heading, level, paragraphsthe same, and span (the heading's, null with no heading)
paragraph.readability_grade, lexical_density, compression_ratioparagraph.metrics.readability_grade, and so on
paragraph.in_blockquoteparagraph.kind: paragraph, quote or list_item
absentspan on every paragraph, sentence and token, in UTF-8 bytes of the analysed text
absentsentence.word_count, tree_depth (null on a cycle), passive
sentence.reportingssentence.clausal_complements
sentence.bare_assertionsentence.root_mood: {root_id, mood}, or null
token.feats: one string of pipe-separated featurestoken.features: {"Mood": "Ind", "Tense": "Past"}
grade12.2density0.53compressionnone

Read the record as the schema asks: ignore a field you do not recognize, treat every string set you receive as open, and reject a major you do not know.

Results that change with no rename#

grade6.1density0.44compressionnone

A few results differ for the same input, and nothing about the call tells you so.

  • Markdown is read by a CommonMark parser. 0.2 dropped tables without a trace, ended the document at a ## References line, read #hashtag as a heading, and passed inline markup such as ** and link syntax to the parser as words. In 0.3 tables, code, front matter and HTML are skipped entries, a ## References heading is a heading like any other, a heading needs a space after its #, list items are paragraphs of kind list_item, and inline markup is removed from the text the parser reads. The counts and measures of a markdown document can change.
  • Plain text splits paragraphs at a line of only whitespace, so \r\n endings and a line of spaces separate paragraphs as an empty line does.
  • Keyphrases with equal scores come in the same order on every run, by first occurrence, where 0.2 followed a hash map's order.
  • TF-IDF scores are the same to the last bit on every run.
  • A binary file is unsupported_format, naming the format when it is a PDF or DOCX. In 0.2 it was an I/O error.
  • The size cap's message reads input too large, where 0.2 read input input too large.
  • Model2Vec.potion_base_8m(dir) looks in dir/potion-base-8M/, where 0.2 looked in dir itself. A 0.2 call that passed the model's own directory still runs, and downloads a second copy one level down. Pass the models directory instead.
  • Model2Vec.potion_base_8m() with no argument reads no config file. It finds the model directory from the environment, as Engine() does, so a model_dir or models.embedding set only in the config file no longer moves it. Model2Vec.from_config() reads the file.
  • Python's constructors announce a download. Engine(), Engine.from_config() and Model2Vec.potion_base_8m() write two lines to sys.stderr before they download a model, the ones the command line prints. Once the model is on disk they write nothing.
  • An empty result says so. matra summarize, keyphrases and clusters still exit 1 when they find nothing, and now also write one line to stderr saying so, unless --json or --quiet is given.

How-to guides

grade6.5density0.52compression0.58

Each guide does one task on one real text. Six show the input, the same call in Rust, Python and the command line, and what matra prints; the three calls print the same result, and CI runs all three and compares what they print with the page. The seventh has no command-line call, because the CLI has no command for it.

GuideInputWhat it reads
Parse a sentence into its dependency treeJane Austen, one sentenceevery word's lemma, part of speech, head and relation
Find negations and modalsthe Constitution, Article I (opening)the negations and modal verbs of each sentence
Compare readability across paragraphsThe Federalist No. 10the three measures of every paragraph
Summarize a long documentDarwin, 2,214 wordsthe three sentences TextRank ranks highest
Extract a document's keyphrasesthe same Darwinthe ten phrases RAKE ranks highest
Read a Markdown document's structurea short Markdown documentsections, headings, and the quote left unparsed
Cluster sentences by meaningyour own textgroups of sentences that say the same thing, Rust and Python only

Before you start#

grade5.8density0.62compressionnone

Install matra the way you mean to use it: the library, the CLI, or the Python package (Install matra). The Rust calls also need serde_json to print, which is the first line of each one: cargo add matra@0.3 serde_json.

grade6.1density0.53compressionnone

Each call reads its input from the directory it runs in, under the file name the page gives. Each page has the command that downloads it. The first call on a machine downloads the English model (about 16 MB), and every later call reads it from disk.

What is not here#

grade8.4density0.61compressionnone

Analyzing a directory as a corpus has no worked example here, because each example pins the output of one input. The CLI, Rust and Python guides cover it.

Parse a sentence into its dependency tree

grade11.5density0.48compressionnone

The first sentence of Pride and Prejudice, parsed: every word with its lemma, its part of speech, the word it depends on, and the relation between them.

Input#

Pride and Prejudice, the first sentence, Jane Austen. Public domain (first published 1813). 23 words.

The calls read it as austen-first-sentence.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/parse-a-sentence/austen-first-sentence.txt

or download austen-first-sentence.txt.

It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.

The call#

Rust

// cargo add matra@0.3 serde_json
use matra::{Engine, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads the English model, downloading it on first use.
    let engine = Engine::new()?;
    let doc = engine.analyze(Input::file("austen-first-sentence.txt"))?;
    println!("{}", serde_json::to_string_pretty(&doc)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()  # downloads the English model on first use
doc = engine.analyze(file="austen-first-sentence.txt")  # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False))

CLI

$ matra analyze --json austen-first-sentence.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "analyze",
  "input": "austen-first-sentence.txt",
  "result": /* the output below */,
  "error": null
}

Output#

Trimmed: only result.sections[0].paragraphs[0].sentences[0].tokens, lists cut to the first 3.

[
  {
    "id": 1,
    "span": {
      "byte_start": 0,
      "byte_end": 2
    },
    "text": "It",
    "lemma": "it",
    "pos": "PRON",
    "xpos": "PRP",
    "features": {
      "Case": "Nom",
      "Gender": "Neut",
      "Number": "Sing",
      "Person": "3",
      "PronType": "Prs"
    },
    "head": 4,
    "dep": "nsubj",
    "deps": "_",
    "misc": "",
    "is_punct": false
  },
  {
    "id": 2,
    "span": {
      "byte_start": 3,
      "byte_end": 5
    },
    "text": "is",
    "lemma": "be",
    "pos": "AUX",
    "xpos": "VBZ",
    "features": {
      "Mood": "Ind",
      "Number": "Sing",
      "Person": "3",
      "Tense": "Pres",
      "VerbForm": "Fin"
    },
    "head": 4,
    "dep": "cop",
    "deps": "_",
    "misc": "",
    "is_punct": false
  },
  {
    "id": 3,
    "span": {
      "byte_start": 6,
      "byte_end": 7
    },
    "text": "a",
    "lemma": "a",
    "pos": "DET",
    "xpos": "DT",
    "features": {
      "Definite": "Ind",
      "PronType": "Art"
    },
    "head": 4,
    "dep": "det",
    "deps": "_",
    "misc": "",
    "is_punct": false
  }
  // … 22 more
]

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

grade2.1density0.40compressionnone

The same 25 tokens as a table, then as arcs drawn from each word's head:

Dependency parse

1It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.

Dependency arcs for: It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.rootdetadvmodpunctamodcaseamodcasedetcopconjdetnmoddetcopcasensubjcaseauxnmodnmodnsubjobjccomppunctItPRONisAUXaDETtruthNOUNuniversallyADVacknowledgedVERB,PUNCTthatPRONaDETsingleADJmanNOUNinADPpossessionNOUNofADPaDETgoodADJfortuneNOUNmustAUXbeAUXinADPwantNOUNofADPaDETwifeNOUN.PUNCT

Scroll sideways to see the whole sentence.

show the data
#WordLemmaPOSHeadRelation
1ItitPRON4truthnsubj
2isbeAUX4truthcop
3aaDET4truthdet
4truthtruthNOUN0rootroot
5universallyuniversallyADV6acknowledgedadvmod
6acknowledgedacknowledgeVERB4truthconj
7,,PUNCT6acknowledgedpunct
8thatthatPRON21wantobj
9aaDET11mandet
10singlesingleADJ11manamod
11manmanNOUN21wantnsubj
12ininADP13possessioncase
13possessionpossessionNOUN11mannmod
14ofofADP17fortunecase
15aaDET17fortunedet
16goodgoodADJ17fortuneamod
17fortunefortuneNOUN13possessionnmod
18mustmustAUX21wantaux
19bebeAUX21wantcop
20ininADP21wantcase
21wantwantNOUN6acknowledgedccomp
22ofofADP24wifecase
23aaDET24wifedet
24wifewifeNOUN21wantnmod
25..PUNCT4truthpunct
  • Core argument
  • Modifier
  • Function word
  • Other

Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.

Input: Pride and Prejudice, the first sentence, Jane Austen. Public domain (first published 1813). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: austen-first-sentence/parse.json.

What to notice#

  • A document is sections of paragraphs of sentences, even when it is one sentence, so the tokens are at sections[0].paragraphs[0].sentences[0].tokens. In Rust, doc.sentences() walks every sentence in document order.
  • Each token carries its CoNLL-U columns under matra's names: id (from 1 within the sentence), text, lemma, pos (the Universal Dependencies tag), head (the id of the word it depends on, 0 for the root) and dep (the relation). Here truth is the root, and It is its nsubj.
  • lemma is the dictionary form: is becomes be, and acknowledged becomes acknowledge.
  • The parse is the model's, reported as the model returned it. The model reads that (token 8) as a pronoun and the object of want, where a grammar would call it the conjunction that opens the clause. matra does not correct the parser; Methodology names the model and its limits.
  • The whole output also carries the structural primitives read off this tree: a modal (must, governing want) and a clausal complement (acknowledge, whose clause is headed by want). The next example reads those across a document.

Find negations and modals

grade11.1density0.55compressionnone

Which sentences of a legal text are negated, and which carry a modal verb, read off each sentence's dependency tree.

Input#

The Constitution of the United States: the Preamble, and Article I, Sections 1 and 2 (opening), The Constitutional Convention, 1787. Public domain (a founding document of the United States, 1787). 165 words.

The calls read it as constitution-article-one.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/negations-and-modals/constitution-article-one.txt

or download constitution-article-one.txt.

The text, 3 paragraphs

We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.

All legislative Powers herein granted shall be vested in a Congress of the United States, which shall consist of a Senate and House of Representatives.

The House of Representatives shall be composed of Members chosen every second Year by the People of the several States, and the Electors in each State shall have the Qualifications requisite for Electors of the most numerous Branch of the State Legislature. No Person shall be a Representative who shall not have attained to the Age of twenty five Years, and been seven Years a Citizen of the United States, and who shall not, when elected, be an Inhabitant of that State in which he shall be chosen.

The call#

Rust

// cargo add matra@0.3 serde_json
use matra::{Engine, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads the English model, downloading it on first use.
    let engine = Engine::new()?;
    let doc = engine.analyze(Input::file("constitution-article-one.txt"))?;
    println!("{}", serde_json::to_string_pretty(&doc)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()  # downloads the English model on first use
doc = engine.analyze(file="constitution-article-one.txt")  # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False))

CLI

$ matra analyze --json constitution-article-one.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "analyze",
  "input": "constitution-article-one.txt",
  "result": /* the output below */,
  "error": null
}

Output#

Trimmed: only result.sections[0].paragraphs[2].sentences, tokens shown as a count, lists cut to the first 2, long strings shortened.

[
  {
    "span": {
      "byte_start": 482,
      "byte_end": 739
    },
    "text": "The House of Representatives shall be composed of Members chosen every …",
    "tokens": [ /* 44 items */ ],
    "word_count": 42,
    "tree_depth": 6,
    "passive": true,
    "negations": [],
    "modals": [
      {
        "aux_id": 5,
        "aux_lemma": "shall",
        "head_id": 7
      },
      {
        "aux_id": 28,
        "aux_lemma": "shall",
        "head_id": 29
      }
    ],
    "clausal_complements": [],
    "root_adverbials": [],
    "hearst_pairs": [],
    "root_mood": null
  },
  {
    "span": {
      "byte_start": 740,
      "byte_end": 982
    },
    "text": "No Person shall be a Representative who shall not have attained to the …",
    "tokens": [ /* 51 items */ ],
    "word_count": 46,
    "tree_depth": 7,
    "passive": true,
    "negations": [
      {
        "cue_id": 1,
        "cue_lemma": "no",
        "head_id": 2
      },
      {
        "cue_id": 9,
        "cue_lemma": "not",
        "head_id": 11
      }
      // … 1 more
    ],
    "modals": [
      {
        "aux_id": 3,
        "aux_lemma": "shall",
        "head_id": 6
      },
      {
        "aux_id": 8,
        "aux_lemma": "shall",
        "head_id": 11
      }
      // … 2 more
    ],
    "clausal_complements": [],
    "root_adverbials": [],
    "hearst_pairs": [],
    "root_mood": null
  }
]

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

grade9.8density0.47compressionnone

Every primitive of every sentence, marked on the word it is read from and the word it attaches to:

Structural primitives

  1. 1

    We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.

  2. 2

    All legislative Powers herein granted shall◇ modal be vested◇ governs in a Congress of the United States, which shall◇ modal consist◇ governs of a Senate and House of Representatives.

  3. 3

    The House of Representatives shall◇ modal be composed◇ governs of Members chosen every second Year by the People of the several States, and the Electors in each State shall◇ modal have◇ governs the Qualifications requisite for Electors of the most numerous Branch of the State Legislature.

  4. 4

    No¬ negation Person¬ negated shall◇ modal be a Representative◇ governs who shall◇ modal not¬ negation have attained¬ negated · ◇ governs to the Age of twenty five Years, and been seven Years a Citizen of the United States, and who shall¬ negated · ◇ modal not¬ negation, when elected, be an Inhabitant◇ governs of that State in which he shall◇ modal be chosen◇ governs.

show the data
#PrimitiveRead fromAttaches toField and detail
1no primitive
2modal6shall8vestedmodals
2modal18shall19consistmodals
3modal5shall7composedmodals
3modal28shall29havemodals
4negation1No2Personnegations
4negation9not11attainednegations
4negation34not33shallnegations
4modal3shall6Representativemodals
4modal8shall11attainedmodals
4modal33shall41Inhabitantmodals
4modal48shall50chosenmodals
  • ¬negation
  • ◇modal

Each primitive is a field matra fills on Sentence from its dependency tree: the tinted word is the one it is read from, the underlined word the one it attaches to. The root mood is read off the whole root clause: the Mood feature on the root or its auxiliary. What any of it means for a text is the reader's call; matra reports the construction.

Input: The Constitution of the United States: the Preamble, and Article I, Sections 1 and 2 (opening), The Constitutional Convention, 1787. Public domain (a founding document of the United States, 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: constitution-article-one/primitives.json.

What to notice#

  • Every sentence has a negations list and a modals list, empty when there is none. A negation names its cue (cue_lemma, at token cue_id) and the word it attaches to (head_id). A modal names its auxiliary (aux_lemma, at aux_id) and the word it governs (head_id). The ids are token ids within the sentence.
  • The second sentence of Section 2 has three negations, no on Person and not twice, and four modals, each shall. To the parser the Constitution's shall is a modal like any other.
  • One not attaches to shall rather than to a verb: in "who shall not, when elected, be an Inhabitant", the parser hung the negation on the auxiliary. head_id is what the tree says, and that is all it says.
  • To collect the negated sentences, keep those whose negations is not empty. What a negation means for your purpose is your code's decision; matra reports where it is.

Compare readability across paragraphs

grade5.4density0.43compressionnone

The three paragraph measures of The Federalist No. 10, to see where the essay is harder going and where it eases.

Input#

The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). 2,995 words.

The calls read it as federalist-10.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/readability-by-paragraph/federalist-10.txt

or download federalist-10.txt.

The text, 23 paragraphs

AMONG the numerous advantages promised by a well constructed Union, none deserves to be more accurately developed than its tendency to break and control the violence of faction. The friend of popular governments never finds himself so much alarmed for their character and fate, as when he contemplates their propensity to this dangerous vice. He will not fail, therefore, to set a due value on any plan which, without violating the principles to which he is attached, provides a proper cure for it. The instability, injustice, and confusion introduced into the public councils, have, in truth, been the mortal diseases under which popular governments have everywhere perished; as they continue to be the favorite and fruitful topics from which the adversaries to liberty derive their most specious declamations. The valuable improvements made by the American constitutions on the popular models, both ancient and modern, cannot certainly be too much admired; but it would be an unwarrantable partiality, to contend that they have as effectually obviated the danger on this side, as was wished and expected. Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority. However anxiously we may wish that these complaints had no foundation, the evidence, of known facts will not permit us to deny that they are in some degree true. It will be found, indeed, on a candid review of our situation, that some of the distresses under which we labor have been erroneously charged on the operation of our governments; but it will be found, at the same time, that other causes will not alone account for many of our heaviest misfortunes; and, particularly, for that prevailing and increasing distrust of public engagements, and alarm for private rights, which are echoed from one end of the continent to the other. These must be chiefly, if not wholly, effects of the unsteadiness and injustice with which a factious spirit has tainted our public administrations.

By a faction, I understand a number of citizens, whether amounting to a majority or a minority of the whole, who are united and actuated by some common impulse of passion, or of interest, adversed to the rights of other citizens, or to the permanent and aggregate interests of the community.

There are two methods of curing the mischiefs of faction: the one, by removing its causes; the other, by controlling its effects.

There are again two methods of removing the causes of faction: the one, by destroying the liberty which is essential to its existence; the other, by giving to every citizen the same opinions, the same passions, and the same interests.

It could never be more truly said than of the first remedy, that it was worse than the disease. Liberty is to faction what air is to fire, an aliment without which it instantly expires. But it could not be less folly to abolish liberty, which is essential to political life, because it nourishes faction, than it would be to wish the annihilation of air, which is essential to animal life, because it imparts to fire its destructive agency.

The second expedient is as impracticable as the first would be unwise. As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed. As long as the connection subsists between his reason and his self-love, his opinions and his passions will have a reciprocal influence on each other; and the former will be objects to which the latter will attach themselves. The diversity in the faculties of men, from which the rights of property originate, is not less an insuperable obstacle to a uniformity of interests. The protection of these faculties is the first object of government. From the protection of different and unequal faculties of acquiring property, the possession of different degrees and kinds of property immediately results; and from the influence of these on the sentiments and views of the respective proprietors, ensues a division of the society into different interests and parties.

The latent causes of faction are thus sown in the nature of man; and we see them everywhere brought into different degrees of activity, according to the different circumstances of civil society. A zeal for different opinions concerning religion, concerning government, and many other points, as well of speculation as of practice; an attachment to different leaders ambitiously contending for pre-eminence and power; or to persons of other descriptions whose fortunes have been interesting to the human passions, have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to co-operate for their common good. So strong is this propensity of mankind to fall into mutual animosities, that where no substantial occasion presents itself, the most frivolous and fanciful distinctions have been sufficient to kindle their unfriendly passions and excite their most violent conflicts. But the most common and durable source of factions has been the various and unequal distribution of property. Those who hold and those who are without property have ever formed distinct interests in society. Those who are creditors, and those who are debtors, fall under a like discrimination. A landed interest, a manufacturing interest, a mercantile interest, a moneyed interest, with many lesser interests, grow up of necessity in civilized nations, and divide them into different classes, actuated by different sentiments and views. The regulation of these various and interfering interests forms the principal task of modern legislation, and involves the spirit of party and faction in the necessary and ordinary operations of the government.

No man is allowed to be a judge in his own cause, because his interest would certainly bias his judgment, and, not improbably, corrupt his integrity. With equal, nay with greater reason, a body of men are unfit to be both judges and parties at the same time; yet what are many of the most important acts of legislation, but so many judicial determinations, not indeed concerning the rights of single persons, but concerning the rights of large bodies of citizens? And what are the different classes of legislators but advocates and parties to the causes which they determine? Is a law proposed concerning private debts? It is a question to which the creditors are parties on one side and the debtors on the other. Justice ought to hold the balance between them. Yet the parties are, and must be, themselves the judges; and the most numerous party, or, in other words, the most powerful faction must be expected to prevail. Shall domestic manufactures be encouraged, and in what degree, by restrictions on foreign manufactures? are questions which would be differently decided by the landed and the manufacturing classes, and probably by neither with a sole regard to justice and the public good. The apportionment of taxes on the various descriptions of property is an act which seems to require the most exact impartiality; yet there is, perhaps, no legislative act in which greater opportunity and temptation are given to a predominant party to trample on the rules of justice. Every shilling with which they overburden the inferior number, is a shilling saved to their own pockets.

It is in vain to say that enlightened statesmen will be able to adjust these clashing interests, and render them all subservient to the public good. Enlightened statesmen will not always be at the helm. Nor, in many cases, can such an adjustment be made at all without taking into view indirect and remote considerations, which will rarely prevail over the immediate interest which one party may find in disregarding the rights of another or the good of the whole.

The inference to which we are brought is, that the CAUSES of faction cannot be removed, and that relief is only to be sought in the means of controlling its EFFECTS.

If a faction consists of less than a majority, relief is supplied by the republican principle, which enables the majority to defeat its sinister views by regular vote. It may clog the administration, it may convulse the society; but it will be unable to execute and mask its violence under the forms of the Constitution. When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens. To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed. Let me add that it is the great desideratum by which this form of government can be rescued from the opprobrium under which it has so long labored, and be recommended to the esteem and adoption of mankind.

By what means is this object attainable? Evidently by one of two only. Either the existence of the same passion or interest in a majority at the same time must be prevented, or the majority, having such coexistent passion or interest, must be rendered, by their number and local situation, unable to concert and carry into effect schemes of oppression. If the impulse and the opportunity be suffered to coincide, we well know that neither moral nor religious motives can be relied on as an adequate control. They are not found to be such on the injustice and violence of individuals, and lose their efficacy in proportion to the number combined together, that is, in proportion as their efficacy becomes needful.

From this view of the subject it may be concluded that a pure democracy, by which I mean a society consisting of a small number of citizens, who assemble and administer the government in person, can admit of no cure for the mischiefs of faction. A common passion or interest will, in almost every case, be felt by a majority of the whole; a communication and concert result from the form of government itself; and there is nothing to check the inducements to sacrifice the weaker party or an obnoxious individual. Hence it is that such democracies have ever been spectacles of turbulence and contention; have ever been found incompatible with personal security or the rights of property; and have in general been as short in their lives as they have been violent in their deaths. Theoretic politicians, who have patronized this species of government, have erroneously supposed that by reducing mankind to a perfect equality in their political rights, they would, at the same time, be perfectly equalized and assimilated in their possessions, their opinions, and their passions.

A republic, by which I mean a government in which the scheme of representation takes place, opens a different prospect, and promises the cure for which we are seeking. Let us examine the points in which it varies from pure democracy, and we shall comprehend both the nature of the cure and the efficacy which it must derive from the Union.

The two great points of difference between a democracy and a republic are: first, the delegation of the government, in the latter, to a small number of citizens elected by the rest; secondly, the greater number of citizens, and greater sphere of country, over which the latter may be extended.

The effect of the first difference is, on the one hand, to refine and enlarge the public views, by passing them through the medium of a chosen body of citizens, whose wisdom may best discern the true interest of their country, and whose patriotism and love of justice will be least likely to sacrifice it to temporary or partial considerations. Under such a regulation, it may well happen that the public voice, pronounced by the representatives of the people, will be more consonant to the public good than if pronounced by the people themselves, convened for the purpose. On the other hand, the effect may be inverted. Men of factious tempers, of local prejudices, or of sinister designs, may, by intrigue, by corruption, or by other means, first obtain the suffrages, and then betray the interests, of the people. The question resulting is, whether small or extensive republics are more favorable to the election of proper guardians of the public weal; and it is clearly decided in favor of the latter by two obvious considerations:

In the first place, it is to be remarked that, however small the republic may be, the representatives must be raised to a certain number, in order to guard against the cabals of a few; and that, however large it may be, they must be limited to a certain number, in order to guard against the confusion of a multitude. Hence, the number of representatives in the two cases not being in proportion to that of the two constituents, and being proportionally greater in the small republic, it follows that, if the proportion of fit characters be not less in the large than in the small republic, the former will present a greater option, and consequently a greater probability of a fit choice.

In the next place, as each representative will be chosen by a greater number of citizens in the large than in the small republic, it will be more difficult for unworthy candidates to practice with success the vicious arts by which elections are too often carried; and the suffrages of the people being more free, will be more likely to centre in men who possess the most attractive merit and the most diffusive and established characters.

It must be confessed that in this, as in most other cases, there is a mean, on both sides of which inconveniences will be found to lie. By enlarging too much the number of electors, you render the representatives too little acquainted with all their local circumstances and lesser interests; as by reducing it too much, you render him unduly attached to these, and too little fit to comprehend and pursue great and national objects. The federal Constitution forms a happy combination in this respect; the great and aggregate interests being referred to the national, the local and particular to the State legislatures.

The other point of difference is, the greater number of citizens and extent of territory which may be brought within the compass of republican than of democratic government; and it is this circumstance principally which renders factious combinations less to be dreaded in the former than in the latter. The smaller the society, the fewer probably will be the distinct parties and interests composing it; the fewer the distinct parties and interests, the more frequently will a majority be found of the same party; and the smaller the number of individuals composing a majority, and the smaller the compass within which they are placed, the more easily will they concert and execute their plans of oppression. Extend the sphere, and you take in a greater variety of parties and interests; you make it less probable that a majority of the whole will have a common motive to invade the rights of other citizens; or if such a common motive exists, it will be more difficult for all who feel it to discover their own strength, and to act in unison with each other. Besides other impediments, it may be remarked that, where there is a consciousness of unjust or dishonorable purposes, communication is always checked by distrust in proportion to the number whose concurrence is necessary.

Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic,--is enjoyed by the Union over the States composing it. Does the advantage consist in the substitution of representatives whose enlightened views and virtuous sentiments render them superior to local prejudices and schemes of injustice? It will not be denied that the representation of the Union will be most likely to possess these requisite endowments. Does it consist in the greater security afforded by a greater variety of parties, against the event of any one party being able to outnumber and oppress the rest? In an equal degree does the increased variety of parties comprised within the Union, increase this security. Does it, in fine, consist in the greater obstacles opposed to the concert and accomplishment of the secret wishes of an unjust and interested majority? Here, again, the extent of the Union gives it the most palpable advantage.

The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States. A religious sect may degenerate into a political faction in a part of the Confederacy; but the variety of sects dispersed over the entire face of it must secure the national councils against any danger from that source. A rage for paper money, for an abolition of debts, for an equal division of property, or for any other improper or wicked project, will be less apt to pervade the whole body of the Union than a particular member of it; in the same proportion as such a malady is more likely to taint a particular county or district, than an entire State.

In the extent and proper structure of the Union, therefore, we behold a republican remedy for the diseases most incident to republican government. And according to the degree of pleasure and pride we feel in being republicans, ought to be our zeal in cherishing the spirit and supporting the character of Federalists.

The call#

Rust

// cargo add matra@0.3 serde_json
use matra::{Engine, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads the English model, downloading it on first use.
    let engine = Engine::new()?;
    let doc = engine.analyze(Input::file("federalist-10.txt"))?;
    println!("{}", serde_json::to_string_pretty(&doc)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()  # downloads the English model on first use
doc = engine.analyze(file="federalist-10.txt")  # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False))

CLI

$ matra analyze --json federalist-10.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "analyze",
  "input": "federalist-10.txt",
  "result": /* the output below */,
  "error": null
}

Output#

Trimmed: sentences shown as a count, lists cut to the first 3, long strings shortened.

{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "plain_text",
    "source": {
      "path": "federalist-10.txt",
      "bytes": 17880,
      "sha256": "e074108c9cdf3d26db71dd52b22ab9f1e1af47b94618e3707daee74673b79a00"
    }
  },
  "sections": [
    {
      "heading": null,
      "level": 0,
      "span": null,
      "paragraphs": [
        {
          "span": {
            "byte_start": 0,
            "byte_end": 2351
          },
          "text": "AMONG the numerous advantages promised by a well constructed Union, non…",
          "kind": "paragraph",
          "sentences": [ /* 9 items */ ],
          "metrics": {
            "readability_grade": 20.724682332463008,
            "lexical_density": 0.4699738903394256,
            "compression_ratio": 0.4564015312632922
          }
        },
        {
          "span": {
            "byte_start": 2353,
            "byte_end": 2644
          },
          "text": "By a faction, I understand a number of citizens, whether amounting to a…",
          "kind": "paragraph",
          "sentences": [ /* 1 item */ ],
          "metrics": {
            "readability_grade": 24.8921568627451,
            "lexical_density": 0.43137254901960786,
            "compression_ratio": 0.563573883161512
          }
        },
        {
          "span": {
            "byte_start": 2646,
            "byte_end": 2775
          },
          "text": "There are two methods of curing the mischiefs of faction: the one, by r…",
          "kind": "paragraph",
          "sentences": [ /* 1 item */ ],
          "metrics": {
            "readability_grade": 10.690000000000001,
            "lexical_density": 0.45454545454545453,
            "compression_ratio": null
          }
        }
        // … 20 more
      ]
    }
  ],
  "metrics": {
    "total_words": 3000,
    "total_sentences": 88,
    "paragraph_count": 23,
    "mean_sentence_length": 34.09090909090909,
    "sentence_length_std": 17.36930597256701,
    "vocabulary_ttr": 0.25233333333333335,
    "nominalization_ratio": 0.042333333333333334,
    "passive_ratio": 0.42045454545454547
  },
  "skipped": []
}

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

grade6.0density0.50compressionnone

Each measure over the essay's 23 paragraphs, one panel each:

Paragraph measures across one document

Mean readability
18.20
Vocabulary TTR
0.25
Nominalization ratio
0.04
Passive ratio
0.42
Paragraph measures across the documentReadability gradeFlesch-Kincaid; higher reads harder010203040mean 18.2Lexical densitycontent words over all words0.300.400.500.60Compression ratiocompressed over original bytes; lower repeats more0.300.400.500.600.701357911131517192123¶
show the data
#Opens withWordsGradeDensityCompression
1AMONG the numerous advantages promised by a well…38420.720.470.46
2By a faction, I understand a number of…5124.890.430.56
3There are two methods of curing the mischiefs…2210.690.45none
4There are again two methods of removing the…4018.890.40none
5It could never be more truly said than…7912.900.460.52
6The second expedient is as impracticable as the…15714.590.440.45
7The latent causes of faction are thus sown…26318.680.530.42
8No man is allowed to be a judge…26712.920.470.46
9It is in vain to say that enlightened…8013.540.470.55
10The inference to which we are brought is,…3212.870.35none
11If a faction consists of less than a…17216.100.430.45
12By what means is this object attainable? Evidently…12113.740.470.50
13From this view of the subject it may…17921.570.450.48
14A republic, by which I mean a government…6113.910.440.58
15The two great points of difference between a…5023.030.48none
16The effect of the first difference is, on…17520.220.490.48
17In the first place, it is to be…12326.530.450.42
18In the next place, as each representative will…7632.060.420.55
19It must be confessed that in this, as…10317.280.460.52
20The other point of difference is, the greater…21624.700.440.45
21Hence, it clearly appears, that the same advantage…16914.140.490.48
22The influence of factious leaders may kindle a…12819.860.480.51
23In the extent and proper structure of the…5214.750.480.54

One panel per paragraph measure, paragraphs in document order along the bottom. A cross marks a paragraph matra declined to measure: the field is None because the paragraph falls outside the measure's applicability conditions above, and the line breaks there rather than inventing a value. The dashed line is the document's mean readability: the mean of the grades of the paragraphs matra measured. matra computes no document-level lexical density or compression ratio, so those panels have no line.

Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: federalist-10/metrics.json.

What to notice#

  • Each paragraph carries readability_grade, lexical_density and compression_ratio in its metrics. The document's own metrics, beside its sections, carry the counts, the sentence-length statistics, vocabulary_ttr, nominalization_ratio and passive_ratio.
  • The grade runs from 10.69 (paragraph 3) to 32.06 (paragraph 18). It is the Flesch-Kincaid formula over sentence length and syllables per word (Methodology), so an eighteenth-century paragraph of a few long sentences scores far above any school grade. Compare paragraphs with each other, not with a grade band.
  • compression_ratio is null for four paragraphs, the ones of 50 words or fewer. null means the measure declined to run, not that it measured zero; which measure needs what is in the Rust guide.

Summarize a long document

grade9.2density0.58compressionnone

The three sentences TextRank ranks highest in the opening of Darwin's chapter on the struggle for existence, 2,214 words.

Input#

On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859). 2,214 words.

The calls read it as origin-struggle.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/summarize/origin-struggle.txt

or download origin-struggle.txt.

The text, 10 paragraphs

Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.

Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.

We will now discuss in a little more detail the struggle for existence. In my future work this subject shall be treated, as it well deserves, at much greater length. The elder De Candolle and Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. In regard to plants, no one has treated this subject with more spirit and ability than W. Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.

I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.

A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.

There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.

But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.

In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.

The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.

In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.

The call#

Rust

// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, SummaryMethod};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = Engine::new()?;
    let text = std::fs::read_to_string("origin-struggle.txt")?;
    let doc = engine.analyze(Input::text(text, Format::PlainText))?;
    // Parse once; a summary is a function of the document.
    let summary = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
    println!("{}", serde_json::to_string_pretty(&summary)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
    doc = engine.analyze(f.read())  # read as plain text
# Parse once; a summary is a function of the document.
summary = matra.summarize(doc, n=3, method="textrank")
print(json.dumps(summary, indent=2, ensure_ascii=False))

CLI

$ matra summarize --method textrank -n 3 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "summarize",
  "input": "origin-struggle.txt",
  "result": /* the output below */,
  "error": null
}

Output#

Trimmed: long strings shortened.

{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "plain_text",
    "source": {
      "path": null,
      "bytes": 12808,
      "sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
    },
    "method": "textrank",
    "parameters": {
      "n": 3
    }
  },
  "sentences": [
    {
      "index": 32,
      "span": {
        "byte_start": 6000,
        "byte_end": 6328
      },
      "text": "Every being, which during its natural lifetime produces several eggs or…",
      "score": 0.029215311359903445
    },
    {
      "index": 33,
      "span": {
        "byte_start": 6329,
        "byte_end": 6591
      },
      "text": "Hence, as more individuals are produced than can possibly survive, ther…",
      "score": 0.03081881474196398
    },
    {
      "index": 58,
      "span": {
        "byte_start": 12049,
        "byte_end": 12444
      },
      "text": "In looking at Nature, it is most necessary to keep the foregoing consid…",
      "score": 0.030089482622730077
    }
  ]
}

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

grade10.4density0.69compressionnone

Every sentence's score in document order, with the three the summary keeps marked:

TextRank: the summary and the scores it came from

TextRank score of each sentence, in document order0.0000.0100.0200.0300.040333459sentence 1sentence 61document order; bands alternate by paragraph
  1. 33Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product.
  2. 34Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life.
  3. 59In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals.
show the data
#¶ScoreSummarySentence
110.0155Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection.
210.0145It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed.
310.0182It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted.
410.0180But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature.
510.0148How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected?
610.0155We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.
720.0123Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species?
820.0089How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise?
920.0168All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life.
1020.0240Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring.
1120.0153The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive.
1220.0102I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection.
1320.0192We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature.
1420.0113But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.
1530.0114We will now discuss in a little more detail the struggle for existence.
1630.0060In my future work this subject shall be treated, as it well deserves, at much greater length.
1730.0025The elder De Candolle and
1830.0055Lyell have largely and philosophically shown that all organic beings are exposed to severe competition.
1930.0103In regard to plants, no one has treated this subject with more spirit and ability than W.
2030.0046Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge.
2130.0121Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind.
2230.0123Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood.
2330.0237We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.
2440.0202I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny.
2540.0152Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live.
2640.0161But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture.
2740.0237A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground.
2840.0121The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die.
2940.0153But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other.
3040.0216As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants.
3140.0120In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.
3250.0197A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase.
3350.0292yesEvery being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product.
3450.0308yesHence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life.
3550.0129It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage.
3650.0154Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.
3760.0142There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair.
3860.0098Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny.
3960.0173Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants.
4060.0152The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.
4170.0199But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons.
4270.0154Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible.
4370.0153So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years.
4470.0093Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery.
4570.0198In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree.
4670.0112The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed.
4770.0109In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.
4880.0213In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair.
4980.0245Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life.
5080.0205Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.
5190.0179The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large.
5290.0115The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world.
5390.0173One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district.
5490.0182A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number.
5590.0237But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one.
5690.0247If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct.
5790.0180It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place.
5890.0225So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.
59100.0301yesIn looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals.
60100.0183Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount.
61100.0120The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.
  • in the summary (3 sentences, matra's default)
  • not picked

TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.

Input: On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: origin-struggle/textrank.json.

What to notice#

  • The summary carries the text's provenance with what made it: method is textrank and parameters is {"n": 3}. Each of its sentences has index, the sentence's place among the document's sentences counting from 0, its span in the input, its text and its score. The three come back in document order (indices 32, 33 and 58), not by score: the highest score, 0.0308, is the second. The figure numbers sentences from 1, so there they are 33, 34 and 59.
  • All three calls hand matra the text rather than the file, so the provenance records no path: the Python method takes text, and the three must record the same input.
  • The summary is extractive: the sentences are as Darwin wrote them, and nothing is rephrased.
  • matra has a second summarizer, TF-IDF, and it is the default: matra::summarize in Rust, and the command line's summary.method as shipped. This example names TextRank in all three calls (SummaryMethod::TextRank, method="textrank", --method textrank), so they agree whatever a configuration file says.

Extract a document's keyphrases

grade6.8density0.60compressionnone

The ten phrases RAKE ranks highest in the same Darwin text as the summary example.

Input#

On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859). 2,214 words.

The calls read it as origin-struggle.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/keyphrases/origin-struggle.txt

or download origin-struggle.txt.

The text, 10 paragraphs

Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.

Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.

We will now discuss in a little more detail the struggle for existence. In my future work this subject shall be treated, as it well deserves, at much greater length. The elder De Candolle and Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. In regard to plants, no one has treated this subject with more spirit and ability than W. Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.

I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.

A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.

There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.

But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.

In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.

The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.

In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.

The call#

Rust

// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, KeyphraseMethod};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = Engine::new()?;
    let text = std::fs::read_to_string("origin-struggle.txt")?;
    let doc = engine.analyze(Input::text(text, Format::PlainText))?;
    // Parse once; keyphrases are a function of the document.
    let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Rake)?;
    println!("{}", serde_json::to_string_pretty(&phrases)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
    doc = engine.analyze(f.read())  # read as plain text
# Parse once; keyphrases are a function of the document.
phrases = matra.keyphrases(doc, n=10, method="rake")
print(json.dumps(phrases, indent=2, ensure_ascii=False))

CLI

$ matra keyphrases --method rake -n 10 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "keyphrases",
  "input": "origin-struggle.txt",
  "result": /* the output below */,
  "error": null
}

Output#

Trimmed: lists cut to the first 5.

{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "plain_text",
    "source": {
      "path": null,
      "bytes": 12808,
      "sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
    },
    "method": "rake",
    "parameters": {
      "n": 10
    }
  },
  "phrases": [
    {
      "phrase": "larger domestic animal tend",
      "score": 12.636363636363637,
      "occurrences": [
        {
          "byte_start": 10132,
          "byte_end": 10161
        }
      ]
    },
    {
      "phrase": "elder de candolle",
      "score": 9,
      "occurrences": [
        {
          "byte_start": 3228,
          "byte_end": 3245
        }
      ]
    },
    {
      "phrase": "fulmar petrel lay",
      "score": 9,
      "occurrences": [
        {
          "byte_start": 10791,
          "byte_end": 10809
        }
      ]
    },
    {
      "phrase": "mere theoretical calculation",
      "score": 8.5,
      "occurrences": [
        {
          "byte_start": 8016,
          "byte_end": 8045
        }
      ]
    },
    {
      "phrase": "great horticultural knowledge",
      "score": 8.2,
      "occurrences": [
        {
          "byte_start": 3501,
          "byte_end": 3530
        }
      ]
    }
    // … 5 more
  ]
}

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

gradenonedensity0.56compressionnone

RAKE against YAKE on the same text, as ranks:

RAKE and YAKE on the same text

RAKE rank against YAKE rank for the top phrases of eachRAKE rankYAKE rank1101001000larger domestic animal tend1elder de candolle2fulmar petrel lay2mere theoretical calculation4great horticultural knowledge5probable minimum rate6astonishingly rapid increase7great destruction fal8slow -breeding man9numerous record case10domestic animal11rapid increase121immaterial whether multitude2sub-specy varietie rank3enter subject4enter subject chapter5enter6whether multitude doubtful7cling hair quadruped8hair quadruped feather9structure beetle dive10beetle dive water11immaterial whether12whether multitude
show the data
PhraseRAKE rankRAKE scoreYAKE rankYAKE score
immaterial whether multitudenot ranked145079.3
larger domestic animal tend112.636not ranked
elder de candolle29.00016590.548
fulmar petrel lay29.00024210.241
sub-specy varietie ranknot ranked226693.1
enter subjectnot ranked38987.0
enter subject chapternot ranked48544.3
mere theoretical calculation48.50022450.352
enternot ranked52233.5
great horticultural knowledge58.20019190.459
probable minimum rate67.60023070.323
whether multitude doubtfulnot ranked61735.7
astonishingly rapid increase77.35721760.377
cling hair quadrupednot ranked71679.6
great destruction fal86.91423790.281
hair quadruped feathernot ranked81615.5
slow -breeding man96.66718100.496
structure beetle divenot ranked91383.8
beetle dive waternot ranked101334.5
numerous record case105.97223610.292
domestic animal114.63615470.590
immaterial whethernot ranked111313.1
rapid increase124.35716120.561
whether multitudenot ranked121214.9
  • in RAKE's top 12
  • in YAKE's top 12
  • only one method produces the phrase

Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 321 RAKE and 2430 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.

Input: On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: origin-struggle/keyphrases.json.

What to notice#

  • The result carries the text's provenance with method rake and parameters {"n": 10}. Each of its phrases has phrase, score and occurrences, the span of every place in the input it was found, highest score first, and a higher score means more relevant. RAKE's score sums each word's co-occurrence degree over its frequency.
  • Phrases are built from lemmas, so the parser's lemma errors show through. great destruction fal is "great destruction falling", with falling read as fal, and slow -breeding man is "slow-breeding man" split at the hyphen. RAKE takes the words between stopwords and punctuation, which is why larger domestic animal tend runs "animals tends" together.
  • Two phrases tie at 9.0. Equal scores come back in the order the phrases first occur, the same on every run, so a cut that falls inside a tie is decided by position in the text, not by merit. This example asks for ten, where the tenth scores 5.972 and no phrase outside the list ties it, so no tie decides the list.
  • The figure shows how little RAKE and YAKE agree on a text like this one. Neither method is right; they rank by different evidence (Methodology).

Read a Markdown document's structure

grade8.4density0.60compressionnone

A short Markdown document with two headings and a quote, read into sections and paragraphs.

Input#

Field notes: a short Markdown document with a heading, a quote and two sections, matra contributors. MIT, as part of the matra repository. 103 words.

The calls read it as field-notes.md in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/markdown-structure/field-notes.md

or download field-notes.md.

# Field notes

The survey team reached the northern ridge on the third day. Snow had closed the lower pass, so the route ran along the eastern face instead.

> Radio contact was lost for six hours.

## Findings

Three of the five markers were intact and matched their recorded positions. The fourth had been moved about ten metres downhill by a rockfall, and its base plate was cracked. The fifth could not be found, although the team searched the slope below it for most of an afternoon. A replacement marker will be set on the next visit, once the pass is open.

The call#

Rust

// cargo add matra@0.3 serde_json
use matra::{Engine, Input};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Loads the English model, downloading it on first use.
    let engine = Engine::new()?;
    let doc = engine.analyze(Input::file("field-notes.md"))?;
    println!("{}", serde_json::to_string_pretty(&doc)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()  # downloads the English model on first use
doc = engine.analyze(file="field-notes.md")  # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False))

CLI

$ matra analyze --json field-notes.md

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "analyze",
  "input": "field-notes.md",
  "result": /* the output below */,
  "error": null
}

Output#

Trimmed: sentences shown as a count, long strings shortened.

{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "markdown",
    "source": {
      "path": "field-notes.md",
      "bytes": 564,
      "sha256": "bbf7f1c703c0bf2b998b1a8fc61e1fe0f0e6aeb670b169fd5e2da372c62fe55f"
    }
  },
  "sections": [
    {
      "heading": "Field notes",
      "level": 1,
      "span": {
        "byte_start": 0,
        "byte_end": 13
      },
      "paragraphs": [
        {
          "span": {
            "byte_start": 15,
            "byte_end": 156
          },
          "text": "The survey team reached the northern ridge on the third day. Snow had c…",
          "kind": "paragraph",
          "sentences": [ /* 2 items */ ],
          "metrics": {
            "readability_grade": 4.91076923076923,
            "lexical_density": 0.6538461538461539,
            "compression_ratio": null
          }
        },
        {
          "span": {
            "byte_start": 158,
            "byte_end": 197
          },
          "text": "Radio contact was lost for six hours.",
          "kind": "quote",
          "sentences": [],
          "metrics": {
            "readability_grade": null,
            "lexical_density": null,
            "compression_ratio": null
          }
        }
      ]
    },
    {
      "heading": "Findings",
      "level": 2,
      "span": {
        "byte_start": 199,
        "byte_end": 210
      },
      "paragraphs": [
        {
          "span": {
            "byte_start": 212,
            "byte_end": 563
          },
          "text": "Three of the five markers were intact and matched their recorded positi…",
          "kind": "paragraph",
          "sentences": [ /* 4 items */ ],
          "metrics": {
            "readability_grade": 6.8750000000000036,
            "lexical_density": 0.484375,
            "compression_ratio": 0.5897435897435898
          }
        }
      ]
    }
  ],
  "metrics": {
    "total_words": 91,
    "total_sentences": 6,
    "paragraph_count": 3,
    "mean_sentence_length": 15.166666666666666,
    "sentence_length_std": 3.3714487489307423,
    "vocabulary_ttr": 0.6923076923076923,
    "nominalization_ratio": 0.01098901098901099,
    "passive_ratio": 0.5
  },
  "skipped": []
}

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

grade7.8density0.50compressionnone

Programming model steps this same document through each stage of the pipeline, to show what each stage adds.

What to notice#

  • sections follow the headings: each has heading, the heading's text, level, the number of # marks, and span, where the heading lies in the file, # marks included.
  • The quote is kept as a paragraph whose kind is quote, and it is never parsed: its sentences list is empty and all three of its measures are null. Its span covers the > markers its text leaves out.
  • The Markdown reader is chosen by the .md extension in all three calls. Read as plain text, the same file is one section with no heading, and the heading lines and the quote are parsed as sentences, # marks and all.

Cluster sentences by meaning

grade10.8density0.55compression0.57

Group the sentences of a document that say the same thing in different words, for example to audit LLM output for restatement, which lexical overlap cannot see. In Rust this needs the model2vec feature for the reference model; the Python package has it. What the clustering threshold does explains what a cluster means, and Semantic clusters lists the value, the model and the bounds.

What you get#

grade17.5density0.53compressionnone

A Clusters value: the groups of sentences whose pairwise cosine similarity cleared the threshold you pass, the pairs that cleared it, and a provenance recording the threshold and the identity of the model that scored them.

grade7.0density0.55compressionnone

Start around 0.85 with the reference model on sentences, and calibrate on your own corpus: the threshold does not travel between models, domains or text lengths. Comparing whole documents shows how to read the raw scores before you choose one.

The model#

grade8.8density0.64compressionnone

Take the reference model, potion-base-8M. The first call downloads it (30.9 MB) and verifies it against a digest compiled into matra; every later call loads it from disk:

use matra::config::Config;
use matra::embed::model2vec::Model2Vec;

let model = Model2Vec::from_config(&Config::resolve()?)?;
from matra import Model2Vec

model = Model2Vec.potion_base_8m()
grade14.2density0.64compression0.47

It goes into a directory named for the configured embedding model (models.embedding, potion-base-8M as shipped) inside the configured model directory. To keep it somewhere else, name a models directory: Model2Vec::potion_base_8m(dir) in Rust, Model2Vec.potion_base_8m(dir) in Python. dir is the directory that holds every model, the one MATRA_MODEL_DIR names, not the model's own: the three files land in dir/potion-base-8M/, beside the parsing model in dir, so passing the directory matra config show reports as model_dir reuses a model that is already there. The call refuses a directory that already holds another model's files and names it; remove them, or name another directory, or point MATRA_MODEL_DIR or models.embedding somewhere else.

grade16.0density0.57compressionnone

To use files you already have, or a different model2vec artifact, put the three files in one directory and load it with Model2Vec::from_dir(dir), which never reaches the network:

$ mkdir -p ~/models/potion-base-8M && cd ~/models/potion-base-8M
$ for f in model.safetensors tokenizer.json config.json; do
    curl -sSfLO "https://huggingface.co/minishlab/potion-base-8M/resolve/main/$f"
  done
grade6.9density0.52compressionnone

Nothing verifies files you load this way. To check that you have the reference model, compare model.model_hash with the digest in the reference.

Rust#

use matra::config::Config;
use matra::embed::model2vec::Model2Vec;
use matra::{Engine, Input};

let cfg = Config::resolve()?;
let engine = Engine::from_config(&cfg)?;
let model = Model2Vec::from_config(&cfg)?;

let doc = engine.analyze(Input::text(text, matra::domain::Format::PlainText))?;
let clusters = matra::clusters(&doc, &model, 0.85)?;

for c in &clusters.clusters {
    println!("restated {} times: sentences {:?}", c.members.len(), c.members);
}
grade13.9density0.46compressionnone

With your own Embedder implementation in place of Model2Vec, clusters needs no feature at all; an embedder that returns vectors you already hold is how they get in.

Python#

import matra

model = matra.Model2Vec.potion_base_8m()
engine = matra.Engine()

doc = engine.analyze(text, format="markdown")   # or plain_text; a str alone reads as plain text
result = matra.clusters(doc, model, threshold=0.85)
for cluster in result["clusters"]:
    print("restated:", cluster["members"])
grade8.1density0.56compression0.56

Name the format the text is in. Read as plain text, a markdown document's heading markers (#, ##) become sentences of their own, and identical short sentences embed identically, so they come back as one cluster of "restatements". Read as markdown, a heading is a heading and never a sentence. engine.analyze(file=path) detects the format from the file.

grade6.3density0.60compressionnone

Already hold embeddings? Pass an object whose embed returns them, one per sentence, and whose identity names the model that made them, as in Rust. And model.embed(texts) returns the raw vectors when you want to do something else with them.

grade15.1density0.55compressionnone

The threshold comes back in result["provenance"]["parameters"]["threshold"] exactly as you passed it, beside the embedder's identity. An edge's score is computed in the vectors' f32 and widened, so compare scores at a few decimal places rather than for equality.

The command line#

matra clusters notes.md --threshold 0.85 --json
grade7.9density0.60compression0.54

matra clusters builds the reference embedder from the configuration, the model models.embedding names in the model directory, and takes its threshold from --threshold, else clusters.threshold in the config file, 0.85 as shipped. --json writes the same Clusters value inside the envelope; the table lists each cluster's sentence indices and text. It needs a build with model2vec, which the Python package has; a cargo install --features cli build refuses the command and names the feature. The CLI guide has the flags.

Comparing whole documents#

grade7.2density0.51compressionnone

clusters works over the sentences of one document, in every language. There is no cross-document primitive. Build one out of what you have: embed each document as a single text with model.embed, then compare the vectors yourself.

grade9.4density0.55compression0.51

One bound decides what the answer means. Model2Vec caps every text at 512 tokens, pre-truncating on bytes and then truncating the token ids, so "embed each document as a single text" embeds roughly the first 512 tokens of it and nothing after. Tokens are not words, and how many words 512 tokens buys depends on what the file holds. Say which basis a figure is on. Swept over the 26 pages of this book long enough to reach it, with the call below, which embeds the raw file text, the cap ran out between 141 and 378 words. Diagrams, code fences and tables pull the low end down; the recipe below reads files off disk, so the low end is the one that applies to it. Two long texts that agree for as few as their first 141 words can already embed to byte-identical vectors. That cuts both ways: documents sharing a boilerplate opening score as near-duplicates on the opening alone, and two real paraphrases that diverge inside their first few hundred words never get compared on the part that matters.

grade9.6density0.54compression0.54

If the tail carries the content, split each document into chunks and compare the chunks, and size the chunks from a token count rather than a word count. A word count is not a safe proxy on raw markup, and the gap is not small: sliding a window across the same pages, a window of 54 words landed inside an inline SVG block and had already filled the cap. Any word figure has to be measured against your own files, because getting it wrong does not raise. It returns a score computed on less text than you handed it.

grade7.6density0.50compressionnone

Read every pairwise score before you choose a cutoff. That is how you calibrate:

import math
from itertools import combinations
from pathlib import Path

from matra import Model2Vec

model = Model2Vec.potion_base_8m()
docs = [
    Path("site/content/guides/cli.md"),
    Path("site/content/guides/rust.md"),
    Path("site/content/roadmap.md"),
]
vectors = model.embed([p.read_text() for p in docs])


def cosine(a: list[float], b: list[float]) -> float:
    norm = math.hypot(*a) * math.hypot(*b)
    return sum(x * y for x, y in zip(a, b)) / norm if norm else 0.0


for a, b in combinations(range(len(docs)), 2):
    print(f"{cosine(vectors[a], vectors[b]):.4f}  {docs[a].name} <-> {docs[b].name}")
grade9.3density0.44compressionnone

Those three are pages of this book, run from a checkout of the repository, so the numbers below are yours to reproduce. Two of them cover the same ground for different surfaces, and the third is unrelated. It prints:

0.8281  cli.md <-> rust.md
0.5800  cli.md <-> roadmap.md
0.6121  rust.md <-> roadmap.md
grade7.4density0.42compression0.56

The separation is decisive, and the sentence-level starting point does not carry over: against a cutoff of 0.85 no pair clears, so the near-duplicate pair would have been reported as unrelated. This is what "the threshold does not travel" costs when it is taken on faith. Calibrate on scores you have read.

grade9.4density0.51compression0.58

Two things carry over from clusters, and on this route they are yours to keep. Attribution: the vectors do not carry the model identity, so record model.model_hash beside every score you keep, as clusters records it in parameters["embedder"]["sha256"]. And a zero-magnitude vector, which is what an empty document embeds to, makes no claim: the cosine above returns 0.0 for it rather than dividing by zero.

Hand matra to an agent

grade9.3density0.50compressionnone

Give an agent matra's own instructions, printed by the version it will run, rather than pointing it at these pages.

Print the skill#

grade4.9density0.46compressionnone

If the agent has a shell and nothing installed, give it this line:

uvx 'matra>=0.3' --skill
grade9.5density0.44compressionnone

If matra is already installed, matra --skill prints the same text. It says what matra is for, when to reach for it, every command with its JSON shape, how to read each number and what it does not license the agent to conclude, the limits, and the errors.

grade1.4density0.49compressionnone

Keep the version floor in the uvx line. The skill describes the command line of the version that prints it, and 0.2's spells its commands and its JSON differently from the 0.3 these pages describe. 0.1.0's has no --skill at all.

Give it the deeper references#

matra --skill -r          # the references, one per line: name, then summary
matra --skill -r json     # one reference
grade10.4density0.51compressionnone

The text is compiled into the binary and the wheel, so it matches the version that is running, and the test suite executes every command in it against the command line. The CLI reference has the flag's exit codes and its --json shapes.

Load it as a plugin#

grade9.1density0.64compressionnone

The same files install as a Claude Code plugin. Run claude --plugin-dir <checkout> over a clone of the repository, and plugin discovery picks up skills/matra/.

Point a reader of this site at llms.txt#

grade9.4density0.65compressionnone

An agent reading the documentation instead of running matra starts at llms.txt, served at the site's root: every page with its one-line summary.

Install the model behind a TLS-intercepting proxy

grade7.7density0.59compressionnone

Place the parsing model by hand when a proxy that re-signs TLS stops matra from downloading it.

Recognise the failure#

grade14.2density0.57compression0.58

matra verifies TLS against root certificates compiled into the binary and never reads the system trust store. That is why it needs no ca-certificates package, on any platform, and it is also why a proxy that re-signs TLS cannot be trusted by installing its CA anywhere on the machine. The first run that needs the model fails with:

matra: io error: download https://lindat.mff.cuni.cz/...: the TLS certificate offered for
lindat.mff.cuni.cz was rejected. matra verifies TLS against root certificates compiled into
it and never reads the system trust store, so a proxy that re-signs TLS cannot be trusted by
installing its CA. Fetch english-ewt-ud-2.5-191206.udpipe by hand and put it in the model
directory instead. Underlying failure: io: invalid peer certificate: ...
grade6.1density0.56compressionnone

In Python the same failure is a matra.IoError, an OSError, from matra.Engine(), with the URL in details["url"] and details["retryable"] false, since retrying meets the same proxy. Errors lists every way a download can fail.

Place the model by hand#

grade11.2density0.49compression0.53

The artifact is pinned by name, size and SHA-256, so a hand-placed file is exactly as trustworthy as a fetched one: it goes through the same verification on load. A file that is not the pinned model is never used and never deleted. matra tries to download the pinned model in its place, and where that download cannot get through, which is the reason to place the file by hand, the call fails and the file stays exactly as you left it.

  1. Find the model directory this machine resolves, and create it. matra config show prints it as model_dir.
  2. Download the model with a client that trusts the proxy, such as curl with the system store.
  3. Check its SHA-256 before moving it in, so a truncated or proxy-rewritten download fails here rather than on the next run. The digest must read 784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957.
  4. Move it into the model directory.
mkdir -p "$(matra config show | awk -F\" '/^model_dir/ {print $2}')"
curl -L -o english-ewt-ud-2.5-191206.udpipe \
  "https://lindat.mff.cuni.cz/repository/server/api/core/bitstreams/handle/11234/1-3131/english-ewt-ud-2.5-191206.udpipe?sequence=17&isAllowed=y"
shasum -a 256 english-ewt-ud-2.5-191206.udpipe
# 784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957
mv english-ewt-ud-2.5-191206.udpipe "<the model_dir above>/"
grade9.9density0.40compressionnone

MATRA_MODEL_DIR points at a directory of your own if the resolved one is not writable.

Check it worked#

grade3.7density0.54compressionnone

Run any command that parses. The command line writes matra: downloading ... to standard error before a fetch and nothing when the model is already in place, so a run that prints no such line has loaded the file you placed.

The embedding model#

grade16.4density0.60compressionnone

The reference embedding model has the same route: three artifacts into a directory loaded with Model2Vec::from_dir, described in semantic clusters.

matra

grade9.3density0.49compression0.53

matra reports the structure of text and measurements over it. It parses a document into a typed tree of sections, paragraphs, sentences, and tokens, measures that tree, and ranks what is in it. Every value it returns is structure or a number over structure. What any of it means for your purpose is your code's decision, which is what makes the output reusable across purposes.

Four tiers of output#

  • Structure. Sections, paragraphs, sentences, and full CoNLL-U tokens, plus the structural primitives read off each dependency tree: negations, modals, clausal complements, root adverbials, Hearst pairs, and the root clause's mood.
  • Measures. Readability, lexical density, and compression ratio per paragraph; counts, sentence length, vocabulary TTR, nominalization ratio, and passive ratio per document. Every document also records where it came from: the parser and its model, and a digest of the analysed text.
  • Extraction. Two extractive summarizers (TF-IDF, TextRank) and two keyphrase extractors (RAKE, YAKE).
  • Semantic clusters. Sentence embeddings grouped by cosine similarity, behind the model2vec feature, carrying the identity of the model that produced them.

End to end#

import matra

engine = matra.Engine()
text = (
    "The committee approved the proposal without debate. "
    "Three amendments were submitted by the working group."
)
doc = engine.analyze(text)

print(doc["metrics"]["passive_ratio"])
print(doc["sections"][0]["paragraphs"][0]["metrics"]["readability_grade"])
gradenonedensity0.40compressionnone

The same analysis from Rust, and from the command line:

use matra::domain::Format;
use matra::{Engine, Input};

let engine = Engine::new()?;

let text = "The committee approved the proposal without debate. \
            Three amendments were submitted by the working group.";
let doc = engine.analyze(Input::text(text, Format::PlainText))?;

println!("{:?}", doc.metrics.passive_ratio);
$ matra analyze essay.md
essay.md
  sentences          35
  words              332
  mean sentence len  9.5
  sentence len sd    7.4
  passive ratio      0.057
grade7.1density0.52compressionnone

All three return the same shape: a struct in Rust, a read-only Document that reads as a dict in Python, a table or --json at the command line. The second sentence in the example is passive, and amendments carries the relation nsubj:pass. matra records that and stops there.

Where to go next#

grade8.8density0.53compressionnone

Understand the model. Concepts covers the document tree, the four tiers, and what a structural primitive is. Situation model covers what the output is for and what it deliberately withholds.

grade6.2density0.63compressionnone

Write against it. Programming model is the surface: Input, Engine, ports, bounds, and what Python exposes. Pragmatics answers the choices you have to make.

gradenonedensity0.67compressionnone

Get it running. Installation, then Rust, Python, or CLI.

grade7.0density0.73compressionnone

Look something up. What matra gives you lists every field and function the pipeline returns.

Concepts

The document tree#

grade6.5density0.59compressionnone

Analysis returns one nested value. Each level owns the level below it and carries its own data.

LevelTypeCarries
documentDocumentthe section list, plus vocabulary_ttr, nominalization_ratio, passive_ratio
sectionSectionheading, level, and its paragraphs in document order
paragraphParagraphits span, text and kind, its sentences, plus readability_grade, lexical_density, compression_ratio
sentenceSentenceits text, id-sorted tokens, and the structural primitive fields
tokenTokenthe ten CoNLL-U columns (text, lemma, pos, xpos, features, head, dep, deps, misc, plus id) and the derived is_punct
grade7.1density0.49compression0.55

Paragraph.text is the text the parser reads, a markdown paragraph's inline markup removed. Sentence.text is rebuilt from the tokens, so its spacing can differ from the source. Both are normalized, and each carries a span, in UTF-8 bytes, saying where in the analysed text it came from. Each paragraph that is not a quote is parsed on its own, so a sentence always belongs to exactly one paragraph, rather than being matched back by text.

grade6.4density0.56compressionnone

Inside a sentence, exactly one token has head equal to 0: the root. Every other token names its governor in head and the relation in dep. Those two columns are what turn a flat token list into a tree.

Dependency tree for: The system was built by the team root depth 1 depth 2 root, head = 0 nsubj:pass aux:pass obl det case det The system was built by the team 1 DET 2 NOUN 3 AUX 4 VERB 5 ADP 6 DET 7 NOUN heavier lines are the arcs that make passive true

Scroll sideways to see the whole diagram.

grade5.3density0.53compressionnone

Words keep their reading order left to right; height is depth in the tree. built is the root, everything else hangs off it directly or through one more hop, so its tree_depth is 2.

grade9.1density0.60compression0.58

The same two columns, read from matra's output for five short sentences. The table is what a Sentence holds; the arcs below it draw each head and dep. The first sentence carries a negation, the second a modal, the third a clausal complement, the fourth a root adverbial, and the fifth two Hearst pairs: the structural primitives described further down.

Dependency parse

Sentence

1The committee did not approve the budget.

  1. The det of committee
  2. committee nsubj of approve
  3. did aux of approve
  4. not advmod of approve
  5. approve root
  6. the det of budget
  7. budget obj of approve
  8. . punct of approve
Dependency arcs for: The committee did not approve the budget.rootdetadvmoddetauxobjnsubjpunctTheDETcommitteeNOUNdidAUXnotPARTapproveVERBtheDETbudgetNOUN.PUNCT
show the data
#WordLemmaPOSHeadRelation
1ThetheDET2committeedet
2committeecommitteeNOUN5approvensubj
3diddoAUX5approveaux
4notnotPART5approveadvmod
5approveapproveVERB0rootroot
6thetheDET7budgetdet
7budgetbudgetNOUN5approveobj
8..PUNCT5approvepunct
  • Core argument
  • Modifier
  • Function word
  • Other

Each arc runs from a word's head to the word, whose arrow it ends at, labelled with the relation. The line groups the relations the way Universal Dependencies does: the core arguments of a predicate, the modifiers, and the function words. Point at or tap a word, here or in the table, to keep its arcs and dim the rest.

Input: Five sentences, one structural primitive each: a negation, a modal, a reporting, a root adverbial, a Hearst pair, matra contributors. MIT, as part of the matra repository. Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: primitives/parse.json.

The four output tiers#

grade8.4density0.40compressionnone

The pipeline produces them in this order, and each is a different kind of result.

  1. Structure, read and parsed by Engine::analyze. Sections, paragraphs, sentences, tokens, and the structural primitive fields. Derived from the parse, checkable against the source bytes.
  2. Measures, filled by Engine::analyze as its last step. Six Option<f64> slots filled by the metric suite: three per paragraph, two per document, plus the materialized passive_ratio. None means not computed, which is distinct from a computed zero.
  3. Outcomes, from free functions over the Document: summarize and keyphrases, with _with forms that name the method. They return Summary and Keyphrases values, each carrying the document's provenance with the method and its parameters, never fields on the tree.
  4. Semantic, with the reference embedder behind the model2vec feature. clusters returns a Clusters value whose provenance carries the embedder's identity and the threshold you supplied. This tier depends on the model, so by design it never becomes a field on Document or Sentence.

Structural primitives#

grade9.6density0.55compression0.52

A structural primitive is a construction read off the dependency graph and reported as a field. Six of them are computed once and travel with the sentence into every language binding: five at Sentence construction, and hearst_pairs as the engine parses each paragraph, because its detector lives outside the domain (a Sentence built by hand has an empty hearst_pairs). They name an arc shape. They never name a judgment.

FieldThe arc shapeGrounded example
negationsnot, never, no, neither, nor on an advmod, det, or cc arcIt was never reviewed. gives one Negation: cue never, head reviewed
modalsone of ten modal lemmas on an aux arc or with the AUX part of speechYou must complete the form by Friday. gives one Modal: must on complete
clausal_complementsa verb governing a clausal complement (ccomp), with its subject when the parse has oneSmith reported that the effect vanished. gives one ClausalComplement: verb report, complement head vanished, subject Smith
root_adverbialsevery advmod arc into the rootReportedly, the deal closed. gives one RootAdverbial: reportedly
hearst_pairsthe six Hearst (1992) lexico-syntactic patterns, matched as dependency arcsAnimals such as dogs and cats need daily care. gives two HearstPair values, both tagged such_as, hypernym animal, hyponyms dog and cat
root_moodthe Mood feature of the root clause, on the root or on its cop, aux, or aux:pass child, with the root's token idThe committee approved it. gives Ind on approved; It might have been done. gives none, because neither the participle root nor its auxiliaries carry a mood
gradenonedensity0.75compressionnone

Two consequences of reporting arcs rather than readings:

grade6.5density0.61compression0.62

root_mood consults neither negation nor modals. The detector reads the root and its finiteness carriers. Nothing else. The sky is not blue reports Ind and also carries a negation; an indicative root that no modal governs is Ind with no modal whose head_id is the root's id. Combining the facts is your code's job.

grade8.4density0.48compressionnone

Overlap is intentional. never in the example above is both a Negation and a RootAdverbial, because it satisfies both arc shapes. Neither field is assigning it a meaning, so neither has to win.

grade18.6density0.57compressionnone

Passive voice is the seventh construction, a flag rather than a list of arcs: Sentence::passive is true when a token carries nsubj:pass, nsubjpass, or aux:pass, and its document-level share is the passive_ratio field of DocumentMetrics.

grade13.8density0.64compressionnone

Reporting verbs and root adverbials are open word classes, so matra ships no lexicon for either. Sentence::clausal_complements_in and Sentence::root_adverbials_in filter by a list you supply.

Where to go next#

grade8.2density0.56compressionnone

Situation model covers what this output is for. Programming model covers the surface that produces it.

Situation model

What matra is for#

grade7.0density0.52compressionnone

matra reports the structure of text and measurements over it. Text goes in; a typed tree, a set of measures, and a set of ranked extractions come out. Everything it returns is a value your code can read, store, and reason over.

grade5.1density0.35compressionnone

It is a library, not an application. The analysis on top of it is yours to write.

What it reports and what it leaves to you#

grade4.1density0.47compressionnone

matra reports passive_ratio = 0.4. Whether 0.4 is too passive for the document at hand is your call, and the answer differs between a statute, a lab report, and a landing page.

gradenonedensity0.67compressionnone

That split runs through every tier:

matra reportsYour code decides
a sentence carries nsubj:passwhether the missing agent matters here
clausal_complements names a verb governing a ccompwhether that verb is evidential and whether the source is credible
hearst_pairs names a construction that conventionally signals hypernymywhether the hypernymy relation actually holds
modals names must on an aux arcwhether that must is obligation or inference
a Cluster groups sentences whose vectors cleared your thresholdwhether those sentences say the same thing
grade4.8density0.43compressionnone

The reason for the split is reuse. A judgment is fitted to one purpose and is dead weight to the next caller; the arc it was computed from is not.

Where it sits in a pipeline#

gradenonedensity0.67compressionnone

Upstream of interpretation, downstream of ingestion.

files or strings  ->  matra  ->  rule evaluation
                                 scoring and dashboards
                                 retrieval and chunking
                                 LLM prompt construction
grade12.8density0.53compression0.57

Two properties make it usable that far downstream. Every value is serde-serializable, so the parse crosses into Python as a plain dict and can be stored as JSON. And every value has a place in the tree: a token id is scoped to its sentence, a sentence belongs to the paragraph whose text was parsed to produce it, and every heading, paragraph, sentence and token carries a span into the analysed text, so a value can be traced back to the bytes it came from.

What it needs from the environment#

grade9.4density0.58compression0.57

Nothing has to be set up first. Engine::new() in Rust, matra.Engine() in Python, and any matra command resolve where models live and fetch what is missing. The directory comes from an explicit argument if you pass one, else MATRA_MODEL_DIR, else the models subdirectory of the data root (MATRA_DATA_DIR, else $XDG_DATA_HOME/matra, else ~/.local/share/matra), with a non-empty ~/.matra/models from an older install used when the new location does not exist yet. The config file does not move the model directory, and it does not choose the parsing model, which is pinned in the source. Programming model has the resolution order.

grade11.9density0.58compression0.53

Models are fetched, not bundled, and only against a pinned digest. The UDPipe English model arrives on the first call to Engine::new or Engine::from_config, in Rust or in Python; the reference embedding model arrives on the first call to Model2Vec::potion_base_8m or Model2Vec::from_config. Both fetch from URLs written in the source and load nothing whose SHA-256 does not equal a constant written beside them. A download that fails verification is fetched once more, in memory, and a second failed download raises without anything having been written. Exactly one artifact set can arrive this way for each.

grade7.8density0.50compression0.63

Bringing your own embedding model is a first-class path. Model2Vec::from_dir loads what you supply and touches no network, whatever the directory holds. The parsing model is a pin. For those, the digest is identity rather than proof: it says which artifacts produced a value, not that they are the ones you meant to fetch.

grade5.8density0.56compressionnone

Verification and load share one read. Bytes are hashed in memory and the model is loaded from those same bytes. Nothing re-reads the disk between the two.

grade10.4density0.54compressionnone

Those pinned model fetches are the only network access anywhere in the library.

The split, on five sentences#

grade8.3density0.53compressionnone

The primitive fields from the table above, as matra fills them for five short sentences written to carry one each. Every mark is a construction in the tree; whether it matters for a text stays with the reader.

Structural primitives

  1. 1

    The committee did not¬ negation · √ root adverbial approve¬ negated the budget.

    ⊢root mood Ind
  2. 2

    Members must◇ modal submit◇ governs revised estimates by Friday.

  3. 3

    The chair“ subject reported“ governs a clause that the vote was postponed“ complement.

    ⊢root mood Ind
  4. 4

    Reportedly√ root adverbial, the new figures could◇ modal arrive◇ governs next week.

  5. 5

    Agencies⊂ hypernym such as the treasury⊂ hyponym and the audit office⊂ hyponym publish quarterly reports.

    ⊢root mood Ind
show the data
#PrimitiveRead fromAttaches toField and detail
1negation4not5approvenegations
1root adverbial4notnoneroot_adverbials
1root mood5approvenoneroot_mood, mood Ind
2modal2must3submitmodals
3clausal complement3reported8postponedclausal_complements, subject 2 chair
3root mood3reportednoneroot_mood, mood Ind
4modal6could7arrivemodals
4root adverbial1Reportedlynoneroot_adverbials
5Hearst pair5treasury1Agencieshearst_pairs, such_as, hyponym span 4 to 5
5Hearst pair9office1Agencieshearst_pairs, such_as, hyponym span 7 to 9
5root mood10publishnoneroot_mood, mood Ind
  • ¬negation
  • ◇modal
  • “clausal complement
  • √root adverbial
  • ⊂Hearst pair
  • ⊢root mood

Each primitive is a field matra fills on Sentence from its dependency tree: the tinted word is the one it is read from, the underlined word the one it attaches to. The root mood is read off the whole root clause: the Mood feature on the root or its auxiliary. What any of it means for a text is the reader's call; matra reports the construction.

Input: Five sentences, one structural primitive each: a negation, a modal, a reporting, a root adverbial, a Hearst pair, matra contributors. MIT, as part of the matra repository. Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: primitives/primitives.json.

Programming model

Two values: Input into Engine#

gradenonedensity0.50compressionnone

Input is one document. Engine is what happens to it.

ValueConstructorsMeaning
Inputtext(string, format), bytes(data, format), file(path), file_as(path, format), blocks(blocks)one document: text in a named format, bytes from anywhere, a file read as its name says or as a format you name, or headings and paragraphs you supply
Inputsdir(path)the files in a directory, an iterator of Input::file
Enginenew(), from_config(cfg), from_config_with_notice(cfg, notice)the assembled pipeline: UDPipe with its pinned model, and the readers this build ships
grade12.6density0.56compressionnone

Source variation lives in the data, not in the function namespace, which is why Engine::analyze is one function rather than one per source and format.

gradenonedensity0.60compressionnone

Two ways to run it:

CallDoes
analyze(input)one document, end to end, returning a Document
analyze_all(inputs)a lazy iterator, one result per input, in order; partitioning successes from failures is the caller's
grade8.1density0.40compressionnone

And what to ask of the result, with free functions over the Document that need no engine and no parser:

CallReturns
summarize(&doc, n), summarize_with(&doc, n, SummaryMethod)Summary: the n sentences that best represent the document, in document order
keyphrases(&doc, n), keyphrases_with(&doc, n, KeyphraseMethod)Keyphrases: at most n phrases, highest score first, each with every place it occurs
clusters(&doc, &embedder, threshold)Clusters: sentences grouped by the similarity of their embeddings, with the embedder you pass
grade15.7density0.56compressionnone

The method is optional and never in a function's name; every result records the method that ran and its parameters in its provenance, because which method ran decides what a score means.

grade10.7density0.65compression0.60

Reads are lazy. An Input reads nothing until the engine reaches it, and Inputs::dir lists entries up front, a listing failure being its Err, but reads no file until the iterator is pulled. A per-file failure (unreadable, oversized, a symlink, bytes that are not text) arrives as an Err item carrying the path, so one bad file cannot abort a directory walk.

grade16.1density0.59compression0.59

Behind analyze, the input side is cut by decision, one crate-private module each: fetching the bytes (with the symlink and size guards), detecting the format (a PDF or DOCX signature, then the format you named, then the extension), reading one format into blocks with their spans, building the section tree from blocks, and parsing. A new format is a reader and a registration line; no published signature moves.

grade12.5density0.54compression0.59

Inside the engine, analysis is two stages: reading and parsing (annotate), then measuring (compose). Both are crate-private, since a public parse-only call would publish a second shape of Document with every metric empty, and the library's tests pin that analysis is exactly the one followed by the other. Measuring is total: it reads what is attached and skips what is not, so it has no failure path.

grade6.9density0.64compressionnone

One short Markdown document at each stage, as matra produces them:

One document through the pipeline

1 Input Input::text adds the text and its format.

Input: format markdown, 564 bytes

# Field notes

The survey team reached the northern ridge on the third day. Snow had closed the lower pass, so the route ran along the eastern face instead.

> Radio contact was lost for six hours.

## Findings

Three of the five markers were intact and matched their recorded positions. The fourth had been moved about ten metres downhill by a rockfall, and its base plate was cracked. The fifth could not be found, although the team searched the slope below it for most of an afternoon. A replacement marker will be set on the next visit, once the pass is open.

2 Read and parse Engine::analyze adds sections, paragraphs, sentences, tokens.

  • section # Field notes
    • ¶1 The survey team reached the northern ridge on… 2 sentences 29 tokens measures not computed yet
    • ¶2 Radio contact was lost for six hours.… quote, not parsed
  • section ## Findings
    • ¶3 Three of the five markers were intact and… 4 sentences 72 tokens measures not computed yet

3 Measure Engine::analyze adds the measures.

  • section # Field notes
    • ¶1 The survey team reached the northern ridge on… 2 sentences 29 tokens grade 4.91 density 0.65 compression none
    • ¶2 Radio contact was lost for six hours.… quote, not parsed grade none density none compression none
  • section ## Findings
    • ¶3 Three of the five markers were intact and… 4 sentences 72 tokens grade 6.88 density 0.48 compression 0.59

Document: vocabulary TTR 0.69, nominalization ratio 0.01, passive ratio 0.50

show the data
¶QuoteSentencesTokensGradeDensityCompression
12294.910.65none
2yes00nonenonenone
34726.880.480.59

Each stage is what matra holds at that point, run by matra on the text shown: one call, Engine::analyze, reads and parses, then measures, and returns the last. The quote is kept as a paragraph but never parsed, so it has no sentences and no measures. A measure whose applicability conditions a paragraph does not meet stays None after measuring, shown as none.

Input: Field notes: a short Markdown document with a heading, a quote and two sections, matra contributors. MIT, as part of the matra repository. Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: field-notes/pipeline.json.

Configuration#

grade5.3density0.52compressionnone

The engine needs to know where the UDPipe model lives. Config answers that question once so no caller has to invent a path.

let engine = Engine::new()?;
let engine = Engine::from_config(&Config::resolve()?)?;
grade16.8density0.63compressionnone

Engine::new() resolves the locations from the environment alone and reads no configuration file; Engine::from_config(&Config::resolve()?) reads the file as well. An explicit directory, Config::with_model_dir, wins over everything below.

grade8.7density0.59compression0.57

Config resolves locations and defaults, never behavior. It carries the model directory, the embedding model's directory name, and the default counts, methods and cluster threshold the command line uses, which are all things a caller could pass as arguments instead. It does not carry which metrics run or how output is shaped. A key 0.3.0 renamed or removed ([summarize], algorithm, [semantic], models.udpipe) is refused with Error::InvalidInput naming what replaced it, so a config file from 0.2 fails to load rather than half-load.

grade11.5density0.56compression0.53

Resolution order, per key: an explicit argument, then the environment, then the config file, then the defaults compiled into the crate from config/default.toml. The argument rung has one producer, Config::with_model_dir, which returns the same configuration with the model directory replaced and that one key's source changed. It is the rung a command line's --model-dir lands on, so the flag reaches the adapter through Config rather than past it.

VariableOverrides
MATRA_CONFIG_FILEthe config file path
MATRA_DATA_DIRthe data root
MATRA_MODEL_DIRthe model directory
gradenonedensity0.60compressionnone

Paths follow the XDG conventions on Linux and on macOS.

WhatDefaultEnv override
config file$XDG_CONFIG_HOME/matra/config.toml, else ~/.config/matra/config.tomlMATRA_CONFIG_FILE
data root$XDG_DATA_HOME/matra, else ~/.local/share/matraMATRA_DATA_DIR
modelsthe data root's models subdirectoryMATRA_MODEL_DIR
grade10.3density0.56compression0.55

One exception to the model directory rule: if the data root has no models subdirectory yet, and ~/.matra/models exists and is not empty, that older location is used. matra never creates ~/.matra. When an existing, non-empty legacy cache is selected it is used as the model directory, downloads and re-downloads included, because the resolved directory is the one handed to the UDPipe adapter and that is where it writes. Create the new location, or set MATRA_MODEL_DIR, to move off it.

grade6.9density0.55compressionnone

An empty ~/.matra/models is not selected. There is no cache in it to keep working, and picking it would send every later download into a directory matra would otherwise never create. A fresh install therefore lands in the new place.

grade10.1density0.55compression0.58

A missing config file is not an error; the built-in defaults stand. A malformed one is Error::InvalidInput naming the file and the line, and an algorithm name the build does not know is rejected the same way at resolve time rather than at call time. Every resolved value records which rung it came from, readable through Config::sources as a ValueSource.

annotate is the only route to the parser#

grade14.2density0.55compression0.56

The engine's reading stage, annotate, runs the size check on every Input, then hands each paragraph that is not a quote to the parser (NlpProvider::parse, a crate-private trait), and maps the token and sentence spans the parser reports against the paragraph's text into the analysed text. Nothing else in the library calls the parser. That is what makes the 8 MiB cap a property of the pipeline rather than of each entry point: no text over MAX_INPUT_BYTES can reach the parser, whichever call or constructor you started from.

grade6.6density0.54compressionnone

It is also why the outcomes are cheap. Analyze once, and hand the same document to every outcome function: none of them parses anything. Nothing is parsed twice.

Ports and adapters#

grade7.9density0.61compressionnone

The pipeline depends on port traits, and adapters implement them. The domain depends on neither: the ports import the domain, never the reverse.

PortTraitAdapters this build shipsFeature
source/Source (crate-private)FileSource; DirectorySource lists a directory for it and reads nothingnone
decompose/Decomposer (crate-private)the markdown, plain-text and block-input readersnone
nlp/NlpProvider (crate-private)Udpipeudpipe (default)
embed/EmbedderModel2Vecmodel2vec
grade9.3density0.67compressionnone

Features are additive. Enabling udpipe adds UDPipe; disabling it removes UDPipe and nothing else. cargo check --no-default-features compiles. The other two features are cli (the command line, which implies udpipe) and python (the PyO3 bindings, which implies cli so that a wheel carries the command).

grade12.7density0.55compression0.51

Every port is open to your own adapter. clusters takes any &dyn Embedder, and with your own implementation it needs no feature flag at all. Engine::with_parser takes a parser of your own (NlpProvider), in any build; with_reader registers a reader (Decomposer) for a format, a Format::Other you name included; and with_source reads the paths inputs name from somewhere else (Source). The engine checks what a parser or a reader returns before it reaches a Document, so a part of yours cannot hand the outcomes a tree with a cycle or a span outside its text. Without implementing anything, bytes you fetch yourself go in through Input::bytes, and a format matra has no reader for goes in as headings and paragraphs through Input::blocks.

Determinism and bounds#

grade8.2density0.57compressionnone

Every gate returns a typed variant of Error. There is no Result<T, String> anywhere in the library, and library code does not panic: UDPipe panics at the C boundary are caught in nlp/udpipe.rs and converted.

GateCapVariant
reading, every Input8 MiB (MAX_INPUT_BYTES)InputTooLarge, what is "input"
file read8 MiBInputTooLarge, what is "file_read"
summarize, summarize_with2,000 sentencesInputTooLarge, what is "tfidf" or "textrank"
clusters2,000 sentencesInputTooLarge, what is "clusters"
keyphrases, keyphrases_with200,000 tokensInputTooLarge, what is "rake" or "yake"
the pinned UDPipe model being downloaded64 MiBInputTooLarge, what is "parser_model_download"
one pinned embedding artifact being downloaded64 MiBInputTooLarge, what is "embedding_model_download"
the config file, checked against its metadata64 KiBInputTooLarge, what is "config_file_read"
grade14.4density0.50compressionnone

The what discriminator names which gate fired, so a caller can route a document-too-big differently from a corpus-too-big.

grade13.4density0.59compressionnone

Cycle safety is a value, not a ceiling: Sentence::tree_depth is None on a malformed parse containing a cycle rather than silently truncated.

Fields cross languages, methods do not#

grade5.2density0.47compressionnone

Values reach Python through serde and pythonize. A field has a serde representation and crosses with its name unchanged. A method has none, so there is nothing for it to cross with.

grade25.4density0.81compressionnone

Rust-only, therefore: the Document iterators (paragraphs, sentences, tokens); the Sentence tree walks and filters (root_token, head_of, children_of, subtree, content_tokens, clausal_complements_in, root_adverbials_in); Paragraph::word_count and sentence_count; and Token::feat.

grade12.1density0.51compressionnone

When an aggregate needs to be visible cross-language it is materialized as a field. The document's counts, sentence-length statistics and ratios are the fields of DocumentMetrics, which the metric suite fills, and a sentence's word_count, tree_depth and passive are fields set when it is built.

What Python exposes#

grade10.5density0.61compressionnone

The same words Rust has, in Python's syntax: Engine, the read-only Document, the outcome functions, Model2Vec, and the entry point the matra console script launches. A Rust Input constructor is a Python keyword, and a Rust _with form is Python's method=.

PythonRust equivalent
Engine(), Engine.from_config(config_file=None)Engine::new(), and Engine::from_config over Config::from_file(path), or over Config::resolve() with no file named
engine.analyze(text, format=)analyze(Input::text(text, format)), plain text when no format is named
engine.analyze(file=), bytes=, blocks=analyze on Input::file (Input::file_as with format=), Input::bytes or block input
engine.analyze_all(dir=), texts=analyze_all over Inputs::dir, or over Input::text per text, lazily
doc["metrics"], doc.to_dict()the fields of Document, through serde
summarize(doc, n=3, method=), keyphrases(doc, n=10, method=)summarize or summarize_with, keyphrases or keyphrases_with
clusters(doc, embedder, threshold=0.85)clusters
span_text(text, span)slicing the analysed text by a Span
Model2Vec.from_dir, .model_hash, .dimensions, .embed, .identityModel2Vec behind the Embedder port
Model2Vec.potion_base_8mModel2Vec::potion_base_8m; with no argument, over the model directory the environment names, reading no config file, as Engine() does
Model2Vec.from_configModel2Vec::from_config over Config::from_file or Config::resolve, as Engine.from_config is Engine::from_config
any object with embed and identityan Embedder implementor, wrapped by an adapter in the binding
_core.cli_main(argv)cli::run, which is what the matra console script launches
grade13.8density0.58compression0.53

A stream crosses item by item: analyze_all yields a Document or a DocumentError dict per input, so a failure on one file is one item rather than an exception that ends the walk. And clusters takes any object with embed and identity: the Embedder port was always the extension point, and the binding now reaches it. Model2Vec carries both methods too, so it satisfies the same protocol a caller's own object does rather than sitting beside it as a second accepted type. In the extension module that arm needs no embedding adapter compiled in. The matra package itself still requires an extension built with model2vec: its __init__ imports Model2Vec and raises ImportError naming the feature when it is absent, and every published wheel includes it.

grade11.7density0.58compression0.57

An outcome takes a document you already analyzed, in both languages, so a text is parsed once however many outcomes are asked of it. Python's outcome functions take only a Document, never a dict: a dict could carry provenance no matra run produced. Parsing is the one thing no caller substitutes, in either language, because the size cap and the panic boundary live behind it. The Python binding lives in src/python/ and uses only the public surface, which boundary rule 7 holds it to.

grade7.6density0.68compressionnone

Engine is unsendable in Python, because the UDPipe model holds C-side state that is not thread safe. Multi-process Python is fine; sharing one engine across threads is not.

One core, three install routes

grade6.9density0.51compression0.57

matra is one Rust core published three ways: a library on crates.io, a command-line binary built from the same crate, and a Python package on PyPI. The routes differ in what they compile and what they print about themselves, and not in what an analysis returns. This page explains why the install commands look as they do. Install matra has the steps and Platforms and models the facts.

One program behind every matra command#

grade6.4density0.52compression0.51

The matra command the Python package installs is not a Python program. From 0.2.0 it is the Rust command line, compiled into the library and reached through the extension module, so pip install and cargo install give you the same program with two launchers. 0.1.0 had a second command line written in Python, with a --json shape of its own and a model directory hardcoded to the pre-0.2.0 location. Two implementations of one interface drifted apart, which is why the command line now lives in the library: a binary target cannot be reached from Python, and a library module can.

Why every install line names a version#

grade4.3density0.49compression0.52

An unpinned install line takes whatever the registry hands out. Today that is 0.3.x on both registries, and it will not stay that way. 0.1.0 is also still published, and its matra is the separate Python implementation above. So is 0.2, whose surface 0.3.0 renamed throughout. The failure an unpinned line risks is not an error: the install succeeds and gives you a program these pages do not describe, so its version banner and its output do not match theirs.

grade2.4density0.51compression0.56

The Python lines take a floor, 'matra>=0.3', because what they claim holds for 0.3.0 and every release after it, and an exact pin would still name 0.3.0 after a later release ships. The cargo lines take a caret requirement, matra@0.3 and --version '^0.3', which is how cargo states a version. Cargo reads a change in the first non-zero component as breaking, so the caret accepts 0.3.0 and later 0.3 releases and stops short of 0.4.0, which a floor would not.

Same program, different builds#

grade8.9density0.56compression0.55

The two banners differ, and both are correct. The CLI binary from cargo install is built with the features the command line needs, udpipe and cli. The wheel is built with python and model2vec on top, because the Python package also carries the library, embedding included. matra --version prints the version and then the features of the build that is running, so a report quoting it says which build produced a result. Every analysis command behaves the same in both.

grade7.8density0.57compression0.56

Free-threaded CPython is the one interpreter the wheels do not reach, and the reason is a tag, not a platform. The wheels are built once per platform against the CPython stable ABI and tagged abi3, which every GIL-enabled CPython from 3.12 accepts. A free-threaded interpreter accepts abi3t instead, so it falls back to building from source even where a wheel exists.

Why the first run is quiet in the library#

grade12.8density0.49compression0.54

The first call that needs the parsing model downloads it, about 16 MB from a university server, and that can take from a few seconds to over half a minute. The command line says so on stderr before it starts, because a blank terminal for that long is indistinguishable from a hung process. The library says nothing, because a program embedding matra chooses its own reporting: the constructors that take a notice callback hand it the same facts, and the program decides what to do with them.

What the clustering threshold does

grade7.1density0.57compression0.58

matra can group the sentences of a document that say the same thing in different words. It embeds each sentence as a vector, scores every pair by cosine similarity, and links each pair whose score clears a threshold you choose. The clusters are the connected groups those links make. Cluster sentences by meaning has the calls.

grade10.5density0.49compression0.57

Everything else matra returns is structure read off the text, checkable against its bytes. Clusters are not: they depend on a model's representation of meaning. So they arrive as their own value from a separate call, never as a field on Document, and their provenance carries the identity of the model that produced them and the threshold you chose.

Three things the shape means#

  • Co-membership is transitive, not pairwise. Clusters are connected components, so sentence A and sentence C can share a cluster because both resemble B, without resembling each other. The edges travel in the result so you can see which pairs actually cleared the bar. A missing edge is no claim, not a low score.
  • Singletons are always excluded. A sentence with no pair above the threshold appears in no cluster, so "not in any cluster" is a meaningful count.
  • The threshold is yours, and it does not travel. Published cutoffs for paraphrase detection span 0.67 to 0.9 with no consensus; the working value depends on the model, the domain and the text length. A cutoff calibrated on sentences is not the cutoff for whole documents, because a document vector is the mean over far more tokens.

Watch the threshold move#

grade3.7density0.39compression0.59

Ten short sentences, with the reference model. Before you move the threshold, predict: sentence 8 is about a different decision by the same committee. As the threshold falls from 0.85, at which value do you expect it to join sentences 1, 2 and 3, and through which of them? Then drag the threshold and find out.

Semantic clusters as the threshold moves

  1. 1The committee approved the budget on Tuesday.
  2. 2On Tuesday the committee passed the budget.
  3. 3The budget was approved by the committee.
  4. 4Heavy rain closed the mountain road overnight.
  5. 5The mountain road was shut overnight by heavy rain.
  6. 6The new release fixes a crash when saving large files.
  7. 7Saving large files no longer crashes the program in this release.
  8. 8The committee approved a new park plan on Monday.
  9. 9Bread rises faster in a warm kitchen.
  10. 10The museum opens at nine on weekdays.

At 0.85, matra's default: 3 clusters, 3 of 10 sentences in none.

Sentence similarities above 0.851The committee approved the budget on Tuesday.2On Tuesday the committee passed the budget.3The budget was approved by the committee.4Heavy rain closed the mountain road overnight.5The mountain road was shut overnight by heavy rain.6The new release fixes a crash when saving large files.7Saving large files no longer crashes the program in this release.8The committee approved a new park plan on Monday.9Bread rises faster in a warm kitchen.10The museum opens at nine on weekdays.0.850.690.960.860.880.640.560.72AAABBCCA0.850.690.960.860.880.640.560.72AAABBCCA0.850.690.960.860.880.640.72AAABBCCA0.850.690.960.860.880.72AAABBCCA0.850.960.860.880.72AAABBCCA0.850.960.860.88AAABBCC0.850.960.860.88AAABBCC0.850.960.860.88AAABBCC0.96AA0.96AA
show the data
ThresholdClustersEdges (cosine)
0.50{1, 2, 3, 8} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 2-8 (0.5646), 3-8 (0.6405), 4-5 (0.9592), 6-7 (0.8590)
0.55{1, 2, 3, 8} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 2-8 (0.5646), 3-8 (0.6405), 4-5 (0.9592), 6-7 (0.8590)
0.60{1, 2, 3, 8} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 3-8 (0.6405), 4-5 (0.9592), 6-7 (0.8590)
0.65{1, 2, 3, 8} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 2-3 (0.6923), 4-5 (0.9592), 6-7 (0.8590)
0.70{1, 2, 3, 8} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 1-8 (0.7239), 4-5 (0.9592), 6-7 (0.8590)
0.75{1, 2, 3} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 4-5 (0.9592), 6-7 (0.8590)
0.80{1, 2, 3} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 4-5 (0.9592), 6-7 (0.8590)
0.85{1, 2, 3} {4, 5} {6, 7}1-2 (0.8521), 1-3 (0.8812), 4-5 (0.9592), 6-7 (0.8590)
0.90{4, 5}4-5 (0.9592)
0.95{4, 5}4-5 (0.9592)
  • Ain a cluster, named by its letter
  • in no cluster
  • what you point at or tap, with its cluster

Each arc is a pair of sentences whose cosine similarity cleared the threshold, labelled with the score. A cluster is a connected group, so two sentences can share one through a third without their own pair clearing the bar. The embeddings are fixed; only the threshold moves, over the values matra was run at, and between two of them the figure blends the two runs rather than compute one of its own. matra ships 0.85 as a starting point, not a verdict: which threshold is right depends on the text and the question.

Input: Three things said two or three ways each, and three said once, matra contributors. MIT, as part of the matra repository. Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3) and the embedding model potion-base-8M (SHA-256 81c359215087). Data: paraphrases/clusters.json.

What sentence 8 did

It joins at 0.70, through sentence 1 alone: their pair scores 0.7239 and clears the bar, while its pairs with sentences 2 and 3 do not, yet it shares their cluster. A cluster can grow through a chain of pairs, on shared words ("committee approved") rather than shared meaning, and the clusters dissolve as the bar rises past 0.85.

Why a static model#

grade16.4density0.62compression0.56

The reference model is a static embedding model, a lookup table rather than a transformer: inference is a row gather, a mean, and a normalize. That costs roughly ten percent of a small transformer's benchmark quality, and buys bit-identical vectors on every platform and in every language binding, which is what lets the conformance suite pin exact vectors rather than tolerances.

Pragmatics

grade12.9density0.44compression0.60

Which call to reach for, and how to read what it returns: the choices behind the summarizers, the keyphrase extractors, the document measures, the clustering threshold and the models, and why each answer is what it is. The caps each call enforces are listed under InputTooLarge; handing matra to an agent is its own guide.

Choosing a summarizer#

grade11.9density0.64compressionnone

summarize returns the top N sentences in document order, as a Summary whose sentences carry the score and their index in the document, and whose provenance names the method. summarize_with chooses the method.

grade13.1density0.54compressionnone

TF-IDF (SummaryMethod::TfIdf, the default) scores each sentence by the mean TF-IDF of its lemmatized terms, treating every sentence as its own document for the IDF computation. It runs in linear time, and it favours sentences carrying vocabulary that is rare in the rest of the document.

grade13.2density0.62compression0.56

TextRank (SummaryMethod::TextRank) builds a similarity graph over the sentences (shared content lemmas divided by the log of the two lengths, so long sentences are not favoured) and runs iterative PageRank scoring over it. It favours sentences many others resemble. The dense similarity matrix is quadratic in sentence count, which is why the 2,000-sentence cap exists.

grade11.5density0.59compressionnone

Rare-in-this-document against central-to-this-document is the whole difference. On a document with one repeated theme they tend to agree; where they diverge, TF-IDF has found the unusual sentence and TextRank has found the typical one.

Choosing a keyphrase extractor#

grade9.6density0.68compressionnone

keyphrases returns Keyphrase values, highest score first, each with every place it occurs, and the provenance names the method. keyphrases_with chooses it.

grade11.0density0.57compressionnone

RAKE (KeyphraseMethod::Rake, the default) splits at stop words and punctuation, keeps runs of NOUN, ADJ, and PROPN tokens as candidates, and scores each by the co-occurrence degree over frequency ratio. Multi-word phrases are what it is built to find, and it needs no corpus.

grade14.8density0.63compressionnone

YAKE (KeyphraseMethod::Yake) scores individual terms on position, frequency, and context diversity, then assembles 1-word to 3-word candidates. YAKE's own score is lower-is-better, and matra inverts it so higher is more relevant in the returned values.

grade14.5density0.62compressionnone

Their caps differ from the summarizers': both are bounded on total tokens rather than sentence count, because a corpus of many one-token sentences costs them nothing.

Reading a Document#

QuestionRead
Is it passive-heavy?DocumentMetrics.passive_ratio, the share of sentences carrying a passive construction
Is it dense?ParagraphMetrics.lexical_density for the content-word share, alongside readability_grade
Is it repetitive in wording?ParagraphMetrics.compression_ratio, where a lower value means the text compressed further and so repeats more on the surface
How varied is the vocabulary?DocumentMetrics.vocabulary_ttr, but see the length caveat below
Is it repetitive in paraphrase?none of the above: lexical measures cannot see restatement in different words. That is what Clusters is for
How nominal is the style?DocumentMetrics.nominalization_ratio
grade9.9density0.53compression0.63

Two cautions. vocabulary_ttr is a raw type-token ratio and falls as text grows, so two documents of different lengths are not comparable on it without normalizing first. And any of these slots can be None, which means the metric was not computed (too short, a quote, or over a per-metric byte ceiling) and is distinct from a computed zero.

The semantic threshold#

grade6.4density0.53compressionnone

clusters takes the cosine threshold from you rather than choosing one, and records it in the result's provenance. Published cutoffs for paraphrase detection span 0.67 to 0.9 with no consensus, and the working value depends on the model, the domain, and sentence length.

grade11.1density0.60compression0.56

Raising the threshold admits fewer edges: clusters get smaller and more of them, and more sentences end up in none. Lowering it merges components through chains, because clusters are connected components rather than cliques. That transitivity is why the clearing edges travel beside the clusters: two sentences can share a cluster without a direct edge, and the edge list is where you check.

grade4.2density0.46compressionnone

Start around 0.85 with the reference model and calibrate on your own corpus.

Provisioning models#

grade9.2density0.47compressionnone

Both models work the same way, and the constructor you reach for is what decides whether anything is fetched.

grade9.2density0.48compression0.51

Ask for nothing and you get the pinned model in the resolved place. Engine::new(), Engine::from_config(cfg) and Model2Vec::from_config(cfg) resolve the directory through Config and fetch the pinned model when it is absent. In Python that is matra.Engine() and Model2Vec.potion_base_8m() with no argument, which read the environment and no configuration file, and Engine.from_config() and Model2Vec.from_config(), which read one. This is the path to take unless you have a reason not to.

grade12.6density0.53compressionnone

Naming the directory changes where, not what. Engine::from_config(&cfg.with_model_dir(dir)), Python's matra.Engine() under MATRA_MODEL_DIR, and Model2Vec::potion_base_8m(dir) fetch the same pinned artifacts into the models directory you name, each model in its own place there. Take them when the model belongs somewhere specific.

grade11.6density0.54compression0.49

Downloading is pinned, or it does not happen. Either form fetches from URLs written in the source and loads nothing whose digest does not equal a constant written in the source next to them. Verification happens before the bytes are parsed, and the bytes that were verified are the bytes that get parsed, with no second read of the disk in between. A download that fails the digest is fetched once more into memory, and a second failed download raises; nothing that failed the digest is written at any point, so a failed attempt leaves nothing behind for a later call to pick up. Exactly one artifact set can arrive this way, which is what makes "matra downloads a model" a bounded claim rather than an open one.

grade10.4density0.52compression0.55

A provisioner never deletes what it did not write. The embedding artifacts are named for their format (model.safetensors, tokenizer.json, config.json) rather than for one model, so a directory can already hold somebody else's. Model2Vec::potion_base_8m(dir) therefore downloads only into a directory holding none of the three. Finding all three with a digest that does not match, or finding only some of them, it raises and names the directory, having downloaded nothing and removed nothing. The UDPipe model file is named for its own release, so its provisioner has no such collision to guard against.

grade9.2density0.52compression0.57

Bringing your own. Model2Vec::from_dir(dir) loads what you supply and never reaches the network, whatever the directory holds. That is the path for a model2vec artifact this build does not pin, from either language. The parsing model is a pin on every surface: no surface loads another UDPipe model file. Nothing verifies it, so the model_hash on a result is identity rather than proof: it tells you which artifacts produced a score, not that they are the ones you meant to fetch.

grade10.7density0.64compressionnone

The pinned model is part of the contract in every case. A different model produces a different parse or a different vector space, and the conformance fixtures will fail, correctly. The semantic clusters reference records the embedding digest the conformance suite pins.

grade5.4density0.44compressionnone

The expensive thing is the model, not the call: load an Engine once and reuse it.

How matra runs

gradenonedensity0.57compressionnone

engine.analyze(Input::file("essay.md")) yields a Document or an Error.

One call, end to end#

grade6.3density0.62compressionnone

Take a 40 KB markdown file. Analysing it runs five things in order, each a module hiding one decision, and only the last two are expensive.

grade8.7density0.58compression0.55

Fetch. Engine::annotate reads the file through the engine's source, FileSource unless the caller supplied another with Engine::with_source. FileSource calls symlink_metadata, which does not traverse. A symlink is refused, and a directory is refused naming Inputs::dir. Anything else that is not a regular file is refused. A file whose metadata size exceeds MAX_INPUT_BYTES, 8 MiB, is refused before a byte is read, so a 1 GB file costs one stat call rather than a GB of resident memory. Then all 40 KB are read in as bytes.

grade11.0density0.52compression0.51

Detect. The bytes are checked against the cap again (the same limit under the "input" label, because bytes and text handed in directly never passed the first gate), then given a format: a PDF or DOCX signature first, then a format the caller named, then the extension, so .md is Format::Markdown. A format with no reader in the engine's table is Error::UnsupportedFormat. Otherwise the reader registered for the format gets the bytes; a text reader decodes them as UTF-8, and bytes that are not are Error::UnsupportedFormat too. Unless the reader names an analysed text of its own, as a PDF reader would, the bytes become the analysed text, with no copy.

grade11.5density0.57compression0.50

Read. The markdown reader runs a CommonMark parser over the text once, and turns its events into headings and paragraphs, each with its span in the text and its own text with the inline markup removed: a second copy of the prose. Front matter, code, tables and HTML are recorded in skipped with their spans rather than dropped. The engine then checks what the reader returned against the reader's contract: every span inside the text and on character boundaries, blocks in document order, no two spans overlapping, every heading at level 1 or more, and every paragraph's text lined up with the text its span covers. The structure step then makes the blocks a section tree, a heading opening each section.

grade9.3density0.59compression0.49

Parse and measure. annotate walks every paragraph in document order, skips the quotes, and calls the parser on each remaining paragraph's text. Sixty body paragraphs means sixty calls into UDPipe. Each call returns sentences of tokens, and the engine checks them before it uses them: ids 1..=n, every head in range, one root, no cycle, and every span a span of the paragraph, in order, over its token's form. Then it maps the spans from the paragraph's text into the file's bytes and builds each Sentence itself, its text and the fields derived from its tree included, so no derived field depends on which parser ran. Then compose runs four metric functions over the attached tree; the document-level pair aggregates over the same attachment, so there is no second sentence set to keep in agreement.

What is alive at each stage of one pipeline call fetch detect read parse measure returned all three resident String, the file text the blocks, a second copy Vec<Sentence> per paragraph bar length is how long an allocation is alive, not how large it is

Scroll sideways to see the whole diagram.

grade15.0density0.50compressionnone

At peak, three representations of the same document are resident at once: the original String, the paragraph copies, and the parsed tokens hanging off the paragraphs.

grade7.2density0.62compressionnone

For a 40 KB input none of that matters. For a caller sizing a batch job against a memory ceiling it does, because every Token carries nine owned Strings.

grade11.7density0.62compression0.58

A directory changes the shape by streaming. Inputs::dir lists the entries up front but reads nothing until pulled, and each pull of analyze_all runs one document through the whole pipeline before the next file is read. One document's representations are resident at a time; only what you retain accumulates, and collecting the stream retains everything.

The model is the expensive thing#

grade13.3density0.75compressionnone

Everything above assumes nlp already exists. Constructing it is the one genuinely slow operation matra performs.

grade13.7density0.53compression0.57

Udpipe::english(dir) looks for the English UD-EWT model in the directory you name. If it is absent, the bytes are verified in memory and then written to a temporary subdirectory named for this call, and moved into place with a single rename, so two processes pointing at the same directory cannot leave each other a half-written file. The name carries more than the process id, because in a container every process is pid 1 and two cold starts would otherwise collide.

grade16.0density0.51compression0.51

Udpipe::from_config(&cfg) is the same call with the directory resolved rather than named, and it is what the public constructors reach: Engine::new() with a Config resolved from the environment alone, Engine::from_config(&cfg) with the one you pass. Config is read once, at construction; nothing in the call path above consults it, which is why configuration cannot change what a call computes. Model2Vec::potion_base_8m(dir) and Model2Vec::from_config(&cfg) do the same for the reference embedding model, against a three-file digest rather than a single one.

grade8.0density0.57compression0.56

The cached file is then checked against a size constant, 16,309,608 bytes, and a SHA-256 constant pinned in nlp/udpipe.rs. A size mismatch fails before the file is hashed. A mismatch downloads the model, and a download that also fails the digest is retried once, so a call makes at most two downloads. The file is replaced only if downloaded bytes verify; if both downloads fail, the call returns Error::ModelInvalid and leaves the old file untouched. An unverified model is never loaded.

grade9.4density0.50compression0.56

The verify step returns the bytes it hashed, and those exact bytes go to the loader. Nothing re-reads the disk in between. That ordering is the whole point: a hash-then-reopen sequence leaves a window in which an attacker with write access to the model directory can swap the file after the check and before the read.

grade7.4density0.50compressionnone

No surface loads a model file the caller names. The pinned model is the only one a parse can come from, so the identity on every result is proof as well as a name.

grade4.3density0.51compressionnone

Load once, reuse. After construction, the model is a live handle to a C++ object. Parsing does not reload it. The Engine owns the provider, so a service builds one engine and keeps it for its lifetime.

grade9.0density0.57compression0.57

One engine per thread. The underlying Model is Send and deliberately not Sync: its parse path mutates internal workspace caches. The Engine that owns it is Send and not Sync for the same reason: an engine moves to another thread, but sharing one between two is a compile error rather than a race. To parallelize you build an engine on each thread, paying another 16 MB load for each, or serialize calls behind a mutex.

grade10.4density0.56compressionnone

Python gets the same constraint through a different mechanism. Its Engine is declared unsendable, so PyO3 records the creating thread and refuses access from any other at runtime. A ProcessPoolExecutor is unaffected, because each process loads its own model.

The unit of work is the paragraph#

grade6.7density0.55compressionnone

Parsing per paragraph looks wasteful. Sixty FFI calls where one would do, and a parser that never sees a paragraph's neighbours as context. The reason it is worth that is a defect class the alternative could not avoid.

grade12.3density0.71compressionnone

The earlier implementation joined every paragraph into one string, parsed once, then wired each returned sentence back to a paragraph by matching text prefixes. Three regression tests in src/lib.rs hold the current behaviour in place, one per way that matching failed:

  • Two paragraphs sharing their first thirty characters. A sentence lands in whichever one matched first, and the other paragraph silently loses it.
  • One paragraph containing another paragraph's opening words mid-text. The greedy match steals the sentence from its rightful owner.
  • An empty paragraph with trailing whitespace, which confused the match entirely.
grade10.5density0.45compressionnone

Follow that down and the root is not the matching algorithm. It is that the relationship between a sentence and its paragraph was being reconstructed from text, and text does not carry identity. Two paragraphs can be character-for-character similar and still be different paragraphs.

grade8.6density0.56compressionnone

So any reconstruction from that data has to guess, and a guess in this position fails silently. The Document that comes out is still well-formed, still serializes, still passes every shape check. It is simply wrong about which paragraph said what, and nothing in the output says so.

Matching back by text against parsing per paragraph join, parse once, then match sentences back by text sentences from the joined text, in order The system processes input now. Tail one. The system processes input now. Tail two. paragraph A paragraph B matches both the dashed sentence has no owner, and nothing reports it parse each paragraph on its own paragraph A paragraph B The system processes input now. Tail one. The system processes input now. Tail two. each sentence is the return value of the call on that paragraph

Scroll sideways to see the whole diagram.

grade7.4density0.49compression0.59

The two identical sentences are the ones from the regression fixture in src/lib.rs. On the left nothing in the text distinguishes them, so the assignment is a coin toss and one sentence is left over. On the right the arrows run the other way: the call produced the sentences, so the owner is known before any text is compared.

grade7.9density0.57compressionnone

Per-paragraph parsing deletes the wiring step instead of improving it. A paragraph's sentences are the return value of the call made on that paragraph's text. The relationship comes from the call graph, so there is nothing left to get wrong.

grade14.8density0.52compression0.57

Blockquote paragraphs are skipped at this stage, which is why they reach the end with no sentences and all three metric slots at None. And a parse failure in any single paragraph aborts that document with Error::ParseFailed; the partial document is dropped, not returned, and in a stream the failure travels as that document's DocumentError while the next document proceeds.

Measure fills slots, extract is a separate call#

grade11.3density0.67compression0.61

The metric suite is four functions with one signature, Box<dyn Fn(&mut Document)>, run in sequence by compose. Readability, lexical density, and compression write per-paragraph slots. The document pass writes vocabulary_ttr, nominalization_ratio and passive_ratio. None of the four reads another's output, so the suite is a list, not a chain.

grade11.5density0.47compressionnone

Each carries its own applicability condition, and this is where unexplained None values come from:

SlotWritten when
readability_grademore than 10 words, not a quote
lexical_densityat least one word, not a quote
compression_ratiomore than 50 words, not a quote, paragraph text at most 256 KiB
vocabulary_ttr, nominalization_ratioat least one word in the document
passive_ratioat least one sentence in the document
grade10.2density0.64compressionnone

A word here is a non-punctuation token, the count Paragraph::word_count returns.

grade9.6density0.56compression0.63

None is not zero. A three-word paragraph has no meaningful Flesch-Kincaid grade, and the slot says so rather than reporting a number nobody should use. The compression cap is a CPU bound: the brotli window is 2^18 bytes, and a paragraph larger than one window is skipped instead of pegging a core on adversarial input.

grade10.2density0.48compressionnone

Summarization and keyphrase extraction are not part of this. The pipeline never calls them. They are free functions over the analysed Document, invoked by the caller, and they need no engine:

let doc = engine.analyze(Input::file("essay.md"))?;
let summary = matra::summarize(&doc, 3)?;
let phrases = matra::keyphrases(&doc, 10)?;
grade11.0density0.55compression0.61

Keeping them out of the pipeline is a cost decision. TextRank builds a dense sentence-similarity matrix that is quadratic in sentence count, roughly 32 MB of f64 at its 2,000-sentence ceiling. Folding it into the analysis pass would charge that to every caller who only wanted a readability grade. Inputs above a cap return Error::InputTooLarge rather than allocating.

What can fail, and where it surfaces#

grade4.0density0.62compressionnone

Every guard sits at the earliest point where the check is still cheap.

Where each guard fires, on which call path Engine::new() once per process size constant, then SHA-256 Error::ModelInvalid analyze(Input::file(p)) Document symlink or non-regular file Error::Io file size, on metadata Error::InputTooLarge text size, in annotate Error::InputTooLarge catch_unwind, once per paragraph Error::ParseFailed summarize(&doc, n) Summary per-method cap Error::InputTooLarge

Scroll sideways to see the whole diagram.

grade7.7density0.58compression0.56

Three separate lanes, because these are three separate call paths. Only the middle one runs during a pipeline call. The model check fires once when you construct the engine, and the outcome caps fire only when you call an outcome function, which is why neither shows up in a pipeline stack trace.

grade10.1density0.53compressionnone

The catch_unwind is the one that matters. A panic crossing an FFI boundary does not unwind into your code, it aborts the process: interpreter death in Python, a trap in WASM, from a library that promised typed errors.

grade9.0density0.49compression0.58

The boundary that converts such a panic into an Error lives in nlp/udpipe.rs, and that file is the only one permitted to import the UDPipe crate. A second import path would be a second process-abort surface with no boundary in front of it. That boundary covers parsing only; model loading translates the loader's own error and is not wrapped.

grade13.7density0.54compression0.56

Every cap a gate checks, and every minimum a metric needs, is a constant in domain::limits, defined once and read where it applies. It sits inside the domain, the one module every row may import, so the gates in the adapters, the outcome functions and the metrics all read the same number without boundary rule 5 bending for it.

grade10.6density0.55compression0.49

The parser and the readers are ports anyone can implement, so the engine trusts none of them, matra's own included: what a reader returns is checked once per document, and what the parser returns once per paragraph, before either reaches a Document. A break is Error::ParseFailed with a reason naming the check and a cause naming the token or block (Errors lists the checks). Each check is a pass over what it reads. Measured over the docsite's inputs and its own reference and guide pages, nearly two thousand paragraphs, the parse checks cost about 0.014% of UDPipe's parse time, roughly 3 microseconds a paragraph, and the reader checks about 7% of the readers' own time, under a tenth of a millisecond for a 140 KB page: together 0.015% of reading and parsing. Every document an engine returns therefore holds the invariants Sentence documents, whichever parts built it.

grade12.6density0.53compression0.56

Two guards live in the domain types rather than at an edge, because a document can arrive from outside the engine, read back from JSON or built by hand, and its shape is not guaranteed. Sentence::tree_depth is computed by walking head references with a visited set, and is None when they form a cycle. Sentence::subtree carries the same visited set so it terminates. The None is deliberate: an earlier version capped depth at a magic 20, which silently truncated malformed parses and legitimate deep ones alike, and a sentinel number would read as a very deep tree to anyone who did not check for it.

grade11.4density0.62compression0.61

Directory reads fail differently. analyze_all yields successes and per-document failures side by side, so one unreadable file does not cost you the other ninety-nine; Inputs::dir is Err only when the listing itself fails. Full detail on every gate, every variant, and the Python exception each maps to is in Errors.

What you can replace#

Which module may import which, in the matra crate src/bin/matra.rs launcher, feature: cli calls run() src/cli/ the command line, feature: cli src/lib.rs declares every module, and is the only file that names every adapter and every port src/config.rs locations and defaults, never behavior source/directory.rs source/file.rs decompose/markdown.rs decompose/plain.rs nlp/udpipe.rs the only importer of udpipe_rs feature: udpipe embed/model2vec.rs sole importer of safetensors, tokenizers · feature: model2vec metrics/ extraction/ source/mod.rs trait Source Box<dyn>, with_source decompose/mod.rs trait Decomposer a table by Format, with_reader nlp/mod.rs trait NlpProvider Box<dyn>, with_parser embed/mod.rs trait Embedder &dyn, passed to clusters no port, no trait domain and stopwords only src/domain.rs serde · thiserror · std, and nothing else

Scroll sideways to see the whole diagram.

grade10.3density0.56compression0.48

Every arrow points down, and that is the rule: a module imports from the row below it, never the row above. One arrow runs sideways: src/bin/matra.rs reaches into src/cli/, which is the whole of that file: the command line lives in the library so both launchers run one program, and the binary only collects arguments, locks the streams, and returns the code. One arrow from src/cli/ reaches past lib.rs, and it still points down: matra clusters builds Model2Vec from the configuration, because the library builds no embedder inside an outcome call and choosing the default one is the application's decision. Boundary rule 7 admits embed for the command line as it does for the Python binding; the other three ports are public too, and neither needs them, so the rule keeps them out of both.

grade10.4density0.53compression0.54

src/config.rs sits in the composition-root row beside lib.rs, and it is the one box an adapter is allowed to import from the row above. Udpipe::from_config and Model2Vec::from_config take a &Config, which is the traffic RFC-0011 intends and boundary rule 7 records: Config imports no port and no adapter, so nothing travels back down with it, and an adapter that reads it is reading a resolved path rather than reaching for the environment itself.

grade13.6density0.56compression0.55

Config is not one of the extension points below, and the distinction is worth keeping. An extension point substitutes an implementation; Config only supplies values a caller could have passed as arguments. That is why it carries locations and defaults and never behavior: the moment a file selects which metrics run, configuration stops being adaptability and becomes dispatch that nothing type-checks.

grade12.1density0.55compression0.50

Every port is open to outside implementers, and each is one or two methods over domain types. NlpProvider has two, parse and identity: a paragraph in, sentences of tokens out, and who parsed them, which the engine records in every document's provenance. Decomposer has one, decompose: a format's bytes in, blocks and skipped constructs out, built with Read::new and ReadBlock::heading and ReadBlock::paragraph. Source has one, read: a path in, bytes out. Embedder is the same shape one tier over: clusters takes it as &dyn Embedder, and its identity travels into every result's provenance so the scores stay attributable to the model that produced them. That is what makes UDPipe an implementation detail rather than the library.

grade9.8density0.56compression0.46

The engine holds a parser, a table mapping each Format to a boxed reader, and a source, and a caller replaces any of them without forking matra. Engine::with_parser(parser) builds an engine around a parser of your own, in any build, with the shipped readers and the file source; .with_reader(format, reader) registers a reader for a format, replacing a shipped one, supplying one for Format::Pdf or Format::Docx, or reading a Format::Other you name; .with_source(source) reads the paths inputs name from somewhere else. Both methods compose with the standard constructors, so Engine::new()?.with_reader(Format::Pdf, MyPdf) is UDPipe with your PDF reader. It is one constructor and two consuming methods rather than a builder type: every part but the parser has a default, so there is nothing for a builder to hold that the engine does not already hold. Inputs::dir still lists the local filesystem, and Input::bytes and Input::blocks reach no source and no reader, so a caller with bytes or blocks from anywhere needs neither. Bring your own parser or reader walks through one of each.

grade12.1density0.62compression0.51

The traits are frozen once someone implements them, since a new method would break every implementation, so they ask for only what the engine cannot derive. A parser returns tokens, not Sentence values: the sentence's text, span and every field derived from its tree are the engine's, computed the same way whichever parser ran. A reader returns opaque blocks, so the block vocabulary can grow without breaking one. And neither trait promises well-formed output in its signature; the engine checks it instead, at the one place each kind of output enters.

grade8.2density0.47compression0.57

The metric suite is data as well, inside the crate: a list of functions over &mut Document that compose runs. It is private, so a new measure is a function and a line in the list. A caller who wants another measure writes a function over the Document it was handed, which is all a metric is.

grade9.2density0.52compression0.59

What you gain from the arrangement is concrete and mostly shows up in test suites. Nothing under metrics/ or extraction/ imports a port, which is the one column above that skips the port row entirely, so those functions run with no model loaded. A new metric is testable against hand-built Sentence values, with no 16 MB download and no C++ toolchain. The rules that keep it that way, and how weakly some of them are enforced, are listed in Boundary rules.

Crossing into Python copies everything#

grade8.0density0.59compression0.59

The Rust side runs the same pipeline, then hands the Document to pythonize, which walks the serde representation and builds a Python dict. That is a full deep copy. Every token becomes a dict of eleven keys. For a document with seven thousand tokens, that is seven thousand dicts allocated on the Python heap.

grade10.2density0.59compression0.59

Fields cross. Methods do not. A Rust method has no serde representation, so it has nothing to cross with, which is why every aggregate a reader needs is stored: compose writes the counts, sentence-length statistics and ratios into DocumentMetrics and ParagraphMetrics, and Sentence::new sets each sentence's word_count, tree_depth and passive (RFC-0008). A Python caller reads them rather than recomputing them. Domain types draws that boundary member by member.

grade9.5density0.55compression0.51

Errors cross by type and as data. Each variant is raised as a class of its own, named as the variant is and deriving from matra.MatraError, and the exception carries kind and details, the values Error::kind and Error::details compute, so a method's answer crosses as a field. The choice of class is a match with no wildcard arm, so adding a variant to domain::Error fails to compile until someone decides which class it becomes. A wildcard would let new failure modes fall through to another variant's class unnoticed.

grade8.0density0.64compressionnone

Every Python method that takes text routes through the same pipeline, so the 8 MiB text cap fires uniformly. The outcome methods add their per-method caps on top.

Where to look next#

grade10.4density0.56compressionnone

What matra gives you is the output side: every value the pipeline produces and what each one measures.

grade8.1density0.50compressionnone

Domain types is the type graph: what each type owns, which values are stored and which are computed on demand.

grade10.5density0.45compressionnone

Boundary rules is the enforcement side: the eight rules, why each exists, and which of them have a mechanical gate.

How matra is maintained

grade10.2density0.53compression0.59

matra is maintained by its owner and by Claude, Anthropic's AI, working through a set of agents, skills and checks that live in the repository beside the code. This page explains that arrangement for a reader outside it: someone deciding whether to depend on matra, someone thinking of contributing, or someone who wants to borrow the pattern. The mechanics of opening a pull request are in CONTRIBUTING.md; this page says why the project is run the way it is.

Two roles#

grade10.4density0.48compression0.49

The owner decides, and Claude does most of the engineering. The owner brings what is outside the codebase: what is worth building, when to ship it, and the judgment calls that depend on context no file holds. Claude brings the convergent work inside a frame: drafting, implementing, reviewing and keeping a large codebase consistent with its own rules. Neither substitutes for the other, and the working rule is to think with, not for: a claim that does not hold is challenged, from either side, with the evidence for the challenge.

WhoWhat they doWhat only they do
The ownerSets direction, reviews what matters to them, comments on records in progressAccepts an enhancement proposal by merging the change of its status to accepted, accepts an API change proposal, approves each release
ClaudeWrites code, docs and proposals; reviews; merges; prepares and dispatches releasesNothing the owner cannot also do. Claude merges every pull request except one that accepts a proposal, each with a comment giving its reason

The agents#

grade8.4density0.50compression0.54

The work is split across seven agents, each a Claude Code subagent defined by one file in .claude/agents/. They are not separate people or separate models: each is Claude with a different set of instructions, and what makes one worth having is that it reasons in a way the others do not. Every agent file states the method it owns and what it refuses to do, and no two agents own the same method.

AgentIts methodWhat it leaves to others
maintainerOrchestration: holds the whole codebase in view, makes the call on a change, authors and argues proposals, and hands each part of the work to the agent whose method fitsAccepting its own proposals, which only the owner does
reviewerFalsification: steel-mans a change, then demands evidence for it, against the review gatesResilience findings, and writing the fix
portsmithMinimal-contract design for the four port traits, and the test for moving one into its own crateAdapter internals that leave a trait unchanged
ffi-keeperKeeping the Rust and Python surfaces in agreement: what crosses, how errors map, how the packages are built and pinnedRust-only surface that never crosses
resilienceFailure-mode analysis: size caps at the entry, panic boundaries at the C++ parser, atomic writes, verified bytes used once, cycle-safe tree walksGeneral review. It is the one agent that blocks a merge on these grounds
archivistLockstep: the CHANGELOG, the index of records, statuses and tracking issues move in the same pull request as the code they describeAuthoring proposals
newcomerNaive literal execution: arrives knowing nothing, follows the pages exactly, records what was expected against what happened, and fixes nothingReading the source to explain a failure
grade5.9density0.48compressionnone

The newcomer is the agent most unlike the rest. Its first two passes, before the 0.2.0 release, found six blocking defects that every automated check had passed, the worst of them on the boundary between what a page promised and what a platform did.

The skills#

grade11.3density0.64compression0.55

The judgment the agents apply lives in .claude/skills/, one directory per practice: what good looks like in this codebase, and why. Two are non-negotiable and are read together on every structural change. ACES (adaptable, composable, extensible) is the design test: does the change leave the system easier or harder to change, to recombine and to extend. The resilience floor is the operational one: does it survive hostile input, a crashing dependency and two processes racing. The others cover the hexagonal architecture, Rust design choices, the Python binding, test strategy, keeping records in step, the review gates, and checking that a built artifact installs for a real user.

grade13.3density0.44compressionnone

They are practices rather than tool instructions, so they survive a change of tooling, and they are versioned with the code, so a change to one is reviewed like a change to the code.

Routing#

grade12.2density0.44compression0.52

Each agent's and each skill's description, at the top of its file, says what it is for and what it is not for. Claude Code reads those descriptions and picks by them, and they are the only routing table: there is no second list to drift out of step with the first. A pull request review is where several are convened, and only by what the diff touches: the reviewer always, the portsmith when a port module changes, the ffi-keeper when the Python binding or its packaging changes, resilience when a change reads files, writes them or takes outside input, and the archivist when the public surface or the records change. That rule is written once, in the /review command.

The gate before a merge#

grade4.2density0.50compressionnone

Two layers stand between a pull request and main, and they are not equal.

grade13.6density0.58compression0.53

The deterministic checks are the verification. just check runs them locally. On GitHub, branch protection on main requires twenty status checks to pass, and holds administrators to them: the Rust build and tests on Linux and macOS under each feature set, the minimum supported Rust version, a WebAssembly build check, the API docs, the dependency audit, the semver check, the Python wheels and type check, CodeQL, the boundary rules as tested semgrep rules, and the docsite floor, which builds the site and checks its links, its published URLs, its figures and its worked examples. The practice for a new check is to plant the violation it exists for and watch it fail before trusting it.

grade9.2density0.48compression0.50

The model review is a cold re-read. Every pull request is also read by Claude against the review gates in the pr-review skill. That reader is a model of the same family as the author, so it is a cold re-read, not independent verification: it shares the author's blind spots, and a separate reader of the same family has not been shown to catch more than the author's own second pass would. What it adds is a reading of intent, the part of a change no check can see. Its verdict is not a required status check. Claude does not merge over a "blockers" verdict, and fixes it and asks for a re-read instead; that holds by discipline, and the comment thread on each pull request shows whether it held.

Who is accountable#

grade4.9density0.69compressionnone

Two kinds of act are the owner's alone, whatever trust has built up.

  • Decisions. An enhancement proposal is accepted only when the owner merges the change of its status line to accepted, and an API change proposal only when the owner says so on its issue. Claude writes and argues both, and may merge a proposal while its status is proposed, so it can be read on this site; that decides nothing, and Claude never merges a pull request that accepts one.
  • Releases. Claude prepares a release and dispatches the workflow, which verifies, tags, builds and attests it, then stops at two deployment environments, one per registry, each requiring the owner's approval. That approval is the release decision, because a published version cannot be withdrawn. An agent never approves a deployment.
grade12.7density0.66compression0.58

Everything else merges on Claude's standing authority: once CI is green and the review raises no blockers, Claude merges and leaves a comment giving its reasoning, which is the audit trail. Every commit Claude makes carries a Co-Authored-By trailer naming the model, and every release the release workflow builds carries an attestation that SECURITY.md shows how to verify.

grade8.8density0.55compression0.55

One limit is worth stating plainly. Today every merge, comment and approval is recorded under the owner's GitHub login, so the record alone cannot show which of them the owner made by hand, and no branch rule yet makes an acceptance wait for the owner. The rules above hold by practice. The Identity section of CONTRIBUTING.md sets out how Claude is to get a GitHub identity of its own, and how a required check then holds an acceptance for the owner's review.

Which record a change takes#

grade7.0density0.55compressionnone

The process follows the Rust project's RFC process, and the size of a change sets the weight of its record.

ChangeRecord
A substantial change users notice: the public surface in Rust, Python, the command line or the JSON output; behaviour, including the default model; removing a substantial feature; an architecture boundaryAn enhancement proposal (EPR-NNNN), proposed as a pull request. It may merge as proposed, so it renders for review, and is accepted when the owner merges the change of its status to accepted
A minor addition to the public surface, or a minor removalAn API change proposal: a short issue labelled acp
Tooling, docs, CI, refactors, measured performance work, bug fixesThe pull request itself. A rule it leaves behind lives in the file that governs it
grade10.2density0.59compression0.56

Each accepted proposal gets a tracking issue, labelled tracking, that holds its milestone checklist and links the pull request for each milestone; it closes when the work ships. A plan that several agents carry out in parallel gets an enhancement plan (EPL-NNNN) beside the proposal. An accepted proposal is never rewritten: a change of mind is a new proposal that supersedes it, so the reasoning behind every decision stays readable. The full process and the index of records are in blueprints/README.md.

grade10.1density0.45compression0.63

Proposals and plans are numbered from 0001, from a baseline on 2026-10-04. The records written before it, cited as RFC-NNNN and EP-NNNN, are kept unchanged in blueprints/legacy/: they are the reasons behind the code that exists, and the comments and pages that cite them still mean them.

Where a record converges#

grade12.2density0.46compression0.50

The proposals and plans are rendered on this site, in its Blueprints area, each with its status; it is the one area of the site that shows a status, and the documentation keeps to what ships. Before a record is settled, the owner reads it there in the local development server, where any passage on any page can be selected and commented on. The threads are saved as files in the repository, so the next Claude session reads them, answers in them and changes the record, and the owner reads the answer in the same place. The discussion that shaped a record stays beside it rather than in a chat log that is gone the next day.

How a lesson becomes a check#

grade9.6density0.47compression0.57

When something goes wrong twice, the fix is a check rather than a note, because a note is read only by whoever already knows to look for it. The pattern is the same each time: write the check, plant the failure it exists for and watch it fail, then run it from just check and in CI.

  • The boundary rules were partly a text search and partly review alone, and the search missed whole forms of import. They are now semgrep rules, each tested against a fixture that holds a violation.
  • A check twice passed silently on a machine where the tool it needed was missing. Every check now fails when its tool is absent.
  • When the design records moved to their present process, every citation of a record in the tree changed at once. A check now fails when a cited record has no file of its own kind: a proposal, a plan, or a legacy RFC or EP.
  • The agent and skill files drifted from the code twice: an agent cited a class at a line it had long left, and another counted files that had been removed. A check now fails when a file and line cited under .claude/ points at a missing file or past its end.
grade10.9density0.48compression0.62

What a check cannot see, such as whether a design is sound or whether a paragraph still describes the code it names, stays with review. The repository's own check scripts say which is which: each ends by naming what it examined, so a pass is not mistaken for a reading.

Where the parts live#

PartWhere
The agents.claude/agents/
The skills.claude/skills/
The review command.claude/commands/review.md
The rules Claude works underCLAUDE.md and AGENTS.md
Who decides, and how a release runsCONTRIBUTING.md
The records and the processblueprints/
The boundary rulesBoundary rules
The local checksjustfile and scripts/

What matra gives you

grade6.9density0.65compressionnone

Every value the pipeline returns, by tier. The tables say which language surfaces carry each value: fields cross to Python, methods do not.

1. Structure#

grade10.7density0.50compressionnone

Produced by Engine::analyze, as the document is read and parsed.

Token#

grade3.5density0.75compressionnone

The ten CoNLL-U columns plus one derived flag. All cross to Python.

FieldHolds
id1-based position in the sentence
textthe word as written
lemmadictionary form (approved becomes approve)
posuniversal part of speech (NOUN, VERB, ADJ)
xpostreebank-specific tag, finer grained than pos
featuresmorphology (tense, number, person), as a map from feature to value
headid of the token this one depends on, 0 for the root
depthe dependency relation to that head
depssecondary dependencies, always _ from the UDPipe adapter, whose binding does not surface them
miscannotation
is_punctpunctuation flag
gradenonedensity0.86compressionnone

Token::feat(key) reads one morphological feature. Rust only.

Sentence#

FieldHolds
textsentence text, rebuilt by the provider from its tokens
tokensVec<Token>, id-sorted
word_countnon-punctuation tokens
tree_depthnesting depth, None on a malformed cycle
passivewhether the sentence carries nsubj:pass, nsubjpass, or aux:pass
negationsVec<Negation>: cue id, cue lemma, head id
modalsVec<Modal>: auxiliary id, lemma, head id
clausal_complementsVec<ClausalComplement>: verb id and lemma, ccomp head id, subject when present
root_adverbialsVec<RootAdverbial>: adverbial id and lemma
hearst_pairsVec<HearstPair>: pattern tag plus hypernym and hyponym HearstSpan values
root_moodOption<RootMood>: the root clause's Mood feature and the root's id
grade4.8density0.45compressionnone

All eleven cross to Python. The methods below are Rust only.

MethodReturns
root_token()the token everything else hangs off
head_of(id)the governor of a token
children_of(id)its direct dependents
subtree(id)the whole clause under it
content_tokens()non-punctuation tokens
clausal_complements_in(lexicon)the clausal complements whose verb lemma is in a list you supply
root_adverbials_in(lexicon)the root adverbials whose lemma is in a list you supply

Paragraph, Section, Document#

TypeFields (cross to Python)Methods (Rust only)
Paragraphspan, text, kind, sentences, metricsword_count(), sentence_count()
Sectionheading, level, paragraphsnone
Documentprovenance, sections, metricsparagraphs(), sentences(), tokens()
grade8.0density0.53compressionnone

provenance records the matra version, the schema version ("2.0"), the parser with its model, digest and licence, the format, and the input's path, and the length and SHA-256 of the analysed text.

2. Metrics#

grade12.5density0.53compressionnone

Produced by Engine::analyze, as its last step, into ParagraphMetrics on each paragraph and DocumentMetrics on the document. Every ratio is Option<f64>, and None means not computed, which is distinct from a computed zero. Methodology gives each formula and the exact condition for None.

FieldOnMeasures
readability_gradeParagraphMetricsFlesch-Kincaid grade level, from sentence and syllable length
lexical_densityParagraphMetricscontent words as a share of all words
compression_ratioParagraphMetricsbrotli compressed size over raw size, a repetition proxy
total_words, total_sentences, paragraph_countDocumentMetricsthe counts, as integers
mean_sentence_length, sentence_length_stdDocumentMetricswords per sentence, and its sample standard deviation
vocabulary_ttrDocumentMetricsdistinct lemmas over total words, and it falls as text grows, so documents of different lengths are not comparable on it
nominalization_ratioDocumentMetricsnouns ending in a nominalizing suffix (-tion, -ment, -ness, -ity, -ence, -ance) over all words, a surface proxy for nouns formed from verbs (decide becoming decision)
passive_ratioDocumentMetricsshare of sentences carrying a passive construction
grade11.1density0.63compressionnone

Every measure is per document. Aggregating across documents is the caller's decision: the counts sum, and vocabulary_ttr does not.

3. Summarization#

grade12.6density0.57compressionnone

Free functions over a Document, needing no engine and no parser. Both return a Summary: the document's provenance with the method and n recorded, and sentences, each a SummarySentence (index, span, text, score), in document order, capped at 2,000 sentences.

FunctionRanks by
summarize(doc, n), summarize_with(doc, n, SummaryMethod::TfIdf)mean TF-IDF of the sentence's lemmatized terms, each sentence its own document for IDF (tfidf, the default)
summarize_with(doc, n, SummaryMethod::TextRank)PageRank over a similarity graph of shared content lemmas normalized by log length (textrank)
grade15.5density0.67compressionnone

Python spells them matra.summarize(doc, n=3) and matra.summarize(doc, n=3, method="textrank"), over the Document an engine returned.

grade10.5density0.47compressionnone

TextRank's summary of The Federalist No. 10 at the default length, and every sentence's score it was chosen from:

TextRank: the summary and the scores it came from

TextRank score of each sentence, in document order0.0000.0100.0200.03064748sentence 1sentence 88document order; bands alternate by paragraph
  1. 6Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority.
  2. 47When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens.
  3. 48To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed.
show the data
#¶ScoreSummarySentence
110.0086AMONG the numerous advantages promised by a well constructed Union, none deserves to be more accurately developed than its tendency to break and control the violence of faction.
210.0097The friend of popular governments never finds himself so much alarmed for their character and fate, as when he contemplates their propensity to this dangerous vice.
310.0058He will not fail, therefore, to set a due value on any plan which, without violating the principles to which he is attached, provides a proper cure for it.
410.0112The instability, injustice, and confusion introduced into the public councils, have, in truth, been the mortal diseases under which popular governments have everywhere perished; as they continue to be the favorite and fruitful topics from which the adversaries to liberty derive their most specious declamations.
510.0064The valuable improvements made by the American constitutions on the popular models, both ancient and modern, cannot certainly be too much admired; but it would be an unwarrantable partiality, to contend that they have as effectually obviated the danger on this side, as was wished and expected.
610.0221yesComplaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority.
710.0050However anxiously we may wish that these complaints had no foundation, the evidence, of known facts will not permit us to deny that they are in some degree true.
810.0163It will be found, indeed, on a candid review of our situation, that some of the distresses under which we labor have been erroneously charged on the operation of our governments; but it will be found, at the same time, that other causes will not alone account for many of our heaviest misfortunes; and, particularly, for that prevailing and increasing distrust of public engagements, and alarm for private rights, which are echoed from one end of the continent to the other.
910.0099These must be chiefly, if not wholly, effects of the unsteadiness and injustice with which a factious spirit has tainted our public administrations.
1020.0211By a faction, I understand a number of citizens, whether amounting to a majority or a minority of the whole, who are united and actuated by some common impulse of passion, or of interest, adversed to the rights of other citizens, or to the permanent and aggregate interests of the community.
1130.0145There are two methods of curing the mischiefs of faction: the one, by removing its causes; the other, by controlling its effects.
1240.0191There are again two methods of removing the causes of faction: the one, by destroying the liberty which is essential to its existence; the other, by giving to every citizen the same opinions, the same passions, and the same interests.
1350.0050It could never be more truly said than of the first remedy, that it was worse than the disease.
1450.0068Liberty is to faction what air is to fire, an aliment without which it instantly expires.
1550.0096But it could not be less folly to abolish liberty, which is essential to political life, because it nourishes faction, than it would be to wish the annihilation of air, which is essential to animal life, because it imparts to fire its destructive agency.
1660.0036The second expedient is as impracticable as the first would be unwise.
1760.0095As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed.
1860.0090As long as the connection subsists between his reason and his self-love, his opinions and his passions will have a reciprocal influence on each other; and the former will be objects to which the latter will attach themselves.
1960.0127The diversity in the faculties of men, from which the rights of property originate, is not less an insuperable obstacle to a uniformity of interests.
2060.0100The protection of these faculties is the first object of government.
2160.0182From the protection of different and unequal faculties of acquiring property, the possession of different degrees and kinds of property immediately results; and from the influence of these on the sentiments and views of the respective proprietors, ensues a division of the society into different interests and parties.
2270.0143The latent causes of faction are thus sown in the nature of man; and we see them everywhere brought into different degrees of activity, according to the different circumstances of civil society.
2370.0210A zeal for different opinions concerning religion, concerning government, and many other points, as well of speculation as of practice; an attachment to different leaders ambitiously contending for pre-eminence and power; or to persons of other descriptions whose fortunes have been interesting to the human passions, have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to co-operate for their common good.
2470.0062So strong is this propensity of mankind to fall into mutual animosities, that where no substantial occasion presents itself, the most frivolous and fanciful distinctions have been sufficient to kindle their unfriendly passions and excite their most violent conflicts.
2570.0051But the most common and durable source of factions has been the various and unequal distribution of property.
2670.0093Those who hold and those who are without property have ever formed distinct interests in society.
2770.0028Those who are creditors, and those who are debtors, fall under a like discrimination.
2870.0112A landed interest, a manufacturing interest, a mercantile interest, a moneyed interest, with many lesser interests, grow up of necessity in civilized nations, and divide them into different classes, actuated by different sentiments and views.
2970.0193The regulation of these various and interfering interests forms the principal task of modern legislation, and involves the spirit of party and faction in the necessary and ordinary operations of the government.
3080.0089No man is allowed to be a judge in his own cause, because his interest would certainly bias his judgment, and, not improbably, corrupt his integrity.
3180.0174With equal, nay with greater reason, a body of men are unfit to be both judges and parties at the same time; yet what are many of the most important acts of legislation, but so many judicial determinations, not indeed concerning the rights of single persons, but concerning the rights of large bodies of citizens?
3280.0085And what are the different classes of legislators but advocates and parties to the causes which they determine?
3380.0031Is a law proposed concerning private debts?
3480.0101It is a question to which the creditors are parties on one side and the debtors on the other.
3580.0025Justice ought to hold the balance between them.
3680.0134Yet the parties are, and must be, themselves the judges; and the most numerous party, or, in other words, the most powerful faction must be expected to prevail.
3780.0033Shall domestic manufactures be encouraged, and in what degree, by restrictions on foreign manufactures?
3880.0082are questions which would be differently decided by the landed and the manufacturing classes, and probably by neither with a sole regard to justice and the public good.
3980.0108The apportionment of taxes on the various descriptions of property is an act which seems to require the most exact impartiality; yet there is, perhaps, no legislative act in which greater opportunity and temptation are given to a predominant party to trample on the rules of justice.
4080.0050Every shilling with which they overburden the inferior number, is a shilling saved to their own pockets.
4190.0121It is in vain to say that enlightened statesmen will be able to adjust these clashing interests, and render them all subservient to the public good.
4290.0024Enlightened statesmen will not always be at the helm.
4390.0202Nor, in many cases, can such an adjustment be made at all without taking into view indirect and remote considerations, which will rarely prevail over the immediate interest which one party may find in disregarding the rights of another or the good of the whole.
44100.0110The inference to which we are brought is, that the CAUSES of faction cannot be removed, and that relief is only to be sought in the means of controlling its EFFECTS.
45110.0119If a faction consists of less than a majority, relief is supplied by the republican principle, which enables the majority to defeat its sinister views by regular vote.
46110.0070It may clog the administration, it may convulse the society; but it will be unable to execute and mask its violence under the forms of the Constitution.
47110.0274yesWhen a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens.
48110.0224yesTo secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed.
49110.0093Let me add that it is the great desideratum by which this form of government can be rescued from the opprobrium under which it has so long labored, and be recommended to the esteem and adoption of mankind.
50120.0040By what means is this object attainable?
51120.0052Evidently by one of two only.
52120.0201Either the existence of the same passion or interest in a majority at the same time must be prevented, or the majority, having such coexistent passion or interest, must be rendered, by their number and local situation, unable to concert and carry into effect schemes of oppression.
53120.0045If the impulse and the opportunity be suffered to coincide, we well know that neither moral nor religious motives can be relied on as an adequate control.
54120.0087They are not found to be such on the injustice and violence of individuals, and lose their efficacy in proportion to the number combined together, that is, in proportion as their efficacy becomes needful.
55130.0198From this view of the subject it may be concluded that a pure democracy, by which I mean a society consisting of a small number of citizens, who assemble and administer the government in person, can admit of no cure for the mischiefs of faction.
56130.0219A common passion or interest will, in almost every case, be felt by a majority of the whole; a communication and concert result from the form of government itself; and there is nothing to check the inducements to sacrifice the weaker party or an obnoxious individual.
57130.0087Hence it is that such democracies have ever been spectacles of turbulence and contention; have ever been found incompatible with personal security or the rights of property; and have in general been as short in their lives as they have been violent in their deaths.
58130.0118Theoretic politicians, who have patronized this species of government, have erroneously supposed that by reducing mankind to a perfect equality in their political rights, they would, at the same time, be perfectly equalized and assimilated in their possessions, their opinions, and their passions.
59140.0122A republic, by which I mean a government in which the scheme of representation takes place, opens a different prospect, and promises the cure for which we are seeking.
60140.0085Let us examine the points in which it varies from pure democracy, and we shall comprehend both the nature of the cure and the efficacy which it must derive from the Union.
61150.0210The two great points of difference between a democracy and a republic are: first, the delegation of the government, in the latter, to a small number of citizens elected by the rest; secondly, the greater number of citizens, and greater sphere of country, over which the latter may be extended.
62160.0192The effect of the first difference is, on the one hand, to refine and enlarge the public views, by passing them through the medium of a chosen body of citizens, whose wisdom may best discern the true interest of their country, and whose patriotism and love of justice will be least likely to sacrifice it to temporary or partial considerations.
63160.0088Under such a regulation, it may well happen that the public voice, pronounced by the representatives of the people, will be more consonant to the public good than if pronounced by the people themselves, convened for the purpose.
64160.0041On the other hand, the effect may be inverted.
65160.0112Men of factious tempers, of local prejudices, or of sinister designs, may, by intrigue, by corruption, or by other means, first obtain the suffrages, and then betray the interests, of the people.
66160.0108The question resulting is, whether small or extensive republics are more favorable to the election of proper guardians of the public weal; and it is clearly decided in favor of the latter by two obvious considerations:
67170.0143In the first place, it is to be remarked that, however small the republic may be, the representatives must be raised to a certain number, in order to guard against the cabals of a few; and that, however large it may be, they must be limited to a certain number, in order to guard against the confusion of a multitude.
68170.0165Hence, the number of representatives in the two cases not being in proportion to that of the two constituents, and being proportionally greater in the small republic, it follows that, if the proportion of fit characters be not less in the large than in the small republic, the former will present a greater option, and consequently a greater probability of a fit choice.
69180.0163In the next place, as each representative will be chosen by a greater number of citizens in the large than in the small republic, it will be more difficult for unworthy candidates to practice with success the vicious arts by which elections are too often carried; and the suffrages of the people being more free, will be more likely to centre in men who possess the most attractive merit and the most diffusive and established characters.
70190.0071It must be confessed that in this, as in most other cases, there is a mean, on both sides of which inconveniences will be found to lie.
71190.0154By enlarging too much the number of electors, you render the representatives too little acquainted with all their local circumstances and lesser interests; as by reducing it too much, you render him unduly attached to these, and too little fit to comprehend and pursue great and national objects.
72190.0110The federal Constitution forms a happy combination in this respect; the great and aggregate interests being referred to the national, the local and particular to the State legislatures.
73200.0188The other point of difference is, the greater number of citizens and extent of territory which may be brought within the compass of republican than of democratic government; and it is this circumstance principally which renders factious combinations less to be dreaded in the former than in the latter.
74200.0198The smaller the society, the fewer probably will be the distinct parties and interests composing it; the fewer the distinct parties and interests, the more frequently will a majority be found of the same party; and the smaller the number of individuals composing a majority, and the smaller the compass within which they are placed, the more easily will they concert and execute their plans of oppression.
75200.0207Extend the sphere, and you take in a greater variety of parties and interests; you make it less probable that a majority of the whole will have a common motive to invade the rights of other citizens; or if such a common motive exists, it will be more difficult for all who feel it to discover their own strength, and to act in unison with each other.
76200.0080Besides other impediments, it may be remarked that, where there is a consciousness of unjust or dishonorable purposes, communication is always checked by distrust in proportion to the number whose concurrence is necessary.
77210.0156Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic,--is enjoyed by the Union over the States composing it.
78210.0105Does the advantage consist in the substitution of representatives whose enlightened views and virtuous sentiments render them superior to local prejudices and schemes of injustice?
79210.0051It will not be denied that the representation of the Union will be most likely to possess these requisite endowments.
80210.0132Does it consist in the greater security afforded by a greater variety of parties, against the event of any one party being able to outnumber and oppress the rest?
81210.0114In an equal degree does the increased variety of parties comprised within the Union, increase this security.
82210.0080Does it, in fine, consist in the greater obstacles opposed to the concert and accomplishment of the secret wishes of an unjust and interested majority?
83210.0055Here, again, the extent of the Union gives it the most palpable advantage.
84220.0055The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States.
85220.0109A religious sect may degenerate into a political faction in a part of the Confederacy; but the variety of sects dispersed over the entire face of it must secure the national councils against any danger from that source.
86220.0110A rage for paper money, for an abolition of debts, for an equal division of property, or for any other improper or wicked project, will be less apt to pervade the whole body of the Union than a particular member of it; in the same proportion as such a malady is more likely to taint a particular county or district, than an entire State.
87230.0100In the extent and proper structure of the Union, therefore, we behold a republican remedy for the diseases most incident to republican government.
88230.0067And according to the degree of pleasure and pride we feel in being republicans, ought to be our zeal in cherishing the spirit and supporting the character of Federalists.
  • in the summary (3 sentences, matra's default)
  • not picked

TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.

Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: federalist-10/textrank.json.

4. Keyphrases#

grade10.8density0.59compressionnone

Free functions over a Document. Both return a Keyphrases: the document's provenance with the method and n recorded, and phrases, each a Keyphrase (phrase, score, occurrences, the span of every place it was found), highest score first, capped at 200,000 tokens.

FunctionRanks by
keyphrases(doc, n), keyphrases_with(doc, n, KeyphraseMethod::Rake)co-occurrence degree over frequency, on NOUN, ADJ, and PROPN runs between stop words (rake, the default)
keyphrases_with(doc, n, KeyphraseMethod::Yake)per-term position, frequency, and context diversity, assembled into 1-word to 3-word candidates, score inverted so higher is more relevant (yake)
grade14.4density0.78compressionnone

Python spells them matra.keyphrases(doc, n=10) and matra.keyphrases(doc, n=10, method="yake").

grade8.0density0.50compressionnone

The two rank by different properties of the text, and on The Federalist No. 10 their top twelve lists share no phrase:

RAKE and YAKE on the same text

RAKE rank against YAKE rank for the top phrases of eachRAKE rankYAKE rank1101001000substantial occasion present1many judicial determination2federal constitution form3many lesser interest4lesser interest5adequate control6alone account6american constitution6attractive merit6candid review6civilized nation6dangerous vice61fate contemplate propensity2tendency break3develop tendency break4tendency break control5fate contemplate6character fate contemplate7complaint foundation evidence8contemplate propensity9contemplate propensity dangerous10break control violence11annihilation air essential12tendency
show the data
PhraseRAKE rankRAKE scoreYAKE rankYAKE score
fate contemplate propensitynot ranked132553.6
substantial occasion present19.00026940.451
many judicial determination27.83326420.468
tendency breaknot ranked222722.0
develop tendency breaknot ranked322609.3
federal constitution form36.42929830.335
many lesser interest45.87522730.576
tendency break controlnot ranked416940.2
fate contemplatenot ranked54523.1
lesser interest54.04217220.727
adequate control64.00024260.520
alone account64.00014860.823
american constitution64.00017930.703
attractive merit64.00028490.404
candid review64.00014650.840
character fate contemplatenot ranked63038.8
civilized nation64.00021360.615
dangerous vice64.00012940.967
complaint foundation evidencenot ranked71730.8
contemplate propensitynot ranked8463.1
contemplate propensity dangerousnot ranked9455.5
break control violencenot ranked10362.2
annihilation air essentialnot ranked11218.2
tendency2721.00012158.4
  • in RAKE's top 12
  • in YAKE's top 12
  • only one method produces the phrase

Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 466 RAKE and 3111 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.

Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: federalist-10/keyphrases.json.

5. Semantic clusters#

grade7.5density0.55compressionnone

Not a field on any type above. The port and clusters are always compiled; the Model2Vec adapter is behind the model2vec feature.

ItemIs
Embedderthe port: embed(&[&str]) returning one Embedding per text, plus identity()
Model2Vecthe adapter, reading model.safetensors, tokenizer.json, and config.json from a directory. from_dir(dir) loads what is there; potion_base_8m(dir), with dir a models directory, and from_config(cfg) fetch the pinned reference model into its potion-base-8M subdirectory when it is absent
clusters(doc, embedder, threshold)embeds a document's sentences with the embedder you pass, then clusters them; vectors you already hold arrive through an Embedder that returns them
Clustersthe document's provenance with the method (cosine), the threshold and the embedder's identity recorded, clusters, and edges
Cluster, SimilarityEdgea cluster's members (sentence indices), and a pair a, b whose score cleared the threshold
grade12.5density0.68compressionnone

Capped at 2,000 sentences. Python spells them matra.clusters(doc, embedder, threshold=0.85) and Model2Vec (including Model2Vec.potion_base_8m); matra.clusters takes a Model2Vec or any object with embed and identity.

The pipeline surface#

ValueConstructorsCarries
Inputtext(string, format), bytes(data, format), file(path), blocks(blocks)one document: text, bytes, a file, or headings and paragraphs you supply
Inputsdir(path)the files in a directory, as an iterator of Input
Enginenew(), from_config(cfg), from_config_with_notice(cfg, notice)analyze one input, analyze_all of many
Configresolve(), from_file(path), from_sources(...), with_model_dir(dir)where models live and what the defaults are, plus the ValueSource each value came from
grade12.4density0.58compressionnone

Engine::new() is the no-setup constructor: locate the model directory from the environment, then build the standard pipeline, fetching the pinned English model if it is not on disk. It reads no configuration file; Engine::from_config(&Config::resolve()?) reads one.

grade10.8density0.66compressionnone

engine.analyze_all(inputs) returns a lazy iterator of per-document results, one per input in order, and one bad file does not abort the stream. Python's Engine.analyze_all(dir=path) is the same lazy walk, yielding a Document or a DocumentError dict per file.

Traceability#

grade11.0density0.55compression0.48

Every heading, paragraph, sentence and token carries a span into the analysed text, in UTF-8 bytes, and skipped lists what the reader saw and did not analyse (a table, a code block, front matter), each with its span. Every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span, so nothing is dropped without a trace. The text fields are normalized: Paragraph.text is what the parser reads, a markdown paragraph's inline markup removed, and Sentence.text is rebuilt by the provider, the UDPipe adapter joining the token forms with a space unless a token carries SpaceAfter=No. The span, not the text, says where a value came from. Each paragraph is parsed on its own and its sentences are attached to it, so the chain from a token up through its sentence, paragraph, and section holds by construction rather than by text matching.

Where to go next#

grade9.2density0.73compressionnone

Domain model is the full type graph. Methodology is every formula. Errors is every failure.

Domain types

grade10.8density0.63compressionnone

matra::domain holds every type the library hands back. The module depends on serde, thiserror, and the standard library, and on nothing else, so a caller can depend on these types without inheriting a C++ toolchain.

grade10.6density0.53compressionnone

It is not a copy of the generated item-level documentation. For that, run cargo doc --no-deps --all-features.

The type graph#

grade2.6density0.73compressionnone

Five types nest, each owning the next. Nothing is owned twice.

Containment from Document to Token, stored fields against computed methods stored fields computed on call Document Section Paragraph Sentence Token provenance · sections · metrics (DocumentMetrics) · skipped heading · level · span · paragraphs span · text · kind · sentences · metrics (ParagraphMetrics) word_count · sentence_count span · text · tokens · word_count · tree_depth · passive · six structural fields root_token · head_of · children_of · subtree · content_tokens the ten CoNLL-U columns · span · is_punct

Scroll sideways to see the whole diagram.

grade11.6density0.54compression0.53

The left column is what the pipeline writes into the struct. The right column is recomputed from the tree every time you ask, and an empty right column means the type only holds. So there is no second list of paragraphs anywhere in Document: the flat iterators walk the section tree. Every number a reader needs from a document is stored, in DocumentMetrics and ParagraphMetrics and in a sentence's word_count, tree_depth and passive, because a field crosses the language boundary and a method does not.

grade16.9density0.57compression0.52

The derived structural types (Negation, Modal, ClausalComplement, RootAdverbial, RootMood, HearstPair with its HearstSpan and HearstPattern) sit inside the nest, owned by Sentence, and the metric records DocumentMetrics and ParagraphMetrics are owned by Document and Paragraph. Provenance, with its Identity and ProvenanceSource, is owned by Document, and so is skipped, a list of Skipped records, each a SkippedKind and a Span. Every heading, paragraph, sentence and token holds a Span, and a paragraph its ParagraphKind. The other types sit outside it:

TypeRole
Blocka heading or paragraph a caller supplies, the input to Input::blocks; never returned
Summary, SummarySentenceoutput of summarize and summarize_with
Keyphrases, Keyphraseoutput of keyphrases and keyphrases_with
Clusters, Cluster, SimilarityEdgeoutput of clusters, described in Semantic clusters
Parameterswhat shaped an outcome, recorded in its provenance
Formatwhich reader a document needs
Errorevery failure the library can return
ErrorDetails, DetailValuethe values a caller acts on in an error, by name: what Error::details returns
DocumentErrorone document's failure, an Err item of analyze_all's stream
Embeddingone vector from an Embedder, the input to clustering
ProvisionNoticewhat a model fetch is about to download, handed to the _with_notice constructors' callback, described in Errors
grade7.6density0.38compressionnone

Where each of those enters and leaves the pipeline is drawn in Architecture.

Attributes shared across the module#

TypeDerives#[non_exhaustive]
TokenDebug, Clone, Serialize, Deserializeyes
TokenBuildernoneno
SentenceDebug, Clone, Serialize, Deserializeyes
ParagraphDebug, Clone, Serialize, Deserializeyes
SectionDebug, Clone, Serialize, Deserializeyes
DocumentDebug, Clone, Serialize, Deserializeyes
DocumentMetrics, ParagraphMetricsDebug, Clone, Default, PartialEq, Serialize, Deserializeyes
ProvenanceDebug, Clone, PartialEq, Serialize, Deserializeyes
ParametersDebug, Clone, Default, PartialEq, Serialize, Deserializeyes
Identity, ProvenanceSourceDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
Summary, SummarySentence, Keyphrases, KeyphraseDebug, Clone, PartialEq, Serialize, Deserializeyes
Clusters, Cluster, SimilarityEdgeDebug, Clone, PartialEq, Serialize, Deserializeyes
FormatDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
SpanDebug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserializeyes
Skipped, SkippedKind, ParagraphKindDebug, Clone, Copy, PartialEq, Eq, Serialize, Deserializeyes
BlockDebug, Clone, PartialEq, Eqno
ErrorDebug, thiserror::Error; Serialize by handyes
ErrorDetailsDebug, Clone, Default, PartialEq, Eq; Serialize by handno
DetailValueDebug, Clone, PartialEq, Eq, Serializeyes
Negation, Modal, ClausalComplement, RootAdverbialDebug, Clone, Serialize, Deserializeyes
RootMoodDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
HearstPair, HearstSpanDebug, Clone, PartialEq, Eq, Serialize, Deserializeyes
HearstPatternDebug, Clone, Copy, PartialEq, Eq, Serialize, Deserializeyes
DocumentErrorDebug; Serialize by handyes
EmbeddingDebug, Clone, PartialEq, Serialize, Deserializeno
ProvisionNoticeDebug, Clone, PartialEq, Eqyes
grade5.8density0.55compressionnone

Block is the one input type. It has no serde derives and no public fields, so it needs no #[non_exhaustive]: a caller builds one with Block::heading or Block::paragraph and only matra reads it. Its JSON form is the blocks format, whose schema is spec/schema/v2/blocks.schema.json.

grade14.7density0.60compression0.50

No type in the module derives Hash, and the parse tree derives no equality: two Token, Sentence, Paragraph, Section or Document values cannot be compared with ==. Format derives PartialEq and Eq, which is what the reader table keys on, and the metric records, the provenance types, Span, Skipped, the kind enums, Block, RootMood, the Hearst types, the outcome types, Embedding, ProvisionNotice, ErrorDetails and DetailValue derive PartialEq. Error has no Clone, no equality and no serde derives; its Serialize is written by hand, as DocumentError's and ErrorDetails' are. ErrorDetails has no public fields, like Block, so it needs no #[non_exhaustive]. Embedding is the one public type with public fields and without #[non_exhaustive]: on a tuple struct the attribute would make the constructor private, and an Embedder implemented outside the crate has to build one.

grade10.3density0.63compression0.62

#[non_exhaustive] has two effects outside the crate. Struct literal syntax is unavailable, so you build values through the associated constructors listed below. The outcome types have no public constructor: matra produces them, and serde reads them back. Matches on Format, ParagraphKind, SkippedKind and Error need a catch-all arm, because variants can be added in a minor release.

grade7.6density0.48compressionnone

Every field on these types is pub and readable. TokenBuilder and Block are the exceptions: their fields are private, and you reach them through the builder's setters and the block's constructors.

Token#

grade9.4density0.92compressionnone

One parsed token carrying the full CoNLL-U annotation set plus one derived flag.

FieldTypeCoNLL-U columnContents
idusize1Position within the sentence, 1-based
spanSpannoneWhere the surface form lies in the analysed text, in UTF-8 bytes
textString2Surface form
lemmaString3Dictionary form
posString4Universal POS tag, for example NOUN, VERB, ADJ
xposString5Language-specific POS tag
featuresBTreeMap<String, String>6Morphological features as a map from feature name to value
headusize7Id of the governing token; 0 marks the root
depString8Dependency relation to the head, for example nsubj, obj, aux:pass
depsString9Enhanced dependency graph
miscString10Miscellaneous annotations, for example SpaceAfter=No
is_punctboolderivedWhether the token is punctuation
grade12.3density0.64compressionnone

Provider-specific notes for the UDPipe adapter, the one adapter that ships:

  • is_punct is set when the UPOS tag is PUNCT.
  • deps is always the string _, the CoNLL-U empty marker. The underlying udpipe-rs binding does not surface column 9.
  • xpos and misc carry whatever the model emits, and can be _ or empty. features is converted from the column: Mood=Ind|Tense=Pres becomes two entries, a multi-valued feature (Case=Nom,Acc) keeps its value unsplit, and _ becomes an empty map.
grade12.9density0.48compressionnone

A token id or head that does not fit in usize is rejected at the adapter with Error::ParseFailed rather than being silently coerced.

grade17.1density0.53compression0.50

span comes from aligning each form with the paragraph text the parser was given, then mapping it into the analysed text. A form the adapter cannot find where the text says it should be is Error::ParseFailed naming the token, never a guessed span, and the engine checks every parser's spans by the same rule, so a span from any parser that does not line up with its form is Error::ParseFailed with reason token_form_mismatch. In a markdown paragraph whose markup the reader removed, a token's span can cover markup inside it (foo**bar** is one token from the f to the last r), and a token read from an escape or an entity (&amp;) covers all of it.

Construction#

let token = matra::domain::Token::builder(
        1,                       // id
        "committee".to_string(), // text
        "committee".to_string(), // lemma
        "NOUN".to_string(),      // pos
        3,                       // head
        "nsubj".to_string(),     // dep
    )
    .xpos("NN".to_string())
    .feats("Number=Sing".to_string())
    .is_punct(false)
    .build();
grade10.3density0.71compression0.54

Token::builder(id, text, lemma, pos, head, dep) -> TokenBuilder takes the six CoNLL-U essentials. TokenBuilder carries one setter per optional field (xpos, deps, misc, each taking String; feats, taking the CoNLL-U column 6 string and converting it into features; is_punct taking bool; span taking a Span), each consuming and returning the builder. build() returns the Token. Unset string fields default to empty, features to an empty map, is_punct to false, and span to the empty span at 0.

Methods#

SignatureReturnsBehavior
feat(&self, key: &str)Option<&str>The value of one feature in features, by exact name. None when the feature is absent
grade9.6density0.55compression0.64

feat("Mood") returns Some("Ind") when the token carries Mood=Ind. The value is what the provider emitted, so a multi-valued feature (Case=Nom,Acc) comes back unsplit: matra does not normalise it. A token whose column 6 was _ or empty has no features, so every lookup on it returns None.

grade8.7density0.53compressionnone

feat is Rust-only by design. features crosses FFI as a map, so a lookup over it adds no information to the wire; a Python or TypeScript caller indexes the same map (RFC-0009).

Sentence#

FieldTypeContents
spanSpanWhere the sentence lies in the analysed text: from its first token's start to its last token's end, derived at construction
textStringSentence text rebuilt from the token forms, normalized, not a slice of the input
tokensVec<Token>Tokens in ascending id order
word_countusizeNon-punctuation tokens, derived at construction
tree_depthOption<usize>Longest path from any token to the root, the root at depth 0, derived at construction. None when the head references form a cycle
passiveboolWhether any token's dep is nsubj:pass, nsubjpass, or aux:pass, derived at construction
negationsVec<Negation>Negation cues, derived at construction
modalsVec<Modal>Modal auxiliaries, derived at construction
clausal_complementsVec<ClausalComplement>Verbs governing a clausal complement (ccomp), each with its subject when the parse has one, derived at construction
root_adverbialsVec<RootAdverbial>Adverbial modifiers attached to the root, derived at construction
hearst_pairsVec<HearstPair>Candidate hypernymy pairs from the six Hearst (1992) patterns, filled by the pipeline at the annotate stage
root_moodOption<RootMood>The root clause's Mood feature with the root's token id, derived at construction. None when the root clause carries no mood
grade13.6density0.60compression0.54

Invariants that downstream code relies on: tokens are id-sorted; exactly one token has head == 0; every other head names a token in the same sentence; following head never loops. Every sentence the engine builds holds them, because it checks what the parser returns before it builds one and fails the document with Error::ParseFailed naming the check otherwise. A sentence built by hand with Sentence::new, which trusts its caller, or read back from JSON can break any of them, and the tree walks stay safe when the last one breaks (see tree_depth below).

grade9.3density0.55compression0.59

tree_depth is computed in time linear in token count with a visited set, and is None when the head references form a cycle, which reports a malformed parse as a value a reader must handle instead of truncating it or returning a number that reads as a very deep tree. An empty sentence has depth Some(0). word_count counts tokens the NLP provider identified, minus punctuation. It is not a whitespace split, and the two counts differ on the same text.

grade11.9density0.56compression0.56

The engine builds text by joining the token surface forms a parser returned, inserting a space unless the preceding token carries SpaceAfter=No in misc. The string is a reconstruction, not a slice of the input, so whitespace can differ from the source text; span is where the sentence came from.

Derived structural fields#

grade14.8density0.57compression0.55

The fields after tokens are computed once from the dependency graph and serialized with the sentence, so every language binding and the CLI's JSON read one Rust detection as data (RFC-0008). Most are computed at Sentence::new; hearst_pairs is the exception, filled by the pipeline at the annotate stage because its detector lives in the hearst module outside the domain, so a hand-built Sentence carries an empty vector until the caller runs hearst::hypernymy_pairs. Each field reports structure only; the reading is the caller's.

grade11.3density0.58compression0.59

Negation reports one cue by token id: cue_id, cue_lemma (not, never, no, neither, nor), and head_id, the token the cue attaches to. A cue fires only on its carrying relation (advmod, det, cc), so nothing as a subject does not fire and tokenizer-split cannot fires exactly once, on the not token. Whether the negation reverses the sentence's claim is not matra's call.

grade12.2density0.56compression0.50

Modal reports one modal auxiliary by token id: aux_id, aux_lemma, and head_id, the token the auxiliary attaches to (0 when the modal is itself the root). The closed class is the ten lemmas the UD English treebank tags MD: can, could, may, might, must, ought, shall, should, will, would, matched by lemma on the aux relation or the AUX part of speech. The part-of-speech arm catches the shapes the model does not park on aux: VP ellipsis promotes the modal to root ("Yes, you should.") and coordinated auxiliaries parse as root plus aux or conj ("He can and will succeed."). Precision holds because the model's only other AUX lemmas (be, have, do, get) are outside the class. Contracted forms fire on their lemma (wo in won't lemmatizes to will), the noun will never fires because it never carries aux or AUX, and a multi-auxiliary chain (might have been done) reports its modal with head_id on the content verb. English modals are ambiguous across readings, and resolving must into obligation or inference takes context matra does not model, so no epistemic, deontic, or dynamic category is assigned: the auxiliary and its arc are reported, the reading is the caller's.

grade8.2density0.57compression0.51

RootMood reports the grammatical mood of the root clause: root_id, the root's token id, and mood, the Mood feature as the model emitted it (Ind, Imp, and an open set beyond). The feature is read off the root token or, when the root carries none, off its first cop, aux, or aux:pass child that does, because the model parks finiteness on the auxiliary in copular ("The sky is blue."), do-support ("He did leave.") and passive clauses. A modal carries VerbForm=Fin and no mood, so a clause whose root a modal governs ("You must complete the form.") usually has no root mood; whether a modal governs the root is the separate fact in modals. It reads the root clause only, so a mood in a subordinate clause does not reach it. The bare assertoric form, an indicative root no modal governs, is mood == "Ind" with no modal whose head_id is root_id.

grade8.4density0.57compression0.48

ClausalComplement reports one verb governing a clausal complement by token id: verb_id and verb_lemma for the verb, ccomp_id for the head of the complement, and subject_id plus subject_lemma for the verb's subject (nsubj, nsubj:pass) when the parse has one in the same sentence. The construction fires for every verb that fills it ("Smith reported that the effect vanished.", "These results suggest that the mechanism is shared.", "We show that the method works." all match), because verbs taking a clausal complement are an open class and any list matra shipped would be incomplete while looking authoritative. Which verb lemmas count as reporting or evidential is the caller's lexicon, supplied as a parameter to clausal_complements_in or applied to the field directly in any language. A ccomp under a non-verb head ("I am sure that it works." parses the complement under the adjective) is outside the construction by decision. The subject is optional by observed fact: UDPipe splits "Smith et al. reported that ..." into two sentences at the period in "et al.", stranding the attribution in the previous sentence while the verb keeps its ccomp. That upstream segmentation defect is recorded here and in the test suite rather than papered over.

grade12.0density0.59compression0.50

RootAdverbial reports one adverbial modifier attached to the root by token id: adv_id and adv_lemma. The root-attached advmod arc is where sentence-scope adverbs land ("Reportedly, the deal closed."), but the dependency graph does not distinguish sentence scope from manner ("The team quickly shipped." puts quickly on the same arc), so every root-attached adverbial is reported and the caller's lexicon selects the evidential ones, via root_adverbials_in or a filter over the field. Negation particles on the same arc ("not" in "The plan is not ready.") appear in both negations and root_adverbials; both report arcs, neither assigns a reading, and the overlap is intentional.

grade18.3density0.61compression0.46

HearstPair reports one candidate hypernymy pair from the six lexico-syntactic patterns of Hearst (1992), detected as dependency-arc patterns rather than regex over surface text: NP such as NP, such NP as NP, NP, including NP, NP, especially NP, NP and other NP, NP or other NP. The pattern field carries a HearstPattern tag naming the construction that matched (such_as, such_np_as, including, especially, and_other, or_other on the wire), and hypernym and hyponym each carry a HearstSpan: head_id and head_lemma for the noun phrase head, plus first_id and last_id for the contiguous token range covering the head and its adjacent nominal modifiers, with the pattern's own marker words (such, other) excluded, so traceability holds against tokens. Precision is favoured over recall: each detector requires the full verified arc shape, so a clausal coordination ("She invited John, and other guests brought gifts."), a definite contrastive coordination ("The teacher and the other students laughed."), or an emphasized plain coordination ("France and especially Spain") does not fire, and a sentence the model misparses is missed rather than misreported. The pair is a candidate by design: matra reports that the sentence used a construction which conventionally signals hypernymy, and does not build a taxonomy or assert the relation is true.

Methods#

grade8.8density0.57compressionnone

Most methods below are walks over the head and dep columns; the two _in filters are views over the derived fields against a caller-supplied lexicon. Concepts draws one parsed sentence as the tree these walk.

SignatureReturnsBehavior
new(text: String, tokens: Vec<Token>)SentenceConstructor. Derives span and the structural fields from tokens. Does not validate the invariants
content_tokens(&self)Vec<&Token>Tokens where is_punct is false
root_token(&self)Option<&Token>The token with head == 0, if there is one
children_of(&self, id: usize)Vec<&Token>Tokens whose head equals id. Empty for an unknown id
head_of(&self, id: usize)Option<&Token>The governing token. None for the root and for an unknown id
subtree(&self, id: usize)Vec<&Token>The token and all its descendants, sorted by id. Empty for an unknown id
clausal_complements_in(&self, lexicon: &[&str])Vec<&ClausalComplement>Clausal complements whose verb_lemma is in the caller-supplied lexicon
root_adverbials_in(&self, lexicon: &[&str])Vec<&RootAdverbial>Root adverbials whose adv_lemma is in the caller-supplied lexicon
grade9.1density0.64compressionnone

subtree carries a visited set, so it terminates on cyclic input.

Paragraph#

FieldTypeContents
spanSpanWhere the paragraph lies in the analysed text, trailing whitespace excluded. For markdown it covers the quote and list markers on its lines
textStringThe text the parser reads, normalized: for markdown the inline text with the markup removed (emphasis, link and image syntax, code span backticks, quote and list markers), soft line breaks as \n, escapes and entities decoded; for plain text and block input a verbatim slice
kindParagraphKindParagraph, Quote (inside a block quote at any depth) or ListItem; paragraph, quote, list_item on the wire
sentencesVec<Sentence>Sentences from parsing this paragraph
metricsParagraphMetricsThe paragraph's three measures
gradenonedensity0.75compressionnone

ParagraphMetrics holds the three:

FieldTypeContents
readability_gradeOption<f64>Flesch-Kincaid grade level, when the paragraph holds more than ten words
lexical_densityOption<f64>Content-word ratio, 0.0 to 1.0, when the paragraph holds a word
compression_ratioOption<f64>Brotli compressed size over original size, when the paragraph holds more than fifty words
grade16.4density0.59compressionnone

Each slot holds None until Engine::compose fills it, and stays None when the paragraph does not meet the metric's applicability condition; no outer Option wraps the record, because each slot gates on its own.

grade13.5density0.60compressionnone

Engine::annotate skips quotes when it parses, so a paragraph whose kind is ParagraphKind::Quote reaches compose with no sentences and keeps all three slots at None.

Methods#

SignatureReturnsBehavior
new(text: String, kind: ParagraphKind, span: Span)ParagraphConstructor. Empty sentences, every metric None
word_count(&self)usizeNon-punctuation tokens summed across this paragraph's sentences
sentence_count(&self)usizeNumber of sentences

Section#

FieldTypeContents
headingOption<String>Heading text, its markup removed
levelusizeHeading depth
spanOption<Span>Where the heading lies in the analysed text, markers included; None with no heading
paragraphsVec<Paragraph>Paragraphs in document order
grade7.5density0.53compression0.48

The markdown reader sets level to the heading's level (the number of # characters, or 1 and 2 for a setext heading) and heading to its text. Content before the first heading becomes a section with heading: None and level: 0. Plain text produces at most one section, always with heading: None and level: 0. Block input opens a section at each Block::heading, at the level it names.

grade12.2density0.61compressionnone

Section::new(heading: Option<String>, level: usize, span: Option<Span>, paragraphs: Vec<Paragraph>) -> Section is the constructor. There are no other methods.

Document#

gradenonedensity0.40compressionnone

The output of the pipeline.

FieldTypeContents
provenanceProvenanceWhere the document came from and what produced it, recorded by Engine::annotate
sectionsVec<Section>The section tree, which owns every paragraph, sentence, and token
metricsDocumentMetricsThe document's aggregates, filled by Engine::compose
skippedVec<Skipped>What the reader saw and did not analyse, in document order
grade9.9density0.60compressionnone

DocumentMetrics holds what the command line's table prints and what a binding would otherwise rebuild:

FieldTypeContents
total_wordsusizeNon-punctuation tokens across every sentence
total_sentencesusizeSentences across every paragraph
paragraph_countusizeParagraphs, quotes included
mean_sentence_lengthOption<f64>total_words over total_sentences. None with no sentences
sentence_length_stdOption<f64>Sample standard deviation of sentence length in words. None with fewer than two sentences, where it is undefined
vocabulary_ttrOption<f64>Type-token ratio over lemmas. Falls as text grows, so documents of different lengths are not comparable on it
nominalization_ratioOption<f64>Share of nominalizing nouns
passive_ratioOption<f64>Passive sentences over total sentences. None with no sentences
grade13.1density0.55compressionnone

Before the document is measured every count is 0 and every ratio None; None after it means the document did not meet the metric's condition, never a computed zero. Engine::analyze always measures, so this holds for a document you build by hand with Document::new.

grade11.2density0.55compression0.49

Provenance holds matra, the version that produced the document; schema, the schema version as "major.minor" (SCHEMA_VERSION, "2.0"), which appears nowhere else; parser, the parser's Identity; format, the Format the input was read as; and source, a ProvenanceSource with the input's path (None for anything but a file), and the length in bytes and the sha256 of the analysed text every span points into, computed once at annotation. For markdown and plain text that is the input; for block input, the blocks joined. On a document, method and parameters are None and absent from the JSON; an outcome derived from the document carries a copy of its provenance with both set (see Summary). There is no timestamp, so two analyses of the same input by the same build are identical. Identity names a model-backed component: name (udpipe), model (english-ewt-ud-2.5-191206), sha256 of the model artifact, and license. The UDPipe adapter derives it from the bytes it loaded, so a model matra does not pin reports its digest with model and license set to None; an embedder reports one in the same shape (model2vec for Model2Vec), recorded in a cluster result's parameters.embedder. Provenance::new(parser, format, source) fills matra and schema from the build.

Methods#

SignatureReturnsBehavior
new(provenance: Provenance, sections: Vec<Section>, skipped: Vec<Skipped>)DocumentConstructor. An empty metric record
paragraphs(&self)impl Iterator<Item = &Paragraph>Every paragraph, in document order
paragraphs_mut(&mut self)impl Iterator<Item = &mut Paragraph>The mutable form, which the engine uses as it parses and measures
sentences(&self)impl Iterator<Item = &Sentence>Every sentence across every paragraph
tokens(&self)impl Iterator<Item = &Token>Every token across every sentence

Summary#

grade12.3density0.58compressionnone

What summarize and summarize_with return. The selector is matra::SummaryMethod, at the crate root (TfIdf, the default, and TextRank, #[non_exhaustive]).

FieldTypeContents
provenanceProvenanceThe document's, with method (tfidf or textrank, an open set) and parameters (n)
sentencesVec<SummarySentence>At most n sentences, in document order
gradenonedensity0.71compressionnone

SummarySentence is one sentence a summary keeps:

FieldTypeContents
indexusizeThe sentence's position among the document's sentences (Document::sentences), counting from 0
spanSpanWhere it lies in the analysed text, as on the Sentence
textStringIts text, as on the Sentence
scoref64The method's score, higher is more representative
grade9.8density0.51compressionnone

Both summarizers select the top n by score and then return them in ascending index, so the result reads in document order while score carries the ranking. A score compares only with another from the same method.

grade15.5density0.53compression0.57

Parameters records what shaped an outcome, each field present only when the method takes it: n (Option<usize>) for a summary and keyphrases, threshold (Option<f64>) and embedder (Option<Identity>, the identity the embedder reports, the same shape as the parser's) for clusters. In JSON an absent parameter is an absent key, so a summary's is {"n": 3}.

Keyphrases#

grade13.3density0.60compressionnone

What keyphrases and keyphrases_with return. The selector is matra::KeyphraseMethod (Rake, the default, and Yake, #[non_exhaustive]).

FieldTypeContents
provenanceProvenanceThe document's, with method (rake or yake, an open set) and parameters (n)
phrasesVec<Keyphrase>At most n phrases, highest score first; equal scores in the order the phrases are first met, the same on every run
gradenonedensity0.80compressionnone

Keyphrase is one ranked phrase:

FieldTypeContents
phraseStringThe phrase, built from lowercased lemmas joined by single spaces
scoref64Relevance score, higher is more relevant
occurrencesVec<Span>Every stretch of the analysed text it was found in, in document order, from its first word's start to its last word's end

Clusters#

grade14.2density0.60compressionnone

What clusters returns: provenance (the document's, with method cosine and parameters threshold and embedder), clusters (Vec<Cluster>, each members, sentence indices) and edges (Vec<SimilarityEdge>, each a, b and score: f64). Semantic clusters describes each field and what co-membership does and does not mean.

Format#

gradenonedensity0.60compressionnone

Which reader a document needs.

VariantWire nameAssigned to
MarkdownmarkdownExtension .md or .markdown
PlainTextplain_textEvery other extension, and a file with no extension
BlocksblocksNo extension: a caller names it, and Input::blocks takes it
PdfpdfExtension .pdf, or bytes opening with %PDF
DocxdocxExtension .docx, or a zip archive holding [Content_Types].xml
Other(String)the name it holdsNever detected: a caller names it, built with Format::named
grade10.7density0.49compression0.60

Format::from_path(path: impl AsRef<Path>) -> Format is the extension column, and it is the only copy of it. Reading a file with Input::file applies it, and the command line reads it for the name given to --stdin-filename when --format names none, so a name means the same thing on both routes. When the bytes are read, a PDF or DOCX signature outranks both the extension and a format the caller names.

grade12.9density0.53compression0.51

Pdf, Docx and an Other have no reader that ships, so analyzing such a document returns Error::UnsupportedFormat with that format, unless the engine was given a reader for it with Engine::with_reader. Other names a format matra has no variant for, the one a caller's own reader reads: Format::named("html"). Format::named gives the variant matra has for a name it knows, so Format::named("markdown") is Markdown, and a name read back from JSON does the same, an unknown one reading as Other rather than failing. Input::text and Input::file_as take the format as an argument; Input::bytes takes one optionally; Input::file detects one. Format derives PartialEq and Eq, which is what the reader table keys on.

Block#

grade10.3density0.57compressionnone

A heading or a paragraph a caller supplies, for a document matra has no reader for. Input only: built by Block::heading(level: u8, text: impl Into<String>) and Block::paragraph(text: impl Into<String>), handed to Input::blocks, and never returned. It has no public fields.

grade4.5density0.52compression0.56

The analysed text of block input is the blocks' texts joined by a blank line ("\n\n"), in order, and every span in the result points into it. A heading opens a section at its level; a level of 0 is refused with Error::InvalidInput when the input is analysed. The same blocks as JSON are the blocks format: an array of {"kind": "heading", "level": 1, "text": "..."} and {"kind": "paragraph", "text": "..."} objects, with an unknown kind or field refused by name.

Span#

FieldTypeContents
byte_startusizeThe first byte
byte_endusizeOne past the last byte
grade7.3density0.55compression0.54

A stretch of the analysed text in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is the stretch. Bytes rather than characters: a caller in a language that indexes strings by character converts. The unit is in the field names because a byte span used to slice a character-indexed string, a Python str, is right on ASCII and wrong, with no error, after the first character outside it. Span::new(byte_start, byte_end) is the constructor.

Skipped#

FieldTypeContents
kindSkippedKindWhat the construct is
spanSpanWhere it lies in the analysed text, trailing whitespace excluded
grade10.8density0.68compression0.61

One construct the reader saw and did not analyse. SkippedKind names it: Table, Code, FrontMatter, Html, ThematicBreak, LinkDefinition (table, code, front_matter, html, thematic_break, link_definition on the wire), a set a reader treats as open. Skipping is specified behaviour beside a correct result, not an error. Skipped::new(kind, span) is the constructor.

grade16.6density0.61compressionnone

Together with the heading and paragraph spans, skipped accounts for the input: every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span.

DocumentError#

FieldTypeContents
pathOption<PathBuf>Where the failure occurred, None for input that is not a file
errorErrorWhat went wrong
grade10.6density0.76compression0.60

One per-document failure: the Err item Engine::analyze_all yields in the failed input's place, so documents plus errors equals inputs consumed. Display prints path: error when a path exists and the bare error otherwise; source() returns the wrapped Error. DocumentError::new(path, error) is the constructor. Not Clone, because Error wraps std::io::Error.

grade5.6density0.53compressionnone

It serializes as its error's form plus the path, {"path", "kind", "message", "details"}, with path null for input that is not a file and written lossily where it is not UTF-8. The JSON Schema is spec/schema/v2/error.schema.json.

grade14.5density0.56compressionnone

matra collects nothing across documents: a stream is the collection, and how to combine documents is the caller's decision (a count sums; vocabulary_ttr does not).

Constants, aliases, and the error type#

ItemDefinitionRole
limitsmodule of usize constantsEvery cap and metric minimum, below
SCHEMA_VERSION&str, "2.0"The document schema version this build emits, as "major.minor", written into Provenance::schema. The JSON Schema for each major is in spec/schema/
Result<T>std::result::Result<T, Error>Return type of every fallible library function
Errorenum, #[non_exhaustive]Every failure the library returns. See Errors
ErrorDetailsordered names and values; get(key), iter(), len(), is_empty()What Error::details returns: the values a caller acts on, by name. Serializes as an object. See kinds and details
DetailValueenum, #[non_exhaustive]: Text(String), Integer(u64), Bool(bool)One value in an ErrorDetails. Serializes as the bare value

Limits#

grade10.5density0.54compression0.51

domain::limits holds every bound matra holds its work to, each defined once and read where it applies. They are published so you can size work before a call instead of learning a bound from a refusal: split an input under MAX_INPUT_BYTES, or chunk a document's sentences under the cap of the method you select, each named for that method as SummaryMethod and KeyphraseMethod name it. They are constants, not settings: no configuration changes them. They are this release's values and may change in any 0.x release, so read them from domain::limits rather than copying the numbers. A cap that is exceeded returns InputTooLarge, whose limit detail is the constant and whose what detail names the gate; a paragraph outside a metric's bounds keeps that metric None. Each constant's rustdoc carries the cost model behind its value.

ConstantValueBounds
MAX_INPUT_BYTES8,388,608 (8 MiB)Text one input may hold; the "input" and "file_read" gates
MAX_CONFIG_FILE_BYTES65,536 (64 KiB)The configuration file; the "config_file_read" gate
MAX_DOWNLOAD_BYTES67,108,864 (64 MiB)Any one model download; the "parser_model_download" and "embedding_model_download" gates
MAX_TFIDF_SENTENCES2,000Sentences a TF-IDF summary ranks; "tfidf"
MAX_TEXTRANK_SENTENCES2,000Sentences a TextRank summary ranks; "textrank"
MAX_CLUSTER_SENTENCESMAX_TEXTRANK_SENTENCESSentences or vectors clustered; "clusters". Defined as TextRank's, since both build the same quadratic graph
MAX_RAKE_TOKENS200,000Tokens RAKE reads, punctuation included; "rake"
MAX_YAKE_TOKENS200,000Tokens YAKE reads, punctuation included; "yake"
MAX_COMPRESSION_PARAGRAPH_BYTES262,144 (256 KiB)The largest paragraph given a compression_ratio
MIN_READABILITY_WORDS11The fewest words for a readability_grade (more than 10)
MIN_COMPRESSION_WORDS51The fewest words for a compression_ratio (more than 50)
grade10.5density0.47compressionnone

Two caps with the same value stay two constants where their costs differ, so tuning one does not move the other: TF-IDF's 2,000 is not TextRank's, and YAKE's 200,000 is not RAKE's.

What crosses the language boundary#

grade6.8density0.50compressionnone

The Python surface serializes values with pythonize. Fields have a serde representation and cross with their names unchanged. Methods have none, so there is nothing for them to cross with.

Fields cross the FFI boundary, methods stop at it Rust Python Document Document stored fields dict keys provenance sections metrics skipped provenance sections metrics skipped methods paragraphs() sentences() tokens() FFI boundary pythonize + serde a Python caller walks these in the sections it already has

Scroll sideways to see the whole diagram.

grade15.1density0.53compression0.55

Token::feat, Document::sentences, Sentence::subtree, and every other method in the tables above are available to Rust callers only. The two methods on Error, kind and details, are different: their values are computed into the error's serde form and onto the Python exception, so they cross as fields. Nothing a reader needs is method-only: every aggregate is a field of DocumentMetrics or ParagraphMetrics, and a sentence carries its word_count, tree_depth and passive, so each crosses with the record that holds it.

Rust typePython shape
TokenToken
SentenceSentence
ParagraphParagraph
SectionSection
Documentmatra.Document, read-only, whose to_dict() is a DocumentDict
DocumentMetrics, ParagraphMetricsDocumentMetrics, ParagraphMetrics
Provenance, Identity, ProvenanceSourceProvenance, Identity, ProvenanceSource
ClausalComplement, RootMoodClausalComplement, RootMood
Span, SkippedSpan, Skipped
ParametersParameters
Summary, SummarySentenceSummary, SummarySentence
Keyphrases, KeyphraseKeyphrases, Keyphrase
Clusters, Cluster, SimilarityEdgeClusters, Cluster, SimilarityEdge
DocumentErrorDocumentError
Errormatra.ErrorInfo, and the exception classes under matra.MatraError
ErrorDetailsErrorDetails
BlockBlockDict, a HeadingBlock or a ParagraphBlock
Formatits name, a FormatName
grade9.4density0.59compression0.50

Document and DocumentError are the two items Python's Engine.analyze_all yields. A Document stays a Rust value behind a read-only handle, and every read of it crosses through serde like every shape above it, with one field replaced: its provenance.source.path is decoded with the platform's filesystem decoding (os.fsdecode on Unix), so a name that is not valid UTF-8 crosses exactly instead of as serde's lossy string. A DocumentError crosses through serde too, as {"path", "kind", "message", "details"}, with its path decoded the same way. kind is Error::kind on the domain type, pinned across bindings by spec/tests/corpus/items.json and exported to Python as matra.ERROR_KINDS; message is the Rust error's own Display text; details is Error::details. An Error raised rather than returned crosses as a Python exception carrying the same three, kind, str(exc) and details: a method is computed into a value before it crosses, as an aggregate is. A future binding takes the same serde form.

grade10.7density0.58compressionnone

The Python shapes are TypedDict declarations available at runtime. They are declared in matra.types and re-exported from the package root, so either import works:

from matra.types import DocumentDict, Keyphrases, Summary
from matra import DocumentDict, Keyphrases, Summary
grade8.2density0.84compressionnone

Field names and nesting match the Rust types exactly. Option<f64> becomes float | None; Option<String> becomes str | None; Vec<T> becomes list[T]; usize becomes int.

Platforms and models

grade7.6density0.66compressionnone

What each of matra's install routes needs and installs, and the models matra downloads. Install matra has the steps; One core, three install routes explains the choices behind them.

Install routes#

RouteCommandBuilds from sourceFeatures in the build
Rust librarycargo add matra@0.3yesudpipe (the default)
CLI binarycargo install matra --version '^0.3' --features cliyesudpipe cli
Python packagepip install 'matra>=0.3'only where no wheel fitsudpipe model2vec python cli
grade13.5density0.46compressionnone

The Python package installs the library and the matra command, which is the Rust CLI reached through the extension module, and has no runtime dependencies of its own.

Toolchains#

Needed forRequirement
Every route that builds from sourceRust 1.88 or later, the minimum supported Rust version
Every route that builds from sourceA C++ compiler: UDPipe is a C++ library, which udpipe-rs builds during the cargo build. A C compiler alone is not enough.
The Python packageCPython 3.12 or later

Python wheels#

PlatformWheel
Linux x86_64, Linux aarch64manylinux2014: glibc 2.17 or newer, which covers Debian 11, Ubuntu 20.04, RHEL 8 and Amazon Linux 2
macOS x86_64, macOS arm64yes
Windowsnone; the UDPipe build under MSVC is unverified, so the sdist route there is untested
grade4.7density0.64compression0.57

The wheels are built against the CPython stable ABI and tagged abi3, so one wheel per platform serves 3.12 and every later 3.x on GIL-enabled CPython. A free-threaded interpreter (python3.14t, for one) accepts a different tag, abi3t, so it gets no wheel on any platform and builds the sdist; pyo3 gains the free-threaded stable ABI at 3.15. Wherever no wheel fits, pip builds the sdist, which needs the toolchains above.

grade6.7density0.59compressionnone

Every release file from 0.2.1 on carries a build provenance attestation; Verifying a release has the commands.

Version requirements#

RequirementUsed byAccepts
matra@0.3, --version '^0.3'cargo0.3.0 or a later release semver-compatible with it, so not 0.4.0
'matra>=0.3'pip, uv, uvx0.3.0 and every later release
grade4.5density0.45compression0.60

An install line with no version takes the newest release on the registry. 0.1.0 is still published, and its matra command is a separate Python implementation with its own --json shape and a model directory fixed at ~/.matra/models. 0.2 is published too, with the surface Migrate from 0.2 to 0.3 moves you off.

Build features#

grade7.6density0.54compressionnone

matra --version prints the version, then the features the build was compiled with:

matra 0.3.0
features: udpipe cli
grade11.5density0.67compressionnone

That is the CLI binary from cargo install. The Python package prints features: udpipe model2vec python cli. Every analysis command behaves the same in both builds except matra clusters, which needs model2vec and refuses, naming it, in a build without; cargo install matra --features cli,model2vec builds one with it.

The model directory#

grade9.1density0.61compressionnone

The library, the CLI and the Python package resolve the model directory the same way, first match wins:

  1. A directory passed explicitly: --model-dir on the command line, Config::with_model_dir in Rust, or the directory argument of the constructors that take one, such as Model2Vec::potion_base_8m.
  2. MATRA_MODEL_DIR.
  3. The models directory of the data root, which is MATRA_DATA_DIR, else $XDG_DATA_HOME/matra, else ~/.local/share/matra.
  4. A pre-existing, non-empty ~/.matra/models from an older install, when the location in 3 does not exist yet. matra never creates ~/.matra; a selected legacy cache is used as the model directory, downloads included.
grade8.5density0.63compressionnone

Where matra keeps things has the config file and data root beside it, and matra config show prints which rung each value came from.

The model download#

Parsing modelEmbedding model
Fetched byEngine::new and Engine::from_config, in Rust and in Python, and the command line, on first useModel2Vec::potion_base_8m, Model2Vec::from_config, their Python forms and matra clusters, on first use
Size16.3 MB30.9 MB, three files
Verified againsta SHA-256 pinned in the sourcea SHA-256 over the three files, pinned in the source
Download cap64 MiB64 MiB per file
grade13.1density0.54compression0.50

The download is held in memory, checked against the pinned hash, and written only if it matches, so nothing unverified reaches the model directory and an interrupted transfer leaves nothing behind. A mismatch downloads once more; a second mismatch is an error, and nothing is loaded. A cached parsing model that fails the check is replaced only when new bytes verify, so a re-download that cannot reach the network leaves the file that was there. The embedding model is never written over files already in its directory; Provisioning has what it does instead.

grade6.7density0.57compressionnone

The library prints nothing while it downloads. The command line prints two lines on stderr before either model's download, naming the model, its size, where it is going and its license; --quiet silences them. Errors has what each failure returns.

Model licenses#

grade7.2density0.50compressionnone

matra's code is MIT. The models it downloads are separate files, published by other people under licenses of their own.

ModelUsed forSourceLicense
english-ewt-ud-2.5-191206.udpipeevery parseLINDAT record 11234/1-3131, published by ÚFAL, Charles UniversityCC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)
potion-base-8Msemantic clusters onlyminishlab/potion-base-8M on Hugging FaceMIT, per its model card
grade9.6density0.56compression0.61

The parsing model's license does not permit commercial use. The UDPipe models page states that the UDPipe 1 models it publishes for Universal Dependencies are distributed under CC BY-NC-SA. This page reports what the licenses say and does not interpret them; read the license text for the terms.

grade7.8density0.55compression0.59

The command line and the config file always use this parsing model: --model-dir, MATRA_MODEL_DIR and the other settings change where it is stored, not which model it is. models.udpipe, which earlier releases wrote into the config file, is gone, and a file that still sets it fails to load, saying so. No surface loads a different UDPipe model file: the model is a pin.

Semantic clusters

grade8.4density0.53compressionnone

The value semantic clustering returns, the model it embeds with, and its bounds. Cluster sentences by meaning has the calls; What the clustering threshold does explains what a cluster means.

The value#

grade11.4density0.68compressionnone

clusters(&doc, &embedder, threshold) and Python's matra.clusters(doc, embedder, threshold) return a Clusters, a standalone value and never a field on Document. Its JSON Schema is spec/schema/v2/clusters.schema.json.

Clusters
  provenance   the document's, plus method "cosine" and parameters:
                 threshold   the cutoff you supplied
                 embedder    the embedder's identity: name, model, sha256, license
  clusters     each: member sentence indices
  edges        every pair that cleared the threshold
TypeFieldHolds
ClustersprovenanceThe document's Provenance, with method and parameters. parameters.embedder is the Identity the embedder reports, the shape provenance.parser has: for Model2Vec, name model2vec, sha256 the digest over its three artifacts (the value its model_hash returns), and model potion-base-8M and license MIT when that digest is the pinned model's, None otherwise. parameters.threshold is the cosine cutoff the call was given
clustersThe connected components, ordered by smallest member index
edgesEvery pair whose cosine cleared the threshold, ordered by a, then b; both ends lie in one cluster
ClustermembersSentence indices in document order, sorted, at least two
SimilarityEdgea, bThe two sentence indices, a < b
scoreTheir cosine similarity, an f64
grade5.3density0.45compressionnone

A sentence with no pair above the threshold is in no cluster. Two members of one cluster need not share an edge: they can be joined through a chain of edges.

grade11.5density0.45compressionnone

The threshold is recorded exactly as it was passed. A score is computed in the vectors' own f32 and widened to f64, so it carries more digits than the model resolved: compare scores with a tolerance, not for equality with a number you typed.

grade17.1density0.56compressionnone

Vectors you already hold go through clusters with an Embedder that returns them, in Rust and in Python alike: the clustering always runs over a document's sentences, so its result always carries a provenance.

The model#

grade12.2density0.62compressionnone

The Model2Vec adapter, behind the model2vec feature, loads static embedding models in the model2vec artifact format: an embedding matrix (model.safetensors), a tokenizer.json, and a config.json in one directory. clusters and the Embedder port are always compiled; with your own Embedder implementation, clustering needs no feature at all.

ConstructorRustPythonReaches the network
The configured directoryModel2Vec::from_config(&cfg)Model2Vec.from_config(), or Model2Vec.from_config(config_file)when the pinned model is absent
The directory the environment names, or the default one; no config file readModel2Vec::potion_base_8m(dir) with the directory you resolveModel2Vec.potion_base_8m()when the pinned model is absent
A models directory you name, the model in its potion-base-8M subdirectoryModel2Vec::potion_base_8m(dir)Model2Vec.potion_base_8m(dir)when the pinned model is absent
Files you supplyModel2Vec::from_dir(dir)Model2Vec.from_dir(dir)never
grade18.7density0.69compressionnone

The configured directory is models.embedding (potion-base-8M as shipped) inside the configured model directory.

The reference model#

grade9.2density0.55compressionnone

potion-base-8M, 30.9 MB in three files, MIT per its model card. The SHA-256 over its three files, concatenated in the order above, is a constant compiled into the library:

81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4
grade12.0density0.54compressionnone

It is the digest spec/tests/semantic/reference-model.json pins and the value model_hash reports once the model is loaded. For files loaded with from_dir, model_hash is identity rather than verification: it names the artifacts that produced a score, and says nothing about whether they are the ones you meant to fetch.

Provisioning#

  • The three artifacts are verified as a set, in memory, before anything is parsed. A mismatch downloads once more without writing anything; a second mismatch raises with the directory as the call found it.
  • Downloading happens only into a directory holding none of the three file names. If all three are there and the digest does not match, or only some of them are, the call raises and names the directory, having downloaded nothing and deleted nothing.
  • Each artifact download is capped at 64 MiB.

Inference#

grade11.9density0.53compression0.58

A static model is a lookup table: inference is a row gather, a mean, and a normalize, and the vectors are bit-identical on every platform and in every language binding. Every text is capped at 512 tokens, pre-truncated on bytes and then truncated on token ids, so a longer text is embedded from its opening only. An empty text embeds to the zero vector.

Bounds and failure#

ConditionRustPython
More than 2,000 sentences, checked before the embedding pass runsInputTooLarge, what is "clusters"matra.InputTooLargeError, a ValueError
Vectors that disagree on dimension, a non-finite value, a non-finite thresholdInvalidInputmatra.InvalidInputError, a ValueError
An embedder returning the wrong number of vectorsInvalidInputmatra.InvalidInputError, a ValueError
grade7.9density0.57compression0.65

The 2,000 cap counts the document's sentences. The contract violations mean the call site or the embedder is wrong, never the text. A zero-magnitude vector has no defined cosine with anything, so it gets no edge at any threshold and no cluster. Errors has the full table, the provisioning failures included.

Use matra from Rust

gradenonedensity0.43compressionnone

You already have matra in your Cargo.toml.

grade9.7density0.56compressionnone

Every snippet below assumes it runs inside a function returning matra::domain::Result<()>, which is why ? works.

The no-setup path#

gradenonedensity0.67compressionnone

One line, no arguments, no model on disk yet:

use matra::Engine;

let engine = Engine::new()?;
grade12.3density0.53compression0.58

Engine::new locates the model directory from the environment, downloads the English UDPipe model into it if it is not already there, and wires the readers this build ships. The directory is MATRA_MODEL_DIR if you set it, otherwise the models subdirectory of $XDG_DATA_HOME/matra, which defaults to ~/.local/share/matra. It reads no configuration file. Programming model has the full resolution order.

grade9.3density0.45compressionnone

When you want your configuration file read, or the resolved values but not all of them, hold the Config yourself:

use matra::config::Config;

let cfg = Config::resolve()?.with_model_dir("/opt/matra/models");
let engine = Engine::from_config(&cfg)?;
grade10.9density0.51compression0.57

with_model_dir puts the directory on the Argument rung, which outranks the environment and the defaults, and cfg.sources() reports where every other value came from. Config is also what Model2Vec::from_config reads. Engine::from_config_with_notice takes a closure called once before each model download, with what is about to arrive and its licence, so a caller that renders can say so.

Where the model lives#

grade13.7density0.51compression0.52

The parser is UDPipe with one pinned English model, english-ewt-ud-2.5-191206, and the engine builds it for you, so there is no parser to construct unless you bring your own (below). Name a directory you own: provisioning trusts what it finds there, and a shared one such as /tmp is writable by anybody on the machine. The engine creates the directory if it is missing, downloads the model (about 16 MB) on first use, verifies it against a pinned SHA-256 hash, and loads from the cached file on every call after that. The verified bytes are written to a temporary subdirectory whose name is unique to the call (the process id, the clock in nanoseconds, and a counter) and moved into place with a single rename, so two processes pointed at the same directory cannot corrupt each other's file, even in containers where every process is pid 1.

grade13.4density0.52compression0.59

The path is passed to std::fs::create_dir_all unchanged, and ~ in it is taken literally: neither Rust nor matra expands it, so with_model_dir("~/models") creates a directory literally named ~ under your current working directory. The same holds for MATRA_MODEL_DIR and the other variables when the shell has not already expanded them. Pass an absolute path, or expand the home directory yourself.

grade7.4density0.51compression0.57

Construction is the expensive step: it loads and validates the model once. Build one engine and reuse it for every document. An Engine is Send but not Sync: it moves to another thread, but it is not shared between two, and a stream borrowed from it stays on the thread that holds it. That is a compile error, not a runtime surprise. To parallelize, build one engine on each worker thread, or serialize calls behind a mutex.

The pipeline#

grade7.4density0.70compressionnone

matra's public Rust surface is one pipeline assembled from two values. Input says what one document is; Engine says what happens to it.

use matra::domain::Format;
use matra::{Engine, Input, Inputs};

let engine = Engine::new()?;

// One document.
let doc = engine.analyze(Input::text("Plain text in memory.", Format::PlainText))?;
println!("{} sentences", doc.metrics.total_sentences);

// Many documents, lazily and in order. A failure is one item, and the
// walk goes on past it.
for item in engine.analyze_all(Inputs::dir("./corpus")?) {
    match item {
        Ok(doc) => println!("{:?}", doc.provenance.source.path),
        Err(e) => eprintln!("skipped: {e}"),
    }
}
grade5.8density0.56compressionnone

Input has five constructors. None of them reads anything or fails: the read, the format and the size cap are settled when the engine reaches the input.

ConstructorThe document isFormat from
Input::text(string, format)the text itselfthe argument
Input::bytes(data, format)bytes from anywherea PDF or DOCX signature, else the argument, else plain text
Input::file(path)the file, read when analyseda PDF or DOCX signature, else the extension
Input::file_as(path, format)the file, read when analyseda PDF or DOCX signature, else the argument
Input::blocks(blocks)headings and paragraphs you supply (Block::heading, Block::paragraph)blocks
grade9.1density0.58compression0.55

.md and .markdown read as markdown; any other extension, or none, reads as plain text when the bytes are UTF-8. Format::Blocks names the same headings and paragraphs as a JSON array, for Input::text, Input::bytes or Input::file_as, since no extension implies it, with the schema in spec/schema/v2/blocks.schema.json. Inputs::dir(path) lists one level of a directory, up front, and is an iterator of Input::file; its Err is the listing failing. Input::file on a directory fails with Error::InvalidInput naming Inputs::dir.

gradenonedensity0.71compressionnone

Engine exposes the pipeline at two grains:

MethodTakesReturnsUse it when
analyzeone InputResult<Document>one document, end to end
analyze_allanything yielding Inputa lazy stream of Result<Document, DocumentError>any number of documents; a failure is one item carrying the path, and the stream goes on
grade15.3density0.58compression0.60

The grains always agree: analyze_all is analyze mapped over the inputs, for every Input constructor. Inside, analyze reads and parses the document, then measures it; those two stages are the engine's own, and the library's test suite pins that analysis is exactly the one followed by the other, so the grains cannot drift apart silently.

grade10.9density0.54compression0.56

The readers this build ships are markdown, plain text and block input. Format::Pdf and Format::Docx are recognised, by extension and by signature, and have no reader: analysing one returns Error::UnsupportedFormat with that format, and so do bytes that are not UTF-8 and carry no signature matra recognises (with format: None). The signature outranks the extension and the format you name, so a PDF renamed notes.txt is still refused as a PDF rather than read as text.

What the size gates reject#

grade9.1density0.51compressionnone

Seven caps guard the pipeline and the functions over its output. Each one returns Error::InputTooLarge, and the what field names which gate fired so you can route on the label instead of guessing from context.

whatLimitFires in
"input"8 MiB (domain::limits::MAX_INPUT_BYTES)the engine's reading stage, which is the only route from text to the parser, on the bytes handed in and on the analysed text, so every Input constructor and every pipeline call inherits it
"file_read"8 MiB on diskInput::file, checked against file metadata before any read
"tfidf"2000 sentencessummarize, and summarize_with with SummaryMethod::TfIdf
"textrank"2000 sentencessummarize_with with SummaryMethod::TextRank
"rake"200000 tokens summed across all sentenceskeyphrases, and keyphrases_with with KeyphraseMethod::Rake
"yake"200000 tokens summed across all sentenceskeyphrases_with with KeyphraseMethod::Yake
"clusters"2000 sentencesclusters, before the embedding pass
grade9.4density0.57compressionnone

Three more labels guard what matra reads that is not your text: "parser_model_download" and "embedding_model_download" cap a model download at 64 MiB, and "config_file_read" caps the config file at 64 KiB. Errors lists all ten.

grade9.4density0.48compression0.57

A document from disk passes the gates in sequence: the metadata check when the file is read, then the byte and text checks before it is parsed. Block input is checked on its analysed text, the blocks joined, so blocks each under the cap that join to more than it are refused. A file at exactly the cap is accepted; one byte over is rejected.

grade8.1density0.54compressionnone

The outcome caps are counted over the parsed document, so a document under 8 MiB can still exceed the TF-IDF sentence cap. Check doc.metrics.total_sentences before calling if you need to decide between summarizing and chunking.

Parse once, use many#

grade13.5density0.54compression0.57

The parse inside analyze is the expensive step: it runs the parser once per paragraph. The outcome functions are free functions over the Document it returns, and never parse anything themselves, so they need no engine and run in a build without UDPipe. Analyze once, and hand the same document to as many outcomes as you need:

use matra::{Input, KeyphraseMethod, SummaryMethod};
use matra::domain::Format;

let doc = engine.analyze(Input::text(text, Format::Markdown))?;

let summary = matra::summarize(&doc, 3)?;
let ranked = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Yake)?;
grade8.7density0.56compressionnone

Each outcome reads the same document; nothing re-parses. The sentences it reads are the ones the reader kept, so markdown headings, code, tables and quotes never reach a summary as if they were prose.

grade12.0density0.65compression0.57

The method is optional. summarize runs TF-IDF and keyphrases runs RAKE, the defaults config/default.toml ships; summarize_with and keyphrases_with take a SummaryMethod or KeyphraseMethod, both #[non_exhaustive]. Every result is a record carrying the document's provenance plus the method that ran (tfidf, textrank, rake, yake) and the parameters it ran with (n), because the method decides what a score means: TF-IDF and TextRank, or RAKE and YAKE, score on unrelated scales.

FunctionReturnsHolds
summarize, summarize_withSummarysentences, at most n, in document order: each a SummarySentence with its index among the document's sentences, its span, text and score
keyphrases, keyphrases_withKeyphrasesphrases, at most n, highest score first: each a Keyphrase with its phrase (lowercased lemmas), score and occurrences, the span of every place it was found
clustersClustersclusters of sentence indices and every edges pair that cleared the threshold; see Cluster sentences by meaning
grade14.0density0.53compressionnone

Keyphrases with equal scores come back in the same order on every run, the order the phrases are first met, so a tie at the requested count is decided the same way each time.

Why a metric slot is None#

grade11.9density0.54compressionnone

Every ratio in ParagraphMetrics and DocumentMetrics is an Option, and each metric has its own threshold. A None means the metric declined to run, not that it computed nothing. The counts in DocumentMetrics (total_words, total_sentences, paragraph_count) are plain integers.

SlotFilled when
ParagraphMetrics::readability_gradethe paragraph has more than 10 words and is not a quote
ParagraphMetrics::lexical_densitythe paragraph has at least 1 word and is not a quote
ParagraphMetrics::compression_ratiothe paragraph has more than 50 words, is not a quote, and its text is at most 256 KiB
DocumentMetrics::vocabulary_ttrthe document's attached sentences hold at least one non-punctuation token
DocumentMetrics::nominalization_ratiothe same condition as vocabulary_ttr
DocumentMetrics::passive_ratio, DocumentMetrics::mean_sentence_lengththe document holds at least one sentence
DocumentMetrics::sentence_length_stdthe document holds at least two sentences
grade12.9density0.65compressionnone

"Words" here means Paragraph::word_count(), the non-punctuation token count summed over the paragraph's attached sentences. A paragraph whose sentences vector is empty has a word count of zero and therefore no per-paragraph metrics at all.

grade10.1density0.58compression0.58

Quotes are never parsed. A paragraph inside a block quote, at any depth, has kind ParagraphKind::Quote; the parser never sees it, so its sentences stays empty, its metric slots stay None, and it contributes nothing to total_words or passive_ratio, though it counts in paragraph_count. It stays in the section tree with its text, and its span covers the > markers.

What the markdown reader skips#

grade10.5density0.56compression0.59

A markdown document (by format argument or by .md extension) is read as CommonMark, with tables, strikethrough, task lists and front matter. Headings, paragraphs, list items and quotes become the section tree; a # with no space after it, as in #hashtag, is text, and ## References is a heading like any other. Each paragraph's text is what the parser reads, with the inline markup removed (emphasis, link and image syntax, the backticks of a code span) and escapes and entities decoded.

grade12.3density0.45compressionnone

What it does not analyse it records in Document::skipped, each entry a SkippedKind and a span, never dropping it silently:

  • FrontMatter: YAML between --- lines, or TOML between +++ lines, at the very start
  • Code: fenced and indented code blocks
  • Table: a table, header and rows
  • Html: an HTML block
  • ThematicBreak: ---, *** or ___ on a line of its own
  • LinkDefinition: a link reference definition, [label]: https://example.com
grade10.8density0.65compressionnone

Every byte of the input that is not whitespace lies in exactly one heading span, paragraph span or skipped span. The test suite checks that over every conformance fixture and thousands of generated documents.

Spans#

grade7.3density0.55compression0.53

Every heading (Section::span), paragraph, skipped construct, sentence and token carries a Span { byte_start, byte_end } into the analysed text, in UTF-8 bytes, so &text[span.byte_start..span.byte_end] is what it came from. For markdown and plain text the analysed text is the input; for block input it is the blocks' texts joined by a blank line. The text fields are normalized (a sentence's text is rebuilt from its tokens, a paragraph's has its markup removed); the span is the authority on where a value came from. provenance.source records the analysed text's length and SHA-256.

Handle domain::Error#

grade20.1density0.58compressionnone

analyze and the outcome functions return domain::Result<T>, an alias for Result<T, domain::Error>. The stream wraps the same error with its path: analyze_all yields DocumentError, whose path field is Some for a file input and whose error field is the domain::Error to match on:

use matra::domain::Error;

for outcome in engine.analyze_all(matra::Inputs::dir("notes/")?) {
    match outcome {
        Ok(doc) => { /* doc.metrics */ }
        Err(doc_err) => match doc_err.error {
            Error::InputTooLarge { limit, actual, what } => {
                eprintln!("{what} gate: {actual} bytes over the {limit} cap");
            }
            Error::UnsupportedFormat { format, .. } => {
                eprintln!("no reader for {format:?}");
            }
            Error::Io(e) => eprintln!("could not read the file: {e}"),
            Error::ParseFailed(msg) => eprintln!("the parser failed: {msg}"),
            e => eprintln!("analysis failed: {e}"),
        },
    }
}
grade12.7density0.57compression0.58

Error::ParseFailed also carries panics that crossed the UDPipe FFI boundary. The adapter wraps the C-side call in catch_unwind and converts a panic into ParseFailed with the payload message, so a bug inside UDPipe surfaces as a matchable error instead of aborting your process. It is also what the parser returns when it cannot place a token in the text it was given, rather than reporting a span it guessed.

grade7.5density0.58compression0.52

To pass an error on to another program, use what every channel carries rather than the message. e.kind() is a stable string such as "input_too_large". e.details() holds the values to act on by name (every dynamic value in the message, and on invalid_input the reason naming the check that failed), so e.details().get("limit") reads a cap without matching the variant. serde_json::to_value(&e) is {"kind", "message", "details"}, and a DocumentError serializes the same way with its path beside them. Errors lists the keys for every kind.

grade13.3density0.52compression0.60

domain::Error is #[non_exhaustive], so a trailing _ => (or a named catch-all binding, as above) is required even once you have named every variant that exists today. A future variant becomes a compile requirement for code that matches without a wildcard, and a routed catch-all for code that has one.

grade2.3density0.46compressionnone

The domain structs carry #[non_exhaustive] for the same reason. From outside the crate you cannot write Token { .. } or Sentence { .. } as struct literals. Use the constructors, which matters most when you are building fixtures for tests:

Token::builder(id: usize, text: String, lemma: String, pos: String, head: usize, dep: String) -> TokenBuilder
Sentence::new(text: String, tokens: Vec<Token>) -> Sentence
grade9.7density0.58compression0.58

Every string argument is an owned String, not a &str, so a literal needs .to_string() on the way in. TokenBuilder takes the remaining CoNLL-U columns through xpos, feats, deps and misc, each also a String, plus is_punct for the derived flag and span for where the token lies in its text, and build() hands back the Token. Sentence::new derives the sentence's span from its first and last token. feats takes column 6 as CoNLL-U writes it and converts it into the token's features map:

use matra::domain::Token;

let token = Token::builder(
    1,
    "approved".to_string(),
    "approve".to_string(),
    "VERB".to_string(),
    0,
    "root".to_string(),
)
.feats("Mood=Ind|Tense=Past".to_string())
.build();

assert_eq!(token.feat("Tense"), Some("Past"));
assert_eq!(token.features.get("Mood").map(String::as_str), Some("Ind"));

Analyze a directory#

grade9.9density0.73compressionnone

Inputs::dir lists one level of a directory, and analyze_all streams it, one result per file:

let (documents, errors): (Vec<_>, Vec<_>) = engine
    .analyze_all(matra::Inputs::dir("./corpus")?)
    .partition(Result::is_ok);

let words: usize = documents
    .iter()
    .flatten()
    .map(|doc| doc.metrics.total_words)
    .sum();
println!("{} documents, {words} words", documents.len());
for err in errors.iter().filter_map(|r| r.as_ref().err()) {
    eprintln!("skipped: {err}");
}
grade10.2density0.54compression0.59

Inputs::dir is Err only when the listing itself fails, such as the directory not existing. Everything else is an Err item carrying its path: an unreadable file as Io, a file over the cap as InputTooLarge with what = "file_read", a non-UTF-8 file, a PDF or a DOCX (by signature or by name) as UnsupportedFormat, and a parser failure as ParseFailed. Documents plus errors always equals inputs consumed, and each document names its file in provenance.source.path.

grade13.4density0.54compression0.55

Two kinds of entry are skipped without an error entry, so they will not appear in either list: subdirectories, because the walk is one level deep, and symlinks, which are filtered out with symlink_metadata so a link cannot redirect the read elsewhere. Files are attempted in sorted path order. How to aggregate across documents is yours to decide: a count such as total_words sums, and a ratio such as vocabulary_ttr does not, since it falls as a document grows.

grade8.0density0.42compressionnone

You do not have to collect. The stream from analyze_all is lazy, so a loop that renders each document as it completes holds one document in memory at a time, not the corpus.

Walk a Document#

grade16.4density0.74compressionnone

Document holds a sections: Vec<Section> tree; sections hold paragraphs: Vec<Paragraph>; paragraphs hold sentences: Vec<Sentence>. Flat iterators cut across the tree when you do not need the structure:

for sentence in doc.sentences() {
    println!("{}", sentence.text);
}
gradenonedensity0.56compressionnone

doc.paragraphs() and doc.tokens() flatten at the other two levels.

grade18.5density0.58compressionnone

A Sentence carries its dependency tree in tokens: Vec<Token>, where each token's head field points at another token's id in the same sentence and head == 0 marks the root. Navigate it with the methods Sentence provides rather than walking tokens by hand:

for sentence in doc.sentences() {
    let Some(root) = sentence.root_token() else { continue };
    println!("root: {} ({})", root.text, root.pos);

    for child in sentence.children_of(root.id) {
        println!("  {} --{}--> {}", child.text, child.dep, root.text);
    }
}
grade10.0density0.63compression0.59

Token fields carry the CoNLL-U columns under Rust names: text is the surface form, pos is the universal POS tag, xpos is the language-specific tag, and dep is the dependency relation. features is the FEATS column as a map from feature name to value (token.feat("Mood") reads one), deps and misc hold their raw column strings, and is_punct is set by the parser.

grade20.9density0.57compressionnone

subtree(id) collects every token reachable from a given token, sorted back into document order, which is the tool for pulling a phrase out from under a dependency relation rather than a single token. Pulling the direct object out of every sentence that has one:

for sentence in doc.sentences() {
    for token in &sentence.tokens {
        if token.dep == "obj" {
            let phrase: Vec<&str> = sentence
                .subtree(token.id)
                .iter()
                .map(|t| t.text.as_str())
                .collect();
            println!("object phrase: {}", phrase.join(" "));
        }
    }
}
grade8.4density0.61compressionnone

matra reports the dependency label UDPipe assigned ("obj" here). It does not interpret what the phrase means. That interpretation is your application's job.

grade14.3density0.61compression0.57

The remaining navigation methods: head_of(id) returns the head token, or None for the root and for an id that is not in the sentence; content_tokens() gives you the non-punctuation view. Three counts are fields, computed when the sentence is built: word_count, the non-punctuation token count; passive, whether any token carries nsubj:pass, nsubjpass, or aux:pass; and tree_depth.

grade10.5density0.54compression0.58

tree_depth is the longest root-to-leaf path in the sentence, an Option<usize>. On a malformed parse where tokens form a cycle it is None rather than a depth truncated at some arbitrary ceiling, so a broken parse is a value the compiler makes you handle rather than a number that reads as a deep tree. subtree and children_of stay safe on the same malformed input; both use a visited set.

Bring your own parser or reader#

grade15.3density0.53compression0.58

The engine is assembled from three parts, and each is a trait you can implement: the parser (matra::nlp::NlpProvider), a reader for one format (matra::decompose::Decomposer), and the source that reads the paths inputs name (matra::source::Source). Plug one in and the rest of the pipeline, every metric and every outcome function, runs on what it produces, with no fork of matra.

grade13.7density0.61compression0.59

A parser takes one paragraph's text and returns its sentences, each a Vec<Token>. It does not build Sentence values: the engine builds them from your tokens, text, span and every derived field included, so passive, negations and mood mean the same thing whichever parser ran. Engine::with_parser takes it, in any build, udpipe or not:

use matra::domain::{Format, Identity, Result, Span, Token};
use matra::nlp::NlpProvider;
use matra::{Engine, Input};

struct Words;

impl NlpProvider for Words {
    fn parse(&self, text: &str) -> Result<Vec<Vec<Token>>> {
        let mut tokens = Vec::new();
        let mut at = 0;
        for word in text.split_whitespace() {
            let start = at + text[at..].find(word).unwrap_or(0);
            at = start + word.len();
            let id = tokens.len() + 1;
            let head = if id == 1 { 0 } else { 1 };
            tokens.push(
                Token::builder(id, word.into(), word.to_lowercase(), "X".into(), head, "dep".into())
                    .span(Span::new(start, at))
                    .build(),
            );
        }
        Ok(if tokens.is_empty() { Vec::new() } else { vec![tokens] })
    }

    fn identity(&self) -> Identity {
        Identity::new("words", None, None, None)
    }
}

let doc = Engine::with_parser(Words).analyze(Input::text("The river rose.", Format::PlainText))?;
grade7.6density0.56compression0.52

The contract is on the trait. In short: ids run 1..=n in each sentence; every head is 0 or a token of the sentence, and exactly one is 0; the heads form no cycle; and every span is where the token's form lies in the paragraph text you were handed, in UTF-8 bytes relative to that text, in token order. Set is_punct for punctuation and SpaceAfter=No in misc where no space follows, since the sentence text is rebuilt from the forms. identity names your parser and its model, and lands in every document's provenance.parser. examples/custom_parser.rs is a runnable version that splits sentences too: cargo run --example custom_parser --no-default-features.

grade12.1density0.52compression0.51

A reader takes a format's bytes and returns a Read: its blocks, built with ReadBlock::heading(level, text, span) and ReadBlock::paragraph(kind, text, span), and the constructs it skipped. A paragraph's text is the analysed text at its span. For a format that is not text, extract the text and name it with Read::with_text, and every span points into that. Register a reader for a format with with_reader, which composes with the standard constructors:

let engine = Engine::new()?
    .with_reader(Format::Pdf, MyPdfReader)          // matra recognises PDF and ships no reader
    .with_reader(Format::named("html"), MyHtmlReader); // a format matra has no name for
let doc = engine.analyze(Input::file_as("page.html", Format::named("html")))?;
grade8.9density0.50compressionnone

A PDF or DOCX is detected by its bytes, so a registered reader for it runs on Input::file with no format named. A Format::Other is never detected: name it on the input, as above. Replacing a shipped reader, say Format::Markdown, works the same way.

grade8.2density0.54compression0.55

A source takes a path and returns the bytes there. with_source swaps it in for the local filesystem, so Input::file("bucket/notes.md") reads through it, the extension still decides the format, and the path is still what provenance.source.path records. Inputs::dir still lists the local filesystem, and Input::bytes and Input::blocks need no source at all.

grade10.8density0.50compression0.49

The engine trusts none of the three. What a reader returns is checked before the section tree is built, and what the parser returns is checked for each paragraph before a sentence is built from it. A break fails that document with Error::ParseFailed, whose reason names the check (head_out_of_range, token_span_order, block_span_invalid, and the rest Errors lists) and whose cause names the paragraph, sentence and token, or the block. That is also how to test an implementation: run your own texts through Engine::analyze, and it conforms when no document fails with one of those reasons. tests/providers.rs in the repository holds a broken parser and a broken reader for each check. A failure of your own is ParseFailed with reason None, or the kind the trait lists for it, and never a panic: a parser wrapping native code catches a panic at its own boundary, as the UDPipe adapter does.

grade1.5density0.38compressionnone

These traits are Rust only in 0.3.0. Python takes an embedder of your own, and not yet a parser, a reader or a source.

Use matra from Python

grade6.2density0.52compression0.56

You installed matra with uv add 'matra>=0.3' or pip install 'matra>=0.3'. That gave you two things: the library described on this page, and the matra command, which is the Rust CLI reached through the extension module rather than a Python program of its own. Everything the command does is in the CLI guide; the Python package adds no behavior to it.

grade7.6density0.56compressionnone

The Python surface spells what Rust spells. You ask an Engine to analyze a document, then ask for an outcome of the result: summarize, keyphrases or clusters.

import matra

engine = matra.Engine()
doc = engine.analyze(file="README.md")
summary = matra.summarize(doc, n=3)
phrases = matra.keyphrases(doc, n=10, method="yake")
print(doc["metrics"]["total_words"])

The no-setup path#

gradenonedensity0.57compressionnone

No argument, no model on disk yet:

import matra

engine = matra.Engine()
grade9.2density0.57compressionnone

Engine() downloads the English UDPipe model on first use, verifies it against a pinned SHA-256 hash, and loads from the cache after that. Before the download it writes two lines to sys.stderr, the ones the command line prints for the same wait:

matra: downloading english-ewt-ud-2.5-191206.udpipe (16.3 MB) into /home/you/.local/share/matra/models
matra: the model is licensed CC BY-NC-SA 4.0 (non-commercial), separately from matra: https://creativecommons.org/licenses/by-nc-sa/4.0/
grade8.8density0.54compression0.56

Once the model is on disk nothing is written. Engine.from_config(), Model2Vec.potion_base_8m() and Model2Vec.from_config() announce a download the same way. Redirect sys.stderr if a notice must not reach your terminal. It finds the model directory from the environment alone: MATRA_MODEL_DIR if you set it, otherwise the models subdirectory of $XDG_DATA_HOME/matra, which defaults to ~/.local/share/matra. A pre-existing ~/.matra/models is used instead when the new location does not exist yet and that one is not empty, so a cache from 0.1.0 keeps working. Programming model has the full resolution order.

grade9.5density0.49compression0.66

To put the model somewhere specific, set MATRA_MODEL_DIR before you build the engine. Name a directory you own: provisioning trusts what it finds there, and a shared one such as /tmp is writable by anybody on the machine. The value is used as a path unchanged, so ~ is taken literally.

grade8.0density0.54compressionnone

Engine.from_config() is the same engine with a configuration file read and checked as well: the one the environment names, or the one you pass, Engine.from_config("matra.toml"), which must exist. The file does not move the model directory, so most callers want Engine().

grade9.1density0.56compression0.53

The two ways provisioning can fail raise different exceptions, and the split is the one the errors reference describes. A download that never arrived, or a directory that could not be written, raises matra.IoError, an OSError. Bytes that arrived and then failed the hash raise matra.ModelInvalidError, a RuntimeError. A bootstrap that wants to survive both catches matra.MatraError. A download that did not arrive carries details["retryable"], which says whether calling again may succeed.

grade4.3density0.50compressionnone

The parser model is pinned. There is no way to load another model file from Python, as there is none from Rust.

grade4.2density0.69compressionnone

Construction is the expensive step. Build one Engine and reuse it across calls.

Threads and processes#

grade7.0density0.53compression0.55

The underlying model holds C-side state that is not thread-safe, so the Rust binding marks the class unsendable. Touching one Engine from any thread other than the one that created it raises pyo3_runtime.PanicException. That exception derives from BaseException, not Exception, so a bare except Exception will not catch it and a thread pool will not report it the way you expect. Do not share an engine across threads. A Document is not tied to a thread.

grade19.8density0.51compressionnone

Separate processes are fine, because each process gets its own model state. Build the engine once per worker with an initializer rather than once per task, so the model is loaded as many times as you have workers and not as many times as you have chunks:

from concurrent.futures import ProcessPoolExecutor

import matra

_engine: matra.Engine | None = None


def _init() -> None:
    global _engine
    _engine = matra.Engine()


def analyze_chunk(text: str) -> matra.DocumentDict:
    assert _engine is not None
    return _engine.analyze(text).to_dict()


if __name__ == "__main__":
    chunks = ["The report was filed.", "Nobody read it.", "The committee met again on Friday."]
    with ProcessPoolExecutor(initializer=_init) as pool:
        results = list(pool.map(analyze_chunk, chunks))
    print(len(results), [r["metrics"]["total_words"] for r in results])
grade8.8density0.56compression0.54

The if __name__ == "__main__": guard is required. On macOS and Windows a worker process starts by importing your script afresh (the spawn start method), so a pool created at module level would start again inside every worker, and Python stops that with a RuntimeError about the bootstrapping phase. Under the guard, only the parent builds the pool. A Document does not pickle, so a worker returns to_dict(). Concurrent first runs against the same model directory are safe. Each process downloads into its own temporary subdirectory and moves the file into place with a single rename.

Analyze one document#

grade13.4density0.73compressionnone

Engine.analyze takes exactly one input, by keyword, and returns a Document:

CallReads
analyze(text)a str, as plain text, or as format when you name one
analyze(file=path)a file, its format detected from its name and bytes, or as format when you name one
analyze(bytes=data)bytes, as format, or detected when you name none
analyze(blocks=[...])headings and paragraphs you supply
doc = engine.analyze("The committee approved the proposal without debate.")
doc = engine.analyze(text, format="markdown")
doc = engine.analyze(file="notes.md")
grade5.8density0.56compression0.53

A str is always text, never a path: engine.analyze("notes.md") analyzes the eight characters notes.md. Name a file with file=. Passing no input, or two, raises TypeError at the call. format is one of markdown, plain_text and blocks, and it applies to text, bytes and file. With no format, a file is read by what it is: .md as markdown, anything else that is UTF-8 as plain text, and a PDF, a DOCX or bytes that are not text as matra.UnsupportedFormatError. A JSON file of blocks is engine.analyze(file="notes.json", format="blocks"), since no extension implies it; a PDF or DOCX signature still outranks the format you name.

grade7.8density0.59compression0.62

A file's path travels into the result as doc["provenance"]["source"]["path"]. On Unix it is decoded the way Python decodes any filesystem path, so os.fsencode on it hands back the bytes the name came from, even when the name is not valid UTF-8. Every path argument takes a str or a pathlib.Path alike.

Text with its own section breaks#

grade12.6density0.46compressionnone

Text that comes from somewhere matra does not read (a PDF you extracted, a CMS, a transcript) can arrive as headings and paragraphs, with no markdown in between:

doc = engine.analyze(blocks=[
    {"kind": "heading", "level": 1, "text": "Field notes"},
    {"kind": "paragraph", "text": "The river rose overnight."},
])
grade6.6density0.54compression0.58

Each block is a BlockDict, a heading (kind, level, text) or a paragraph (kind, text). It is the same blocks input format the JSON schema in spec/schema/v2/blocks.schema.json publishes, read by the same reader, so a kind or a key matra does not know raises matra.InvalidInputError naming it. The analysed text is the blocks' texts joined by a blank line, and every span points into it.

Analyze many documents#

grade15.8density0.65compressionnone

Engine.analyze_all takes a directory or an iterable of texts and returns an iterator, one item per document, in input order:

for item in engine.analyze_all(dir="docs/"):
    if isinstance(item, matra.Document):
        print(item["provenance"]["source"]["path"], item["metrics"]["vocabulary_ttr"])
    else:
        print(f"{item['path']}: {item['kind']}: {item['message']}")
grade8.2density0.62compression0.56

Nothing is read or parsed until you pull the next item, so a large directory costs nothing up front. One unreadable file costs one item, not the walk. A document that analyzed arrives as a Document. One that did not arrives as a DocumentError dict (path, kind, message, details), holding the position the document would have had. Test isinstance(item, matra.Document) to tell them apart; a type checker narrows the union on that test.

grade3.6density0.55compressionnone

analyze_all(texts=[...]) does the same for texts you hold, read as plain text or as format. A str alone raises TypeError, since it would otherwise be read one character at a time.

grade12.9density0.50compressionnone

The walk is not recursive, and symlinks and subdirectories are skipped rather than followed. Only a failure listing the directory itself raises: a missing directory is matra.IoNotFoundError, a FileNotFoundError, because there is no per-document result for it to travel in.

Read a document#

grade4.4density0.53compressionnone

A Document holds the parse matra made, and you read it as the JSON reads:

doc["metrics"]["total_words"]
doc["sections"][0]["heading"]
doc["provenance"]["schema"]          # "2.0"
whole = doc.to_dict()                 # the whole document, as a dict
grade5.0density0.50compression0.64

The keys are provenance, sections, metrics and skipped. list(doc), len(doc) and "metrics" in doc work as they do on a dict. Each read builds a new value from the parse, so changing what a read returned changes nothing in the document. Hold a field in a variable when you read it in a loop.

grade8.1density0.51compressionnone

A Document is read-only. There is nothing to assign to, and the outcome functions take a Document and never a dict, so a summary cannot carry provenance no matra run produced. A Document does not pickle; to_dict() is what you store or send.

Summaries, keyphrases and clusters#

grade11.2density0.50compressionnone

The outcome functions run on a Document you already have, so a text is parsed once however many outcomes you ask for:

FunctionReturns
matra.summarize(doc, n=3, *, method=None)Summary
matra.keyphrases(doc, n=10, *, method=None)Keyphrases
matra.clusters(doc, embedder, threshold=0.85)Clusters
summary = matra.summarize(doc, n=3)
summary = matra.summarize(doc, n=3, method="textrank")
phrases = matra.keyphrases(doc, n=10, method="yake")
grade5.8density0.55compression0.61

method names how the ranking is done: tfidf or textrank for a summary, rake or yake for keyphrases. With none, matra's default runs, which is tfidf and rake today and the same on the command line. An unknown name raises matra.InvalidInputError. The defaults for n are the command line's.

grade12.8density0.53compression0.52

Each result is a dict with a provenance, the document's own with the method that ran and the parameters it ran with ({"n": 3}), beside the result itself. A summary returns its sentences in document order, not score order: read index to see where a sentence sits among the document's sentences, span for where it lies in the text, and score to see how it ranked. Keyphrases return their phrases in score order, highest first, each with occurrences, the span of every place the phrase was found.

grade10.9density0.63compression0.61

Keyphrases come back as lowercased lemmas joined with spaces, not as the surface text. A document about "Dependency Parses" yields the phrase dependency parse; its occurrences' spans point at the surface text. Phrases with identical scores come back in the same order on every run, so a tie straddling the n cutoff is decided the same way each time.

grade11.1density0.48compression0.49

The score on a Keyphrase is an ordering key and nothing more. RAKE's is a sum of per-word ratios that are each at least 1, so a phrase of k words scores at least k, which is a floor per length and not an order across lengths; YAKE's is the reciprocal of a product of per-word scores, so it is unbounded above, carries no unit, and does not track phrase length in either direction. The two scales are unrelated to each other, and neither is comparable across documents or with another tool's. Do not surface either number to a reader as a strength without saying which method produced it. The CLI guide works through what the ranking does and does not tell you, and Methodology carries the formulas.

Bring your own embeddings#

grade8.0density0.58compressionnone

matra.clusters takes the embedder as an argument, always. Model2Vec is one, and so is any object with two methods:

class MyEmbedder:
    def embed(self, texts: list[str]) -> list[list[float]]:
        return my_service.encode(texts)     # one vector per text, in order

    def identity(self) -> matra.types.Identity:
        # Names the geometry, not the run, in the shape the parser uses.
        return {"name": "my-service", "model": "v3", "sha256": None, "license": None}


found = matra.clusters(doc, MyEmbedder(), threshold=0.85)
assert found["provenance"]["parameters"]["embedder"]["model"] == "v3"
grade8.4density0.48compressionnone

Embedder in matra.types is the protocol those two methods satisfy, and it is what clusters is annotated with. Importing it is optional and buys you a type checker's opinion, not a runtime check.

grade14.7density0.50compressionnone

The object is asked for its identity before anything is embedded, so an embedder that cannot name its geometry is refused for the price of one method call.

grade9.5density0.54compression0.52

The contract is the one the Rust port carries: exactly one vector per sentence, in order, every vector the same length. Break it and the call raises matra.InvalidInputError, a ValueError, with the same message a Rust implementor gets, because the check is in the library and not in the binding. An exception raised inside your embed arrives as matra.InvalidInputError too, with your exception's own text inside the message.

grade9.2density0.45compression0.54

identity returns the shape provenance["parser"] has, matra.types.Identity: a name, and the model, its sha256 and its license, each None when you do not know it. It is read once, when the object is handed over, and travels into the result's provenance as parameters["embedder"], so the model's licence travels with the scores as the parsing model's does with the parse. A bare string is refused. Two embedders that can disagree must not report the same identity, or scores end up attributed to a geometry that did not produce them.

Size limits#

grade8.7density0.54compression0.60

Every input over 8 MiB is rejected with matra.InputTooLargeError, a ValueError, whose details carry what, limit and actual. Text, bytes and blocks are checked as input; a file is checked before it is read, as file_read. The gate lives in the one step every input passes through to reach the parser, so there is no input that skips it. In analyze_all, an oversized input is one item whose kind is input_too_large, and the stream goes on.

grade9.2density0.45compressionnone

The outcome functions add caps of their own: 2000 sentences for a summary and for clusters, and 200000 tokens for keyphrases, each raising matra.InputTooLargeError with the method in details["what"].

grade20.4density0.51compressionnone

Every cap, and the word counts a paragraph needs before it gets a readability grade or a compression ratio, is a constant in matra.limits, under the name Rust's domain::limits gives it, read from the extension so it is the value the library enforces:

from pathlib import Path

import matra

data = Path("transcript.txt").read_bytes()
if len(data) > matra.limits.MAX_INPUT_BYTES:
    ...  # split it before analysing it
grade46.1density0.80compressionnone

The eleven are MAX_INPUT_BYTES, MAX_CONFIG_FILE_BYTES, MAX_DOWNLOAD_BYTES, MAX_TFIDF_SENTENCES, MAX_TEXTRANK_SENTENCES, MAX_CLUSTER_SENTENCES, MAX_RAKE_TOKENS, MAX_YAKE_TOKENS, MAX_COMPRESSION_PARAGRAPH_BYTES, MIN_READABILITY_WORDS and MIN_COMPRESSION_WORDS.

The shapes of the data#

grade13.8density0.66compressionnone

A Document reads as a dict, and every outcome returns one. DocumentDict, Section, Paragraph, Sentence, Token, Summary, Keyphrases and Clusters are TypedDict definitions in matra.types that describe the shapes at both runtime and type-check time:

from matra import Clusters, DocumentDict, Keyphrases, Paragraph, Section, Sentence, Summary, Token
gradenonedensity0.63compressionnone

A document nests down to the token level:

for section in doc["sections"]:               # heading, level, span, paragraphs
    for para in section["paragraphs"]:        # span, text, kind, sentences,
                                              # metrics
        print(para["metrics"]["readability_grade"])  # float or None
        for sentence in para["sentences"]:    # span, text, tokens, word_count,
                                              # tree_depth, passive, negations,
                                              # modals, clausal_complements,
                                              # root_adverbials, hearst_pairs,
                                              # root_mood
            for token in sentence["tokens"]:
                print(token["lemma"], token["pos"], token["dep"])

print(doc["metrics"]["vocabulary_ttr"])        # float or None
print(doc["metrics"]["nominalization_ratio"])  # float or None
print(doc["skipped"])                          # [{"kind": "table", "span": {...}}, ...]
grade10.3density0.56compression0.55

Every heading, paragraph, skipped construct, sentence and token has a span, {"byte_start": int, "byte_end": int}, into the analysed text. It counts UTF-8 bytes, not characters, as the field names say: text[span["byte_start"]:span["byte_end"]] on a str is right on ASCII and wrong, with no error, after the first character outside it. Slice with matra.span_text(text, span), which encodes the text and decodes the stretch for you. The text fields are normalized (a sentence's text is rebuilt from its tokens, a markdown paragraph's has its markup removed); the span is where the value came from.

grade7.2density0.62compression0.64

A Token carries the ten CoNLL-U columns under these keys: id, text, lemma, pos, xpos, features, head, dep, deps, misc, plus its span and the derived is_punct. features is a dict from feature name to value, {"Number": "Plur"}. head is the id of the governing token in the same sentence, and 0 marks the root.

grade9.4density0.58compressionnone

Summary is provenance and sentences, each a SummarySentence of index, span, text and score. Keyphrases is provenance and phrases, each a Keyphrase of phrase, score and occurrences. Clusters is provenance, clusters and edges.

Why a metric field is None#

grade9.1density0.48compressionnone

Each metric declines to run below its own threshold, and a None records that decision rather than a failed computation. The first three are in each paragraph's metrics, the rest in the document's.

FieldFilled when
readability_gradethe paragraph has more than 10 words and its kind is not quote
lexical_densitythe paragraph has at least 1 word and its kind is not quote
compression_ratiothe paragraph has more than 50 words, its kind is not quote, and the paragraph text is at most 256 KiB
vocabulary_ttrthe document has at least one non-punctuation token
nominalization_ratiothe same condition as vocabulary_ttr
passive_ratio, mean_sentence_lengththe document has at least one sentence
sentence_length_stdthe document has at least two sentences
grade10.6density0.62compressionnone

"Words" means the count of tokens with is_punct false across the paragraph's sentences. Quotes (a kind of quote) are never parsed at all: their sentences list is empty, their three paragraph metrics are None, and they contribute nothing to any document total.

grade10.1density0.58compression0.59

Markdown is read as CommonMark, and what matra does not analyse it lists in skipped rather than dropping it: front matter, fenced and indented code, tables, HTML blocks, thematic breaks and link reference definitions, each with its kind and span. List items are paragraphs of kind list_item. ## References is a heading like any other, and a #hashtag is text.

The document-level aggregates#

grade12.6density0.57compressionnone

Every document-level aggregate is a field of metrics. Read them rather than recomputing them:

doc = engine.analyze("The report was filed. The committee met.")
print(list(doc))
# ['provenance', 'sections', 'metrics', 'skipped']
print(doc["metrics"])
# {'total_words': 7, 'total_sentences': 2, 'paragraph_count': 1,
#  'mean_sentence_length': 3.5, 'sentence_length_std': 0.7071067811865476,
#  'vocabulary_ttr': 0.8571428571428571, 'nominalization_ratio': 0.0,
#  'passive_ratio': 0.5}
grade10.0density0.59compressionnone

The counts are always integers. Each ratio is None when its condition was not met: mean_sentence_length and passive_ratio need a sentence, sentence_length_std (the sample standard deviation, as statistics.stdev computes it) needs two.

grade11.7density0.53compression0.61

Methods do not cross the FFI boundary, only fields do, which is why these are fields: the metric suite materializes them so the aggregates cross as data instead of being re-derived once per language binding (RFC-0008). The same holds per sentence: word_count, tree_depth and passive are keys on each sentence dict.

passive = [
    s["text"]
    for section in doc["sections"]
    for para in section["paragraphs"]
    for s in para["sentences"]
    if s["passive"]
]
grade10.7density0.63compression0.56

doc["provenance"] says where the document came from: matra (the version), schema ("2.0", the schema version), parser (name, model, sha256, license), format (markdown, plain_text or blocks), and source (path, bytes, sha256 of the analysed text). Read schema before anything else when a result was stored by another process: ignore keys you do not recognize, and reject a major version you do not know.

Exceptions#

grade7.6density0.52compression0.61

Every domain::Error variant crosses the FFI boundary as a class of its own, named for the variant with PEP 8's Error suffix; the name it shares with Rust and the JSON is its kind. Each derives from matra.MatraError and from a builtin class, the one matra raised before 0.3.0, so an except ValueError written then still catches what it caught:

SituationPython classAlso a
A model artifact is missing from the directory Model2Vec.from_dir was givenModelNotFoundErrorFileNotFoundError
Input exceeds the 8 MiB cap, or an outcome's capInputTooLargeErrorValueError
Input format has no reader, or the bytes are not UTF-8 textUnsupportedFormatErrorValueError
A caller broke a documented contract, such as an unknown method or an embedder returning the wrong number of vectorsInvalidInputErrorValueError
File I/O error, or a download that did not arriveIoErrorOSError
A path that does not existIoNotFoundErrorFileNotFoundError
A path matra may not read or writeIoPermissionErrorPermissionError
Model file corrupt or wrong format, or bytes that arrived and failed the pinned hashModelInvalidErrorRuntimeError
NLP parsing failed, including a panic caught at the UDPipe boundaryParseFailedErrorRuntimeError
grade8.6density0.57compression0.56

IoNotFoundError and IoPermissionError derive from IoError, so except matra.IoError catches every I/O failure. matra.IoError is not the builtin IOError, which is another name for OSError. Every exception carries kind, the stable string from matra.ERROR_KINDS, and details, a dict of the values to act on: what, limit and actual on an InputTooLargeError, path on a missing file, url and retryable on a download that did not arrive. The errors reference lists the keys for every kind.

grade8.4density0.51compressionnone

A call with the wrong arguments (no input, two inputs, format with blocks) raises the builtin TypeError, as any Python function does. It is a mistake in the call, not a failure of the analysis.

grade10.9density0.64compressionnone

Loading the model and analyzing text fail in different ways, so catch them separately:

import matra

try:
    engine = matra.Engine()
except matra.MatraError as e:
    raise SystemExit(f"cannot load the model: {e.kind}: {e}")

try:
    doc = engine.analyze(file=path)
except FileNotFoundError as e:
    print(f"no such file: {e.details['path']}")
except matra.InputTooLargeError as e:
    print(f"{e.details['what']} gate: {e.details['actual']} > {e.details['limit']}")
except matra.MatraError as e:
    print(f"{e.kind}: {e}")
grade15.3density0.54compression0.51

The message on each exception is the Rust error's own Display output, for a person to read. Branch on the class or on kind, and read values from details. An InputTooLargeError names its gate in details["what"]: input and file_read for an input, tfidf, textrank, rake, yake and clusters for an outcome, and config_file_read, parser_model_download and embedding_model_download for the constructors that read a configuration or download a model. An InvalidInputError names the check that failed in details["reason"], a stable name such as method_unknown, embedder_raised or config_malformed, so a caller tells a bad argument from a broken embedder or a config file to fix without reading the message.

grade8.4density0.58compression0.62

The mapping is exhaustive and enforced at compile time on the Rust side. The match that picks the class has no wildcard arm over the variants, so adding a new domain::Error variant without also giving it a class fails to build. A new variant cannot fall through to another variant's class by accident.

grade13.2density0.54compressionnone

The one failure that does not follow this table is the cross-thread access described above, which arrives as pyo3_runtime.PanicException and bypasses except Exception entirely.

Use the matra CLI

grade7.9density0.56compressionnone

matra reads a file, a directory or stdin and writes to stdout. matra --help describes the tool in the line built into the command: "matra parses text into CoNLL-U structure and measures it. It reports what is there; interpretation is yours."

Install#

gradenonedensity1.00compressionnone

Nothing installed, nothing configured:

uvx 'matra>=0.3' analyze essay.md
grade4.3density0.54compression0.56

The version floor is load-bearing. A bare uvx matra takes whatever release is newest, which is 0.3.x today and will not stay that way, and 0.1.0 and 0.2 are still on PyPI for anything pinned to them. 0.2's command line spells its JSON and its config file differently; Migrate from 0.2 to 0.3 lists each change. 0.1.0's analyze is a separate Python implementation rather than this one missing a flag: --json prints a bare document with no envelope, the human table is a different renderer, and the model cache lands in ~/.matra/models instead of the resolved model directory. It does not fail, it succeeds and gives you something else, so pin it.

grade10.0density0.61compression0.47

With the floor in place the command ships two ways beyond that, and all three run the same program. The Python package's matra command is the Rust CLI reached through the extension module, not a second implementation, so the flags, the output, and the exit codes are the same whichever route you took. One thing does differ: which features were compiled in. cargo install matra --version '^0.3' --features cli reports features: udpipe cli, while the published wheel reports features: udpipe model2vec python cli, so the embedding adapter is present on the Python route and absent from a --features cli build, whose matra clusters refuses and names the feature. cargo install matra --features cli,model2vec builds one with it. matra --version prints the list of the binary in front of you on its second line.

# Rust binary, no Python involved
cargo install matra --version '^0.3' --features cli

# or through the Python package
uv add 'matra>=0.3'   # then: matra --help
grade11.4density0.55compression0.57

Every route caches the model at the resolved model directory (about 16 MB) and downloads it the first time any command needs it. No flag and no environment variable is required to make that happen. If the cached file fails its SHA-256 check, matra never deletes it: a download that verifies replaces it through an atomic rename, and a download that fails leaves the file you had exactly as it was.

Where matra keeps things#

WhatPathOverride
config file$XDG_CONFIG_HOME/matra/config.toml, else ~/.config/matra/config.tomlMATRA_CONFIG_FILE
data root$XDG_DATA_HOME/matra, else ~/.local/share/matraMATRA_DATA_DIR
modelsthe data root's models directoryMATRA_MODEL_DIR, or --model-dir on the command line
grade9.1density0.60compression0.58

A pre-existing ~/.matra/models from an older install is still used when the new location does not exist yet. matra never creates ~/.matra; when a non-empty legacy cache is selected it is the resolved model directory, downloads and re-downloads included. Create the new location, or set MATRA_MODEL_DIR, to move off it.

grade9.8density0.60compression0.50

Per key, --model-dir outranks every environment variable, which outranks the config file, which outranks the defaults compiled into the crate. The config file has no key for the model directory, so that value comes from the flag, the environment, or the defaults. matra config show prints which rung each value actually came from.

Commands#

grade9.6density0.62compression0.59

analyze, summarize, keyphrases and clusters each read a file, or - for stdin. analyze also reads a directory, one result per file. The other three report on one document, so they refuse a directory and name analyze, which reads one; picking one file out of it would answer a question nobody asked.

grade10.9density0.58compression0.60

Every subcommand routes through the library pipeline, which rejects symlinks, rejects files over 8 MiB before reading them, and picks a reader from the file's bytes and extension. --format markdown|plain_text|blocks names the format instead, which is how a JSON file of blocks is read, since no extension implies one. A PDF or DOCX signature in the bytes still outranks it.

matra analyze notes.json --format blocks --json

matra analyze#

gradenonedensity0.57compressionnone

Analyzes a file and reports its metrics.

matra analyze essay.md
matra analyze essay.md --sections
matra analyze essay.md --json
grade13.3density0.67compressionnone

--sections adds a per-section breakdown under the metric table: one row per section with its heading level, heading, and the paragraph, sentence, and word counts underneath it.

grade8.7density0.53compression0.54

Given a directory, analyze reads every regular file in it, one level deep and in name order, and reports each one: a table per file, or under --json one envelope per line. A file that fails is reported where it happens, on stderr and under --json as its own line with its error, and the walk goes on to the next file. The exit code is 2 if any file failed. Symlinks and subdirectories are skipped, as the library's Inputs::dir skips them. --format is refused with a directory, because each of its files is read as its own name says.

matra analyze docs/ --json --depth document | jq -c '{input, words: .result.metrics.total_words}'
grade10.6density0.54compression0.48

--depth decides how much of the parse --json writes. document writes the provenance, the metrics and skipped, and no sections; paragraph adds the sections and their paragraphs, with no sentences; sentence adds the sentences, with no tokens; and token, the default, writes everything. A level below the depth is absent as a key, never an empty array, because an empty sentences would say a paragraph has none, and what remains keeps the whole document's key order, so the two diff line for line. Tokens are most of a document's JSON, so a caller that reads only the measures pays for no more than it reads. --depth without --json is a usage error.

grade7.6density0.59compressionnone

On a markdown file, front matter, code blocks, tables, HTML blocks, thematic breaks and link reference definitions are not analyzed. They are listed in the JSON's skipped, each with its kind and span, rather than dropped. ## References is a heading like any other.

grade12.9density0.65compressionnone

--json emits the full Document (provenance, sections, and metrics, which carries the counts and ratios the table prints) inside the envelope described below:

matra analyze essay.md --json | jq '.result.sections[0].paragraphs[0].metrics.readability_grade'
grade12.1density0.54compressionnone

That query returns null more often than you might expect. Per-paragraph metrics have thresholds: readability_grade needs more than 10 words, compression_ratio needs more than 50, and quotes are skipped entirely. Filter for non-null before aggregating:

matra analyze essay.md --json \
  | jq '[.result.sections[].paragraphs[].metrics.readability_grade | select(. != null)] | add / length'

matra summarize#

gradenonedensity0.63compressionnone

Extracts the top-N sentences as an extractive summary.

matra summarize essay.md
matra summarize essay.md -n 5 --method textrank
OptionDefaultValues
-nsummary.n from the config, 3 as shippedany positive integer
--methodsummary.method from the config, tfidf as shippedtfidf, textrank
grade8.0density0.54compression0.47

TF-IDF scores each sentence by term frequency against the document. TextRank scores by a graph-coherence measure over sentence similarity. TF-IDF is the default because it is cheap and needs no graph, not because it picks better sentences. It favours sentences dense in the document's rarer terms, and on narrative prose that can mean short sentences that make little sense on their own. Reach for TextRank when the summary should carry the document's main thread, when sentences restate and build on each other, or when the TF-IDF picks read as fragments. Read both on your own text before you rely on either. Both reject input over 2000 sentences.

grade10.3density0.60compressionnone

Both methods select their top-N sentences by score internally, then return that selection re-sorted into document order, not score order. The score field tells you how each sentence ranked; the list order tells you where each sentence sits in the source text.

grade9.9density0.56compression0.56

The human table prints each score at three decimals. Two sentences whose scores differ in the fourth therefore print the same number, and a summary of three sentences can arrive looking as though the ranker scored them identically when it did not. --json carries the full float, so reach for it whenever the ordering has to be defended rather than read.

matra summarize essay.md --json | jq '.result.sentences[0]'
# {"index": 4, "span": {"byte_start": 812, "byte_end": 951}, "text": "...", "score": 0.312}
matra summarize essay.md --json | jq '.result.provenance.method'
# "tfidf"
grade17.0density0.43compressionnone

Under --json the result is a summary: the document's provenance, with the method that ran and its parameters ({"n": 3}), and the sentences, each with its index among the document's sentences and its span in the input.

matra keyphrases#

gradenonedensity1.00compressionnone

Extracts ranked keyphrases.

matra keyphrases essay.md
matra keyphrases essay.md -n 20 --method yake
OptionDefaultValues
-nkeyphrases.n from the config, 10 as shippedany positive integer
--methodkeyphrases.method from the config, rake as shippedrake, yake
grade9.8density0.59compression0.57

RAKE splits on stop words and scores the remaining word runs by a co-occurrence degree-to-frequency ratio. It is rule-based and fast. YAKE scores individual terms by position, frequency, and how varied their surrounding context is, then builds one-word to three-word candidates from those scores. It tends to surface longer, more specific phrases and to notice terms that first appear later in a long document, at more computation than RAKE. Both reject input over 200000 tokens.

grade12.3density0.57compressionnone

Unlike summarize, keyphrases output stays in score order, highest first. A phrase is not tied to one location the way a sentence is, so under --json each carries occurrences, the span of every place in the input it was found, beside the result's provenance with its method and parameters.

grade8.8density0.57compression0.53

Phrases are printed as lowercased lemmas, not as the surface text of the document. "Dependency Parses" appears as dependency parse, and its occurrences point at the surface text. The lemma is the parser's, read in each sentence's context, and the parser does not always lemmatize one word the same way twice. "Reading room" can come out as reading room in one sentence and read room in another, two phrases for one surface form. Merge such pairs yourself if they matter. Phrases with equal scores come back in the same order on every run, the order they are first met, so a tie at the -n boundary is decided the same way each time.

Reading the scores#

grade8.4density0.56compression0.51

Neither number is a probability, a percentage, or comparable with the other. RAKE sums a per-word degree-over-frequency ratio across the words of the phrase, and that ratio is at least 1 for every word, so a phrase of k words scores at least k. That is a floor per length, not an order across lengths: on this book's methodology page the two-word lexical density scores 5.667 and the three-word model file name scores 5.200. YAKE multiplies its words' term scores and returns the reciprocal, so it is unbounded above and carries no interpretable unit, and phrase length does not predict where a YAKE score lands. Use either score to order phrases within one document under one method. Do not read it as a magnitude, and do not compare a RAKE number with a YAKE number.

grade5.3density0.53compression0.55

Expect the top phrase not to be the document's topic. On this book's errors page, RAKE puts error implement st ::error, read off the code span std::error::Error and the words around it, first at 12.135 and scores the bare error at 1.468, while YAKE's first is move copy at 2257.447. What is low there is the score, not the position: 1.468 sits not far above 1.000, the floor any single word scores, and 229 of the 589 phrases sit flat on that floor underneath it, which puts error around the middle of the ranking, 306th. Both methods are behaving as specified: they rank properties of word co-occurrence, not aboutness. Methodology carries both formulas and every departure from the published versions.

matra keyphrases essay.md --json | jq '.result.phrases[0:3]'

matra clusters#

grade11.9density0.50compressionnone

Groups a document's sentences by the similarity of their embeddings.

matra clusters essay.md
matra clusters essay.md --threshold 0.8 --json
grade8.1density0.55compressionnone

On prose that does not repeat itself, expect the first line to find nothing. 0.85 joins near-restatements, and an essay whose sentences each say something new has none, so the table is empty, stderr says so, and the exit code is 1:

matra: found no clusters in essay.md at --threshold 0.85; a lower threshold joins less similar sentences
grade5.5density0.45compressionnone

That is a result, not a failure. summarize and keyphrases say the same when they find nothing, and under --json or --quiet none of the three writes the line.

OptionDefaultValues
--thresholdclusters.threshold from the config, 0.85 as shippeda cosine similarity
grade9.7density0.55compression0.50

Each sentence is embedded with potion-base-8M, the embedding model models.embedding names, downloaded into the model directory on first use (30.9 MB), with the same two lines on stderr the parsing model's first download prints. Two sentences whose cosine similarity is at least the threshold are joined, and a cluster is a connected component of that graph, so co-membership is transitive and is not pairwise similarity. A sentence with no partner is in no cluster. The table prints each cluster with its members' sentence indices and text; --json writes the Clusters value, with edges, every pair that cleared the threshold and its score, and the embedder's identity in provenance.parameters. Finding no cluster is exit 1.

grade6.4density0.50compressionnone

The threshold does not travel between models, domains or granularities, and 0.85 is a starting point for sentences only. Semantic clusters says how to choose one.

grade8.8density0.54compressionnone

The command needs a build with the model2vec feature, which the Python wheel has. A build without it refuses before it reads anything, with invalid_input naming the feature.

matra config#

grade11.6density0.76compressionnone

matra config show prints every resolved value with the rung it came from, one key per line, in the shape cargo config get --show-origin uses:

matra config show
data_dir = "/home/you/.local/share/matra" # default
model_dir = "/home/you/.local/share/matra/models" # default
models.embedding = "potion-base-8M" # default
summary.n = 5 # /home/you/.config/matra/config.toml
summary.method = "tfidf" # default
keyphrases.n = 10 # default
keyphrases.method = "rake" # default
clusters.threshold = 0.85 # default
grade10.9density0.56compression0.56

models.embedding is the name of the embedding model's directory under model_dir, which matra clusters, Model2Vec::from_config and Python's Model2Vec.from_config() provision into and load from. The file's sections are [models], [summary] and [keyphrases], each of the last two with n and method, and [clusters] with threshold. They configure the command line only: the library and Python take every count, method and threshold as an argument.

grade5.9density0.57compression0.56

A config file from 0.2 does not load. 0.3.0 renamed [summarize] to [summary], algorithm to method, and semantic.threshold to clusters.threshold, and dropped models.udpipe, since the parsing model is pinned in the source beside its digest. Every command that reads such a file fails with invalid_input, naming the file and each key's replacement, so the file can be fixed from the message. matra config init --force writes a fresh one.

grade4.1density0.54compression0.53

The origin is default for a value compiled into the crate, a path for one read from your config file, environment variable ... for one an environment variable set, and command line for one a flag set. With --json, each key carries its value, the rung, and what the rung pointed at.

grade8.4density0.54compression0.48

What that listing does not carry is the config file path. Every key whose origin is default names no file, so a run with no config file on disk prints no path at all, and "where is my config" is not a question the human table answers. Two commands do answer it, once there is a file. matra config show --json puts the config file's path in the envelope's input field when that file exists, and null when it does not: input names what the run read, and a run with no config file read none. matra config init prints the path it wrote to, which is also where config show looks.

matra config show --json | jq -r .input
grade12.1density0.59compression0.56

matra config init writes the shipped defaults (models.embedding, and the summary, keyphrase and cluster keys) to the resolved config path and prints where it wrote them. It creates the parent directories, writes through a temporary file in the same directory so a reader never sees a half-written config, and refuses to overwrite an existing file unless you pass --force.

matra config init
matra config init --force

matra completions#

gradenonedensity0.67compressionnone

Prints a completion script for bash, zsh, or fish.

mkdir -p ~/.zfunc && matra completions zsh > ~/.zfunc/_matra   # ~/.zfunc must be on your fpath
matra completions bash > /etc/bash_completion.d/matra

For an agent#

grade3.8density0.45compression0.54

matra --skill prints the agent skill: what matra is for, when to reach for it, every command with its JSON shape, how to read each number and what it does not mean, the limits, and the errors. It is the whole hand-off. If you are pointing an agent at matra and it has nothing installed, uvx 'matra>=0.3' --skill is the line to give it. Pin the version: --skill arrived in 0.2.0, and a bare uvx matra takes whatever release is newest, which before 0.2.0 was a different command line that does not have the flag.

matra --skill                 # the skill itself
matra --skill -r              # the references, one per line: name, then summary
matra --skill -r json         # one reference
grade7.2density0.43compression0.58

The text is compiled into the program with include_str! from skills/matra/SKILL.md and skills/matra/references/*.md, so what you read is what the installed version does, not a copy of the docs that may have moved on. The same files are what the repository ships as a plugin. Every command in them is executed against the command line by the test suite, so an incantation that no longer runs fails CI.

grade6.2density0.55compression0.60

--skill outranks a subcommand: matra analyze essay.md --skill prints the skill and never looks at the file. -r without --skill is an error that says so, and an unknown reference name exits 2 naming the ones that exist. Everything else exits 0. --quiet does not apply, as it does not to completions: the text is the command's output, not a rendering of a result.

grade12.3density0.62compressionnone

Under --json the skill uses the same envelope every other command uses, with input null because the command reads no document:

{
  "format_version": 2,
  "command": "skill",
  "input": null,
  "result": { "name": "SKILL", "body": "---\nname: matra\n..." },
  "error": null
}
grade5.8density0.64compressionnone

result.name is SKILL for the top level and the reference's own name for matra --skill -r <name> --json. The list replaces result with one key:

{
  "format_version": 2,
  "command": "skill",
  "input": null,
  "result": { "references": [{ "name": "json", "summary": "The JSON envelope every command but completions emits, ..." }] },
  "error": null
}
grade14.4density0.45compressionnone

The list is every file under references/, in file-name order, and each summary is read from that file's own frontmatter rather than from a second list that could drift from it.

grade10.2density0.57compression0.64

The same files install as a plugin: claude --plugin-dir <checkout> over a clone of the repository picks up skills/matra/, which is where plugin discovery looks. And this documentation site serves llms.txt at its root, the map of every page with its one-line summary, for an agent reading the human door instead.

Reading from stdin#

grade6.8density0.49compression0.54

- in place of the path reads stdin, as bytes, and the format is detected the way it is for a file. --stdin-filename gives that input a name, which is the label in the output and the envelope, and whose extension stands in for a file's. --format names the format outright. With neither, the input is read as plain text and labelled <stdin>. Bytes that are not UTF-8 text fail as unsupported_format, as a file's do.

pandoc notes.docx -t markdown | matra analyze - --stdin-filename notes.md
pandoc notes.docx -t markdown | matra analyze - --format markdown
cat essay.txt | matra keyphrases -
grade9.7density0.47compressionnone

The 8 MiB cap applies to stdin as it does to a file, and it is enforced while reading rather than after, so an unbounded stream is refused instead of buffered. Stdin is read before the model loads, so an oversized stream costs no download.

Color, quiet, and version#

grade11.5density0.61compressionnone

--color takes auto (the default), always, or never. Under auto, matra colors an interactive terminal and nothing else, and honors NO_COLOR: if that variable is present and not empty, color is off whatever its value. --color always and --color never are explicit requests and outrank NO_COLOR in both directions.

grade10.2density0.52compression0.52

--quiet suppresses the human-readable output while leaving the exit code alone, so a script can branch on "found something" without collecting the table. It has no effect on --json. It also silences the one line the command prints to standard error before it downloads a model, which names the artifact, its size and where it is going. That line is the only output a first run produces during a wait that can reach half a minute, so silence it deliberately rather than by habit.

grade6.7density0.46compressionnone

--version prints the version and then the features this build was compiled with:

matra 0.3.0
features: udpipe model2vec python cli

The JSON envelope#

grade9.7density0.75compressionnone

--json emits one object for every command but completions, success or failure:

{
  "format_version": 2,
  "command": "analyze",
  "input": "essay.md",
  "result": { "provenance": {}, "sections": [], "metrics": {}, "skipped": [] },
  "error": null
}
grade11.1density0.51compression0.46

command is analyze, summarize, keyphrases, clusters, config, skill or schema. input is the path that was read, or the name --stdin-filename gave it. For config it is the config file the run read, and null when there is none, and for skill and schema, which read no document, it is always null. result is the serde form of the domain value the command produced: a Document for analyze, a Summary for summarize, a Keyphrases for keyphrases and a Clusters for clusters, each of the last three carrying the document's provenance with the method that ran and its parameters. For config it is an object keyed by setting, each carrying value, source and origin, and for skill the shapes shown above.

grade18.3density0.63compressionnone

All five keys are always present, and exactly one of result and error is null. A failure writes the envelope with result null and error holding the failure's kind, message and details, the same three Rust serializes and Python raises:

{
  "format_version": 2,
  "command": "analyze",
  "input": "missing.md",
  "result": null,
  "error": {
    "kind": "io",
    "message": "io error: cannot read missing.md: No such file or directory (os error 2)",
    "details": { "io_kind": "not_found", "path": "missing.md", "cause": "cannot read missing.md: No such file or directory (os error 2)" }
  }
}
grade8.4density0.57compression0.49

The message still goes to stderr and the exit code is still 2, so a script that reads only the exit code is unchanged. Branch on error.kind and read values from error.details, never from the message. The command line's own refusals carry the kind whose remedy matches, which for every one of them is invalid_input, a call to change, with error.details.reason naming which refusal it is: a directory given to a command that reads one document, --format with a directory, -r without --skill, an unknown reference or schema name, config init over an existing file, and clusters in a build without an embedder. A usage error under --json, such as an unknown flag, is an envelope too, with command null when the arguments name none. Errors lists every kind.

grade13.0density0.70compressionnone

analyze DIR --json writes one envelope per line, one line per file, each compact rather than indented, so the output is a stream of JSON values that jq reads one at a time.

grade12.4density0.60compression0.56

Stability: format_version increments on any change to the envelope, and whenever the document schema's major version moves; the result value is the serde form of the documented domain types. That is the same promise cargo makes for cargo metadata --format-version. Pin your consumer to a format_version you have tested against.

grade6.9density0.56compression0.56

A Document also names its own schema version, in result.provenance.schema ("2.0"), and that is the only place it appears. A minor version adds optional fields only; anything else is a major. So ignore fields you do not recognize, treat every string set you receive as open (format, pos, dep, mood, pattern, the feature names), and reject a major you do not know.

grade9.6density0.54compression0.56

The JSON Schemas for this major are compiled into the program. matra --schema lists them with their titles, and matra --schema NAME prints one: document, summary, keyphrases, clusters and error for the outputs, and blocks for block input. Like --skill it outranks a subcommand and reads no document, and under --json it is the envelope with command schema. The same files are in the repository under spec/schema/v2/.

matra --schema
matra --schema document > document.schema.json

Exit codes#

grade12.7density0.49compression0.51

0 on success when the command found something, 1 on success when it found nothing (an empty summary, no keyphrases, no clusters, a file with zero sentences), and 2 when an error occurred. Model load failure, a missing or unreadable file, a directory where one document is read, input over the size cap, a bad argument, and a parse failure all land on 2, with the message on stderr prefixed matra: and, under --json, the envelope with its error on stdout. For a directory the code is 2 if any file failed, 0 if none failed and any had a sentence, and 1 otherwise. A broken pipe, which is what you get piping into head, is treated as success and exits 0.

matra keyphrases notes.txt > /dev/null
case $? in
  0) echo "phrases found" ;;
  1) echo "no phrases" ;;
  2) echo "failed" ;;
esac

Methodology

grade13.2density0.42compression0.56

What each number matra reports is computed from, how to cite the work it comes from, and what it does not license you to conclude. Every formula on this page is transcribed from the implementation, not from the publication it descends from; where matra departs from the published method, the departure is stated.

grade10.4density0.61compressionnone

Related pages: Domain types for the fields these values live in, Errors for the caps each algorithm enforces.

The parse layer#

Model identity#

gradenonedensity0.57compressionnone

The English model is pinned in src/nlp/udpipe.rs.

PropertyValue
Fileenglish-ewt-ud-2.5-191206.udpipe
Size16,309,608 bytes
SHA-256784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957
TreebankEnglish Web Treebank, Universal Dependencies 2.5, release 191206
DistributorLINDAT/CLARIAH-CZ repository, handle 11234/1-3131
grade11.2density0.49compression0.51

The engine's constructors, Engine::new and Engine::from_config, download the file into the model directory when it is absent, checks the size, computes the SHA-256, and loads the same bytes it hashed. There is no second read from disk between verification and load. A cached file that fails verification is not loaded: the model is downloaded, and a download that also fails the digest is retried once, so a call makes at most two downloads. The file is replaced only by bytes that verify. If both downloads fail the digest the call returns Error::ModelInvalid, nothing is loaded, and the file that was there is still there.

grade13.2density0.62compressionnone

That path is the only way a parsing model arrives: no surface loads another model file, so every result names the pinned model it came from.

grade12.1density0.64compressionnone

Changing the model means editing the pinned constants, which makes the change visible in version control and in any caller that pins a matra version.

What the parse produces#

grade13.4density0.70compressionnone

One Token per token with all ten CoNLL-U columns, grouped into sentences. Two properties of the shipped adapter matter when you report results:

  • deps (CoNLL-U column 9) is always _. The udpipe-rs binding does not surface enhanced dependencies.
  • Sentence::text is built from token surface forms, with a space inserted unless the preceding token carries SpaceAfter=No in misc. It is a reconstruction, not a substring of your input.

Which text is parsed#

grade9.1density0.55compressionnone

The route through the library changes what gets segmented into sentences.

grade11.7density0.57compression0.57

Every pipeline route parses each paragraph that is not a quote separately. Quotes are not parsed, so they contribute nothing to any metric. Calling a provider's parse directly parses the whole string in one call with no decomposition, and its segmentation can differ from the pipeline's at paragraph boundaries.

grade7.6density0.50compressionnone

Input::file picks the format from the file's bytes and then its extension, so the same text segments differently under .md and under .txt; Input::text takes the format as an argument.

grade11.4density0.66compressionnone

Paragraph-at-a-time parsing means a sentence can never be attributed to the wrong paragraph. It also means sentence segmentation cannot run across a paragraph boundary. Report which format your numbers were produced under.

Citation#

grade5.3density0.66compressionnone

Straka, M., & Straková, J. (2017). Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe. Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, 88-99.

grade5.1density0.76compressionnone

Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., & Manning, C.D. (2014). A Gold Standard Dependency Corpus for English. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014).

grade11.7density0.79compressionnone

Zeman, D., et al. (2019). Universal Dependencies 2.5. LINDAT/CLARIAH-CZ digital library, Institute of Formal and Applied Linguistics, Charles University.

Limitations#

grade10.8density0.60compressionnone

UDPipe is a statistical syntactic parser. It performs no named entity recognition, no coreference resolution, and no semantic role labeling. Tags and relations are predictions, not ground truth, and error rates on text unlike the training corpus are higher than on text like it. The shipped configuration is English only.

Paragraph metrics#

grade9.7density0.61compressionnone

Two different counts appear below, and confusing them changes the answer. Paragraph::word_count counts non-punctuation tokens from the parse, and every applicability gate is stated in those terms. The formulas themselves split the paragraph text on whitespace. A gate stated in word_count terms is not a gate on whitespace words.

Flesch-Kincaid grade level#

grade = 0.39 * (words / sentences) + 11.8 * (syllables / words) - 15.59
InputAs computed
wordsCount of whitespace-separated pieces of the paragraph text
sentencesCount of ., !, and ? characters in the paragraph text, floored at 1
syllablesSum over words of the syllable estimate below. A word with no alphabetic characters contributes 0
grade22.6density0.54compression0.59

The syllable estimate for one word: keep only alphabetic characters, lowercase them; if the result is three bytes or shorter, return 1; otherwise count runs of the letters a, e, i, o, u, y, where adjacent vowels count once; subtract 1 if the word ends in e and the count is above 1; return at least 1.

grade10.0density0.50compression0.57

Applied to paragraphs that are not quotes and whose word_count is above 10. Stored in ParagraphMetrics::readability_grade, the paragraph's metrics.readability_grade. The value is unbounded and can be negative on short, simple text. The formula is computed inside the metric suite only; matra publishes no function that computes it on a string.

grade7.2density0.82compressionnone

Citation. Kincaid, J.P., Fishburne, R.P., Rogers, R.L., & Chisholm, B.S. (1975). Derivation of New Readability Formulas for Navy Enlisted Personnel. Research Branch Report 8-75, Naval Technical Training Command.

grade14.3density0.66compression0.53

Limitations. The formula was fitted to Navy training material in 1975 and predicts nothing about conceptual difficulty, argument quality, or required background knowledge. Two paragraphs with the same syllable and length profile score the same whatever they assert. The original publication specifies no syllable-counting algorithm, so grade values from different tools are not comparable; the heuristic above miscounts many words, and non-ASCII letters register as consonants. Counting sentence-ending characters treats abbreviations, decimals, and ellipses as sentence boundaries.

Lexical density#

lexical_density = content_words / total_words
InputAs computed
total_wordsCount of whitespace-separated pieces of the paragraph text
content_wordsThose pieces whose alphabetic characters, lowercased, form a non-empty string that is not in the stop word list
grade10.7density0.47compressionnone

A piece made only of punctuation counts in the denominator and never in the numerator.

grade7.7density0.57compressionnone

Applied to paragraphs that are not quotes, whose word_count is not zero, and whose text yields at least one whitespace-separated piece. Stored in ParagraphMetrics::lexical_density. Range 0.0 to 1.0.

grade8.3density0.77compressionnone

Citation. Ure, J. (1971). Lexical density and register differentiation. In G. Perren & J.L.M. Trim (Eds.), Applications of Linguistics. Cambridge University Press.

grade8.8density0.64compression0.60

Limitations. The value depends entirely on the stop word list, which is matra's own list of 105 English words, not a standard one. Ure's definition counts lexical items by part of speech; matra approximates that with a stop word list over whitespace pieces. Density says nothing about whether the content words are used well, or at all correctly.

Compression ratio#

compression_ratio = compressed_bytes / original_bytes
InputAs computed
original_bytesUTF-8 byte length of the paragraph text
compressed_bytesByte length of the brotli-compressed text at quality 6 with a window parameter of 18, which is a 256 KiB window
grade7.8density0.58compression0.59

Applied to paragraphs that are not quotes, whose word_count is above 50, and whose text is at most 262,144 bytes. Paragraphs above that byte ceiling are skipped and keep None, which bounds worst-case compression time. The slot also stays None if the encoder fails to consume the text. Stored in ParagraphMetrics::compression_ratio. A lower ratio means the text compressed further, which means more surface repetition.

grade7.7density0.82compressionnone

Citation. Alakuijala, J., & Szabadka, Z. (2016). Brotli Compressed Data Format. RFC 7932, Internet Engineering Task Force.

grade12.7density0.64compression0.58

Limitations. This is a redundancy proxy over bytes, not a measure of meaning, quality, or novelty. Precise technical prose that reuses terminology compresses much like repetitive filler. The number is a property of one encoder at two specific parameter settings: a different compressor, or a different version of this one, gives different values.

grade12.0density0.63compressionnone

The three paragraph measures, computed for every paragraph of one longer text, The Federalist No. 10:

Paragraph measures across one document

Mean readability
18.20
Vocabulary TTR
0.25
Nominalization ratio
0.04
Passive ratio
0.42
Paragraph measures across the documentReadability gradeFlesch-Kincaid; higher reads harder010203040mean 18.2Lexical densitycontent words over all words0.300.400.500.60Compression ratiocompressed over original bytes; lower repeats more0.300.400.500.600.701357911131517192123¶
show the data
#Opens withWordsGradeDensityCompression
1AMONG the numerous advantages promised by a well…38420.720.470.46
2By a faction, I understand a number of…5124.890.430.56
3There are two methods of curing the mischiefs…2210.690.45none
4There are again two methods of removing the…4018.890.40none
5It could never be more truly said than…7912.900.460.52
6The second expedient is as impracticable as the…15714.590.440.45
7The latent causes of faction are thus sown…26318.680.530.42
8No man is allowed to be a judge…26712.920.470.46
9It is in vain to say that enlightened…8013.540.470.55
10The inference to which we are brought is,…3212.870.35none
11If a faction consists of less than a…17216.100.430.45
12By what means is this object attainable? Evidently…12113.740.470.50
13From this view of the subject it may…17921.570.450.48
14A republic, by which I mean a government…6113.910.440.58
15The two great points of difference between a…5023.030.48none
16The effect of the first difference is, on…17520.220.490.48
17In the first place, it is to be…12326.530.450.42
18In the next place, as each representative will…7632.060.420.55
19It must be confessed that in this, as…10317.280.460.52
20The other point of difference is, the greater…21624.700.440.45
21Hence, it clearly appears, that the same advantage…16914.140.490.48
22The influence of factious leaders may kindle a…12819.860.480.51
23In the extent and proper structure of the…5214.750.480.54

One panel per paragraph measure, paragraphs in document order along the bottom. A cross marks a paragraph matra declined to measure: the field is None because the paragraph falls outside the measure's applicability conditions above, and the line breaks there rather than inventing a value. The dashed line is the document's mean readability: the mean of the grades of the paragraphs matra measured. matra computes no document-level lexical density or compression ratio, so those panels have no line.

Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: federalist-10/metrics.json.

Document metrics#

grade12.8density0.53compressionnone

Every field of DocumentMetrics is computed over the sentences attached to the document's paragraphs, which excludes quotes. There is exactly one sentence set, and it is derivable from the tree in one way, so the numbers cannot disagree with the structure they describe.

Vocabulary type-token ratio#

vocabulary_ttr = distinct_lemmas / total_lemmas
InputAs computed
total_lemmasCount of non-punctuation tokens across the sentence slice
distinct_lemmasCount of distinct lemma strings in that same set, compared exactly, without case folding
grade10.2density0.61compressionnone

Applied to the whole slice. Stored in DocumentMetrics::vocabulary_ttr, which stays None when the slice contains no non-punctuation token.

grade6.3density0.63compressionnone

Citation. Johnson, W. (1944). Studies in language behavior: A program of research. Psychological Monographs, 56(2), 1-15.

grade13.0density0.74compressionnone

Limitations. Type-token ratio falls as documents get longer, because repetition accumulates. Values from documents of different lengths are not comparable without a length correction, and matra applies none. Because lemmas are compared exactly, a provider that emits case-varying lemmas inflates the distinct count.

Nominalization ratio#

nominalization_ratio = nominalizing_nouns / total_lemmas
InputAs computed
total_lemmasThe same denominator as vocabulary type-token ratio: non-punctuation tokens across the slice
nominalizing_nounsTokens whose pos is NOUN and whose text, lowercased, ends in tion, ment, ness, ity, ence, or ance
grade12.6density0.58compressionnone

The suffix test reads the surface form, not the lemma, so a plural nominalization such as conditions or measurements is missed: the plural -s falls after the suffix, so "conditions".ends_with("tion") is false even though the singular condition matches.

grade12.2density0.61compressionnone

Applied to the whole slice. Stored in DocumentMetrics::nominalization_ratio, which stays None when the slice contains no non-punctuation token.

grade10.4density0.57compression0.55

Limitations. Six suffixes are a heuristic, not a morphological analysis. Words that end in those letters without being nominalizations count as false positives, and nominalizations formed otherwise, such as growth or failure, are missed. The test depends on the POS tagger having assigned NOUN, so tagging errors propagate. The denominator counts every running word rather than every noun, so the value is a share of the text, not a share of the nouns in it.

Passive ratio#

passive_ratio = passive_sentences / total_sentences
grade10.5density0.56compressionnone

A sentence counts as passive when any of its tokens carries a dep of nsubj:pass, nsubjpass, or aux:pass. The first two are the Universal Dependencies and older Stanford spellings of a passive subject; the third is a passive auxiliary.

grade11.8density0.48compressionnone

Stored in DocumentMetrics::passive_ratio when the document is measured, and per sentence in Sentence::passive, which Sentence::new sets; both cross to Python as fields. The field is None when there are no sentences.

grade5.4density0.78compressionnone

Citation. de Marneffe, M.-C., Manning, C.D., Nivre, J., & Zeman, D. (2021). Universal Dependencies. Computational Linguistics, 47(2), 255-308.

grade8.3density0.61compressionnone

Limitations. This counts sentences that contain a passive construction, not passive clauses, so a sentence with three passives counts once. Detection is only as good as the parser's relation labels. Nothing here judges whether a passive is appropriate.

Sentence length statistics#

grade14.5density0.56compression0.49

DocumentMetrics::mean_sentence_length is total_words / total_sentences, where total_words counts non-punctuation tokens. DocumentMetrics::sentence_length_std is the sample standard deviation of per-sentence non-punctuation token counts, with denominator n minus 1. Both are None when undefined: no sentences for the mean, fewer than two for the deviation. Measuring the document stores both, with the counts total_words, total_sentences and paragraph_count, so all of them cross to Python.

Corpus aggregates#

grade11.6density0.52compression0.50

matra computes none. Every measure above is of one document, and how to combine documents is a decision about what the combination is for, so it is the caller's. The counts sum across documents. A ratio pools by summing its numerator and denominator rather than averaging the ratios, so the passive ratio of a set is its passive sentences over its sentences. vocabulary_ttr does not pool at all: it falls as text grows, so documents of different lengths are not comparable on it, alone or summed. A mean of paragraph readability grades averages paragraphs, not documents, so a long document contributes more terms than a short one; leave out the paragraphs whose slot is None.

Summarization#

grade13.6density0.66compressionnone

Both summarizers take a Document and a count, score every sentence, keep the highest N, and return them in ascending index, the sentence's position among the document's sentences. Neither generates text: the output sentences are the input sentences.

Shared inputs#

grade10.0density0.63compression0.56

A term is a token's lemma with punctuation tokens removed and stop words removed. The stop word test lowercases the lemma before looking it up; the term itself is the lemma as the provider produced it, without case folding. The stop word list holds 105 English words and is shared with the paragraph lexical density metric.

TF-IDF#

grade11.2density0.55compressionnone

Each sentence is treated as a document when computing document frequency.

score(s) = ( sum over distinct terms t in s of tf(t, s) * idf(t) ) / distinct_terms(s)

tf(t, s)  = occurrences of t in s / total terms in s
idf(t)    = ln( total_sentences / df(t) )
df(t)     = number of sentences containing t
grade4.9density0.49compressionnone

A sentence with no terms scores 0.0. The mean divides by the number of distinct terms, not by the number of term occurrences. A term present in every sentence has an idf of 0 and contributes nothing.

grade5.1density0.45compressionnone

Cap. 2,000 sentences. Above it, Error::InputTooLarge with what set to "tfidf".

grade7.4density0.61compression0.60

Citation. Luhn, H.P. (1958). The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development, 2(2), 159-165. Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11-21. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24(5), 513-523.

grade11.7density0.59compression0.54

Limitations. Treating sentences as documents is an adaptation of a corpus-level weighting scheme, and the resulting weights are not the weights you would get from a real document collection. Scores measure term rarity within one text, not informativeness. Selected sentences are not guaranteed to read coherently together, and nothing checks for redundancy between them.

TextRank#

similarity(a, b) = shared(a, b) / ( ln(1 + |a|) + ln(1 + |b|) )

shared(a, b) = sum over terms t in both of min(count_a(t), count_b(t))
|a|          = number of distinct terms in a
grade7.6density0.50compressionnone

Similarity is 0 when either sentence has no terms or they share none. The matrix is symmetric and its diagonal stays 0, so a sentence never reinforces itself.

score(i) at step k+1 = (1 - d) / N
                     + d * sum over j not equal to i of
                       ( similarity(j, i) / out_sum(j) ) * score(j) at step k

d          = 0.85
N          = number of sentences
out_sum(j) = sum of row j of the similarity matrix
grade4.4density0.55compressionnone

All scores start at 1 / N. Iteration stops after 50 passes, or earlier when the largest single-sentence change in a pass falls below 1e-6. Sentences whose out_sum is 0 send no weight.

grade5.9density0.52compressionnone

Cap. 2,000 sentences, which bounds the dense similarity matrix at roughly 32 MB of f64. Above it, Error::InputTooLarge with what set to "textrank".

grade5.4density0.69compression0.64

Citation. Mihalcea, R., & Tarau, P. (2004). TextRank: Bringing Order into Text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, 404-411. Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank Citation Ranking: Bringing Order to the Web. Stanford InfoLab Technical Report 1999-66.

grade11.0density0.56compression0.59

Limitations. Similarity here is lemma overlap. Two sentences expressing one idea in different words have similarity 0. The normalization by the logarithm of distinct-term counts is matra's, not the one in the 2004 paper, so scores are not comparable with other TextRank implementations. Reaching the iteration ceiling without converging is possible and is not reported.

Keyphrase extraction#

grade9.2density0.66compressionnone

Both extractors take &[Sentence] and a maximum count, and return Keyphrase values in descending score order. Both cap input at 200,000 tokens counted across the slice with punctuation included.

RAKE#

grade6.6density0.64compressionnone

Candidates. Within each sentence, a candidate is a maximal run of tokens whose pos is NOUN, ADJ, or PROPN and whose lowercased lemma is not a stop word. Punctuation, stop words, and any other part of speech end the run. Candidate words are lowercased lemmas.

for each candidate phrase p of length k, for each word w in p:
    freq(w)   += 1
    degree(w) += k

word_score(w)     = degree(w) / freq(w)
phrase_score(p)   = sum over w in p of word_score(w)
grade5.8density0.48compressionnone

When the same phrase string occurs more than once, the highest score is kept. Phrases are then sorted by score descending and truncated to the requested count.

grade4.1density0.45compressionnone

Cap. 200,000 tokens. Above it, Error::InputTooLarge with what set to "rake".

grade4.7density0.79compressionnone

Citation. Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010). Automatic Keyword Extraction from Individual Documents. In M.W. Berry & J. Kogan (Eds.), Text Mining: Applications and Theory. John Wiley and Sons.

grade11.9density0.62compression0.57

Limitations. Published RAKE delimits candidates with stop words and punctuation over surface words. matra adds a part-of-speech filter and works over lemmas, so scores and candidate sets differ from a reference implementation. A high degree over frequency ratio means a word appears in long phrases relative to how often it appears at all; that is a structural property, not evidence of topical centrality.

YAKE#

grade11.9density0.70compressionnone

Terms. Non-punctuation tokens whose lowercased lemma is not a stop word and is longer than one character. Position is a running index over every non-punctuation token, so stop words and single-character tokens occupy positions without becoming terms.

pos_score(t)         = ln( 1 + mean_position(t) / total_positions )
freq_score(t)        = occurrences(t) / total_positions
context_diversity(t) = distinct_neighbors(t) / neighbor_observations(t)

term_score(t) = ( pos_score(t) + context_diversity(t) ) / ( freq_score(t) + 1.0 )
grade8.4density0.51compression0.54

total_positions is the count of all non-punctuation tokens across the slice, floored at 1. A neighbor is the adjacent non-punctuation token on either side within the same sentence, recorded only when it is not a stop word; a term with no recorded neighbor gets a context diversity of 0.0. Lower term_score means more relevant at this stage.

ngram_score(p) = product over words w in p of term_score(w)
output_score(p) = 1.0 / ngram_score(p)
grade9.3density0.53compression0.57

Candidates are sliding windows of 1, 2, and 3 words over each sentence's term list, with stop words and single-character tokens already removed, so an n-gram can join words that were not adjacent in the source. A sentence with fewer than three terms yields correspondingly shorter windows. When a phrase string occurs more than once, the lowest ngram_score is kept.

grade10.9density0.47compression0.58

Candidates whose score is not strictly positive, or is not finite, are dropped before inversion. A term reaches a score of exactly 0 when it occurs only at position 0 and has no eligible neighbor, and any n-gram containing it is dropped with it. The remainder are sorted by output_score descending, so higher is more relevant in the returned Keyphrase.

grade4.1density0.45compressionnone

Cap. 200,000 tokens. Above it, Error::InputTooLarge with what set to "yake".

grade5.4density0.72compressionnone

Citation. Campos, R., Mangaravite, V., Pasquali, A., Jorge, A., Nunes, C., & Jatowt, A. (2020). YAKE! Keyword extraction from single documents using multiple local features. Information Sciences, 509, 257-289.

grade13.4density0.61compression0.53

Limitations. Published YAKE combines five features, including casing and sentence spread, and applies deduplication over similar candidates. matra implements three features, no deduplication, and a different combination formula. Scores are not comparable with the reference implementation, and the two will not agree on rankings. The inversion means output scores are unbounded above and have no interpretable unit; use them for ordering, not as magnitudes.

Determinism and reproducibility#

What is fixed#

grade7.4density0.54compression0.62

matra draws no random numbers, sets no seeds, samples nothing, and runs the analysis path on one thread. The model is pinned by SHA-256, and the bytes that pass verification are the bytes that get loaded. Given the same model file and the same input, the parse is the same.

grade18.2density0.68compression0.58

Given the same parse, these are reproducible bit for bit on one machine: every paragraph metric, every document metric, every derived method on Document, TextRank scores and selection, RAKE word and phrase scores, YAKE term and n-gram scores, and the order of keyphrases with equal scores. TextRank in particular is order-stable because its shared-term counts are integer sums and its iteration walks sentence indices in order.

What is not bit-stable#

gradenonedensity0.80compressionnone

One place carries hash map iteration order into the result.

  1. TF-IDF sentence scores. The per-sentence score sums over a hash map whose iteration order is not fixed between runs. Floating-point addition is not associative, so the low bits of a score can differ between runs.

    Selection changes only when two sentences differ by less than that rounding difference.

grade20.6density0.52compression0.61

RAKE and YAKE collect their candidates in the order each phrase is first met and sort them by score with a stable sort, so phrases with exactly equal scores come out in the same order on every run, and a tie straddling the truncation point is decided the same way each time.

grade8.8density0.67compressionnone

Two further sources of variation are worth recording in a methods section.

  • Natural logarithms come from the platform's math library. Values can differ by an ulp across operating systems and toolchains, so compare cross-platform results with a tolerance rather than for equality.
  • The compression ratio depends on the brotli encoder. A change in the encoder version can change the value for unchanged text.

What to record#

grade16.3density0.58compression0.57

To let someone reproduce a result, report: the matra version, the model file name and its SHA-256, which format the document was analyzed under, the method, and the parameter values it ran with, such as the requested sentence count or maximum phrase count. A summary, keyphrases or clusters carry every one of these in their provenance: matra, parser, format, method and parameters.

matra <crate version> (https://github.com/mox-labs/matra)
parse: UDPipe, english-ewt-ud-2.5-191206.udpipe
       SHA-256 784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957
format: markdown
method: tfidf
parameters: n = 5
grade15.6density0.57compressionnone

Cite the algorithm publications listed above for the methods themselves, and the UDPipe and Universal Dependencies references for the parse layer.

grade9.6density0.49compression0.47

Cite matra separately from the methods. The CITATION.cff file at the repository root carries the software entry, and it identifies the implementation and its version. The publications identify the method. Because matra departs from several published methods in the ways recorded on this page, a reader who has only the method citation does not know which values you produced, and a reader who has only the software citation does not know what the values are meant to measure. Both are needed.

General limitations#

gradenonedensity0.50compressionnone

These hold for every number on this page.

  • matra reports structure and stops. It measures; your application decides what a measurement means.
  • No output is a judgment of quality, correctness, originality, or authorship. Nothing here detects machine-generated text.
  • No output carries meaning. Passive ratio counts a syntactic pattern, compression ratio counts bytes, keyphrase scores count co-occurrence. None of them read.
  • Metrics that share a name with published measures are matra's implementations of them, with the departures listed per metric. Do not compare matra's values with another tool's values for a like-named metric.
  • Every value downstream of the parse inherits the parser's errors. On text unlike the English Web Treebank, that inheritance is larger.

Errors

grade13.6density0.64compression0.59

Every fallible function in matra returns domain::Result<T>, which is std::result::Result<T, domain::Error>. No function in the library returns Result<T, String>. matra signals failure by returning Error, not by unwinding: a panic raised inside the UDPipe C++ boundary is caught at the adapter and converted into Error::ParseFailed.

grade6.9density0.52compression0.64

Error is #[non_exhaustive]. Variants can be added in a minor release, so a match on it from another crate needs a catch-all arm. Error does not derive Clone; it is moved, not copied. The stream surface wraps it in DocumentError, which adds the path the failure occurred at, and yields it as that input's Err item.

grade6.8density0.50compression0.50

Every error reads the same way on every channel. It has a kind, a stable string to branch on. It has a message, for a person. And it has details: the values a caller acts on, as named data, so nothing has to parse the message. For every kind but invalid_input that is every dynamic value in the message; an invalid_input names the check that failed, and its message adds the values that failed it for a person. In Rust these are Error::kind, Display and Error::details. The serde form is the three together, and Python exceptions carry the same three. See kinds and details.

The variants#

VariantPayloadReturned when
ModelNotFoundPathBufA model file does not exist at the given path
ModelInvalidStringBytes that arrived could not be verified against the pin, or could not be loaded
ParseFailed{ reason: Option<&'static str>, cause: String }The parser or a reader failed on the input, or returned what its contract forbids. reason names the engine's check that failed, when one did
InputTooLarge{ limit: usize, actual: usize, what: &'static str }A size gate rejected the input. what names the gate
UnsupportedFormat{ format: Option<Format>, feature: Option<&'static str> }The input is in a format no reader in this build reads, or is not UTF-8 text
InvalidInput{ reason: &'static str, message: String }The call cannot be honored as made. reason names the check that failed
Iostd::io::ErrorA filesystem operation failed, or a download did not arrive
grade13.3density0.67compressionnone

Error implements std::error::Error and Display through thiserror, and From<std::io::Error>, so ? converts I/O failures automatically.

ModelNotFound#

grade11.6density0.55compression0.63

Returned by Model2Vec::from_dir when any of the three artifacts is absent. The payload is the path of the first one missing. Neither the UDPipe provisioner, behind Engine::new and Engine::from_config in Rust and Python, nor Model2Vec::potion_base_8m returns this variant: both download the model when the file is absent.

ModelInvalid#

gradenonedensity0.50compressionnone

Returned by:

  • The UDPipe provisioner when the bytes it downloaded still fail SHA-256 verification after one refetch. The payload is SHA-256 mismatch after re-download from <url>. Nothing that failed verification is written, so the model directory is left as the call found it.
  • Model2Vec::from_dir when an artifact does not parse, uses an embedding dtype other than f32, or panics the loader.
  • Model2Vec::potion_base_8m when the directory already holds all three artifacts and their digest is not the pinned one, or holds only some of them. The payload names the directory and the ways out; nothing there is downloaded over or removed.
  • Model2Vec::potion_base_8m when the artifacts it downloaded still fail the digest after one re-download. Both attempts verify in memory, so neither writes anything, and the directory is left exactly as the call found it. The payload names the directory and the expected digest.
grade7.8density0.56compressionnone

A file whose hash does not match the pinned constant is treated as untrusted and is never loaded.

grade10.4density0.55compression0.57

ModelInvalid is about bytes that arrived. A download that never arrived, a server that answered about the request rather than with a model, and a filesystem that refused the write are Io, not ModelInvalid (RFC-0015). Before 0.2.0 the UDPipe download funnelled all three here, so a DNS failure reported model_invalid; a consumer that branched on that reads the provisioning failures section below.

ParseFailed#

grade7.6density0.44compression0.58

The parser or a reader failed on the input, or returned what its contract forbids. The remedy is the component, not the call: the same input through the same parser fails the same way. cause is free text for a person, and reason is present when the failure is one of the engine's checks.

grade10.0density0.52compressionnone

A parser or a reader reports its own failure with reason None. The UDPipe adapter's parse does so in four situations:

  • The underlying parser returns an error. The cause is that error's message.
  • The underlying parser panics. The cause is udpipe panicked: <message>. The catch_unwind boundary that produces this lives in nlp/udpipe.rs, and it exists because an unhandled panic crossing the C++ boundary aborts the host process instead of unwinding.
  • A token id or head value cannot be represented as usize. The cause names the token and its sentence.
  • A token's form cannot be found in the paragraph text the parser was given, at or after the previous token, so the token has no span. The cause names the token, its sentence and the byte the search began at. The adapter reports this rather than a guessed span.
grade30.6density0.57compression0.51

The engine trusts no parser and no reader, matra's own included. Before anything a reader returns becomes a section, and before anything the parser returns for a paragraph becomes a sentence, the engine checks it against the contract its trait states (matra::decompose::Decomposer, matra::nlp::NlpProvider), and a break fails the document with reason naming the check and cause naming the block, or the paragraph, sentence and token:

reasonCheck that failedOf
token_id_sequenceA sentence's token ids do not run 1..=n in orderthe parser
head_out_of_rangeA head is neither 0 nor the id of a token in the sentencethe parser
root_countA sentence has no token whose head is 0, or more than one; a sentence with no tokens has nonethe parser
head_cycleFollowing heads from a token never reaches the rootthe parser
token_span_invalidA token's span is reversed, runs past the paragraph, or splits a characterthe parser
token_span_orderA token's span starts before the previous token's ends, within a sentence or across twothe parser
token_form_mismatchThe paragraph text at a token's span is not its form (a run of whitespace in the form matches any run of whitespace)the parser
block_span_invalidA block's or a skipped construct's span is reversed, runs past the analysed text, or splits a charactera reader
block_span_orderThe blocks are out of document order, or two spans among the blocks and skipped constructs overlapa reader
heading_levelA heading is at level 0a reader
paragraph_text_mismatchA paragraph's text is not the analysed text its span coversa reader
grade7.6density0.55compressionnone

A conforming parser or reader is one whose output passes these checks on every input it accepts. The checks are a pass over what they read: on the docsite's own pages they cost about 0.014% of UDPipe's parse time, about 3 microseconds a paragraph.

InputTooLarge#

grade5.4density0.60compressionnone

Ten gate labels produce this variant. The what field carries the label so a caller can route each gate differently.

whatGateLimitMeasured over
"input"The engine's reading stage, the only route from text to the parser, for every Input constructor, and the CLI's stdin read, which applies the same cap firstMAX_INPUT_BYTES, 8 MiB (8,388,608)Byte length of the input handed in, and of the analysed text the reader produces (for block input, the blocks joined); the CLI counts the bytes as they arrive, before decoding them
"file_read"Reading the file an Input::file namesMAX_INPUT_BYTES, 8,388,608 bytesFile size reported by the filesystem, checked before any read
"tfidf"summarize, and summarize_with with SummaryMethod::TfIdfMAX_TFIDF_SENTENCES, 2,000Number of sentences in the document
"textrank"summarize_with with SummaryMethod::TextRankMAX_TEXTRANK_SENTENCES, 2,000Number of sentences in the document
"rake"keyphrases, and keyphrases_with with KeyphraseMethod::RakeMAX_RAKE_TOKENS, 200,000Total tokens across the document's sentences, punctuation included
"yake"keyphrases_with with KeyphraseMethod::YakeMAX_YAKE_TOKENS, 200,000Total tokens across the document's sentences, punctuation included
"clusters"clusters, before anything is embeddedMAX_CLUSTER_SENTENCES, 2,000Number of sentences in the document
"parser_model_download"The UDPipe provisionerMAX_DOWNLOAD_BYTES, 64 MiB (67,108,864)Bytes read from the response, which stops one past the cap
"embedding_model_download"Model2Vec::potion_base_8m, per artifactMAX_DOWNLOAD_BYTES, 64 MiB (67,108,864)Bytes read from the response, which stops one past the cap
"config_file_read"Config::resolve and Config::from_file, reading a config fileMAX_CONFIG_FILE_BYTES, 64 KiB (65,536)File size reported by the filesystem, checked before any read
grade10.5density0.59compressionnone

limit carries the cap and actual carries the measured size, so an error message can be built without hardcoding the constants. Each cap is a published constant in domain::limits, named in the table; the module's documentation carries the cost model behind each value. No configuration changes any of them.

grade6.8density0.58compression0.52

The caps bound worst-case memory and time. TextRank builds a dense similarity matrix that reaches roughly 32 MB of f64 at 2,000 sentences. RAKE and YAKE build phrase-keyed maps whose size follows token count rather than sentence count, which is why their caps are stated in tokens. The two download caps are the gates whose input is not the caller's: they bound what a redirected or misbehaving server can make the process hold, and the read stops at the bound rather than continuing, so actual reports the bound that was breached rather than the response's full length. Both are 64 MiB per file, against the 16.3 MB parsing model and the embedding model's largest file, 30.2 MB of its 30.9 MB set, which is headroom for a later version and still finite.

grade9.2density0.52compressionnone

A document from disk crosses both gates in sequence: "file_read" when the file is read, "input" before it is parsed. In practice "file_read" fires first, since both carry the same 8 MiB limit and the file size is checked before the read.

grade11.6density0.48compressionnone

Every route from text to a parser, a parser of your own included, runs through the engine, so no text bypasses the "input" gate on its way to one.

grade9.4density0.52compression0.59

The label has a second producer. The CLI reads stdin through read_capped in src/cli/mod.rs, which reads at most one byte past MAX_INPUT_BYTES and returns InputTooLarge with what set to "input" before the bytes ever reach the engine. Same limit, same label, so a caller sees one gate however the text arrived. The CLI never decodes what it reads: the bytes go to the engine as Input::bytes, so an oversized pipe is reported as too large, and a pipe that is not UTF-8 is UnsupportedFormat, as a file's bytes would be.

grade7.2density0.52compression0.50

The four ranking methods check their caps after their empty-result checks. summarize_with(&doc, 0, ..) and keyphrases_with(&doc, 0, ..) return an empty result without evaluating the cap, whatever the size of the document. The same holds for a document with no sentences. clusters has no count parameter; its cap is checked first, before anything is embedded, then its contract, and a document with no sentences returns empty clusters.

grade14.7density0.52compressionnone

A metric has no gate of its own. The compression ratio skips any paragraph over 262,144 bytes (MAX_COMPRESSION_PARAGRAPH_BYTES) and leaves its metric slot at None rather than returning an error, and a paragraph under a metric's minimum word count (MIN_READABILITY_WORDS, MIN_COMPRESSION_WORDS) does the same.

InvalidInput#

grade9.8density0.50compressionnone

The call cannot be honored as made, and making it again as it is will fail the same way. The remedy is to change something the caller controls: the arguments, the configuration, the data handed in, or the embedder passed to clusters. Nothing about analysed text produces it.

grade9.7density0.60compression0.57

One kind covers every such check, because they share that remedy, and reason says which check failed: a stable snake_case name, in details["reason"] on every channel, to branch on where the remedy differs. message names the values that failed it, for a person. A reader treats the set of reasons as open, since a later version can add a check.

reasonCheck that failedRaised by
threshold_not_finiteThe threshold is NaN or infiniteclusters
embedder_vector_countThe embedder returned a different number of vectors than there are sentencesclusters
embedder_dimensions_differTwo of the embedder's vectors have different lengthsclusters
embedder_value_not_finiteA vector holds a NaN or an infinityclusters
not_a_directoryThe path is not a directoryInputs::dir
is_a_directoryThe path is a directory where one file is readEngine::analyze of an Input::file or Input::file_as
blocks_malformedBlock input is not an array of heading and paragraph blocksEngine::analyze of input read as Format::Blocks; Python's Engine.analyze(blocks=...)
heading_level_zeroA heading block has level 0Engine::analyze of block input, from Input::blocks, JSON or Python
config_malformedThe config file does not parse, holds an unknown key, or a value of the wrong typeConfig::resolve, Config::from_file
config_key_renamedThe config file sets a key 0.3.0 renamed or removed ([summarize], algorithm, [semantic], models.udpipe); the message names each replacementConfig::resolve, Config::from_file
config_method_unknownsummary.method or keyphrases.method names a method this build does not knowConfig::resolve, Config::from_file
config_threshold_not_finiteclusters.threshold is NaN or infiniteConfig::resolve, Config::from_file
config_not_path_componentmodels.embedding is not a single directory nameConfig::resolve, Config::from_file
config_not_utf8The config file is not valid UTF-8Config::resolve, Config::from_file
config_default_missingThe built-in defaults lack a key: a defect in this build, to reportConfig::resolve, Config::from_file
no_data_directoryNone of MATRA_DATA_DIR, XDG_DATA_HOME or HOME is setConfig::resolve, Engine::new
format_unknownA format name matra does not knowPython's Engine.analyze
method_unknownA method name matra does not knowPython's summarize and keyphrases
embedder_identity_failedThe embedder has no identity(), or it raisedPython's clusters
embedder_identity_shapeidentity() returned something other than an Identity dictPython's clusters
embedder_raisedThe embedder's embed() raisedPython's clusters
embedder_return_shapeembed() returned something other than a sequence of number sequencesPython's clusters
usageThe arguments do not parse: an unknown flag, a missing valuethe command line, under --json
no_commandNo subcommand and neither --skill nor --schemathe command line
reference_without_skill-r without --skillthe command line
reference_unknown--skill -r names no referencethe command line
schema_unknown--schema names no schemathe command line
format_with_directory--format given with a directorymatra analyze
directory_for_one_documentA directory given to a command that reads one documentmatra summarize, keyphrases, clusters
method_unavailableThe configured method is one this build cannot runmatra summarize, keyphrases
embedder_unavailableThe build has no embedder (no model2vec feature)matra clusters
config_file_existsThe config file exists and --force was not givenmatra config init
no_config_file_locationNone of MATRA_CONFIG_FILE, XDG_CONFIG_HOME or HOME is setmatra config init
config_key_unrenderedA resolved key has no renderer: a defect in this build, to reportmatra config show
embedded_file_malformedA schema or skill reference compiled into the binary does not read: a defect in this build, to reportmatra --schema, matra --skill
grade14.3density0.57compressionnone

A config file that is simply absent is not an error for Config::resolve, and the built-in defaults stand; Config::from_file refuses one with Io, because the caller named it.

UnsupportedFormat#

gradenonedensity0.67compressionnone

Returned by Engine::analyze in two cases:

  • The engine has no reader for the input's format: Pdf or Docx, whether named by the caller, read off the extension, or recognised from the bytes (%PDF, or a zip archive holding [Content_Types].xml, which outranks the extension), or a Format::Other the caller named and registered no reader for. format is that format, and feature names the Cargo feature that would read it; no feature of this version reads any of them, so feature is None. A reader registered with Engine::with_reader reads any of them. Markdown, PlainText and Blocks always have a reader.
  • The bytes are not UTF-8 text and carry no signature matra recognises. format and feature are both None. This used to be Io with kind InvalidData; nothing went wrong reading the bytes, so it is now a kind with a remedy.
grade6.1density0.47compressionnone

A UTF-8 file with an extension matra does not know is not this error: it reads as plain text.

Io#

gradenonedensity0.75compressionnone

Wraps std::io::Error, produced by:

  • Reading a file rejecting a symlink, with ErrorKind::Unsupported and the message refusing to read symlink: <path>; Inputs::dir refuses a symlinked directory the same way. The path is also details.path.
  • Reading a file rejecting a path that is neither a regular file nor a directory, with ErrorKind::InvalidInput and the message not a regular file: <path>, and the path in details.path. A directory is InvalidInput instead, naming Inputs::dir.
  • Any read, directory listing, directory creation, write, or rename that fails.
  • The UDPipe provisioner when a download fails at the transport or answers with a non-2xx status. Same shape as the embedding path below: the message names the URL, and the kind is TimedOut past the 300-second fetch budget or the 30-second connect budget, NotConnected for an unreachable host, and whatever the socket reported otherwise. The URL is also details.url, and details.retryable is true, except for a rejected certificate.
  • The UDPipe provisioner when the model directory cannot be created, or when the verified bytes cannot be written or renamed into place. The message names the operation and the path, so a full disk reads cannot write the model to <path>: No space left on device (os error 28) rather than Permission denied (os error 13) with nothing to act on. The path is also details.path.
  • Model2Vec::potion_base_8m when a download fails at the transport or answers with a non-2xx status. The message names the URL, and the kind is TimedOut past the 300-second fetch budget or the 30-second connect budget, NotConnected for an unreachable host, and whatever the socket reported otherwise. The URL and retryable are details, as on the UDPipe path. Bytes that arrived and then failed the digest are ModelInvalid instead; this variant is for the ones that never arrived.
  • Model2Vec::potion_base_8m when the model directory cannot be created, when an artifact cannot be read, or when a verified artifact cannot be written or renamed into place. The message names the operation and the path, exactly as the UDPipe path's does, and the path is details.path. Model2Vec::from_dir reports an unreadable artifact the same way.
  • Model2Vec::potion_base_8m when the temporary file an artifact lands through cannot be created, with the kind the open reported (AlreadyExists when something is already sitting at that path). The temporary is opened exclusively, so a path already there, symlink or not, fails the open rather than being written through.
  • Engine::analyze when reading a file yields no document, with ErrorKind::InvalidData and the message source returned no documents.

Provisioning failures#

grade13.2density0.50compressionnone

The UDPipe provisioner (behind Engine::new and Engine::from_config, in Rust and Python) and Model2Vec::potion_base_8m obtain a pinned artifact over the network. Both fetch through matra's own client, under the same bounds, and both classify a failure the same way.

ConditionVariantKindWhat the message carriesDetails
DNS failure, unreachable host, refused connectionIo, ErrorKind::NotConnectedioThe URLurl, retryable: true
Connect budget exceeded (30 seconds)Io, ErrorKind::TimedOutioThe URLurl, retryable: true
Fetch budget exceeded (300 seconds, lookup through last byte)Io, ErrorKind::TimedOutioThe URLurl, retryable: true
TLS certificate rejectedIoioThe host, why matra cannot be made to trust it, the way out, then the underlying failureurl, retryable: false
Non-2xx statusIoioThe URL and the statusurl, retryable: true
Response past 64 MiBInputTooLargeinput_too_largewhat is "parser_model_download" or "embedding_model_download"what, limit, actual
Directory, write or rename failedIoioThe operation and the pathpath
Bytes arrived and failed the pinned digest, twiceModelInvalidmodel_invalidThe URL, or the directory and the expected digestcause
Bytes arrived, passed the digest, and did not loadModelInvalidmodel_invalidThe loader's messagecause
grade9.8density0.56compressionnone

The rule behind the table: model_invalid is about bytes that arrived. Anything that stopped a fetch from arriving, or a filesystem from accepting it, is io (RFC-0015).

grade6.8density0.45compression0.58

retryable is true when the same call can be made again as it is and may succeed. A failed fetch writes nothing, so a retry starts from where the first call did. A rejected certificate is false: a retry meets the same proxy, and the message names the way out instead.

grade8.7density0.51compression0.53

The rule has a consequence in Python that is easy to miss, because it changes which except clause fires rather than only which string a message carries. IoError is an OSError and ModelInvalidError is a RuntimeError, so from 0.2.0 a DNS failure, a rejected certificate, a timeout or a non-2xx status raises an OSError from the call that builds the engine, where it used to raise RuntimeError. A caller that wrapped a bootstrap in except RuntimeError catches nothing now and the OSError propagates past it. Catch matra.MatraError, which both derive from, and branch on kind.

grade14.1density0.55compression0.44

Nothing that failed the digest is ever written. Both provisioners fetch into memory, verify there, and write only what verified, so a download that fails adds nothing to the model directory and a run killed mid-transfer leaves nothing at all. A failed run removes nothing either. A cached file that is already there and fails verification is not loaded and not deleted: the UDPipe provisioner downloads again, at most twice, and replaces the file only with bytes that verified, through the same atomic rename every install lands through. A run whose downloads fail, at the network or at the digest, leaves that file byte-identical. Model2Vec::potion_base_8m replaces nothing and refuses a directory whose artifacts fail verification, because its three filenames belong to the artifact format rather than to this one model and so may be a caller's own (RFC-0015). A temporary left by a killed process is reclaimed by the next download that finds it older than ten minutes, which is twice the fetch budget and therefore older than any transfer that could still be running. The UDPipe provisioner leaves a temporary directory and Model2Vec::potion_base_8m three temporary files, and each reclaims its own by the same rule.

Behind a TLS-intercepting proxy#

grade25.5density0.63compression0.57

matra verifies TLS against root certificates compiled into the binary and never reads the system trust store, so a proxy that re-signs TLS cannot be made trusted. The fetch fails as Io, kind io, and the message names the host, says why its certificate cannot be trusted, names the file to fetch by hand, and ends with the underlying failure:

matra: io error: download https://lindat.mff.cuni.cz/...: the TLS certificate offered for
lindat.mff.cuni.cz was rejected. matra verifies TLS against root certificates compiled into
it and never reads the system trust store, so a proxy that re-signs TLS cannot be trusted by
installing its CA. Fetch english-ewt-ud-2.5-191206.udpipe by hand and put it in the model
directory instead. Underlying failure: io: invalid peer certificate: ...
grade8.4density0.60compressionnone

Install the model behind a TLS-intercepting proxy has the steps.

The first run says so#

grade9.5density0.48compressionnone

A cold run has to fetch 16.3 MB before it can answer, and how long that takes is the link's business rather than matra's. The command line writes two lines to standard error before the transfer and nothing at all when the model is already there:

matra: downloading english-ewt-ud-2.5-191206.udpipe (16.3 MB) into /home/u/.local/share/matra/models
matra: the model is licensed CC BY-NC-SA 4.0 (non-commercial), separately from matra: https://creativecommons.org/licenses/by-nc-sa/4.0/
grade12.4density0.45compressionnone

The second line is there because the model is a separate file with its own license, which is not matra's MIT; model licenses has both models and their sources.

grade11.1density0.63compressionnone

Standard error, so --json output stays a single object on standard output. --quiet silences it. A library caller gets the same facts as a domain::ProvisionNotice from Engine::from_config_with_notice, and decides the wording itself.

grade8.0density0.59compressionnone

The first matra clusters says the same for the embedding model, whose three files are fetched as one set of 30.9 MB:

matra: downloading potion-base-8M (30.9 MB) into /home/u/.local/share/matra/models/potion-base-8M
matra: the model is licensed MIT, separately from matra: https://opensource.org/license/mit
grade13.7density0.60compression0.47

Python writes the same two lines to sys.stderr before Engine(), Engine.from_config(), Model2Vec.potion_base_8m() or Model2Vec.from_config() downloads; redirect sys.stderr to silence them. A Rust caller gets the facts from Engine::from_config_with_notice and Model2Vec::from_config_with_notice and chooses the wording, since the library writes to no terminal. The other Rust constructors, Engine::new, Engine::from_config, Model2Vec::potion_base_8m and Model2Vec::from_config, take no notice and say nothing for the length of their download.

Per-document failures: DocumentError#

grade12.4density0.59compression0.55

The stream surface never lets one bad document abort the rest. Engine::analyze_all yields Result<Document, DocumentError> per input, where DocumentError pairs the Error with the path it occurred at (None for an input that is not a file, so a path-less document is distinguishable from one at an empty path). Its Display prints path: error when a path exists and the bare error otherwise, and its source() is the wrapped Error.

grade14.0density0.61compression0.60

The stream yields one item per input, in order: the Document, or the DocumentError in its place, so documents plus errors equals inputs consumed, and partitioning them is the caller's. Inputs::dir is Err only when the listing itself fails; both read failures and analysis failures travel as stream items.

grade8.8density0.56compressionnone

Two kinds of entry never appear on either side. The directory listing skips symlinks, so a symlink produces no document and no error. It also skips anything that is not a regular file, and the walk is one level deep, so a subdirectory is skipped the same way.

grade7.3density0.59compression0.51

DocumentError serializes as its error's form plus the path: {"path", "kind", "message", "details"}, with path null for input that never came from disk. Python's Engine.analyze_all yields that form as a dict for each failed document. kind is the stable string Error::kind names for the variant, and the table below carries all seven. Error::kind is a match with no wildcard arm, so a new Error variant fails to compile until someone names it, and spec/tests/corpus/items.json pins the vocabulary with a runner per crust, so every crust spells a failure the same way. Both item shapes come from serde. Either's path (provenance.source.path on a document, path on an error) is decoded with os.fsdecode, so os.fsencode on it names the file the walk read even when that name is not valid UTF-8. The Python call yields the items in order and the caller partitions them, testing isinstance(item, matra.Document).

Kinds and details#

grade9.3density0.53compressionnone

Error::details returns an ErrorDetails: the values a caller acts on, by name, which for every kind but invalid_input is every dynamic value in the message. Each value is a DetailValue, which is text, a whole number or a boolean. A caller reads a value instead of parsing the message.

KindDetails
model_not_foundpath
model_invalidcause
parse_failedcause; reason when what the parser or a reader returned failed one of the engine's checks, from the table above
input_too_largewhat, limit, actual
unsupported_formatformat when one was recognised; feature when a Cargo feature would read it
invalid_inputreason, the check that failed, from the table above
ioio_kind and cause; path when the failure concerns a path matra named; url and retryable when a download did not arrive
  • One word, one meaning. reason is always the name of a check that failed, stable and snake_case, to branch on: on invalid_input always, and on parse_failed when the engine's checks refused what a parser or a reader returned. cause is always free text for a person, like the message: the text the variant holds, or the operating system's account of an I/O failure.
  • io_kind is the I/O failure's class in snake case: not_found, permission_denied, already_exists, unsupported, invalid_input, invalid_data, timed_out, not_connected, connection_refused, connection_reset, unexpected_eof, interrupted, not_a_directory, is_a_directory or storage_full. Any other class is other. Treat the set as open.
  • retryable appears only on a download that did not arrive. Its absence makes no claim either way.
grade7.9density0.53compressionnone

A detail that does not apply is absent, never empty: an unsupported_format for bytes that are not text carries no details at all. A key a kind carries keeps appearing in later versions. A later version may add keys, so read the ones you know and ignore the rest.

gradenonedensity0.71compressionnone

The serde form is {"kind", "message", "details"}:

{ "kind": "input_too_large", "message": "input too large: 9000000 > limit 8388608",
  "details": { "what": "input", "limit": 8388608, "actual": 9000000 } }
grade8.4density0.50compressionnone

spec/schema/v2/error.schema.json is the published JSON Schema for this form and for DocumentError's, with the keys each kind carries.

Display strings and kinds#

grade8.2density0.51compressionnone

These are the strings Display produces, which are also the strings Python's str(exc) returns, alongside the kind Error::kind reports for each variant. A message is for a human to read; a kind is what code branches on.

VariantFormatKind
ModelNotFound(path)model not found: {path}model_not_found
ModelInvalid(s)invalid model: {s}model_invalid
ParseFailed { reason, cause }parse failed: {cause}parse_failed
InputTooLarge { limit, actual, what }{what} input too large: {actual} > limit {limit}, or input too large: {actual} > limit {limit} when what is "input"input_too_large
UnsupportedFormat { format, feature }unsupported format: {format}; no feature of this matra version reads it, unsupported format: {format}; no reader is registered for it for a Format::Other, unsupported format: {format}; enable the `{feature}` feature to read it, or unsupported format: the input is not UTF-8 text and carries no signature matra recognises when format is Noneunsupported_format
InvalidInput { reason, message }invalid input: {message}invalid_input
Io(e)io error: {e}io
grade12.3density0.67compressionnone

UnsupportedFormat renders the format's wire name, so the string reads unsupported format: pdf; no feature of this matra version reads it.

Matching in Rust#

use matra::domain::{Error, Format};
use matra::{Engine, Input};

fn report(text: &str, engine: &Engine) {
    match engine.analyze(Input::text(text, Format::PlainText)) {
        Ok(doc) => println!("{} sentences", doc.metrics.total_sentences),
        Err(Error::InputTooLarge {
            what,
            actual,
            limit,
        }) => eprintln!("{what} gate: {actual} over limit {limit}"),
        Err(Error::UnsupportedFormat { format, .. }) => eprintln!("no reader for {format:?}"),
        Err(other) => eprintln!("{}: {}", other.kind(), other),
    }
}
grade11.5density0.50compressionnone

The details carry the same values without a match on the variant, which is what a caller forwarding an error to another program wants:

use matra::domain::{DetailValue, Error};

fn limit_of(err: &Error) -> Option<u64> {
    match err.details().get("limit") {
        Some(DetailValue::Integer(limit)) => Some(*limit),
        _ => None,
    }
}

Python exception mapping#

grade7.0density0.54compression0.56

The PyO3 binding raises each Error as a class of its own, named for the variant with PEP 8's Error suffix. The name a failure shares across languages is its kind, not its class. Every class derives from matra.MatraError and from the builtin class matra raised before 0.3.0, so except ValueError still catches an InputTooLargeError. The routing is a match with no wildcard arm over the variants, so adding a variant to Error without assigning it a class fails to compile.

VariantPython classAlso a
ModelNotFoundModelNotFoundErrorFileNotFoundError
ModelInvalidModelInvalidErrorRuntimeError
ParseFailedParseFailedErrorRuntimeError
InputTooLargeInputTooLargeErrorValueError
UnsupportedFormatUnsupportedFormatErrorValueError
InvalidInputInvalidInputErrorValueError
Io, any other kindIoErrorOSError
Io, kind NotFoundIoNotFoundErrorFileNotFoundError
Io, kind PermissionDeniedIoPermissionErrorPermissionError
grade8.4density0.52compression0.54

Io routes on the I/O kind it wraps. A missing file is an IoNotFoundError, which is a FileNotFoundError; a refused permission is an IoPermissionError, which is a PermissionError. Both derive from IoError, so except matra.IoError catches all three. matra.IoError is matra's class, not the builtin IOError, which is another name for OSError.

grade17.4density0.57compressionnone

Every exception carries the same three things as the serde form: kind, the string Error::kind names; details, a dict of the details; and the Display string as its message, which str(exc) returns. Branch on the class or on kind, and read values from details:

import matra

try:
    engine.analyze(huge)
except matra.InputTooLargeError as exc:   # also a ValueError
    print(exc.kind, exc.details["what"], exc.details["limit"], exc.details["actual"])
gradenonedensity0.60compressionnone

IoNotFoundError is a FileNotFoundError, so the conventional Python idiom works:

try:
    doc = engine.analyze(file="notes.md")
except FileNotFoundError as exc:
    print(exc.details["path"])  # notes.md
grade9.1density0.55compressionnone

The classes name matra as their module, so a traceback reads matra.InputTooLargeError, and an exception survives pickle with its kind and details.

grade11.1density0.63compressionnone

Every Python input reaches the parser through the engine's reading stage, so the 8 MiB "input" gate applies uniformly, to text, bytes and blocks alike, and a file meets "file_read" first. The outcome functions take a Document, never text, and their own caps apply to it.

grade7.7density0.61compressionnone

One further failure has no Error variant behind it. Python's Engine is #[pyclass(unsendable)], because the loaded model holds C-side state that is not thread-safe. Accessing one engine from a thread other than the one that created it fails at runtime. Multi-process use is unaffected.

At the command line#

grade9.6density0.56compression0.59

One program is installed under the name matra, by either of two routes. The Rust binary comes from cargo install matra --features cli; the Python console script comes from the wheel and calls the same matra::cli::run through the extension module. The exit-code contract below is the program's, so it holds on both.

Exit codeMeaning
0The command succeeded, and where applicable something was found
1The command succeeded and found nothing
2An error occurred
grade12.9density0.58compression0.53

On exit code 2 the command writes matra: followed by the error's Display string to standard error, prefixed with the path when the failure concerns a file. Under --json it also writes the envelope to standard output, with result null and error holding the serde form above, {kind, message, details}: the same three a Rust caller serializes and a Python caller reads off the exception. The envelope's input names the file, so the error object carries no path of its own.

{ "format_version": 2, "command": "analyze", "input": "scan.pdf", "result": null,
  "error": { "kind": "unsupported_format", "message": "unsupported format: pdf; no feature of this matra version reads it", "details": { "format": "pdf" } } }
grade9.1density0.57compressionnone

spec/tests/parity/parity.json pins that equality: the same files through Rust, Python and the command line give the same kind, message and details, and a runner per surface fails on a drift.

grade4.8density0.58compressionnone

A broken pipe, which is what happens when the reading end of matra analyze file.md | head goes away, exits 0 and prints nothing.

grade9.8density0.57compression0.57

matra analyze DIR reads every file in the directory and keeps going past a failure. Each failed file is a line on standard error and, under --json, its own envelope line with its error; the run exits 2 if any file failed, after the walk has finished. This is the one partial success the command line has: per-item failure in a bulk operation.

grade16.1density0.54compression0.51

The table is the whole contract, and no command narrows it. The command line's own refusals carry the kind whose remedy matches, which for each of them is InvalidInput, a call to change: a directory given to summarize, keyphrases or clusters, --format with a directory, matra --skill -r <name> or matra --schema <name> with a name nothing answers to (both name the ones that exist), -r without --skill, config init over an existing file without --force, clusters in a build without the model2vec feature, and a usage error such as an unknown flag. A missing path is not one of them: it is the library's own Io failure with not_found, raised before the engine is built so it costs no model load. matra --skill and matra --schema in any of their accepted forms exit 0; they read no document, so they have nothing to find and never exit 1.

grade13.4density0.55compressionnone

The exception classes above belong to the library surface, not to the command line. A domain::Error reaching the command line is rendered and turned into an exit code and, under --json, an envelope; it never surfaces as a Python exception, whichever launcher started it.

Boundary rules

gradenonedensity0.88compressionnone

Eight rules hold matra's hexagonal architecture in place.

grade4.8density0.64compressionnone

CLAUDE.md carries a summary list and points here for the reasoning.

What enforcement means here#

grade7.4density0.60compression0.59

Rust offers no directional import control between modules inside a single crate, and matra is a single crate by RFC-0004, so the compiler enforces none of these rules except rule 6. Seven of the eight are checked by semgrep rules in .semgrep/, which read the import and path forms each rule forbids. Review remains the gate for what an import cannot show.

RuleChecked byWhat the check catchesWhat review still reads for
1. Domain dependency setboundary-rule1-*any use or inline path in src/domain.rs rooted outside std, core, alloc, serde, thiserrora #[macro_use] macro used unqualified
2. Ports import only domainboundary-rule2-*crate:: or super:: into anything but domain, external crates, the port's own adaptersa trait whose shape leaks an adapter
3. No cross-port importboundary-rule3-*a port named by use, brace group, inline path or trait boundmacro-generated paths
4. Single udpipe_rs importerboundary-rule4-*udpipe_rs named anywhere else, and anything pub in the adapter that names ita private alias made public under another name
5. Metrics and extraction purityboundary-rule5-*any crate module other than domain and stopwordsa function taking text instead of parsed structure
6. No-default-features buildCIthe whole rule, by compiling
7. Composition root knows the wholeboundary-rule7-*src/cli/ reaching past Engine, Input, Inputs, Block, the outcome functions and their selectors, config and domain; src/python/ reaching past the same, embed and cliwiring outside src/lib.rs in any other file
8. No tracing in domain or portsboundary-rule8-*the identifier in code in the five files
grade6.4density0.47compression0.64

Each finding names its rule, says why the boundary exists, and links the section of this page to read. The full table, with every form each check covers and misses and the reasons the checks are built the way they are, is EP-0014. Code inside #[cfg(test)] modules is exempt from rules 1, 2, 3, 5 and 7, and read by rules 4 and 8.

grade6.2density0.53compression0.54

scripts/check-boundaries.sh runs the checks: first each rule file against its fixture of violating and allowed code, then the scan of src/, then a check that the scan read every Rust file there. It runs from just check, from the pre-commit hook that scripts/install-hooks.sh installs, and from the Boundary check job in ci.yml. The hook is opt-in, runs on every commit regardless of which files are staged, and skips the check with a warning when semgrep is not installed; CI does not skip it.

grade7.0density0.58compression0.53

Rule 6 is the only rule CI verifies by compiling. The rust job runs cargo check, cargo clippy, and cargo test with and without default features, on Linux and macOS. The msrv job runs cargo check on Rust 1.88, once with all features and once with no default features. CI fires on pushes to main and alpha and on pull requests targeting them, so work on a feature branch is ungated until the pull request opens.

Rule 1: the domain dependency set#

grade2.4density0.62compressionnone

The rule. src/domain.rs depends on serde, thiserror, and std, and on nothing else.

gradenonedensity0.78compressionnone

Scope. One file, plus the non-optional entries in [dependencies].

grade9.2density0.62compression0.63

Why it is drawn there. Domain types are what every language surface serializes. A dependency added here enters the closure of every caller on every target. Changing the set takes an RFC. thiserror was admitted that way: it emits no public API and replaced roughly 35 lines of hand-written Display and Error implementations.

grade8.8density0.65compression0.53

Enforcement. boundary-rule1-domain-imports fails on any use or extern crate in src/domain.rs rooted outside std, core, alloc, serde and thiserror, including crate::, super:: and self::: the domain imports nothing from the rest of matra. boundary-rule1-domain-inline-paths fails on an inline path with any other lowercase root, because a crate listed in Cargo.toml can be named inline with no use line; for that reason the domain writes std paths in full (std::fmt::Display). A dependency added to [dependencies] and used in domain.rs is therefore caught wherever domain.rs names it. Review reads for the one route left, a macro or derive brought into scope crate-wide by #[macro_use] in src/lib.rs and invoked unqualified.

Rule 2: ports import only from domain#

grade4.4density0.47compressionnone

The rule. Each port module imports from crate::domain and std, and from no other module or crate.

gradenonedensity1.00compressionnone

Scope. src/source/mod.rs, src/decompose/mod.rs, src/nlp/mod.rs, src/embed/mod.rs.

grade9.1density0.53compressionnone

Why it is drawn there. A port is a contract. Whatever the contract imports becomes a requirement on everyone who implements it, so a domain-only port stays implementable by someone who has never read matra's adapters.

grade10.1density0.61compression0.54

Enforcement. boundary-rule2-port-imports and boundary-rule2-port-inline-paths fail on any import or inline path in those four files that reaches past crate::domain and std: another crate module, a crate-root item such as crate::Engine, a glob, an external crate, or the port's own adapter through self::. In a port's mod.rs, super:: is the crate root and is read as such. Review reads the trait method signatures for what an import cannot show: a contract shaped around one adapter's needs.

Rule 3: no port module imports another port module#

grade0.5density0.50compressionnone

The rule. The four ports are peers and name each other nowhere.

gradenonedensity0.60compressionnone

Scope. The same four files.

grade7.3density0.44compressionnone

Why it is drawn there. Stage order belongs to the composition root, not to the contracts. If Decomposer knew about NlpProvider, the pipeline's shape would be encoded in the traits and a stage could no longer be replaced on its own.

grade9.4density0.57compression0.52

Enforcement. boundary-rule3-port-names-port fails when one of the four files names a port through crate:: or super::: in a use line, a brace group (a comment inside a multi-line group does not hide it), a fully qualified inline path, or a trait bound. A type alias cannot launder the path, because the only module a port may import is the domain, and the domain may import nothing from the crate (rule 1). A crate-root re-export of another port's trait is caught by rule 2.

Rule 4: one file imports udpipe_rs#

grade3.3density0.46compressionnone

The rule. src/nlp/udpipe.rs is the only file in the crate that imports udpipe_rs.

gradenonedensity0.50compressionnone

Scope. All of src/.

grade8.1density0.53compression0.55

Why it is drawn there. This is a resilience rule. UDPipe is C++ across an FFI boundary holding state that is not Send, and a panic on the C side aborts the host process rather than unwinding, which means interpreter death in Python. The catch_unwind boundary in nlp/udpipe.rs converts that into a domain::Error. Confining the import is what makes that boundary the only entrance rather than one entrance among several. A second NLP backend gets its own adapter file with its own panic boundary, which is the pattern working rather than an exception to it.

grade12.3density0.69compression0.52

Enforcement. boundary-rule4-udpipe-rs-outside-adapter fails on udpipe_rs anywhere in code under src/ outside src/nlp/udpipe.rs, test modules included. boundary-rule4-udpipe-rs-exposed fails inside the adapter on anything pub, at any restriction, that names udpipe_rs: a re-export, a type alias, a signature or a field, any of which would let another file hold the C-backed type while the first check stays green. Review reads for the laundering route left: a private alias (use udpipe_rs::Model as M;) made public under another name.

grade11.0density0.51compression0.58

The analog. src/embed/model2vec.rs is the only file that imports safetensors and tokenizers, enforced by the same pair of checks (boundary-rule4-model2vec-crates-*) with the same one route left to review. Those crates are pure Rust, so the confinement is not a C panic boundary; it is what keeps the adapter swappable and the model-format vocabulary out of every other file's reach. A second embedding backend gets its own adapter file with its own confinement line.

Rule 5: metrics, extraction, and structure readers stay pure#

grade8.4density0.57compressionnone

The rule. No file under src/metrics/ or src/extraction/, and no structure-reading module (src/hearst.rs), imports from any crate module other than crate::domain and crate::stopwords.

grade12.1density0.75compressionnone

Scope. Intra-crate imports only. External crates that the computation itself needs are unaffected: the compression metric uses brotli, and several extraction files use std::collections.

grade10.2density0.51compression0.56

Why it is drawn there. These are pure functions over already-parsed structure. Purity is what lets them run with no model loaded, be unit-tested without fixtures, and be called by someone who parsed elsewhere. It is also what makes the parse-once-use-many contract real: a metric that reached for an NlpProvider would re-parse internally, and a caller who had already parsed would pay twice.

grade8.2density0.62compression0.52

Enforcement. boundary-rule5-pure-imports fails on any crate:: path in those trees, in a use line, a brace group or inline, naming something beyond domain and stopwords, and on super::super::, which leaves the tree. boundary-rule5-root-super reads super:: the same way in src/metrics/mod.rs, src/extraction/mod.rs and src/hearst.rs, where it is the crate root. Sibling imports within a tree (super::textrank) and external crates stay allowed. Review reads for what purity means rather than what it imports: any function there taking raw text instead of a Document or its sentences.

Rule 6: the no-default-features build compiles#

gradenonedensity0.83compressionnone

The rule. cargo check --no-default-features succeeds.

gradenonedensity0.75compressionnone

Scope. The whole crate.

grade7.6density0.44compressionnone

Why it is drawn there. This is the mechanical proxy for features being additive and the core standing alone. It proves that the domain and the ports compile with no UDPipe, which is the configuration a type-only caller needs.

grade8.8density0.65compressionnone

Enforcement. CI, as described above. Code that only compiles with udpipe enabled belongs behind #[cfg(feature = "udpipe")].

Rule 7: the composition root knows the whole#

grade0.6density0.43compressionnone

The rule. src/lib.rs is the only place that knows all adapters and all ports.

gradenonedensity0.89compressionnone

Scope. Every file except src/lib.rs. Test modules are exempt.

grade5.9density0.53compression0.62

Why it is drawn there. Knowledge of the full assembly is a cost paid once. Concentrating it in one file means a reader learns how matra is wired by reading one file, and adding an adapter is a one-file change. Two files that both know the wiring drift, and the place that was missed becomes the bug.

grade7.6density0.60compressionnone

Enforcement. For src/cli/ and src/python/, the checks described below. Everywhere else, review: read for any file other than lib.rs importing from two or more adapter modules, and for any helper outside the composition root that matches on Format to pick a reader.

grade7.6density0.57compression0.55

src/config.rs sits in this tier alongside lib.rs. It imports domain, std, serde and toml, and it imports no port and no adapter. The traffic runs the other way: an adapter may import Config to offer a from_config constructor (RFC-0011), which is why Udpipe::from_config lives in src/nlp/udpipe.rs and Model2Vec::from_config in src/embed/model2vec.rs, not in the composition root. That import gives the adapter a default, not a second opinion about the wiring, so rule 7 still holds: lib.rs remains the only file that knows every adapter and every port.

grade13.7density0.56compression0.46

src/cli/ sits above that tier: it is the application, compiled into the library so both launchers run one program. From the crate it uses the public surface lib.rs exports (Engine, Input, Inputs, Block, the outcome functions summarize, summarize_with, keyphrases, keyphrases_with and clusters, and their selectors SummaryMethod and KeyphraseMethod), config, domain and embed, and never another port module, a reader or parser adapter, or a crate-private module such as extraction. It reaches the pipeline through Engine::from_config and the outcomes through the functions at the crate root, so rule 7 holds there too. All four port modules are public since the provider traits opened, but only embed is on the command line's list: it implements no parser, reader or source and plugs none in, so nlp, decompose or source there would be wiring the composition root does, by another name, and the check keeps them out. Model2Vec is public behind its feature. The library builds no embedder inside an outcome call, because building the default one downloads a model, so matra clusters builds it from the configuration with Model2Vec::from_config and hands it to clusters. Choosing that default is the application's decision, and the command line is the application.

grade9.9density0.60compression0.48

boundary-rule7-cli-imports reads both spellings of a violation, because the inline one is the one it would most likely take. The declared form is a use crate::... line in src/cli/; the inline form is a qualified path in a function body. The CLI already reaches Engine, Input and the outcome functions by inline path rather than by use, so the inline form is the ordinary idiom here and the one a violation would blend into. The check is an allowlist: any crate:: path in src/cli/ must start with Engine, Input, Inputs, Block, summarize, summarize_with, keyphrases, keyphrases_with, clusters, SummaryMethod, KeyphraseMethod, config, domain, embed or cli, so another port, a reader or parser adapter, a crate-private module, or a crate-root re-export of an adapter fails it. boundary-rule7-cli-root-super reads super:: in src/cli/mod.rs, where it is the crate root, the same way. Either spelling would put adapter selection in the command line, which is the composition root's job.

grade7.4density0.53compression0.50

src/python/ is the Python binding, a driving adapter that translates and decides nothing, held to the same rule by boundary-rule7-python-imports and boundary-rule7-python-root-super in .semgrep/rule7-python.yml. Its allowlist is the command line's plus cli, because the console script runs cli::run. It reaches embed for its own reason: a Python caller hands clusters an embedder (Model2Vec, or any object the binding wraps as an Embedder). It reaches no other port: Python cannot implement a parser, a reader or a source in 0.3.0, so the binding has nothing to wrap as one, and when it can, the wrapper is the reason to add the port to its list, as embed was added. Its own tree is crate::python. Before 0.3.0 the binding lived inside src/lib.rs, out of this check's reach, and it decided things the command line decided differently: it read markdown as plain text before a summary. Held to the public surface, it cannot.

Rule 8: no tracing in the domain or the ports#

grade3.3density0.54compressionnone

The rule. tracing is forbidden in src/domain.rs and in the four port modules.

gradenonedensity0.75compressionnone

Scope. Those five files.

grade9.0density0.55compression0.58

Why it is drawn there. Observability is an adapter and composition-root concern. A domain type that emits spans holds an opinion about the host's runtime and subscriber configuration, and a port that traces forces that opinion onto every implementor. In domain.rs it is also rule 1 by another route, since it would be a fourth dependency.

grade8.5density0.55compression0.60

Enforcement. boundary-rule8-no-tracing fails on the identifier tracing anywhere in code in those five files, test modules included, whether in a use line, a brace group, a path or an attribute. The rule is preemptive: tracing is not a dependency of matra at all. The line was drawn before the first import could land.

Which rules apply to which files#

File touchedRules in scope
src/domain.rs1, 8
src/source/mod.rs, src/decompose/mod.rs, src/nlp/mod.rs, src/embed/mod.rs2, 3, 8
src/nlp/udpipe.rs4, 6
Other adapters6, 7
src/config.rs6, 7
src/cli/7
src/python/7
src/metrics/, src/extraction/, src/hearst.rs5, 6
src/lib.rs6, 7
Cargo.toml1, 6

Running the checks#

pip install --require-hashes -r .github/requirements/semgrep.txt  # once
just boundary      # scripts/check-boundaries.sh on its own
just check         # every local gate, including the boundary check
just install-hooks # install the pre-commit hook that runs it
grade8.0density0.60compression0.50

The check needs semgrep, at the version .github/requirements/semgrep.txt pins, and must run from the root of a git work tree: semgrep resolves each rule's paths against the work tree, and outside one a path-scoped rule matches nothing. Each finding prints the file, the line and the rule's message. On a clean tree the script ends with check-boundaries: N of N Rust files in src/ read, 0 finding(s), and it exits non-zero on any finding, any failed rule test, any Rust file under src/ the scan did not read, or any rule scoped to a path git no longer has. A rule scoped to a moved file would scan nothing and pass, so when a file moves, the rules that name it move with it.

grade5.2density0.47compression0.61

A violation is a merge blocker. The remedy is a change to the structure, or an RFC that changes the rule deliberately. It is never a change to the check. A check that flags correct code is a bug in the check: fix the rule and add the case to its fixture, as EP-0014 describes.

Roadmap

grade4.2density0.29compressionnone

What matra does not do yet, why, and the condition that would change that.

grade8.4density0.45compressionnone

Every other page of this documentation describes what ships today. This page is the only one that describes what does not, which is what lets you trust the rest without auditing each claim.

grade17.6density0.60compression0.54

matra follows a discipline borrowed from road-building: dirt road first, then cobblestone when traffic justifies it, then tarmac when the route is established. Each capability below is deferred not because it is unimportant, but because adding it before the trigger condition fires would add complexity that nobody needs yet, or that requires design decisions that need real usage patterns to make correctly.

grade9.8density0.50compressionnone

The discipline is Chesterton's fence applied to features: before you add a capability, be able to state what problem it solves for which caller today. If the answer is "we might need this eventually," the fence stays up.

grade10.2density0.61compressionnone

Every entry below names the trigger condition. When the condition is met, write an enhancement proposal and proceed.

The scoping principle: lowest verifiable tier#

grade23.7density0.67compression0.54

A survey of the computational landscape (internal, 2026-08-21) sorts NLP affordances into three tiers: deterministic tooling whose output is checkable against the source (tokenizers, morphology, dependency parsers, document-structure extraction), specialist models (coreference, discourse relations), and general model judgment, which has the widest coverage and the lowest verifiability. The engineering rule it supports: route each operation to the lowest tier that meets the accuracy and verifiability requirement, and escalate only when the cheaper tier provably fails.

grade12.0density0.55compressionnone

matra is the first tier, deliberately. Everything it ships is deterministic and grounds back to the bytes it came from, which is what makes its output usable as data rather than as interpretation. Two consequences bound the roadmap:

grade17.9density0.64compression0.57

Capabilities with no reliable affordance stay out, and the gap is recorded rather than simulated. Pragmatic enrichment (implicature, purport, illocutionary force) and claim or fallacy certification have no deterministic path and no reliable model path either; the honest output there is a structural gap marker, not simulated confidence. matra's idiom already encodes this (Option slots that mean "declined to compute", a tree_depth of None on a cycle, UnsupportedFormat, DocumentError, the skipped list), and new capabilities inherit it.

grade15.1density0.62compression0.58

Capabilities that need a specialist model are adapter work, not extensions of the parse. Coreference is the clearest case: it sits a tier above UDPipe, the research synthesis places it above matra beside SRL and NLI, and pretending otherwise would put unverifiable output behind a surface that promises verifiability. If a specialist coreference adapter ever lands, it arrives as a new parser adapter behind its own feature flag, with its tier stated plainly.

After 0.3.0#

grade3.5density0.56compression0.60

0.3.0 settled the public surface: one set of outcome names in Rust, Python and the command line, input through Input, a span and provenance on every result, schema 2.0 with a published JSON Schema, and an error kind on every channel. Migrate from 0.2 to 0.3 lists every change. Four things are planned for 0.4.0, each an addition to that surface:

  • A feed for text arriving over time, in Rust, Python and the command line at once. See Streaming source.
  • Caller extensions: predicates and the findings they produce, over a Document and its schema, in their own proposal. See Rule evaluation over parsed structure.
  • TypeScript: types generated from the published schema, and a web component that renders a parse from matra's JSON. The spike on compiling UDPipe to WebAssembly returned a go, and EPR-0007 proposes a package that parses in the browser and in Node.js; it is proposed, not accepted. See WASM binding for TypeScript/browser.
  • A permissively licensed parsing model, as a second pinned model chosen by id. See A permissively licensed parsing model.

Rule evaluation over parsed structure#

grade9.1density0.52compression0.59

What it is. Declarative rules expressed as predicates over Document. A Rule names a structural pattern (a dependency arc, a token sequence, a metric threshold) and fires when the pattern is present. The output is a Finding with a SourceSpan that points back to the bytes in the original text where the rule matched.

grade12.3density0.64compressionnone

This is the capability that bridges the record tier (what matra exposes today) and the abstract tier: relation extraction, modality detection, speech act classification, voice signature analysis. The record tier gives you the tokens and arcs. Rule evaluation gives you the named patterns over them.

grade8.2density0.62compression0.51

Where it lands. Inside matra, in a new src/rules/ module, occupying the abstract extension point RFC-0007 reserves between structure and purpose-fitted output. The vocabulary is locked by RFC-0006: Rule, Predicate, Finding, SourceSpan. Rule evaluation is not a separate crate; callers compose against one surface. The Finding type's shape (trait vs enum) is deferred to Phase 2; the name is locked. Since 0.3.0 every heading, paragraph, sentence and token carries a Span into the analysed text, so a finding can point at bytes without a new addressing scheme. Caller extensions are planned for 0.4.0, in their own proposal, on schema 2. Rules are deterministic predicates over structure, so their findings stay at the verifiable tier: a Finding names the pattern and points at the bytes, and whether the pattern matters remains the caller's judgment.

grade10.1density0.75compressionnone

Trigger condition. FIRED, 2026-05-23. The condition was at least one concrete caller pattern that direct Document field access and application-side logic cannot adequately serve.

grade15.5density0.67compression0.63

An internal research synthesis dated 2026-05-23 names five such patterns and states the case directly: "these five primitives are the consumer sites. The deferral has fired." They are negation, modal classification, evidentiality, Hearst patterns, and typed morphological features, each grounded in a distinct literature (FactBank polarity, CoNLL-2010 hedges, Aikhenvald evidentiality, Hearst 1992).

grade12.2density0.63compression0.61

A second, sharper piece of evidence sat inside this repository. Sentence::is_passive was a method, and methods do not cross FFI, so python/matra/cli.py re-implemented passive detection over raw tokens: matra's own language binding duplicated its own primitive. RFC-0008 settled the channel (derived structural facts cross FFI as fields, computed once in Rust at a single point in the pipeline) and the Python re-implementation is deleted.

grade11.2density0.50compressionnone

The five primitives shipped, 2026-08-21, in blueprints/legacy/eps/0007-structural-primitives.md, which carries the milestones, their rubrics, and the reasoning trail. Rule and Predicate are designed after them, from the shape the five actually took, which is what "pulled from real use, not anticipated" asked for.

grade6.4density0.55compressionnone

What the five revealed about that shape. Written from the landed code, not from anticipation. A Predicate must be able to name:

  • A single dependency arc, selected by relation and lemma. Negation is one advmod arc carrying a closed-class cue; a modal is one aux arc whose lemma sits in the ten-item UD closed class, widened by the AUX part of speech to catch modals the parser promotes under ellipsis and coordination. Both landed as structs of token ids and lemmas (Negation, Modal). Arc-plus-lemma is the smallest pattern unit, and two of the five primitives are exactly one of them.
  • A morphological feature at a tree position. The root-mood detector reads Mood through Token::feat on the root or on a cop or aux child of it. A predicate vocabulary with arcs but without feature lookups cannot express the test that separates asserted from modalized clauses.
  • A multi-arc construction with optional participants. The reporting construction is a verb governing a ccomp, plus its nsubj when the sentence has one (ClausalComplement, named for what the detector finds). The subject slot is optional because sentence segmentation strands attribution across boundaries; a construction predicate must match with an absent argument rather than fail.
  • A lexicon as a parameter, for open classes. Reporting verbs and evidential adverbs have no closed list, so the detectors report every structural match and Sentence::clausal_complements_in and Sentence::root_adverbials_in filter by a caller-supplied lexicon. A Rule over an open class carries its lexicon as data; embedding one would ship an incomplete list that looks authoritative.
  • Span pairs with token-id traceability, as output. A Hearst match returns a pattern tag plus two spans, each a head token and a token-id range (HearstPair, HearstSpan). SourceSpan therefore has to address a range, not a single token, and a Finding has to be able to carry more than one of them.
grade15.1density0.63compression0.60

Two mechanism facts carry over. Predicates read materialized fields, not the raw parse: RFC-0008 computes each derivation once at a single point in the pipeline and serializes it, so rule evaluation is field access over Document before it is graph walking. And that line held under real pressure: every discriminant that landed names a construction (HearstPattern variants, modal lemmas, the reporting shape), never a semantic judgment, which is exactly the Frame-3 contract RFC-0006 imposes on Finding.

WASM binding for TypeScript/browser#

grade10.1density0.60compression0.57

What it is. A wasm-bindgen surface that exposes the same Document types to TypeScript running in a browser or Node.js environment. The same names cross; the same methods-don't-cross constraint applies. This makes matra usable in frontend applications, browser-based document editors, and Node.js pipelines without requiring a Python or Rust runtime.

grade8.2density0.57compression0.55

What is blocking it. A decision, not the parser. This entry used to say that C FFI cannot run in a WASM sandbox, and that a parser in the browser would need a different parser or a server. A spike run between 2026-11-14 and 2026-11-18 found otherwise. UDPipe's C++, compiled with Emscripten, gave output byte-identical to the native build over matra's twelve example texts, in Node.js and in the Chromium, Firefox and WebKit builds that Playwright ships, at about half the native speed. EPR-0007 records every measurement and what was not measured, which includes Safari itself, any phone, and a model fetched over a real network.

grade12.3density0.55compressionnone

The NlpProvider trait required no change. In the spike matra's core compiled for WebAssembly without the udpipe feature, with the compiled parser handed in through Engine::with_parser, and the outcome functions over a Document need no parser at all.

grade11.8density0.51compression0.54

What remains open is the design, which EPR-0007 proposes and the owner has not accepted: where the binding lives, what a caller sees when something fails, which browsers the parser build admits, what the package is called on npm, what a second reader for one format does, and whether text is detected as bytes are. One question it lays out without a recommendation: who serves the parsing model to a browser. The model is licensed CC BY-NC-SA 4.0, so the package cannot carry it, and its publisher's host refuses requests from other sites, so a page cannot fetch it from there.

grade11.6density0.50compression0.48

Planned, in steps, each usable alone. EPR-0006 placed three in 0.4.0: TypeScript types generated from the published JSON Schema in spec/schema/v2/ with a web component that renders a parse from matra's JSON, the pure-Rust core in WASM, and the spike, which has now returned a go. EPR-0007 orders what follows as eight milestones, in the order they are built: the types, for TypeScript and Python from the same schema; the ports and the names, so that a reader or parser written outside matra can do what matra's own do, every document names the reader that read it, and each method's name is written once; one copy in Rust of the mapping from the parser's output to tokens, the core in WebAssembly with the outcome functions, the parser, the release to npm, the component that renders a parse, and the playground that shows its result with it. The component needs only the types, so it may be built any time after them. None starts before the proposal is accepted, and nothing reaches npm without the owner's approval of that release.

grade9.3density0.56compressionnone

Trigger condition. The owner accepts EPR-0007. The condition this entry used to carry, an NlpProvider implementation that compiles to WASM with sufficient parse quality, is met as far as the spike measured: it is the same parser giving the same output.

PDF and DOCX decomposers#

grade11.3density0.60compression0.61

What they are. Readers (crate-private Decomposer implementations) for PDF and DOCX documents. The variants already exist in domain::Format, and detection already recognises both, by extension and by their bytes (%PDF, or a zip archive holding [Content_Types].xml). Analysing one returns Error::UnsupportedFormat today, whose feature is None because no feature adds a reader; there is no reader to hand the bytes to.

grade9.7density0.69compression0.58

What is blocking them. The Rust PDF extraction ecosystem. PDF text extraction requires handling layout ordering (columns, reading order), encoding diversity (Type1, CFF, CID fonts), and embedded-font subsetting. No current Rust crate handles all of these consistently and without correctness holes on real-world documents. DOCX extraction is more tractable but has not been required by any caller yet.

grade11.9density0.59compression0.58

Trigger condition. A documented caller requirement for PDF or DOCX analysis, paired with a Rust extraction crate that is stable (post-1.0, maintained, no known correctness holes on the document shapes the caller needs). When a reader lands, no published signature changes: it is a reader file and a registration line in the composition root, behind a feature that UnsupportedFormat then names, emitting the same headings and paragraphs with spans as the markdown reader.

Recursive directory walk#

grade12.9density0.63compressionnone

What it is. Inputs::dir descending into subdirectories rather than stopping at depth one. This enables processing nested document collections (a repository of markdown files organized by directory, for example) without the caller having to walk the tree manually.

grade9.0density0.60compression0.55

What is blocking it. No caller has required it. Recursive walks have non-obvious behavior: symlink cycles, permission errors mid-walk, and large directory trees with heterogeneous content all require explicit policy decisions. What is the depth limit? How are symlinks handled at nested levels? How do per-file errors propagate: do they abort the entire walk or continue? Building these policies into the API without a caller to validate them risks building the wrong policies.

grade15.8density0.69compressionnone

Trigger condition. A caller that documents a requirement to process nested directory trees, with enough specificity to make the symlink, error-tolerance, and depth-limit policy decisions correctly.

Streaming source#

grade4.1density0.41compressionnone

What it is. A feed: text that arrives over time, such as a transcript or a message queue, analysed as it comes rather than handed over whole. It is planned for 0.4.0, in Rust, Python and the command line at once, so no surface gets it before the others.

grade8.5density0.50compression0.54

What is in place. Engine::analyze_all is already lazy: it takes any iterator of Input, yields one result per input in order, and Inputs::dir reads each file only when the stream reaches it. So a caller can already stream documents that are each complete. What the feed adds is text that is not yet complete when analysis starts. The readers and the parser are private to the crate, so the feed arrives as an addition to the public surface, not a change to a signature that ships.

grade8.6density0.55compressionnone

What is blocking it. The design: where a paragraph or a sentence ends when more text may still arrive, and what a partial result promises. The async reactor decision (deferred per RFC-0004) bears on the most useful form, async processing; a synchronous feed is possible without it.

grade3.5density0.56compressionnone

Trigger condition. FIRED. EPR-0006 places the feed in 0.4.0 (decision 2), as an addition in all three syntaxes.

Separate matra-nlp-api port crate#

grade10.8density0.58compression0.55

What it is. Extracting the NlpProvider trait, public since 0.3.0, into a separately published minimal crate, so third-party NLP provider implementors can depend on the port contract without depending on all of matra. This follows the Pattern 6 criterion: separately publish a minimal port crate if and only if an external implementor ecosystem exists that needs to pin the contract independently of the main crate's version churn.

grade12.0density0.72compressionnone

What is blocking it. No third-party NLP provider implementor crate exists today. The criterion is ecosystem size, not architectural elegance. Extracting matra-nlp-api before any external implementor exists would add a publishing burden (two crates to version, two changelogs, two sets of CI) for zero current benefit.

grade11.8density0.60compressionnone

Trigger condition. A published Rust crate that depends on matra solely for NlpProvider (a matra-stanza, matra-spacy, or similar, shipped by a non-matra maintainer). At that point, extract matra-nlp-api as a minimal port crate, have matra depend on it, and keep matra as the caller-facing surface.


grade8.5density0.52compression0.57

The pattern across all of these: capability waits for the caller that justifies it. Adding structure in advance of use is complexity that the next collaborator pays for. The trigger conditions are the guard against that. When one fires, write the enhancement proposal, explain why this trigger and not the next one, and proceed.

HTML report#

grade11.5density0.54compression0.59

What it is. A self-contained HTML rendering of a Document: the parse, the metrics, and the extracted phrases in a form suitable for visual inspection, notebook display, and paper supplementary material. The surface would be Document::to_html_report() in Rust, report(text, format="html") in Python, and matra report on the CLI.

grade4.2density0.50compressionnone

What is blocking it. Nothing structural. It is unbuilt because no caller has needed it yet.

grade10.4density0.57compressionnone

Trigger condition. A caller that needs to eyeball a parse rather than consume it programmatically, most likely a researcher working in a notebook.

A permissively licensed parsing model#

grade4.7density0.62compressionnone

What it is. A second pinned English parsing model whose licence permits commercial use, trained on UD English EWT, chosen by id beside the model that ships. The model that ships, english-ewt-ud-2.5-191206, is CC BY-NC-SA 4.0, which does not.

grade6.3density0.48compression0.60

Where it lands. As a pin, like the first: a URL, a SHA-256 and a licence compiled into the adapter, and an id a caller names. Every document already records the parser and its model's digest in provenance.parser, so a reader sees which model produced a parse rather than discovering it as a shift in the numbers. Choosing by id is an addition; 0.3.0 removed loading a model file by path so that this would not be a break.

grade3.0density0.55compressionnone

Trigger condition. FIRED. EPR-0006 places it in 0.4.0. What remains is training the model and measuring it against the one that ships.

Browser playground#

grade3.0density0.39compressionnone

What it is. A page where text can be pasted and its structure rendered live, with no install.

grade9.0density0.51compression0.58

What is blocking it. It depends on the WASM binding above, and on a decision about who serves the parsing model to a browser. The model's licence is non-commercial and its publisher's host refuses requests from other sites, so a page needs the model at an address it may fetch. EPR-0007 lays out three options and what each costs: the docs site serves the current model with attribution, the playground waits for the permissively licensed model, or each visitor supplies the file. The choice is the owner's.

grade2.6density0.64compressionnone

Trigger condition. The WASM binding ships and that choice is made.

Voice fingerprint metrics#

grade13.9density0.62compressionnone

What it is. Three additions that make matra's existing metrics usable as a stylometric signature rather than as per-document readouts: a length-normalized lexical diversity measure, burstiness as a first-class value, and a contraction count.

grade7.7density0.60compression0.54

Why these three specifically. An internal note dated 2026-04-21 records a writing-assistant agent blocked on a calibration loop against a measured baseline: 26.6 word mean sentence length, 36.9 percent passive, 0 percent contractions, 7.7 percent nominalisation, 0.76 burstiness. Checked against what ships, three of those five are already Document fields. Burstiness is derivable from metrics.sentence_length_std and metrics.mean_sentence_length, both fields since 0.3.0, but is not itself exposed. Contractions are not counted at all. That note estimated the remaining work as a small configuration layer rather than a rewrite.

grade10.1density0.49compression0.57

One of them is a defect, not an addition. vocabulary_ttr is a raw type-token ratio and TTR falls mechanically as text grows. Measured on this repository: README scores 0.690 against architecture/design.md at 0.227, but README has 35 sentences and the other has 266, so most of that gap is length rather than voice. As a cross-document feature it is unsound, which matters most when a directory is analyzed (Engine::analyze_all over Inputs::dir, or analyze_all(dir=) in Python), because the result invites the comparison. The limitation is documented on the field; what remains is a length-normalized measure beside it (MTLD, MATTR, or standardized TTR).

grade13.1density0.60compression0.56

Why this is the strongest candidate of anything on this page. The four faces of voice all map onto shipping output with no hole: agentive onto nsubj and nsubj:pass, modal onto aux and feats, structural onto section hierarchy and sentence length, stylistic onto lexical density and compression. That is unlike claim atomization, which needs an LLM, and unlike deriving a text's situation model, for which no shipping output is a foundation at all.

grade9.0density0.57compressionnone

Trigger condition. Met in substance: a named caller with a documented baseline exists and has been blocked for months. What is missing is confirmation that the caller still wants matra rather than the thin-wrapper alternative it also considered. Confirm that, then proceed.

Self-similarity and redundancy metrics#

grade15.0density0.61compressionnone

What it is. Structural detection of a document repeating itself: clusters of sentences restating the same content, with quantitative measures over them. The concrete caller pattern is auditing LLM-generated text, whose characteristic failure is high-lexical-overlap restatement.

grade6.9density0.55compressionnone

The shape, all deterministic and traceable back to the source bytes:

OutputKind
Similarity clusters: sentence groups whose pairwise content-lemma overlap (Jaccard or TF-IDF cosine) exceeds a caller-supplied threshold, returned as span sets with the shared lemmas as evidencequalitative
Redundancy ratio: share of sentences in clusters of size above onequantitative
rep-n and distinct-n: n-gram repetition rates, the established NLG-literature measuresquantitative
Skeleton repetition: sentences sharing root-verb lemma plus subject and object lemmasqualitative
Opener formulae: repeated sentence-initial lemma sequencesquantitative
Document-scope compression ratio, closing the per-paragraph measure's cross-paragraph blind spotquantitative
POS-sequence compression ratio: the same compression measure over the tag sequence, catching structural repetition independent of vocabulary (Shaib et al. 2024)quantitative
Syntactic template rate: recurring POS n-grams, the published operationalization of templated LLM output (Shaib et al., EMNLP 2024)quantitative
Span recurrence: long token spans repeating verbatim, the loop-degeneration signaturequalitative
grade11.2density0.57compression0.51

matra reports the clusters and the numbers; whether they constitute fluff is the caller's reading. The word never appears in the output. That restraint is now empirical as well as principled: the 2026 slop-measurement literature found repetition and templatedness are weak predictors of human quality judgments on their own, so these are structural facts for a caller's argument, never a judgment. Synonym-level paraphrase (different vocabulary, same meaning) is out of reach of lexical overlap by design; catching it needs semantic similarity, which sits above the verifiable tier. That half shipped 2026-09-05 via EP-0009: the Embedder port, the model2vec static adapter, and the clustering that clusters runs (see the semantic clusters guide). The deterministic family above remains the open half of this entry. Published similarity thresholds for that task span 0.67 to 0.9 with no consensus, which is why every threshold here is caller-supplied.

grade11.3density0.60compressionnone

Where it lands. Mostly paid for already: extraction/textrank.rs builds a pairwise sentence-similarity matrix and projects it down to centrality ranks; re-projecting the same matrix as clusters is the core of this capability. Thresholds are caller-supplied parameters, not constants matra pretends to know.

grade11.5density0.69compression0.64

Trigger condition. FIRED, 2026-08-21. A concrete caller pattern was named that field access cannot serve: quantifying restatement across a document for LLM-output auditing. Design against EP-0007's primitives and the rule-vocabulary shape; an enhancement proposal settles whether this is a metric family, an extractor, or the first rule pack.

Information density#

grade14.3density0.70compression0.52

What it is. Deterministic measures of how much a text says per word, the complement of the redundancy family: redundancy catches the same thing said twice, density catches little said at length. The anchor is propositional idea density, a validated psycholinguistics measure (propositions per word) with two published computational forms. CPIDR counts propositions from Penn Treebank tags, which UDPipe already emits in Token::xpos; DEPID counts them from dependency relations, which is matra's core output, and agrees with human raters slightly better than CPIDR does. DEPID-R, the variant counting distinct relation-lemma triples rather than tokens of them, catches fluent-but-repetitive text and costs nothing extra once lemmas are in hand. Around the anchor sit smaller settled measures that are direct tree walks: mean dependency distance, modifiers per noun phrase, words before the main verb, content-to-function ratio.

grade14.8density0.62compression0.51

Where it lands. Metric functions over Document, same family as the shipping metrics suite. The DEPID relation inventory is Stanford-style, so an explicit mapping to UD relations is part of the design; the CPIDR path additionally holds only for English models, since Token::xpos carries whatever tagset the provider's model emits and PTB tags are the English case. Three counting rules the literature leaves genuinely unsettled (direct objects, coordinating conjunctions, the denominator) get settled by an enhancement proposal rather than inherited silently, and the proposal also states the non-English story. The reference implementations are GPL, so the derivation is clean-room from the published rules, and exact numeric parity with them is a non-goal. No implementation of idea density exists for CoNLL-U input in any language; this would be the first.

grade15.1density0.61compression0.64

What stays out, and why. Hedging and booster density need a lexicon, and no canonical machine-readable hedge list exists; per the open-class precedent (clausal_complements_in), any such lexicon is caller-supplied data, never shipped as if authoritative. Surprisal needs frequency norms or a language model, and sycophancy detection has no deterministic path at all; both gaps are recorded rather than simulated.

grade12.5density0.61compression0.56

Trigger condition. FIRED, 2026-09-04. The same LLM-output-auditing caller pattern that fired the redundancy entry names low density as the second failure mode, and the 2026-09-04 landscape survey (internal, not in this repository) established implementability from matra's existing parse output. Design together with the redundancy family; the same proposal should place both.

Configuration-driven invocation#

grade9.8density0.47compressionnone

What it is. A configuration file selecting which metrics run, which extractors run, and how output is shaped, so both the library and the CLI are driven by declaration rather than by argument lists.

grade6.4density0.49compression0.55

What is blocking it. The metric suite is private to the crate since 0.3.0, so choosing metrics from a file would first need a public way to name them. What is absent beyond that is the surface: a format, a resolution order, and a decision about whether configuration is a library concern or belongs only to the application tier.

grade10.3density0.62compressionnone

That last question is the real one. matra's discipline is that the library returns typed data and the binary decides presentation. A config format that reaches into the library risks putting policy where the composition root belongs.

grade11.1density0.64compression0.52

Shipped, 2026-09-05, via EP-0010: Config resolves locations and defaults per key from argument, environment, config file, then the defaults compiled into the crate; Engine::with_defaults() (since 0.3.0 Engine::new(), which reads no configuration file, and matra.Engine() in Python, which replaced Matra.english()) and the from_config constructors are the no-setup path on every surface; matra config show and matra config init are the command-line half. The question this entry asks is answered narrowly: the file carries locations and defaults, never which metrics run or how output is shaped. Selecting behavior from a file remains unbuilt, and reopening it needs its own enhancement proposal rather than a wider schema here.

grade12.4density0.60compression0.54

Trigger condition. FIRED, 2026-09-05. The condition was a caller running matra repeatedly with the same non-default selection, with agent-driven use the likely first instance. It fired as owner direction: matra works with no setup on every surface, follows developer-tool conventions for config and paths, and keeps Rust as the core with Python and TypeScript as thin reach layers. RFC-0011 settles the library-or-application question the paragraph above asks (a resolver for locations and defaults in the library, behavior selection in the application), and blueprints/legacy/eps/0010-foundations.md carries the milestones.

Agent surface#

grade8.2density0.52compression0.57

What it is. A --skill flag on the CLI that prints a self-contained description of matra's semantics for an agent: what it is for, when to reach for it, the incantations with their JSON shapes, how to read the numbers, and the limits. Progressive disclosure follows the shape of a skill on disk: --skill prints the short top level, --skill -r <name> prints one deeper reference. --help stays the framework-generated reference for humans. The same file is what a plugin marketplace distributes.

grade8.5density0.46compression0.54

Why a flag and not only a docs page. Human attention is the scarce input. The people who would read the whole docsite are few; the agents that will run matra on their behalf are many, and an agent that can print the semantics it needs, from the tool it is about to run, needs no link and no prior knowledge. The docs stay for human comprehension, with citations and readable benchmarks; the skill is the other door, derived from the same code and tested against it.

grade9.4density0.58compression0.57

Precedent. A survey of seventeen tools (docs/surveys/2026-09-05-conventions.md) found one that prints its own agent-facing instructions with a second tier of detail, Vercel Labs' agent-browser, whose stated reason is that instructions served from the installed binary always match its version. The same survey records the surrounding conventions this entry adopts alongside the flag: an llms.txt on the docsite, an AGENTS.md in the repository, and a CITATION.cff so the research behind each measure is citable from the repository page.

grade8.0density0.61compression0.56

Shipped, 2026-09-06, via EP-0011: matra --skill prints the top level, matra --skill -r lists the references and matra --skill -r <name> prints one. The text is embedded with include_str!, so an installed program's instructions are the ones that match it, and a test executes the commands the skill names, so an incantation that stops working fails a gate rather than a reader. --help is unchanged and stays the reference for humans. The surrounding conventions the precedent paragraph names landed alongside it: llms.txt on the docsite with a currency gate, AGENTS.md in the repository, a plugin manifest, and CITATION.cff.

grade9.5density0.60compression0.62

Trigger condition. FIRED, 2026-09-05, by owner direction, sequenced after EP-0010: the skill documents the CLI contract, so the CLI had to be one implementation with a pinned JSON shape first. EP-0010 landed on 2026-09-06; RFC-0012 records the decisions and blueprints/legacy/eps/0011-agent-surface.md carries the milestones.

Terminal UI for the Rust CLI#

grade11.4density0.55compressionnone

What it is. An interactive terminal interface over the same cli module: browse a parsed document, its sections, sentences, and dependency trees, and the metrics beside them, without leaving the terminal.

grade4.6density0.47compressionnone

Trigger condition. FIRED, 2026-09-06. EP-0010 and the agent surface have both met their acceptance gates. The TUI is a renderer over a contract that has to be stable first; building it before that means re-doing it.

github issues discussions api reference all pages single page llms.txt the mark

matra is MIT licensed. Type: Alegreya and IBM Plex, both SIL OFL 1.1.