Migrate from 0.2 to 0.3

on this page
grade5.3density0.50compression0.53

0.3.0 changes Rust, Python, the command line, the JSON and the config file at once. Nothing keeps its old name. There are no deprecated aliases, no hidden flags, no fields written under both names and no setting that brings back the 0.2 output. A renamed or removed call fails at once in 0.3 rather than running with a different meaning: a compile error in Rust, an AttributeError or TypeError in Python, an unknown key in the config file, and a new format_version and schema major in the JSON. A few results do change for the same call, and the last section lists them.

grade-0.7density0.48compressionnone

This page lists every change a 0.2 caller meets, with the 0.2 form and the 0.3 form side by side. Work through the section for each surface you use.

Update the version

grade5.7density0.63compressionnone

Move the version requirement first. The old requirement keeps you on 0.2, because cargo reads a change in the first non-zero component as breaking.

cargo add matra@0.3                                   # was matra@0.2
cargo install matra --version '^0.3' --features cli   # was '^0.2'
pip install 'matra>=0.3'                              # was 'matra>=0.2'
uvx 'matra>=0.3' --skill                              # the skill for 0.3

Every rename at a glance

0.20.3Where
Engine::new(nlp, decomposers), Engine::with_defaults()Engine::new(), Engine::from_config(&Config::resolve()?)Rust
Ingest::text, Ingest::pathInput::text, Input::file, Inputs::dirRust
Engine::analyze(Ingest), Engine::analyze_oneEngine::analyze(Input), Engine::analyze_all(inputs)Rust
Engine::annotate, Engine::composenone: analyze is the callRust
Ingested, RawDocument, CorpusEntry, Corpus, CorpusResultnone: Document, its path in provenance.source.pathRust
extraction::tfidf_summarize, textrank_summarizesummarize, summarize_with(.., SummaryMethod::TextRank)Rust
extraction::rake_keyphrases, yake_keyphraseskeyphrases, keyphrases_with(.., KeyphraseMethod::Yake)Rust
embed_and_cluster, extraction::semantic_clustersclustersRust
ScoredSentence, SemanticClusters, SemanticCluster, SemanticEdgeSummarySentence in a Summary, Clusters, Cluster, SimilarityEdgeRust, Python types
domain::MAX_INPUT_BYTESdomain::limits::MAX_INPUT_BYTESRust
Error::InvalidInput(message)Error::InvalidInput { reason, message }Rust
Error::ParseFailed(message)Error::ParseFailed { reason, cause }Rust
Embedder::identity() -> &strEmbedder::identity() -> IdentityRust
Config::summarize_n, summarize_algorithm, keyphrases_algorithmConfig::summary_n, summary_method, keyphrases_methodRust
Config::udpipe_model, Config::semantic_thresholdnone; Config::clusters_threshold is newRust
Engine::new(Box::new(parser), decomposers)Engine::with_parser(parser), then .with_reader(format, reader) per readerRust
NlpProvider::parse -> Vec<Sentence>NlpProvider::parse -> Vec<Vec<Token>>, one list of tokens per sentenceRust
Decomposer::decompose(&str) -> Vec<Section>Decomposer::decompose(&[u8]) -> Result<Read>, built from ReadBlock::heading and ReadBlock::paragraphRust
Source::read -> Vec<RawDocument>, Source::acceptsSource::read -> Vec<u8>; accepts is goneRust
Udpipe, Decomposers, MarkdownDecomposer, PlainTextDecomposer, FileSource, DirectorySource, standard_decomposers, metrics, hearst, extractionprivate to the crateRust
Matra.english(), Matra.from_path(path)matra.Engine()Python
Matra.analyze(text), analyze_markdown(text)engine.analyze(text), engine.analyze(text, format="markdown")Python
Matra.analyze_path(path)engine.analyze(file=path), engine.analyze_all(dir=path)Python
Matra.tfidf_summarize(text, n), textrank_summarizematra.summarize(doc, n, method=...)Python
Matra.rake_keyphrases(text, max_phrases), yake_keyphrasesmatra.keyphrases(doc, n, method=...)Python
Matra.semantic_clusters(text, threshold, model)matra.clusters(doc, model, threshold)Python
matra.semantic_clusters(vectors, threshold, model_hash)none: a cosine over the vectors you holdPython
an embedder's identity() -> str, Model2Vec.identity()identity() returning an Identity dict, {name, model, sha256, license}Python
Model2Vec.potion_base_8m() reading the config fileModel2Vec.from_config()Python
Document, CorpusEntry, CorpusItem (types)matra.Document (a class), DocumentDict, and Document or DocumentError itemsPython
paragraph["readability_grade"] and its siblingsparagraph["metrics"]["readability_grade"]JSON, Python
doc["vocabulary_ttr"] and its siblingsdoc["metrics"]["vocabulary_ttr"]JSON, Python
Paragraph.in_blockquoteParagraph.kind ("quote")all
Sentence.reportings, Reporting, reportings_inSentence.clausal_complements, ClausalComplement, clausal_complements_inall
Sentence.bare_assertionSentence.root_moodall
Token.feats (a string)Token.features (a map)all
Sentence.tree_depth() returning usize::MAX on a cycleSentence.tree_depth, None on a cycleall
SemanticClusters.model_hashClusters.provenance.parameters.embedder.sha256all
InputTooLarge what: file_source, config_file, udpipe_download, embedding_downloadfile_read, config_file_read, parser_model_download, embedding_model_downloadall
invalid_input details.reason, the message textdetails.reason, the name of the check that failedall
model_invalid, parse_failed and io details.reason, the error's textdetails.causeall
Format serialized as "Markdown", "PlainText""markdown", "plain_text", "blocks"Rust serde, JSON
[summarize], algorithm, [semantic] threshold, [models] udpipe[summary], method, [clusters] threshold, noneconfig file
--json envelope format_version: 1format_version: 2, with an error keycommand line

Rust

Build the engine

grade8.0density0.57compressionnone

Engine::new takes no arguments and builds the pinned parser itself. It reads no configuration file. Engine::with_defaults is gone, because it read one under a name that did not say so.

// 0.2
let engine = Engine::with_defaults()?;
let engine = Engine::new(Box::new(Udpipe::english(dir)?), standard_decomposers());
// 0.3
let engine = Engine::new()?;
let engine = Engine::from_config(&Config::resolve()?)?;                     // reads the config file
let engine = Engine::from_config(&Config::resolve()?.with_model_dir(dir))?; // a model directory you name
grade6.8density0.48compressionnone

There is no provider to construct and no reader table to pass for the standard pipeline. No surface loads a UDPipe model file of your choosing: the model is a pin.

grade5.9density0.44compressionnone

A parser, reader or source of your own still plugs in, through the engine rather than a table:

// 0.2
let engine = Engine::new(Box::new(MyParser), standard_decomposers().with(Format::Markdown, Box::new(MyReader)));
// 0.3
let engine = Engine::with_parser(MyParser).with_reader(Format::Markdown, MyReader);
let engine = Engine::new()?.with_reader(Format::Pdf, MyPdfReader);   // UDPipe, plus a reader of yours
let engine = Engine::new()?.with_source(MyStore);                    // paths read from somewhere else
grade10.2density0.61compression0.51

The traits changed shape. NlpProvider::parse returns one Vec<Token> per sentence instead of Sentence values: the engine builds each sentence from the tokens, its text and every derived field included, so drop your Sentence::new call and return the tokens. Spans are required now, relative to the paragraph text parse was handed. Decomposer::decompose takes the format's bytes and returns a Result<Read> of ReadBlock values with their spans, where it returned sections; a text reader decodes the bytes itself. Source::read returns the document's bytes, and accepts is gone, since the engine never asked it. Whatever a parser or a reader returns is checked against its trait's contract, and a break is Error::ParseFailed naming the check (Errors); 0.2 passed it through unchecked. Bring your own parser or reader has the contracts.

Analyze one document

grade8.4density0.52compressionnone

Input is one document. Engine::analyze returns its Document or the error. There is no stream to unwrap and no CorpusEntry around the document.

// 0.2
let analysis = engine
    .analyze(Ingest::path("essay.md")?)
    .next()
    .unwrap()
    .map_err(|e| e.error)?
    .analysis;
let doc = engine
    .analyze_one(RawDocument::new(text, None, Format::Markdown))
    .map_err(|e| e.error)?
    .analysis;
// 0.3
let doc = engine.analyze(Input::file("essay.md"))?;
let doc = engine.analyze(Input::text(text, Format::Markdown))?;
grade11.4density0.60compressionnone

Input has five constructors: text, bytes, file, file_as (a file read as the format you name) and blocks (headings and paragraphs you supply for a format matra has no reader for). Input::file on a directory fails with InvalidInput, naming Inputs::dir.

Analyze a directory

grade7.9density0.59compressionnone

Ingest::path on a directory is Inputs::dir, and analyze_all streams one result per file. A failed file is an Err item carrying its path. The path of a document that analyzed is doc.provenance.source.path.

// 0.2
for item in engine.analyze(Ingest::path("docs/")?) {
    match item {
        Ok(entry) => println!("{:?}: {}", entry.path, entry.analysis.total_words()),
        Err(e) => eprintln!("{:?}: {}", e.path, e.error),
    }
}
// 0.3
for item in engine.analyze_all(Inputs::dir("docs/")?) {
    match item {
        Ok(doc) => println!("{:?}: {}", doc.provenance.source.path, doc.metrics.total_words),
        Err(e) => eprintln!("{e}"),
    }
}
grade10.2density0.56compressionnone

Corpus and CorpusResult are gone, with Corpus::total_words, passive_ratio and mean_readability. Collect the stream yourself. A count such as total_words sums across documents. A ratio such as vocabulary_ttr does not, because it falls as a document grows.

Read the measures

grade9.2density0.50compressionnone

The aggregate methods are fields now, so they reach Python and the JSON too. The paragraph measures moved into Paragraph.metrics, and the document measures into Document.metrics.

// 0.2
doc.total_words();
doc.passive_ratio();            // 0.0 on a document with no sentences
doc.vocabulary_ttr;
paragraph.readability_grade;
sentence.word_count();
sentence.is_passive();
sentence.tree_depth();          // usize::MAX on a cycle
// 0.3
doc.metrics.total_words;
doc.metrics.passive_ratio;      // None on a document with no sentences
doc.metrics.vocabulary_ttr;
paragraph.metrics.readability_grade;
sentence.word_count;
sentence.passive;
sentence.tree_depth;            // None on a cycle
grade11.9density0.57compressionnone

DocumentMetrics also carries total_sentences, paragraph_count, mean_sentence_length and sentence_length_std. In 0.2 these were methods, so they never reached Python or the JSON.

Read the record facts

grade6.7density0.45compressionnone

The facts on a sentence are named for what the detector finds. ClausalComplement is a verb governing a clausal complement. Which of those verbs report speech is your lexicon, as before.

// 0.2
sentence.reportings;
sentence.reportings_in(&["say", "claim"]);
sentence.bare_assertion;
token.feats;                    // "Mood=Ind|Tense=Past"
// 0.3
sentence.clausal_complements;
sentence.clausal_complements_in(&["say", "claim"]);
sentence.root_mood;             // Option<RootMood { root_id, mood }>
token.features;                 // BTreeMap: {"Mood": "Ind", "Tense": "Past"}
grade9.4density0.57compressionnone

The 0.2 bare_assertion flag is root_mood.mood == "Ind" with no modal whose head_id is root_mood.root_id. Token::feat("Mood") reads one feature, as before. Paragraph.in_blockquote is Paragraph.kind: in_blockquote: true is ParagraphKind::Quote, and a list item is now ParagraphKind::ListItem.

Summaries, keyphrases and clusters

grade7.2density0.51compression0.59

The outcomes are free functions at the crate root. They take the analysed &Document, not a slice of its sentences, and need no engine, so they run in a build without UDPipe. The method is optional and never in the name. Each returns a record whose provenance names the method that ran and its parameters.

// 0.2
let sentences: Vec<Sentence> = doc.sentences().cloned().collect();
let top = extraction::tfidf_summarize(&sentences, 3)?;          // Vec<ScoredSentence>
let ranked = extraction::textrank_summarize(&sentences, 3)?;
let phrases = extraction::yake_keyphrases(&sentences, 10)?;     // Vec<Keyphrase>
let groups = embed_and_cluster(&doc, &model, 0.85)?;           // SemanticClusters
// 0.3
let top = matra::summarize(&doc, 3)?;                                    // Summary
let ranked = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Yake)?; // Keyphrases
let groups = matra::clusters(&doc, &model, 0.85)?;                      // Clusters
gradenonedensity0.60compressionnone

What moved inside the results:

  • ScoredSentence { text, score, position } is SummarySentence { index, span, text, score }, in Summary.sentences. position is index.
  • Keyphrase gains occurrences, every span the phrase was found in, and sits in Keyphrases.phrases.
  • SemanticClusters { model_hash, threshold, clusters } is Clusters { provenance, clusters, edges }. The embedder's identity is provenance.parameters.embedder, an Identity like the parser's (the old hash is its sha256), and the threshold provenance.parameters.threshold. The edges moved out of each cluster into one list, ordered by a and then b. Each Cluster holds only members.
  • The threshold and every score are f64, not f32.
  • Model2Vec::potion_base_8m(dir) takes the models directory, the one MATRA_MODEL_DIR names, and keeps the three files in dir/potion-base-8M/. In 0.2 dir was the model's own directory. Pass the parent: potion_base_8m(cfg.model_dir()) where 0.2 took potion_base_8m(cfg.model_dir().join("potion-base-8M")). Python's Model2Vec.potion_base_8m(dir) changed the same way.

Errors

gradenonedensity0.78compressionnone

Error keeps its variants and kind(). Three shapes changed.

// 0.2
Error::UnsupportedFormat(format)
Error::InvalidInput(message)
Error::ParseFailed(message)
domain::MAX_INPUT_BYTES
// 0.3
Error::UnsupportedFormat { format, feature }   // both Option
Error::InvalidInput { reason, message }       // reason: "threshold_not_finite", ...
Error::ParseFailed { reason, cause }          // reason: None, or "head_out_of_range", ...
domain::limits::MAX_INPUT_BYTES
grade13.1density0.61compression0.50

InvalidInput's reason names the check that failed, a stable name the errors reference lists, and details["reason"] carries it, where 0.2 put the message text there. On model_invalid, parse_failed and io the error's own text moves from details["reason"] to details["cause"], so reason names a check wherever it appears. Four InputTooLarge labels name what you read rather than matra's modules: file_source is file_read, config_file is config_file_read, udpipe_download is parser_model_download and embedding_download is embedding_model_download. A match on the old strings no longer fires.

grade6.5density0.59compression0.54

Error::details() is new. It returns every value in the message by name, so no caller parses a message. Error and DocumentError serialize as {kind, message, details}, and DocumentError adds path. A failed read now names its path: io error: cannot read essay.md: No such file or directory (os error 2). A file whose bytes are not UTF-8 is UnsupportedFormat, not Io. Every cap is in domain::limits, with the same values as 0.2. Errors lists every kind and its details.

What left the public surface

grade12.4density0.59compression0.51

These modules and items are private to the crate in 0.3: the Udpipe adapter, metrics with Metric, default_suite, run_suite and readability::flesch_kincaid_grade, hearst, extraction, the readers MarkdownDecomposer and PlainTextDecomposer with the Decomposers table, FileSource and DirectorySource, and standard_decomposers. The port modules nlp, decompose and source stay public with their traits, reshaped as above. The measures stay public as fields of the Document. For another measure, write a function over &Document. Embedder stays public and keeps its shape. Its identity returns a domain::Identity, the shape the parser reports, where it returned &str: Identity::new("my-service", Some("v3".into()), None, None) in place of "my-service/v3". Return ModelInvalid, Io or InputTooLarge for your own failures, as the trait documents.

Configuration

gradenonedensity0.60compressionnone

Config reads the renamed sections, so its accessors follow them.

// 0.2
cfg.summarize_n();
cfg.summarize_algorithm();
cfg.keyphrases_algorithm();
cfg.semantic_threshold();   // f32
cfg.udpipe_model();
// 0.3
cfg.summary_n();
cfg.summary_method();
cfg.keyphrases_method();
cfg.clusters_threshold();   // f64
grade10.5density0.47compressionnone

Config::from_file(path) is new: the configuration with a file you name, where a missing file is an error.

Python

Build the engine

grade7.4density0.55compressionnone

Matra is Engine. A model directory is named by MATRA_MODEL_DIR, or by a config file you pass to Engine.from_config. Matra.from_path is gone: no surface loads a model file of your choosing.

# 0.2
from matra import Matra

m = Matra.english()
m = Matra.english("/opt/matra/models")
# 0.3
import matra

engine = matra.Engine()                                  # MATRA_MODEL_DIR, else the default
engine = matra.Engine.from_config("matra.toml")          # a config file you name
engine = matra.Engine.from_config()                      # the config file the environment names

Analyze

grade5.2density0.63compressionnone

engine.analyze takes exactly one input. A str alone is text, never a path, and reads as plain text unless you name a format. It returns a matra.Document, not a dict.

# 0.2
doc = m.analyze(text)
doc = m.analyze_markdown(text)
items = m.analyze_path("essay.md")              # a list of one
items = m.analyze_path("docs/")                 # a list, built in full
# 0.3
doc = engine.analyze(text)
doc = engine.analyze(text, format="markdown")
doc = engine.analyze(file="essay.md")           # read as its extension and bytes say
for item in engine.analyze_all(dir="docs/"):    # lazy, one item per file
    ...
grade14.2density0.74compressionnone

analyze_path returned CorpusEntry dicts ({"path", "analysis"}) and DocumentError dicts ({"path", "error": {"kind", "message"}}). analyze_all yields a matra.Document or a DocumentError dict, {"path", "kind", "message", "details"}. Tell them apart with isinstance(item, matra.Document). The path of a document is item["provenance"]["source"]["path"].

# 0.2
for item in m.analyze_path("docs/"):
    if "error" in item:
        print(item["path"], item["error"]["kind"])
    else:
        print(item["path"], item["analysis"]["passive_ratio"])
# 0.3
for item in engine.analyze_all(dir="docs/"):
    if isinstance(item, matra.Document):
        print(item["provenance"]["source"]["path"], item["metrics"]["passive_ratio"])
    else:
        print(item["path"], item["kind"])

Read a document

grade6.1density0.59compressionnone

A matra.Document reads like the dict it replaced: doc["metrics"]["total_words"], list(doc) and len(doc) work. It is read-only. Call doc.to_dict() where your code needs a dict, changes the result, or pickles it. The keys follow the schema 2.0 record, so the measures moved:

# 0.2
doc["passive_ratio"]
doc["sections"][0]["paragraphs"][0]["readability_grade"]
doc["sections"][0]["paragraphs"][0]["in_blockquote"]
# 0.3
doc["metrics"]["passive_ratio"]
doc["sections"][0]["paragraphs"][0]["metrics"]["readability_grade"]
doc["sections"][0]["paragraphs"][0]["kind"] == "quote"

Summaries, keyphrases and clusters

grade8.8density0.56compression0.55

The four algorithm-named methods and semantic_clusters are module functions over a Document. A text is parsed once however many outcomes you ask of it, and in the format you analysed it in: 0.2's methods read every text as plain text, so a README's headings could land in its summary. Analyse a markdown file with file=, or a markdown string with format="markdown", and its headings stay headings. max_phrases is n. Each returns a dict whose provenance names the method and its parameters.

# 0.2
top = m.tfidf_summarize(text, 3)                   # [{"text", "score", "position"}]
ranked = m.textrank_summarize(text, 3)
phrases = m.yake_keyphrases(text, max_phrases=10)  # [{"phrase", "score"}]
groups = m.semantic_clusters(text, 0.85, model)
# 0.3
doc = engine.analyze(text)
top = matra.summarize(doc, 3)                       # {"provenance", "sentences": [{"index", "span", "text", "score"}]}
ranked = matra.summarize(doc, 3, method="textrank")
phrases = matra.keyphrases(doc, 10, method="yake")  # {"provenance", "phrases": [{"phrase", "score", "occurrences"}]}
groups = matra.clusters(doc, model, 0.85)
grade10.4density0.68compressionnone

The vectors-in matra.semantic_clusters(vectors, threshold, model_hash) is gone, because Rust never had it. To compare vectors you already hold, compute the cosine yourself, as the semantic clusters guide shows.

Exceptions

grade7.2density0.59compressionnone

Every exception is a matra.MatraError subclass with kind and details. Each also derives from the builtin class 0.2 raised, so except ValueError and except OSError catch what they caught.

# 0.2
try:
    m.analyze(text)
except ValueError as e:          # which ValueError: read the message
    ...
# 0.3
try:
    engine.analyze(text)
except matra.InputTooLargeError as e:   # also a ValueError
    print(e.kind, e.details["limit"], e.details["actual"])
grade10.5density0.42compressionnone

A missing file was a bare OSError in 0.2. It is now matra.IoNotFoundError, which is a FileNotFoundError and an OSError, so except FileNotFoundError now catches it. A permission failure is matra.IoPermissionError, a PermissionError.

Types

grade15.5density0.63compressionnone

matra.types follows the record. ScoredSentence is SummarySentence inside Summary; SemanticClusters, SemanticCluster and SemanticEdge are Clusters, Cluster and SimilarityEdge; Reporting is ClausalComplement; CorpusEntry and CorpusItem are gone. The TypedDict that was named Document is DocumentDict, the shape to_dict() returns, and matra.Document is the class.

Command line

The config file

grade4.6density0.61compressionnone

A config file from 0.2 does not load. Every command that reads one fails with invalid_input, naming the file and each key's replacement. Edit the file, or write a fresh one with matra config init --force.

# 0.2
[models]
udpipe = "english-ewt-ud-2.5-191206"
embedding = "potion-base-8M"

[semantic]
threshold = 0.85

[summarize]
n = 3
algorithm = "tfidf"

[keyphrases]
n = 10
algorithm = "rake"
# 0.3
[models]
embedding = "potion-base-8M"

[summary]
n = 3
method = "tfidf"

[keyphrases]
n = 10
method = "rake"

[clusters]
threshold = 0.85
grade8.5density0.65compressionnone

models.udpipe goes without a replacement, because the parsing model is pinned in the source beside its digest.

The JSON envelope

grade6.1density0.53compressionnone

format_version is 2, and the envelope gains a fifth key, error. All five are always present, and exactly one of result and error is null. A consumer pinned to format_version: 1 should fail, and then move.

{ "format_version": 1, "command": "analyze", "input": "essay.md", "result": {} }
{ "format_version": 2, "command": "analyze", "input": "essay.md", "result": {}, "error": null }
gradenonedensity0.67compressionnone

result changed too:

  • analyze: a schema 2.0 Document. See The JSON record below.
  • summarize: a Summary record, {provenance, sentences}, not a bare list of {text, score, position}.
  • keyphrases: a Keyphrases record, {provenance, phrases}, not a bare list of {phrase, score}.
# 0.2
matra summarize essay.md --json | jq '.result[0].text'
matra keyphrases essay.md --json | jq '.result[].phrase'
# 0.3
matra summarize essay.md --json | jq '.result.sentences[0].text'
matra keyphrases essay.md --json | jq '.result.phrases[].phrase'

Failures under --json

grade4.2density0.51compression0.56

In 0.2 a failure under --json wrote nothing to stdout. In 0.3 it writes the envelope there, with result null and error holding the kind, message and details. It still writes matra: <message> to stderr and still exits 2, so a script that reads only the exit code is unchanged. A usage error under --json is an envelope too.

# 0.3
matra analyze missing.md --json | jq -r '.error.kind'      # io
grade2.4density0.58compressionnone

The message for a missing file changed with it: io error: cannot read missing.md: ... where 0.2 printed no such file: missing.md. Branch on error.kind and read error.details, never the message.

Standard input

grade6.3density0.47compression0.61

Standard input is read as bytes, and its format is detected as a file's is. --stdin-filename still names it, and its extension still picks the format. Bytes that are not UTF-8 fail as unsupported_format, as a file's do, where 0.2 failed with a read error that had no kind.

What is new

grade6.7density0.61compressionnone

These do not break a 0.2 command, and are listed so you know they exist: matra analyze DIR, one result per file; matra clusters; --format on every document command; --depth on analyze --json; and matra --schema. The CLI guide has each.

The JSON record

grade7.3density0.55compressionnone

The same record is the Rust Document, the Python matra.Document and analyze --json's result. It now names its own schema version in provenance.schema, "2.0". The schema is published: matra --schema document prints it.

0.20.3
no provenanceprovenance: the matra version, the schema, the parser and its model's SHA-256, the format, and the input's path, and the size and SHA-256 of the analysed text
vocabulary_ttr, nominalization_ratio, passive_ratiometrics.vocabulary_ttr, metrics.nominalization_ratio, metrics.passive_ratio
absentmetrics.total_words, total_sentences, paragraph_count, mean_sentence_length, sentence_length_std
absentskipped: each construct the reader saw and did not analyse, with its kind and span
sections[].heading, level, paragraphsthe same, and span (the heading's, null with no heading)
paragraph.readability_grade, lexical_density, compression_ratioparagraph.metrics.readability_grade, and so on
paragraph.in_blockquoteparagraph.kind: paragraph, quote or list_item
absentspan on every paragraph, sentence and token, in UTF-8 bytes of the analysed text
absentsentence.word_count, tree_depth (null on a cycle), passive
sentence.reportingssentence.clausal_complements
sentence.bare_assertionsentence.root_mood: {root_id, mood}, or null
token.feats: one string of pipe-separated featurestoken.features: {"Mood": "Ind", "Tense": "Past"}
grade12.2density0.53compressionnone

Read the record as the schema asks: ignore a field you do not recognize, treat every string set you receive as open, and reject a major you do not know.

Results that change with no rename

grade6.1density0.44compressionnone

A few results differ for the same input, and nothing about the call tells you so.

  • Markdown is read by a CommonMark parser. 0.2 dropped tables without a trace, ended the document at a ## References line, read #hashtag as a heading, and passed inline markup such as ** and link syntax to the parser as words. In 0.3 tables, code, front matter and HTML are skipped entries, a ## References heading is a heading like any other, a heading needs a space after its #, list items are paragraphs of kind list_item, and inline markup is removed from the text the parser reads. The counts and measures of a markdown document can change.
  • Plain text splits paragraphs at a line of only whitespace, so \r\n endings and a line of spaces separate paragraphs as an empty line does.
  • Keyphrases with equal scores come in the same order on every run, by first occurrence, where 0.2 followed a hash map's order.
  • TF-IDF scores are the same to the last bit on every run.
  • A binary file is unsupported_format, naming the format when it is a PDF or DOCX. In 0.2 it was an I/O error.
  • The size cap's message reads input too large, where 0.2 read input input too large.
  • Model2Vec.potion_base_8m(dir) looks in dir/potion-base-8M/, where 0.2 looked in dir itself. A 0.2 call that passed the model's own directory still runs, and downloads a second copy one level down. Pass the models directory instead.
  • Model2Vec.potion_base_8m() with no argument reads no config file. It finds the model directory from the environment, as Engine() does, so a model_dir or models.embedding set only in the config file no longer moves it. Model2Vec.from_config() reads the file.
  • Python's constructors announce a download. Engine(), Engine.from_config() and Model2Vec.potion_base_8m() write two lines to sys.stderr before they download a model, the ones the command line prints. Once the model is on disk they write nothing.
  • An empty result says so. matra summarize, keyphrases and clusters still exit 1 when they find nothing, and now also write one line to stderr saying so, unless --json or --quiet is given.

This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 43 paragraphs, in the margin. the data