Enhancement proposal EPR-0002 The data model proposed

0 of 2 decisions settled 1 assumption to confirm pinned at 4fcfb4a pull request #133 tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track) started: 2026-10-04feature name: data_model decides: the owner, who alone sets the status to accepted

EPR-0002 0 of 2 settled 1 to confirm

Enhancement proposal EPR-0002 The data model proposed

pinned at 4fcfb4a pull request #133 tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track) started: 2026-10-04feature name: data_model decides: the owner, who alone sets the status to accepted

EPR-0002: The data model

enhancement proposal: proposed

on this record: read, decide, confirm
Started
2026-10-04
Pinned at
4fcfb4a, the commit every code citation below opens at
Decides
the owner, who alone sets the status to accepted
Proposal PR
#133
Tracking issue
none (a baseline proposal describes code that already ships, so there is no milestone to track)
Feature name
data_model

Summary

Everything matra hands back is a type in src/domain.rs, which depends on serde, thiserror and std and nothing else. A parsed document is a tree: Document holds Sections, which hold Paragraphs, which hold Sentences, which hold CoNLL-U Tokens. Structural facts derived from the parse (negation cues, modal auxiliaries, the bare-assertion flag, reporting constructions, root adverbials, Hearst pairs) and the document-level passive ratio are stored as fields, so they reach Python and the JSON the command line emits as data. A failure is a domain::Error variant with a stable kind string. Names reserved for the rule-evaluation tier stay reserved.

This is a baseline proposal. It describes the code at the pinned commit and carries forward the reasons that still hold from RFC-0002, RFC-0006, RFC-0008, RFC-0009 and EP-0007, with the error-routing points RFC-0011 and RFC-0015 settled. A later proposal that changes any of it says so. Unmarked statements about the code are observed at the pinned commit through the link beside them; a claim that is inferred or assumed says so.

Motivation

The data model is the contract that crosses languages. Rust callers read the types; Python callers read the dicts pythonize builds from their serde form observed lib.rs:452-455; command-line callers read the same form inside the JSON envelope. Three decisions make that contract hold.

Fields cross the language boundary, methods do not. The Python surface is the Serialize derive and nothing else, so a Rust method on a domain type does not exist for Python or the JSON. Before RFC-0008, the Python command line re-implemented passive detection over raw tokens because Sentence::is_passive was a method, and it disagreed with the Rust one. The rule settled then: derivations of the parse are stored as fields, computed once in Rust; views over data already crossing stay methods.

The record is structure, not interpretation. Every derived field names a construction in the parse (a cue and its arc, an auxiliary and its head, a pattern and its spans), never a category such as "hedged" or "deontic". EP-0007 held this line for all five primitives, because the moment matra decides must is deontic it has taken a position on meaning the caller is better placed to take.

A small, pure domain. Keeping domain.rs to three dependencies means a caller can depend on the types without pulling the parser, the embedding stack or Python, and it keeps the wire form exactly the struct form.

Guide-level explanation

A Document is the parse made queryable. In Rust:

for sentence in doc.sentences() {
    for neg in &sentence.negations {
        // neg.cue_id, neg.cue_lemma, neg.head_id: token ids in this sentence
    }
    let verb = sentence.tokens.iter().find(|t| t.pos == "VERB");
    let mood = verb.and_then(|t| t.feat("Mood")); // Some("Ind"), say
}
let ratio = doc.passive_ratio; // Option<f64>, filled by compose

In Python the same document is a dict with the same keys (doc["sections"][0]["paragraphs"][0]["sentences"][0]["negations"]), typed by TypedDicts in matra.types observed types.py:155-201.

What a caller should hold on to:

  • Metric slots are Option. None means compose has not run or the metric does not apply (a paragraph with no words, a blockquote). There is no separate "not measured" marker.
  • Ids are sentence-scoped. Every derived fact points into sentence.tokens by token id, so provenance is checkable against the parse.
  • Lexicons are the caller's. Reporting verbs and evidential adverbs are open classes, so matra reports every construction and lets the caller filter: sentence.reportings_in(&["claim", "report"]) in Rust, a filter on verb_lemma elsewhere observed domain.rs:822-851.
  • Errors are matched, not read. A caller branches on the Error variant in Rust, on the exception class in Python, and on kind wherever an error is materialized as data.

Reference-level explanation

The tree

TypeFieldsFilled by
Documentsections, vocabulary_ttr, nominalization_ratio, passive_ratiosections by annotate; the three ratios by compose
Sectionheading, level, paragraphsthe decomposer
Paragraphtext, in_blockquote, sentences, readability_grade, lexical_density, compression_ratiotext and flag by the decomposer; sentences by annotate; slots by compose
Sentencetext, tokens, negations, modals, bare_assertion, reportings, root_adverbials, hearst_pairsthe provider, through Sentence::new; hearst_pairs by annotate
Tokenthe ten CoNLL-U columns plus is_punctthe provider

Sentence documents four invariants (ids sorted, one root, heads in range, no cycle) and does not validate them; the NlpProvider contract states them for providers observed domain.rs:706-790. The tree walks stay safe on input that breaks the last one: tree_depth returns usize::MAX on a cycle, using a visited set rather than a depth ceiling, and subtree carries a visited set too observed domain.rs:875-956 domain.rs:706-790.

The structural primitives

FieldReportsDetector
negationscue (not, never, no, neither, nor) on advmod, det or cc, and its headdetect_negations
modalsan auxiliary from the closed ten-lemma class, on aux or tagged AUX, and its head; never its readingdetect_modals
bare_assertionroot clause finite indicative with no modal governing itdetect_bare_assertion
reportingseach verb governing a ccomp, with its subject when parseddetect_reportings
root_adverbialseach advmod attached to the rootdetect_root_adverbials
hearst_pairscandidate hypernym and hyponym spans from the six Hearst (1992) patterns, with the pattern taghearst::hypernymy_pairs (types)

Five are computed in Sentence::new observed domain.rs:792-820. hearst_pairs is filled at the annotate stage because its detector lives in src/hearst.rs, outside the domain, which imports only domain types observed lib.rs:235-243 hearst.rs:28. That is the amendment RFC-0008 recorded for EP-0007's fifth milestone: the field still crosses as data; only the point that fills it moved. A hand-built Sentence carries an empty hearst_pairs until the caller runs the detector.

Every derived field is #[serde(default)], so a document serialized before the field existed still deserializes observed domain.rs:706-790. Derived fields reflect tokens as passed to Sentence::new; mutating tokens afterwards does not recompute them, and keeping them consistent is the caller's side of the documented contract observed domain.rs:792-820.

Views stay methods

Token::feat looks up one key in the feats string by a linear scan with no allocation, and returns the raw value, so Case=Nom,Acc comes back unsplit observed domain.rs:287-308. It is Rust-only by design (RFC-0009): feats already crosses as a string, so the lookup adds nothing to the wire. reportings_in, root_adverbials_in, is_passive, tree_depth, subtree, the Document iterators and the Corpus aggregates are views of the same kind.

Document::passive_ratio() the method and Document::passive_ratio the field both exist observed domain.rs:1178-1191 domain.rs:1095-1120. The method computes the ratio on any document; the metric suite stores it in the field so it crosses observed metrics/document.rs:12-22.

Errors

Error has seven variants, each with a stable kind string from Error::kind, an exhaustive match with no wildcard so a new variant cannot inherit another's key observed domain.rs:151-191 domain.rs:193-217. The vocabulary is pinned for every binding in spec/tests/corpus/items.json and mirrored in Python as ERROR_KINDS observed items.json:4-12 types.py:15-31. InputTooLarge carries a what label naming which gate fired.

In Python each variant maps to one exception class through an exhaustive match: ModelNotFound to FileNotFoundError; InputTooLarge, UnsupportedFormat and InvalidInput to ValueError; Io to OSError; ModelInvalid and ParseFailed to RuntimeError observed lib.rs:417-450. A new variant fails to compile until it is routed.

DocumentError and CorpusResult are not Serialize, because Error wraps std::io::Error observed domain.rs:1448-1464. Where a per-document failure crosses, it is materialized as {kind, message} by hand observed lib.rs:1036-1055.

Forward compatibility

Every public struct with public fields and every public enum in the library is #[non_exhaustive], observed by listing them at the pinned commit, and the one exception is Embedding, a tuple struct whose constructor external Embedder implementors must call observed domain.rs:93; RFC-0010 records why (EPR-0005). Token::builder exists because the attribute forbids struct literals outside the crate observed domain.rs:261-285.

Reserved vocabulary still in force

RFC-0006 reserved names for the rule-evaluation tier before any of it exists, so that tier cannot arrive under other names. None of these is a type in src/ today.

ConceptReserved name
Declarative rule wrapper; predicate over DocumentRule, Predicate
Umbrella over rule outputFinding (not Frame)
Pointer back to the source bytesSourceSpan
Relation triple; typed entity; modal marker; illocutionary force; stylometric profileRelation, Schema, Modality, SpeechAct, Stylometry
Paragraph kind; orthogonal paragraph roleParagraphKind, ParagraphRole
Grouped metric slotsParagraphMetrics, DocumentMetrics

Frame and the verb frame stay reserved for frame semantics, as RFC-0002 first set and RFC-0006 kept. The five Finding rules RFC-0006 states (provenance mandatory, confidence shape mandatory, a discriminant that names a structural fact, only FFI-safe types, #[non_exhaustive]) bind whichever shape lands. If metric slots are ever grouped, an outer Option<ParagraphMetrics> is ruled out, because each slot gates independently. ROADMAP.md describes the rule-evaluation entry against this vocabulary (roadmap).

No longer in force

  • The Analysis alias. RFC-0006 renamed Analysis to Document with a deprecated alias due for removal before 0.1.0. The alias is gone: no type named Analysis exists in src/.
  • in_blockquote as a deprecation signal. RFC-0006 says the field's rustdoc signals its replacement by ParagraphKind. The rustdoc now says the replacement is planned and that nothing is deprecated, because the field's job is still binary observed domain.rs:1019-1029.
  • The measure and extract split as a type distinction. RFC-0002 called extraction a peer of measurement; RFC-0007 superseded it. Extraction output (ScoredSentence, Keyphrase) remains plain domain types; the stage vocabulary is EPR-0001's.

Drawbacks

  • Every crossing field is a schema commitment in Rust, Python and the JSON. A rename is a breaking change in all three (RFC-0008).
  • Everyone pays for every primitive. Each sentence stores its derived facts whether or not the caller reads them. That storage is small beside the tokens already on the wire assumed: RFC-0008 judged so, and no benchmark at the pinned commit measures it.
  • Derived fields go stale under mutation of tokens, by the documented contract rather than by a check.
  • Two spellings of one value. passive_ratio is both a method and a field on Document, and they disagree on a document with no sentences: the method returns 0.0 and the field stays None even after compose observed domain.rs:1178-1191 metrics/document.rs:12-22.

Rationale and alternatives

  • Methods only: no schema lock-in and no storage, but invisible to every binding, so each binding re-implements each primitive with nothing checking they agree. Rejected in RFC-0008, with the Python passive fold as the live case.
  • Methods with a hand-written Serialize: the same wire at a worse price, a wire that no longer equals the struct and graph walks inside serialization, in the most protected file. Rejected in RFC-0008.
  • Exhaustive typed enums for feats: the Universal Dependencies feature inventory is large, language-dependent and grows by release. Rejected in RFC-0009, as was a per-token HashMap, for an allocation on the hot path with no benchmark asking for it.
  • Resolving modal readings or shipping evidential word lists: rejected in EP-0007, because a reading is interpretation and an incomplete open-class list that looks authoritative is worse than none.
  • Routing primitives through Finding: would make the primitives wait on a shape RFC-0006 defers until rule evaluation exists. Rejected in RFC-0008.

Prior art

  • CoNLL-U, the Universal Dependencies format, is the token shape: one field per column.
  • Hearst (1992), "Automatic acquisition of hyponyms from large text corpora", is the source of the six patterns, implemented here as dependency-arc patterns rather than surface regexes.
  • RFC-0006, RFC-0008, RFC-0009 and EP-0007 hold the full history.

Unresolved questions

For the owner to settle in this proposal's review. Each is a decision of its own.

CorpusEntry.analysis still carries the rejected name. The field is analysis, holding a Document observed domain.rs:1338-1346 grounds for decision 1. RFC-0006's deferred list, extended by RFC-0010, records that the field should be renamed, and that the rename became a breaking change once 0.1.0 shipped. When, and folded into what, is open.

Decision · open · costly to reverse

1. When is CorpusEntry.analysis renamed?

A With the next release that breaks the schema recommended

The rename is folded into a release whose callers migrate anyway, so they migrate once.

B In a release of its own

The rename ships as soon as possible, on its own, with the old key still accepted when a document is read back.

C Never

The field keeps analysis, and its documentation says it holds a Document.

Claude recommends A, With the next release that breaks the schema. inferred, a judgment

Every crossing field is a schema commitment in Rust, Python and the JSON, so the rename costs every caller a migration whenever it lands. Landing it where callers migrate anyway pays that cost once.

The strongest case against it

Until then every new caller learns a name the records already rejected, and each release that ships it adds callers who will have to move.

The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.

Error::Io routes to OSError whatever its kind. Every Io variant becomes OSError observed lib.rs:417-450 grounds for decision 2. A missing directory arrives in Python as OSError, not FileNotFoundError, although the wrapped io::ErrorKind is NotFound (RFC-0011, unresolved questions). Routing on the wrapped kind would change a shipped mapping.

Decision · open · costly to reverse

2. Does Error::Io route on the kind it wraps?

A Route on the wrapped kind recommended

NotFound becomes FileNotFoundError, PermissionDenied becomes PermissionError, and every other kind stays OSError.

B Keep one class

Every Error::Io stays OSError, and the mapping's documentation says so on purpose.

Claude recommends A, Route on the wrapped kind. inferred, a judgment

Python's FileNotFoundError and PermissionError are subclasses of OSError, so an except OSError clause written today keeps catching them, and a caller who expected FileNotFoundError for a missing path gets it.

The strongest case against it

It changes a shipped mapping: code that tests an exception's exact class, rather than catching it, sees a different class. And the kind string stays io for all of them, so the exception class would name a failure more finely than the kind beside it.

The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.

What the owner should confirm or strike, collected from the claims above that stand on no grounds yet:

1 claim stands on no grounds yet. Confirm or strike each with a comment on it.

  1. That storage is small beside the tokens already on the wire where: Drawbacks bears on no decision

Future possibilities

  • The rule-evaluation tier, using the reserved names, reading these fields rather than walking the raw parse. Its Finding shape (trait or enum) is decided when the first concrete consumer appears.
  • ParagraphKind replacing in_blockquote once a paragraph needs more than parse-or-skip.