Enhancement proposalEPR-0002The data model○ proposed
pinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: data_modeldecides: the owner, who alone sets the status to accepted
Everything matra hands back is a type in src/domain.rs, which depends on
serde, thiserror and std and nothing else. A parsed document is a tree:
Document holds Sections, which hold Paragraphs, which hold
Sentences, which hold CoNLL-U Tokens. Structural facts derived from the
parse (negation cues, modal auxiliaries, the bare-assertion flag, reporting
constructions, root adverbials, Hearst pairs) and the document-level passive
ratio are stored as fields, so they reach Python and the JSON the command
line emits as data. A failure is a domain::Error variant with a stable
kind string. Names reserved for the rule-evaluation tier stay reserved.
This is a baseline proposal. It describes the code at the pinned commit and
carries forward the reasons that still hold from RFC-0002, RFC-0006,
RFC-0008, RFC-0009 and EP-0007, with the error-routing points RFC-0011 and
RFC-0015 settled. A later proposal that changes any of it says so. Unmarked
statements about the code are observed at the pinned commit through the link
beside them; a claim that is inferred or assumed says so.
The data model is the contract that crosses languages. Rust callers read the
types;
Python callers read the dicts pythonize builds from their serde form ● observedlib.rs:452-455;
command-line callers read the same form inside the JSON envelope. Three
decisions make that contract hold.
Fields cross the language boundary, methods do not. The Python surface
is the Serialize derive and nothing else, so a Rust method on a domain
type does not exist for Python or the JSON. Before
RFC-0008, the
Python command line re-implemented passive detection over raw tokens because
Sentence::is_passive was a method, and it disagreed with the Rust one.
The rule settled then: derivations of the parse are stored as fields,
computed once in Rust; views over data already crossing stay methods.
The record is structure, not interpretation. Every derived field names a
construction in the parse (a cue and its arc, an auxiliary and its head, a
pattern and its spans), never a category such as "hedged" or "deontic".
EP-0007 held this line for
all five primitives, because the moment matra decides must is deontic it
has taken a position on meaning the caller is better placed to take.
A small, pure domain. Keeping domain.rs to three dependencies means a
caller can depend on the types without pulling the parser, the embedding
stack or Python, and it keeps the wire form exactly the struct form.
for sentence in doc.sentences() { for neg in &sentence.negations { // neg.cue_id, neg.cue_lemma, neg.head_id: token ids in this sentence } let verb = sentence.tokens.iter().find(|t| t.pos == "VERB"); let mood = verb.and_then(|t| t.feat("Mood")); // Some("Ind"), say}let ratio = doc.passive_ratio; // Option<f64>, filled by compose
In Python the same document is a dict with the same keys (doc["sections"][0]["paragraphs"][0]["sentences"][0]["negations"]), typed by TypedDicts in matra.types● observedtypes.py:155-201.
python/matra/types.py lines 155 to 201, at 4fcfb4a
155 class Sentence(TypedDict):
156 """One parsed sentence. Mirrors `matra::domain::Sentence`."""
157158 text: str
159 tokens: list[Token]
160 negations: list[Negation]
161 modals: list[Modal]
162 bare_assertion: bool
163 reportings: list[Reporting]
164 root_adverbials: list[RootAdverbial]
165 hearst_pairs: list[HearstPair]
166167168 class Paragraph(TypedDict):
169 """One paragraph with optional metric slots. Mirrors `matra::domain::Paragraph`."""
170171 text: str
172 in_blockquote: bool
173 sentences: list[Sentence]
174 readability_grade: float | None
175 lexical_density: float | None
176 compression_ratio: float | None
177178179 class Section(TypedDict):
180 """One section (heading + paragraphs). Mirrors `matra::domain::Section`."""
181182 heading: str | None
183 level: int
184 paragraphs: list[Paragraph]
185186187 class Document(TypedDict):
188 """Top-level analysis output. Mirrors `matra::domain::Document`.
189190 Document-level aggregates that cross the FFI boundary do so as
191 fields filled by the metric suite (RFC-0008): `passive_ratio`
192 arrives materialized, like `vocabulary_ttr`. Remaining aggregate
193 methods on the Rust `Document` (`mean_sentence_length`,
194 `total_words`, etc.) do not cross; consumers compute them from
195 `sections` if needed.
196 """
197198 sections: list[Section]
199 vocabulary_ttr: float | None
200 nominalization_ratio: float | None
201 passive_ratio: float | None
Metric slots are Option.None means compose has not run or the
metric does not apply (a paragraph with no words, a blockquote). There is
no separate "not measured" marker.
Ids are sentence-scoped. Every derived fact points into
sentence.tokens by token id, so provenance is checkable against the
parse.
Lexicons are the caller's.Reporting verbs and evidential adverbs are open classes, so matra reports every construction and lets the caller filter: sentence.reportings_in(&["claim", "report"]) in Rust, a filter on verb_lemma elsewhere ● observeddomain.rs:822-851.
Errors are matched, not read. A caller branches on the Error
variant in Rust, on the exception class in Python, and on kind wherever
an error is materialized as data.
src/domain.rs lines 822 to 851, at 4fcfb4a
822 /// The reporting constructions whose verb lemma is in the
823 /// caller-supplied lexicon.
824 ///
825 /// Reporting verbs are an open class, so matra ships no default
826 /// lexicon: an incomplete list that looks authoritative is worse
827 /// than none. The structural detection ([`Sentence::reportings`])
828 /// crosses FFI as data; this is a Rust-side convenience view over
829 /// it, per RFC-0008's criterion (derivations cross as fields,
830 /// views over crossing data stay methods). Non-Rust consumers
831 /// filter the field by `verb_lemma` the same way.
832 pub fn reportings_in(&self, lexicon: &[&str]) -> Vec<&Reporting> {
833 self.reportings
834 .iter()
835 .filter(|r| lexicon.contains(&r.verb_lemma.as_str()))
836 .collect()
837 }
838839 /// The root-attached adverbials whose lemma is in the
840 /// caller-supplied lexicon.
841 ///
842 /// Evidential adverbs are an open class, so matra ships no default
843 /// lexicon; see [`Sentence::reportings_in`] for the reasoning and
844 /// the RFC-0008 classification. Non-Rust consumers filter
845 /// [`Sentence::root_adverbials`] by `adv_lemma` the same way.
846 pub fn root_adverbials_in(&self, lexicon: &[&str]) -> Vec<&RootAdverbial> {
847 self.root_adverbials
848 .iter()
849 .filter(|a| lexicon.contains(&a.adv_lemma.as_str()))
850 .collect()
851 }
Sentence documents four invariants (ids sorted, one root, heads in range, no cycle) and does not validate them; the NlpProvider contract states them for providers ● observeddomain.rs:706-790.
The tree walks stay safe on input that breaks the last one: tree_depth returns usize::MAX on a cycle, using a visited set rather than a depth ceiling, and subtree carries a visited set too ● observeddomain.rs:875-956domain.rs:706-790.
src/domain.rs lines 706 to 790, at 4fcfb4a
706 /// One parsed sentence: its text plus its ordered tokens.
707 ///
708 /// Invariants downstream code relies on:
709 /// - `tokens` are id-sorted ascending.
710 /// - Exactly one token has `head == 0` (the syntactic root).
711 /// - All `head` references point to another token in the same sentence
712 /// or to `0`.
713 /// - Following `head` references never loops back on itself.
714 ///
715 /// Nothing validates these: [`Sentence::new`] trusts its caller, and the
716 /// `NlpProvider` contract states them for providers. The tree walks stay
717 /// safe on input that breaks the last one: [`Sentence::tree_depth`]
718 /// returns `usize::MAX` on a cycle and [`Sentence::subtree`] carries a
719 /// visited set.
720 #[derive(Debug, Clone, Serialize, Deserialize)]
721 #[non_exhaustive]
722 pub struct Sentence {
723 /// Sentence text as the NLP provider reports it. The UDPipe adapter
724 /// rebuilds it from the token forms, joined by a space unless a token
725 /// carries `SpaceAfter=No` in `misc`, so its whitespace can differ
726 /// from the source.
727 pub text: String,
728 /// CoNLL-U tokens in id-sorted order.
729 pub tokens: Vec<Token>,
730 /// Negation cues derived from the dependency graph at construction
731 /// (see [`Sentence::new`]). Serialized with the sentence so the
732 /// detection crosses FFI as data (RFC-0008). Defaults to empty when
733 /// deserializing sentences serialized before this field existed.
734 #[serde(default)]
735 pub negations: Vec<Negation>,
736 /// Modal auxiliaries derived from the dependency graph at
737 /// construction (see [`Sentence::new`]): tokens from the closed
738 /// class `can`, `could`, `may`, `might`, `must`, `ought`,
739 /// `shall`, `should`, `will`, `would`, matched by lemma on the
740 /// `aux` relation or the `AUX` part of speech (the latter catches
741 /// modals the model promotes to root or `conj` under VP ellipsis
742 /// and coordination). Serialized with the sentence so the
743 /// detection crosses FFI as data (RFC-0008). Defaults to empty
744 /// when deserializing sentences serialized before this field
745 /// existed.
746 #[serde(default)]
747 pub modals: Vec<Modal>,
748 /// Bare-assertion discriminator, derived at construction: true
749 /// when the root clause is finite indicative (`Mood=Ind` on the
750 /// root token or on a `cop`, `aux`, or `aux:pass` child of it)
751 /// and no modal auxiliary governs it. Reports the surface form
752 /// only; what the assertion commits its speaker to is the
753 /// consumer's reading. Defaults to false when deserializing
754 /// sentences serialized before this field existed.
755 #[serde(default)]
756 pub bare_assertion: bool,
757 /// Reporting constructions derived from the dependency graph at
758 /// construction (see [`Sentence::new`]): each verb governing a
759 /// clausal complement (`ccomp`), with its subject when the parse
760 /// has one. The construction is reported for every verb that
761 /// fills it; which verbs count as evidential is the consumer's
762 /// lexicon ([`Sentence::reportings_in`]). Serialized with the
763 /// sentence so the detection crosses FFI as data (RFC-0008).
764 /// Defaults to empty when deserializing sentences serialized
765 /// before this field existed.
766 #[serde(default)]
767 pub reportings: Vec<Reporting>,
768 /// Adverbial modifiers attached to the root, derived at
769 /// construction (see [`Sentence::new`]): the arc sentence-scope
770 /// adverbs land on. Every root-attached `advmod` is reported;
771 /// which lemmas read as evidential is the consumer's lexicon
772 /// ([`Sentence::root_adverbials_in`]). Serialized with the
773 /// sentence so the detection crosses FFI as data (RFC-0008).
774 /// Defaults to empty when deserializing sentences serialized
775 /// before this field existed.
776 #[serde(default)]
777 pub root_adverbials: Vec<RootAdverbial>,
778 /// Candidate hypernymy pairs from the six Hearst (1992) patterns.
779 ///
780 /// Unlike the other derived fields, this one is not computed by
781 /// [`Sentence::new`]: the detector lives in `matra::hearst`, a
782 /// module outside the domain (EP-0007 M5 boundary), so the pipeline
783 /// fills the field at the annotate stage. A hand-built `Sentence`
784 /// carries an empty vector until the caller runs the detector.
785 /// Serialized with the sentence so the detection crosses FFI as
786 /// data (RFC-0008). Defaults to empty when deserializing
787 /// sentences serialized before this field existed.
788 #[serde(default)]
789 pub hearst_pairs: Vec<HearstPair>,
790 }
875 /// Maximum depth of the dependency tree (longest path from any token
876 /// to root, where root is the token with `head = 0`).
877 ///
878 /// O(n) per sentence: builds a `HashMap<id, head>` once, then walks
879 /// each token's ancestor chain with depth memoization in a second
880 /// `HashMap<id, usize>`. Each token's depth is computed at most once;
881 /// the visited set inside each walk detects cycles and returns
882 /// `usize::MAX` for that token (and any token transitively rooted in
883 /// the cycle), surfacing the malformed parse loudly rather than
884 /// silently truncating.
885 ///
886 /// On a malformed parse where some tokens form a cycle, the depth
887 /// returned for the sentence is `usize::MAX` (the max over per-token
888 /// depths). On a well-formed tree of any depth (no artificial
889 /// ceiling), the depth is the true tree depth.
890 pub fn tree_depth(&self) -> usize {
891 use std::collections::{HashMap, HashSet};
892893 // Build id → head once. `0` head means root.
894 let head_by_id: HashMap<usize, usize> =
895 self.tokens.iter().map(|t| (t.id, t.head)).collect();
896897 // Memoize depth per token id. usize::MAX is the cycle sentinel.
898 let mut depth_by_id: HashMap<usize, usize> = HashMap::new();
899900 // Compute depth for one token. Walks up the head chain, using
901 // a visited set to detect cycles in this single chain. On hit,
902 // memoizes via the cache to amortize across tokens that share
903 // ancestors.
904 fn depth_of(
905 id: usize,
906 head_by_id: &HashMap<usize, usize>,
907 depth_by_id: &mut HashMap<usize, usize>,
908 ) -> usize {
909 if let Some(&d) = depth_by_id.get(&id) {
910 return d;
911 }
912 let mut visited: HashSet<usize> = HashSet::new();
913 let mut chain: Vec<usize> = Vec::new();
914 let mut cur = id;
915916 // Walk to root or to a memoized ancestor or to a cycle.
917 let base_depth = loop {
918 if !visited.insert(cur) {
919 // Cycle detected — every token in the chain is malformed.
920 for &cid in &chain {
921 depth_by_id.insert(cid, usize::MAX);
922 }
923 return usize::MAX;
924 }
925 let head = head_by_id.get(&cur).copied().unwrap_or(0);
926 if head == 0 {
927 // cur is root or its head is missing: depth contribution from this point is 0.
928 chain.push(cur);
929 break 0usize;
930 }
931 if let Some(&d) = depth_by_id.get(&head) {
932 chain.push(cur);
933 break d.saturating_add(1);
934 }
935 chain.push(cur);
936 cur = head;
937 };
938939 // Walk back down the chain, assigning depths in order.
940 // chain[last] is the closest-to-root token; chain[0] is the
941 // original `id`. Depth of chain[last] = base_depth; each step
942 // away from root adds 1.
943 let n = chain.len();
944 for (i, &cid) in chain.iter().enumerate() {
945 let d = base_depth.saturating_add(n - 1 - i);
946 depth_by_id.insert(cid, d);
947 }
948 depth_by_id[&id]
949 }
950951 self.tokens
952 .iter()
953 .map(|t| depth_of(t.id, &head_by_id, &mut depth_by_id))
954 .max()
955 .unwrap_or(0)
956 }
706 /// One parsed sentence: its text plus its ordered tokens.
707 ///
708 /// Invariants downstream code relies on:
709 /// - `tokens` are id-sorted ascending.
710 /// - Exactly one token has `head == 0` (the syntactic root).
711 /// - All `head` references point to another token in the same sentence
712 /// or to `0`.
713 /// - Following `head` references never loops back on itself.
714 ///
715 /// Nothing validates these: [`Sentence::new`] trusts its caller, and the
716 /// `NlpProvider` contract states them for providers. The tree walks stay
717 /// safe on input that breaks the last one: [`Sentence::tree_depth`]
718 /// returns `usize::MAX` on a cycle and [`Sentence::subtree`] carries a
719 /// visited set.
720 #[derive(Debug, Clone, Serialize, Deserialize)]
721 #[non_exhaustive]
722 pub struct Sentence {
723 /// Sentence text as the NLP provider reports it. The UDPipe adapter
724 /// rebuilds it from the token forms, joined by a space unless a token
725 /// carries `SpaceAfter=No` in `misc`, so its whitespace can differ
726 /// from the source.
727 pub text: String,
728 /// CoNLL-U tokens in id-sorted order.
729 pub tokens: Vec<Token>,
730 /// Negation cues derived from the dependency graph at construction
731 /// (see [`Sentence::new`]). Serialized with the sentence so the
732 /// detection crosses FFI as data (RFC-0008). Defaults to empty when
733 /// deserializing sentences serialized before this field existed.
734 #[serde(default)]
735 pub negations: Vec<Negation>,
736 /// Modal auxiliaries derived from the dependency graph at
737 /// construction (see [`Sentence::new`]): tokens from the closed
738 /// class `can`, `could`, `may`, `might`, `must`, `ought`,
739 /// `shall`, `should`, `will`, `would`, matched by lemma on the
740 /// `aux` relation or the `AUX` part of speech (the latter catches
741 /// modals the model promotes to root or `conj` under VP ellipsis
742 /// and coordination). Serialized with the sentence so the
743 /// detection crosses FFI as data (RFC-0008). Defaults to empty
744 /// when deserializing sentences serialized before this field
745 /// existed.
746 #[serde(default)]
747 pub modals: Vec<Modal>,
748 /// Bare-assertion discriminator, derived at construction: true
749 /// when the root clause is finite indicative (`Mood=Ind` on the
750 /// root token or on a `cop`, `aux`, or `aux:pass` child of it)
751 /// and no modal auxiliary governs it. Reports the surface form
752 /// only; what the assertion commits its speaker to is the
753 /// consumer's reading. Defaults to false when deserializing
754 /// sentences serialized before this field existed.
755 #[serde(default)]
756 pub bare_assertion: bool,
757 /// Reporting constructions derived from the dependency graph at
758 /// construction (see [`Sentence::new`]): each verb governing a
759 /// clausal complement (`ccomp`), with its subject when the parse
760 /// has one. The construction is reported for every verb that
761 /// fills it; which verbs count as evidential is the consumer's
762 /// lexicon ([`Sentence::reportings_in`]). Serialized with the
763 /// sentence so the detection crosses FFI as data (RFC-0008).
764 /// Defaults to empty when deserializing sentences serialized
765 /// before this field existed.
766 #[serde(default)]
767 pub reportings: Vec<Reporting>,
768 /// Adverbial modifiers attached to the root, derived at
769 /// construction (see [`Sentence::new`]): the arc sentence-scope
770 /// adverbs land on. Every root-attached `advmod` is reported;
771 /// which lemmas read as evidential is the consumer's lexicon
772 /// ([`Sentence::root_adverbials_in`]). Serialized with the
773 /// sentence so the detection crosses FFI as data (RFC-0008).
774 /// Defaults to empty when deserializing sentences serialized
775 /// before this field existed.
776 #[serde(default)]
777 pub root_adverbials: Vec<RootAdverbial>,
778 /// Candidate hypernymy pairs from the six Hearst (1992) patterns.
779 ///
780 /// Unlike the other derived fields, this one is not computed by
781 /// [`Sentence::new`]: the detector lives in `matra::hearst`, a
782 /// module outside the domain (EP-0007 M5 boundary), so the pipeline
783 /// fills the field at the annotate stage. A hand-built `Sentence`
784 /// carries an empty vector until the caller runs the detector.
785 /// Serialized with the sentence so the detection crosses FFI as
786 /// data (RFC-0008). Defaults to empty when deserializing
787 /// sentences serialized before this field existed.
788 #[serde(default)]
789 pub hearst_pairs: Vec<HearstPair>,
790 }
Five are computed in Sentence::new● observeddomain.rs:792-820.
hearst_pairs is filled at the annotate stage because its detector lives in src/hearst.rs, outside the domain, which imports only domain types ● observedlib.rs:235-243hearst.rs:28.
That is the amendment
RFC-0008
recorded for EP-0007's fifth milestone: the field still crosses as data;
only the point that fills it moved. A hand-built Sentence carries an empty
hearst_pairs until the caller runs the detector.
src/domain.rs lines 792 to 820, at 4fcfb4a
792 impl Sentence {
793 /// Construct a `Sentence` from a text string and a token vector.
794 ///
795 /// The caller is responsible for upholding the invariants documented
796 /// on [`Sentence`]; this constructor does not validate them. Derived
797 /// fields (`negations`, `modals`, `bare_assertion`, `reportings`,
798 /// `root_adverbials`) are computed here and reflect `tokens` as
799 /// passed in; mutating `tokens` afterwards does not recompute them,
800 /// and keeping them consistent is part of the same caller contract.
801 /// `hearst_pairs` starts empty and is filled at the annotate stage
802 /// by the pipeline, because its detector lives outside the domain
803 /// (see the field's documentation).
804 pub fn new(text: String, tokens: Vec<Token>) -> Self {
805 let negations = detect_negations(&tokens);
806 let modals = detect_modals(&tokens);
807 let bare_assertion = detect_bare_assertion(&tokens, &modals);
808 let reportings = detect_reportings(&tokens);
809 let root_adverbials = detect_root_adverbials(&tokens);
810 Self {
811 text,
812 tokens,
813 negations,
814 modals,
815 bare_assertion,
816 reportings,
817 root_adverbials,
818 hearst_pairs: Vec::new(),
819 }
820 }
Every derived field is #[serde(default)], so a document serialized before the field existed still deserializes ● observeddomain.rs:706-790.
Derived fields reflect tokens as passed to Sentence::new; mutating tokens afterwards does not recompute them, and keeping them consistent is the caller's side of the documented contract ● observeddomain.rs:792-820.
src/domain.rs lines 706 to 790, at 4fcfb4a
706 /// One parsed sentence: its text plus its ordered tokens.
707 ///
708 /// Invariants downstream code relies on:
709 /// - `tokens` are id-sorted ascending.
710 /// - Exactly one token has `head == 0` (the syntactic root).
711 /// - All `head` references point to another token in the same sentence
712 /// or to `0`.
713 /// - Following `head` references never loops back on itself.
714 ///
715 /// Nothing validates these: [`Sentence::new`] trusts its caller, and the
716 /// `NlpProvider` contract states them for providers. The tree walks stay
717 /// safe on input that breaks the last one: [`Sentence::tree_depth`]
718 /// returns `usize::MAX` on a cycle and [`Sentence::subtree`] carries a
719 /// visited set.
720 #[derive(Debug, Clone, Serialize, Deserialize)]
721 #[non_exhaustive]
722 pub struct Sentence {
723 /// Sentence text as the NLP provider reports it. The UDPipe adapter
724 /// rebuilds it from the token forms, joined by a space unless a token
725 /// carries `SpaceAfter=No` in `misc`, so its whitespace can differ
726 /// from the source.
727 pub text: String,
728 /// CoNLL-U tokens in id-sorted order.
729 pub tokens: Vec<Token>,
730 /// Negation cues derived from the dependency graph at construction
731 /// (see [`Sentence::new`]). Serialized with the sentence so the
732 /// detection crosses FFI as data (RFC-0008). Defaults to empty when
733 /// deserializing sentences serialized before this field existed.
734 #[serde(default)]
735 pub negations: Vec<Negation>,
736 /// Modal auxiliaries derived from the dependency graph at
737 /// construction (see [`Sentence::new`]): tokens from the closed
738 /// class `can`, `could`, `may`, `might`, `must`, `ought`,
739 /// `shall`, `should`, `will`, `would`, matched by lemma on the
740 /// `aux` relation or the `AUX` part of speech (the latter catches
741 /// modals the model promotes to root or `conj` under VP ellipsis
742 /// and coordination). Serialized with the sentence so the
743 /// detection crosses FFI as data (RFC-0008). Defaults to empty
744 /// when deserializing sentences serialized before this field
745 /// existed.
746 #[serde(default)]
747 pub modals: Vec<Modal>,
748 /// Bare-assertion discriminator, derived at construction: true
749 /// when the root clause is finite indicative (`Mood=Ind` on the
750 /// root token or on a `cop`, `aux`, or `aux:pass` child of it)
751 /// and no modal auxiliary governs it. Reports the surface form
752 /// only; what the assertion commits its speaker to is the
753 /// consumer's reading. Defaults to false when deserializing
754 /// sentences serialized before this field existed.
755 #[serde(default)]
756 pub bare_assertion: bool,
757 /// Reporting constructions derived from the dependency graph at
758 /// construction (see [`Sentence::new`]): each verb governing a
759 /// clausal complement (`ccomp`), with its subject when the parse
760 /// has one. The construction is reported for every verb that
761 /// fills it; which verbs count as evidential is the consumer's
762 /// lexicon ([`Sentence::reportings_in`]). Serialized with the
763 /// sentence so the detection crosses FFI as data (RFC-0008).
764 /// Defaults to empty when deserializing sentences serialized
765 /// before this field existed.
766 #[serde(default)]
767 pub reportings: Vec<Reporting>,
768 /// Adverbial modifiers attached to the root, derived at
769 /// construction (see [`Sentence::new`]): the arc sentence-scope
770 /// adverbs land on. Every root-attached `advmod` is reported;
771 /// which lemmas read as evidential is the consumer's lexicon
772 /// ([`Sentence::root_adverbials_in`]). Serialized with the
773 /// sentence so the detection crosses FFI as data (RFC-0008).
774 /// Defaults to empty when deserializing sentences serialized
775 /// before this field existed.
776 #[serde(default)]
777 pub root_adverbials: Vec<RootAdverbial>,
778 /// Candidate hypernymy pairs from the six Hearst (1992) patterns.
779 ///
780 /// Unlike the other derived fields, this one is not computed by
781 /// [`Sentence::new`]: the detector lives in `matra::hearst`, a
782 /// module outside the domain (EP-0007 M5 boundary), so the pipeline
783 /// fills the field at the annotate stage. A hand-built `Sentence`
784 /// carries an empty vector until the caller runs the detector.
785 /// Serialized with the sentence so the detection crosses FFI as
786 /// data (RFC-0008). Defaults to empty when deserializing
787 /// sentences serialized before this field existed.
788 #[serde(default)]
789 pub hearst_pairs: Vec<HearstPair>,
790 }
792 impl Sentence {
793 /// Construct a `Sentence` from a text string and a token vector.
794 ///
795 /// The caller is responsible for upholding the invariants documented
796 /// on [`Sentence`]; this constructor does not validate them. Derived
797 /// fields (`negations`, `modals`, `bare_assertion`, `reportings`,
798 /// `root_adverbials`) are computed here and reflect `tokens` as
799 /// passed in; mutating `tokens` afterwards does not recompute them,
800 /// and keeping them consistent is part of the same caller contract.
801 /// `hearst_pairs` starts empty and is filled at the annotate stage
802 /// by the pipeline, because its detector lives outside the domain
803 /// (see the field's documentation).
804 pub fn new(text: String, tokens: Vec<Token>) -> Self {
805 let negations = detect_negations(&tokens);
806 let modals = detect_modals(&tokens);
807 let bare_assertion = detect_bare_assertion(&tokens, &modals);
808 let reportings = detect_reportings(&tokens);
809 let root_adverbials = detect_root_adverbials(&tokens);
810 Self {
811 text,
812 tokens,
813 negations,
814 modals,
815 bare_assertion,
816 reportings,
817 root_adverbials,
818 hearst_pairs: Vec::new(),
819 }
820 }
Token::feat looks up one key in the feats string by a linear scan with no allocation, and returns the raw value, so Case=Nom,Acc comes back unsplit ● observeddomain.rs:287-308.
It is Rust-only by design
(RFC-0009): feats already
crosses as a string, so the lookup adds nothing to the wire.
reportings_in, root_adverbials_in, is_passive, tree_depth,
subtree, the Document iterators and the Corpus aggregates are views of
the same kind.
src/domain.rs lines 287 to 308, at 4fcfb4a
287 /// Look up one morphological feature by exact key in the CoNLL-U
288 /// `feats` string (column 6): `feat("Mood")` is `Some("Ind")` when
289 /// `feats` is `"Mood=Ind|Tense=Pres"`.
290 ///
291 /// A linear scan over the pipe-separated pairs, first exact-key
292 /// match, no allocation. The value is borrowed raw from `feats`,
293 /// so multi-valued features (`Case=Nom,Acc`) come back unsplit:
294 /// matra exposes what the provider emitted and does not normalise
295 /// it. Both the empty string and the CoNLL-U placeholder `"_"`
296 /// contain no `key=value` pair, so every lookup on them returns
297 /// `None` by construction.
298 ///
299 /// Rust-only by design: `feats` already crosses FFI as a string,
300 /// so this view adds no information to the wire (RFC-0009).
301 pub fn feat(&self, key: &str) -> Option<&str> {
302 self.feats
303 .split('|')
304 .find_map(|pair| match pair.split_once('=') {
305 Some((k, v)) if k == key => Some(v),
306 _ => None,
307 })
308 }
Document::passive_ratio() the method and Document::passive_ratio the field both exist ● observeddomain.rs:1178-1191domain.rs:1095-1120.
The method computes the ratio on any document; the metric suite stores it in the field so it crosses ● observedmetrics/document.rs:12-22.
src/domain.rs lines 1178 to 1191, at 4fcfb4a
1178 /// Fraction of sentences containing a passive-voice construction.
1179 /// Returns `0.0` when there are no sentences.
1180 ///
1181 /// This is the computation behind the [`Document::passive_ratio`]
1182 /// field, which the metric suite fills so the value crosses FFI
1183 /// (RFC-0008). The method stays for Rust callers who want the
1184 /// ratio on an unmeasured document.
1185 pub fn passive_ratio(&self) -> f64 {
1186 let total = self.total_sentences();
1187 if total == 0 {
1188 return 0.0;
1189 }
1190 self.sentences().filter(|s| s.is_passive()).count() as f64 / total as f64
1191 }
1095 /// The full analysis output. `Engine::annotate` builds the section tree
1096 /// and attaches each paragraph's parsed sentences; `Engine::compose` fills
1097 /// the metric slots.
1098 ///
1099 /// Paragraphs live in sections (single source of truth). Derived methods
1100 /// compute document-level metrics on the fly from the section tree.
1101 #[derive(Debug, Clone, Serialize, Deserialize)]
1102 #[non_exhaustive]
1103 pub struct Document {
1104 /// Section tree (the single source of truth for paragraph ownership).
1105 pub sections: Vec<Section>,
1106 /// Document-level vocabulary type-token ratio, if `Engine::compose`
1107 /// ran and the document holds a non-punctuation token.
1108 pub vocabulary_ttr: Option<f64>,
1109 /// Document-level nominalization ratio, if `Engine::compose` ran and
1110 /// the document holds a non-punctuation token.
1111 pub nominalization_ratio: Option<f64>,
1112 /// Fraction of sentences containing a passive-voice construction,
1113 /// if `Engine::compose` ran and the document holds a sentence.
1114 /// Materialized so the aggregate crosses FFI as data instead of
1115 /// being re-derived per language binding (RFC-0008). Defaults
1116 /// to `None` when deserializing documents serialized before this
1117 /// field existed.
1118 #[serde(default)]
1119 pub passive_ratio: Option<f64>,
1120 }
src/metrics/document.rs lines 12 to 22, at 4fcfb4a
12 /// Populate `Document::vocabulary_ttr` (type-token ratio over lemmas,
13 /// excluding punctuation), `Document::nominalization_ratio`
14 /// (share of NOUN tokens ending in a nominalizing suffix), and
15 /// `Document::passive_ratio` (share of sentences with a passive-voice
16 /// construction, materialized so the aggregate crosses FFI per
17 /// RFC-0008).
18 pub fn compute(analysis: &mut Document) {
19 if analysis.total_sentences() > 0 {
20 let ratio = analysis.passive_ratio();
21 analysis.passive_ratio = Some(ratio);
22 }
Error has seven variants, each with a stable kind string from Error::kind, an exhaustive match with no wildcard so a new variant cannot inherit another's key ● observeddomain.rs:151-191domain.rs:193-217.
The vocabulary is pinned for every binding in spec/tests/corpus/items.json and mirrored in Python as ERROR_KINDS● observeditems.json:4-12types.py:15-31.
InputTooLarge carries a what label naming which gate fired.
src/domain.rs lines 151 to 191, at 4fcfb4a
151 /// All errors matra can produce. Matchable, not opaque.
152 #[derive(thiserror::Error, Debug)]
153 #[non_exhaustive]
154 pub enum Error {
155 /// Model file does not exist at the given path.
156 #[error("model not found: {}", .0.display())]
157 ModelNotFound(PathBuf),
158 /// Model file exists but could not be loaded (corrupt, wrong format).
159 #[error("invalid model: {0}")]
160 ModelInvalid(String),
161 /// NLP parsing failed on the input text.
162 #[error("parse failed: {0}")]
163 ParseFailed(String),
164 /// Input exceeded a bounded limit (e.g. too many sentences for
165 /// an O(n^2) algorithm like TextRank).
166 #[error("{what} input too large: {actual} > limit {limit}")]
167 InputTooLarge {
168 /// The size cap that was exceeded.
169 limit: usize,
170 /// The actual size that exceeded the cap.
171 actual: usize,
172 /// Discriminator naming which gate fired, so consumers can route
173 /// differently per gate. The labels in use: `"input"`,
174 /// `"file_source"`, `"tfidf"`, `"textrank"`, `"rake"`, `"yake"`,
175 /// `"semantic_clusters"`, `"udpipe_download"`,
176 /// `"embedding_download"` and `"config_file"`.
177 what: &'static str,
178 },
179 /// The document format has no registered decomposer in this build.
180 /// Seen when analyzing a Pdf/Docx file without the relevant adapter.
181 #[error("unsupported format: {0:?}")]
182 UnsupportedFormat(Format),
183 /// A caller violated a documented API contract (mismatched argument
184 /// lengths, disagreeing dimensions, a non-finite parameter). Names
185 /// the violation; fix the call site, not the input data.
186 #[error("invalid input: {0}")]
187 InvalidInput(String),
188 /// File I/O error.
189 #[error("io error: {0}")]
190 Io(#[from] std::io::Error),
191 }
193 impl Error {
194 /// The stable kind string for this variant: the key a consumer
195 /// branches on instead of matching against a message.
196 ///
197 /// The vocabulary is part of the public contract. A binding that
198 /// cannot carry a Rust enum across (`DocumentError` has no serde
199 /// form, because it wraps `std::io::Error`) materializes this string
200 /// instead, so every language binding names a failure the same way.
201 /// `spec/tests/corpus/items.json` pins the list.
202 ///
203 /// Exhaustive with no wildcard: a new variant fails to compile until
204 /// someone names it here, which is what stops a new failure mode
205 /// from quietly inheriting another one's key.
206 pub fn kind(&self) -> &'static str {
207 match self {
208 Error::ModelNotFound(_) => "model_not_found",
209 Error::ModelInvalid(_) => "model_invalid",
210 Error::ParseFailed(_) => "parse_failed",
211 Error::InputTooLarge { .. } => "input_too_large",
212 Error::UnsupportedFormat(_) => "unsupported_format",
213 Error::InvalidInput(_) => "invalid_input",
214 Error::Io(_) => "io",
215 }
216 }
217 }
15 ERROR_KINDS: Final[tuple[str, ...]] = (
16 "model_not_found",
17 "model_invalid",
18 "parse_failed",
19 "input_too_large",
20 "unsupported_format",
21 "invalid_input",
22 "io",
23 )
24 """Every value ``ErrorInfo["kind"]`` can take, in the order the Rust
25 enum declares them.
2627 Mirrors ``matra::domain::Error::kind``, the one place the vocabulary is
28 written on the Rust side. ``spec/tests/corpus/items.json`` pins the list
29 and both crusts assert against it, so a kind added on one side and not
30 the other fails the corpus conformance lane.
31 """
In Python each variant maps to one exception class through an exhaustive match: ModelNotFound to FileNotFoundError; InputTooLarge, UnsupportedFormat and InvalidInput to ValueError; Io to OSError; ModelInvalid and ParseFailed to RuntimeError● observedlib.rs:417-450.
A new variant fails to compile until it is routed.
src/lib.rs lines 417 to 450, at 4fcfb4a
417 /// Routes domain::Error variants to the appropriate Python exception
418 /// class, preserving variant identity across the FFI boundary.
419 ///
420 /// The mapping follows pyo3 conventions: file-not-found maps to
421 /// PyFileNotFoundError so Python `try ... except FileNotFoundError`
422 /// works as expected; oversized or unsupported inputs are PyValueError;
423 /// I/O errors are PyOSError; everything else is PyRuntimeError.
424 struct MatraError(domain::Error);
425426 impl From<domain::Error> for MatraError {
427 fn from(e: domain::Error) -> Self {
428 MatraError(e)
429 }
430 }
431432 impl From<MatraError> for PyErr {
433 fn from(e: MatraError) -> PyErr {
434 use domain::Error::*;
435 use pyo3::exceptions::*;
436 let msg = e.0.to_string();
437 // domain::Error is #[non_exhaustive] from outside the crate but
438 // the compiler sees the full variant set in here, so this match
439 // is exhaustive without a wildcard. A new variant will become a
440 // compile error — exactly what we want for routing fidelity.
441 match e.0 {
442 ModelNotFound(_) => PyFileNotFoundError::new_err(msg),
443 InputTooLarge { .. } | UnsupportedFormat(_) | InvalidInput(_) => {
444 PyValueError::new_err(msg)
445 }
446 Io(_) => PyOSError::new_err(msg),
447 ModelInvalid(_) | ParseFailed(_) => PyRuntimeError::new_err(msg),
448 }
449 }
450 }
DocumentError and CorpusResult are not Serialize, because Error wraps std::io::Error● observeddomain.rs:1448-1464.
Where a per-document failure crosses, it is materialized as {kind, message} by hand ● observedlib.rs:1036-1055.
src/domain.rs lines 1448 to 1464, at 4fcfb4a
1448 /// The outcome of analyzing a stream of documents: every success in a
1449 /// [`Corpus`], every per-document failure alongside it.
1450 ///
1451 /// The partition invariant: entries plus errors equals documents
1452 /// consumed. Nothing is silently dropped.
1453 ///
1454 /// Not `Serialize`: [`DocumentError`] wraps [`Error`], which wraps
1455 /// `std::io::Error`. Crossing a language boundary needs a projection
1456 /// with stable kind strings, not a serialization of `io::Error`.
1457 #[derive(Debug)]
1458 #[non_exhaustive]
1459 pub struct CorpusResult {
1460 /// Every successfully analyzed document.
1461 pub corpus: Corpus,
1462 /// Every per-document failure, in consumption order.
1463 pub errors: Vec<DocumentError>,
1464 }
1036 /// Project a [`domain::DocumentError`] into a dict.
1037 ///
1038 /// `DocumentError` is deliberately not `Serialize`: it wraps
1039 /// `domain::Error`, which wraps `std::io::Error`, and no stable wire
1040 /// shape exists for that. The two fields are materialized instead,
1041 /// the error as the `kind` [`domain::Error::kind`] names plus the
1042 /// `Display` text a human reads.
1043 fn document_error_dict<'py>(
1044 py: Python<'py>,
1045 err: &domain::DocumentError,
1046 ) -> PyResult<Bound<'py, PyAny>> {
1047 let error = PyDict::new(py);
1048 error.set_item("kind", err.error.kind())?;
1049 error.set_item("message", err.error.to_string())?;
10501051 let item = PyDict::new(py);
1052 item.set_item("path", fs_path_item(py, err.path.as_deref())?)?;
1053 item.set_item("error", error)?;
1054 Ok(item.into_any())
1055 }
Every public struct with public fields and every public enum in the library is #[non_exhaustive], observed by listing them at the pinned commit, and the one exception is Embedding, a tuple struct whose constructor external Embedder implementors must call ● observeddomain.rs:93;
RFC-0010 records why
(EPR-0005).
Token::builder exists because the attribute forbids struct literals outside the crate ● observeddomain.rs:261-285.
RFC-0006 reserved
names for the rule-evaluation tier before any of it exists, so that tier
cannot arrive under other names. None of these is a type in src/ today.
Frame and the verb frame stay reserved for frame semantics, as
RFC-0002 first set and
RFC-0006 kept. The five Finding rules RFC-0006 states (provenance
mandatory, confidence shape mandatory, a discriminant that names a
structural fact, only FFI-safe types, #[non_exhaustive]) bind whichever
shape lands. If metric slots are ever grouped, an outer
Option<ParagraphMetrics> is ruled out, because each slot gates
independently. ROADMAP.md describes the rule-evaluation entry against this
vocabulary (roadmap).
The Analysis alias.RFC-0006
renamed Analysis to Document with a deprecated alias due for removal
before 0.1.0. The alias is gone: no type named Analysis exists in
src/.
in_blockquote as a deprecation signal. RFC-0006 says the field's
rustdoc signals its replacement by ParagraphKind.
The rustdoc now says the replacement is planned and that nothing is deprecated, because the field's job is still binary ● observeddomain.rs:1019-1029.
The measure and extract split as a type distinction.RFC-0002 called extraction
a peer of measurement; RFC-0007
superseded it. Extraction output (ScoredSentence, Keyphrase) remains
plain domain types; the stage vocabulary is
EPR-0001's.
src/domain.rs lines 1019 to 1029, at 4fcfb4a
1019 /// Whether the paragraph is inside a blockquote (never parsed, never
1020 /// measured).
1021 ///
1022 /// **Planned change:** this boolean may be replaced with
1023 /// `kind: ParagraphKind` once the variant inventory (Body / Quote /
1024 /// Code / List / Caption) is justified by real consumer semantics.
1025 /// Nothing is deprecated today: the boolean stays while its job
1026 /// (parse and measure the paragraph or not) is binary. See
1027 /// [RFC-0006](https://github.com/mox-labs/matra/blob/main/blueprints/legacy/rfcs/0006-abstract-tier-vocabulary-lock.md)
1028 /// for the abstract-tier vocabulary lock.
1029 pub in_blockquote: bool,
Every crossing field is a schema commitment in Rust, Python and the
JSON. A rename is a breaking change in all three
(RFC-0008).
Everyone pays for every primitive. Each sentence stores its derived
facts whether or not the caller reads them.
That storage is small beside the tokens already on the wire○ assumed:
RFC-0008 judged so, and no benchmark at the pinned commit measures it.
Derived fields go stale under mutation of tokens, by the documented
contract rather than by a check.
Two spellings of one value.passive_ratio is both a method and a
field on Document, and
they disagree on a document with no sentences: the method returns 0.0 and the field stays None even after compose● observeddomain.rs:1178-1191metrics/document.rs:12-22.
src/domain.rs lines 1178 to 1191, at 4fcfb4a
1178 /// Fraction of sentences containing a passive-voice construction.
1179 /// Returns `0.0` when there are no sentences.
1180 ///
1181 /// This is the computation behind the [`Document::passive_ratio`]
1182 /// field, which the metric suite fills so the value crosses FFI
1183 /// (RFC-0008). The method stays for Rust callers who want the
1184 /// ratio on an unmeasured document.
1185 pub fn passive_ratio(&self) -> f64 {
1186 let total = self.total_sentences();
1187 if total == 0 {
1188 return 0.0;
1189 }
1190 self.sentences().filter(|s| s.is_passive()).count() as f64 / total as f64
1191 }
src/metrics/document.rs lines 12 to 22, at 4fcfb4a
12 /// Populate `Document::vocabulary_ttr` (type-token ratio over lemmas,
13 /// excluding punctuation), `Document::nominalization_ratio`
14 /// (share of NOUN tokens ending in a nominalizing suffix), and
15 /// `Document::passive_ratio` (share of sentences with a passive-voice
16 /// construction, materialized so the aggregate crosses FFI per
17 /// RFC-0008).
18 pub fn compute(analysis: &mut Document) {
19 if analysis.total_sentences() > 0 {
20 let ratio = analysis.passive_ratio();
21 analysis.passive_ratio = Some(ratio);
22 }
Methods only: no schema lock-in and no storage, but invisible to every
binding, so each binding re-implements each primitive with nothing
checking they agree. Rejected in
RFC-0008,
with the Python passive fold as the live case.
Methods with a hand-written Serialize: the same wire at a worse
price, a wire that no longer equals the struct and graph walks inside
serialization, in the most protected file. Rejected in RFC-0008.
Exhaustive typed enums for feats: the Universal Dependencies feature
inventory is large, language-dependent and grows by release. Rejected in
RFC-0009, as was a
per-token HashMap, for an allocation on the hot path with no benchmark
asking for it.
Resolving modal readings or shipping evidential word lists: rejected
in EP-0007, because a
reading is interpretation and an incomplete open-class list that looks
authoritative is worse than none.
Routing primitives through Finding: would make the primitives wait
on a shape RFC-0006 defers until rule evaluation exists. Rejected in
RFC-0008.
CoNLL-U, the Universal Dependencies format, is the token shape: one field
per column.
Hearst (1992), "Automatic acquisition of hyponyms from large text
corpora", is the source of the six patterns, implemented here as
dependency-arc patterns rather than surface regexes.
For the owner to settle in this proposal's review. Each is a decision of its
own.
CorpusEntry.analysis still carries the rejected name.The field is analysis, holding a Document● observeddomain.rs:1338-1346grounds for decision 1.
RFC-0006's deferred list, extended by RFC-0010, records that the field
should be renamed, and that the rename became a breaking change once 0.1.0
shipped. When, and folded into what, is open.
src/domain.rs lines 1338 to 1346, at 4fcfb4a
1338 /// A single entry in a corpus: one analyzed document.
1339 #[derive(Debug, Clone, Serialize, Deserialize)]
1340 #[non_exhaustive]
1341 pub struct CorpusEntry {
1342 /// Source path, if the document came from disk.
1343 pub path: Option<PathBuf>,
1344 /// The document's analysis output.
1345 pub analysis: Document,
1346 }
A With the next release that breaks the schema recommended
The rename is folded into a release whose callers migrate anyway, so they migrate once.
B In a release of its own
The rename ships as soon as possible, on its own, with the old key still accepted when a document is read back.
C Never
The field keeps analysis, and its documentation says it holds a Document.
Claude recommends A, With the next release that breaks the schema. ◐ inferred, a judgment
Every crossing field is a schema commitment in Rust, Python and the JSON, so the rename costs every caller a migration whenever it lands. Landing it where callers migrate anyway pays that cost once.
The strongest case against it
Until then every new caller learns a name the records already rejected, and each release that ships it adds callers who will have to move.
The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.
Error::Io routes to OSError whatever its kind.Every Io variant becomes OSError● observedlib.rs:417-450grounds for decision 2.
A missing directory arrives in Python as OSError, not
FileNotFoundError, although the wrapped io::ErrorKind is NotFound
(RFC-0011, unresolved questions).
Routing on the wrapped kind would change a shipped mapping.
src/lib.rs lines 417 to 450, at 4fcfb4a
417 /// Routes domain::Error variants to the appropriate Python exception
418 /// class, preserving variant identity across the FFI boundary.
419 ///
420 /// The mapping follows pyo3 conventions: file-not-found maps to
421 /// PyFileNotFoundError so Python `try ... except FileNotFoundError`
422 /// works as expected; oversized or unsupported inputs are PyValueError;
423 /// I/O errors are PyOSError; everything else is PyRuntimeError.
424 struct MatraError(domain::Error);
425426 impl From<domain::Error> for MatraError {
427 fn from(e: domain::Error) -> Self {
428 MatraError(e)
429 }
430 }
431432 impl From<MatraError> for PyErr {
433 fn from(e: MatraError) -> PyErr {
434 use domain::Error::*;
435 use pyo3::exceptions::*;
436 let msg = e.0.to_string();
437 // domain::Error is #[non_exhaustive] from outside the crate but
438 // the compiler sees the full variant set in here, so this match
439 // is exhaustive without a wildcard. A new variant will become a
440 // compile error — exactly what we want for routing fidelity.
441 match e.0 {
442 ModelNotFound(_) => PyFileNotFoundError::new_err(msg),
443 InputTooLarge { .. } | UnsupportedFormat(_) | InvalidInput(_) => {
444 PyValueError::new_err(msg)
445 }
446 Io(_) => PyOSError::new_err(msg),
447 ModelInvalid(_) | ParseFailed(_) => PyRuntimeError::new_err(msg),
448 }
449 }
450 }
NotFound becomes FileNotFoundError, PermissionDenied becomes PermissionError, and every other kind stays OSError.
B Keep one class
Every Error::Io stays OSError, and the mapping's documentation says so on purpose.
Claude recommends A, Route on the wrapped kind. ◐ inferred, a judgment
Python's FileNotFoundError and PermissionError are subclasses of OSError, so an except OSError clause written today keeps catching them, and a caller who expected FileNotFoundError for a missing path gets it.
The strongest case against it
It changes a shipped mapping: code that tests an exception's exact class, rather than catching it, sees a different class. And the kind string stays io for all of them, so the exception class would name a failure more finely than the kind beside it.
The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.
What the owner should confirm or strike, collected from the claims above
that stand on no grounds yet:
○ 1 claim stands on no grounds yet. Confirm or strike each with a comment on it.
That storage is small beside the tokens already on the wirewhere: Drawbacksbears on no decision
The rule-evaluation tier, using the reserved names, reading these fields
rather than walking the raw parse. Its Finding shape (trait or enum) is
decided when the first concrete consumer appears.
ParagraphKind replacing in_blockquote once a paragraph needs more than
parse-or-skip.