Enhancement proposalEPR-0001The pipeline and its ports○ proposed
● 0 of 2 decisions settled○ 1 assumption to confirmpinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: pipeline_and_portsdecides: the owner, who alone sets the status to accepted
EPR-0001● 0 of 2 settled○ 1 to confirm
Enhancement proposalEPR-0001The pipeline and its ports○ proposed
pinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: pipeline_and_portsdecides: the owner, who alone sets the status to accepted
matra is one crate whose surface is one pipeline. A caller turns input into
an Ingest stream and hands it to an Engine, which runs each document
through two stages: annotate (decompose the text into sections and
paragraphs, then parse each paragraph) and compose (run the metric suite).
annotate is the only route from text to the parser, so the input size cap
is a property of the pipeline rather than of each entry point. Four ports
(Source, Decomposer, NlpProvider, Embedder) declare what the pipeline
needs from the outside; adapters implement them; src/lib.rs is the
composition root that wires them; eight boundary rules keep the dependency
arrows pointing inward.
This is a baseline proposal. It describes the code at the pinned commit and
carries forward the reasons that still hold from RFC-0002, RFC-0003,
RFC-0004, RFC-0007, EP-0008 and EP-0014. A later proposal that changes any of
it says so. Unmarked statements about the code are observed at the pinned
commit through the link beside them; a claim that is inferred or assumed
says so.
Why this is the foundation, rather than any other shape, comes down to four
reasons, each paid for by a defect or a near miss.
One implementation of each invariant. Before 0.1.0 the public Rust
surface was six free functions, each a partial application of the same
chain, and each restated the invariants itself. Two live defects came from
that shape: the input size cap held on four Python methods and not on the
other four, and analyze_from returned documents whose paragraph metrics
were always None because the sentence set travelled twice with nothing
enforcing agreement (EP-0008,
"The two defects"). RFC-0007 replaced
the six with one pipeline so each invariant has one home.
Today the size check sits in check_input_size, called first thing in Engine::annotate, and the metric suite takes the document alone (Metric), so the sentence set exists once ● observedlib.rs:23-38lib.rs:212-253metrics/mod.rs:22-57.
src/lib.rs lines 23 to 38, at 4fcfb4a
23 /// Reject text whose byte length exceeds [`MAX_INPUT_BYTES`].
24 ///
25 /// Returns `Error::InputTooLarge { what: "input", .. }` so consumers can
26 /// distinguish the bound check from per-extractor caps (which use distinct
27 /// `what` labels). [`Engine::annotate`] runs this gate, and annotate is
28 /// the only route from text to the parser.
29 fn check_input_size(text: &str) -> domain::Result<()> {
30 if text.len() > MAX_INPUT_BYTES {
31 return Err(domain::Error::InputTooLarge {
32 limit: MAX_INPUT_BYTES,
33 actual: text.len(),
34 what: "input",
35 });
36 }
37 Ok(())
38 }
212 /// Decompose and parse one document into an unmeasured [`Document`]:
213 /// structure from the format's decomposer, sentences attached
214 /// per-paragraph, every metric slot still `None`.
215 ///
216 /// This is the only route from text to the parser, which is what
217 /// makes the input size cap a property of the pipeline rather than
218 /// of each entry point: no text over [`MAX_INPUT_BYTES`] reaches
219 /// [`NlpProvider::parse`].
220 ///
221 /// Structure materializes here (RFC-0008): derived facts whose
222 /// detectors live outside the domain are filled onto each parsed
223 /// sentence at this choke point. Today that is
224 /// [`domain::Sentence::hearst_pairs`], computed by
225 /// [`hearst::hypernymy_pairs`]; sentence-level facts whose
226 /// detectors live in the domain are computed by `Sentence::new`
227 /// inside the provider.
228 pub fn annotate(&self, raw: &domain::RawDocument) -> domain::Result<Document> {
229 check_input_size(&raw.text)?;
230 let decomposer = self
231 .decomposers
232 .get(&raw.format)
233 .ok_or_else(|| domain::Error::UnsupportedFormat(raw.format.clone()))?;
234 let mut doc = Document::new(decomposer.decompose(&raw.text));
235 for para in doc.paragraphs_mut() {
236 if para.in_blockquote {
237 continue;
238 }
239 para.sentences = self.nlp.parse(¶.text)?;
240 for sentence in &mut para.sentences {
241 sentence.hearst_pairs = hearst::hypernymy_pairs(&sentence.tokens);
242 }
243 }
244 Ok(doc)
245 }
246247 /// Run the metric suite over an annotated document. Total: metrics
248 /// read what is attached and skip what is not; there is no failure
249 /// path.
250 pub fn compose(&self, doc: &mut Document) {
251 let suite = metrics::default_suite();
252 metrics::run_suite(doc, &suite);
253 }
22 /// A metric reads NLP output and enriches the analysis.
23 /// `Box<dyn Fn>` allows closures with captured config/state.
24 ///
25 /// One parameter, one sentence set: every metric reads the sentences
26 /// attached to the document's paragraphs ([`Document::sentences`]).
27 /// The old two-parameter shape carried the sentence set twice (a flat
28 /// slice plus the paragraph attachment) with nothing enforcing
29 /// agreement, which is how `analyze_from` shipped half-populated
30 /// documents.
31 pub type Metric = Box<dyn Fn(&mut Document)>;
3233 /// Default metric suite.
34 ///
35 /// The suite is a list, not a chain. No metric reads a slot another metric
36 /// writes, so the order here is not load-bearing and a caller may drop,
37 /// reorder, or add entries freely.
38 ///
39 /// Per-paragraph metrics (readability, lexical, compression) read from
40 /// `paragraph.sentences`, which the composition root populates by parsing
41 /// each paragraph individually. Document-level metrics aggregate over
42 /// the same attachment via [`Document::sentences`].
43 pub fn default_suite() -> Vec<Metric> {
44 vec![
45 Box::new(readability::compute),
46 Box::new(lexical::compute),
47 Box::new(document::compute),
48 Box::new(compression::compute),
49 ]
50 }
5152 /// Run every metric in the suite, in order.
53 pub fn run_suite(analysis: &mut Document, suite: &[Metric]) {
54 for metric in suite {
55 metric(analysis);
56 }
57 }
Per-paragraph parsing. An earlier pipeline joined paragraphs, parsed
once, and wired sentences back to paragraphs by prefix match. Two paragraphs
sharing their first 30 characters had their sentences silently reassigned
(the failure the project calls FM1).
annotate parses each paragraph on its own, and a regression test, parse_per_paragraph_scopes_sentences_to_originating_paragraph, pins the case ● observedlib.rs:235-243lib.rs:1475-1496.
src/lib.rs lines 235 to 243, at 4fcfb4a
235 for para in doc.paragraphs_mut() {
236 if para.in_blockquote {
237 continue;
238 }
239 para.sentences = self.nlp.parse(¶.text)?;
240 for sentence in &mut para.sentences {
241 sentence.hearst_pairs = hearst::hypernymy_pairs(&sentence.tokens);
242 }
243 }
1475 #[test]
1476 fn parse_per_paragraph_scopes_sentences_to_originating_paragraph() {
1477 // FM1 regression: two paragraphs with the same first 30 chars.
1478 // Pre-fix, attach_sentences would prefix-match and could assign
1479 // either paragraph's first sentence to the wrong paragraph
1480 // (and silently drop the other). Per-paragraph parse makes the
1481 // assignment unambiguous by construction.
1482 let text = "The system processes input now. Tail one.\n\n\
1483 The system processes input now. Tail two.";
1484 let analysis = law_engine().analyze_one(raw_plain(text)).unwrap().analysis;
14851486 let paras: Vec<_> = analysis.paragraphs().collect();
1487 assert_eq!(paras.len(), 2);
1488 assert_eq!(paras[0].sentences.len(), 2);
1489 assert_eq!(paras[1].sentences.len(), 2);
14901491 // Each paragraph keeps its own tail; no leak.
1492 assert!(paras[0].sentences.iter().any(|s| s.text.contains("one")));
1493 assert!(paras[1].sentences.iter().any(|s| s.text.contains("two")));
1494 assert!(!paras[0].sentences.iter().any(|s| s.text.contains("two")));
1495 assert!(!paras[1].sentences.iter().any(|s| s.text.contains("one")));
1496 }
Ports factored by dependency and failure mode. A port exists where the
outside world differs in what it depends on and how it fails: reading bytes
(Source), cutting text into structure (Decomposer, total), parsing
(NlpProvider, native code and a panic boundary), embedding (Embedder,
model output). RFC-0007 kept the
trait names RFC-0002 chose
for this reason: they are ports, not stages.
One crate.RFC-0004 kept
matra a single crate because the test for splitting out a port crate is
whether external implementors need to pin the contract apart from the main
crate's version churn, and no such implementors exist.
There is one [package] in Cargo.toml and no workspace ● observedCargo.toml:1-14.
You hold an Engine and feed it documents. A string is a stream of one, a
file is a stream of one, a directory is a stream of many; all three are an
Ingest, one concrete type (Ingest).
use matra::domain::{CorpusResult, Format};use matra::{Engine, Ingest};let engine = Engine::with_defaults()?;// One document in memory: a stream of one.let one: CorpusResult = engine .analyze(Ingest::text("The report was filed.", Format::PlainText)) .collect();// A directory: each file is read when the stream is pulled, and a file// that fails becomes an error item rather than ending the walk.let many: CorpusResult = engine.analyze(Ingest::path("docs/")?).collect();for failure in &many.errors { /* failure.path, failure.error */ }
analyze is lazy: nothing is read or parsed until the iterator is pulled, and each pull runs one document to completion ● observedlib.rs:163-192.
So Ingest::path(dir)? returning Ok means the directory listed, not that
every file read.
Per-file failures arrive as DocumentError items carrying the path ● observedlib.rs:89-113,
and collecting into CorpusResult partitions successes from failures.
src/lib.rs lines 163 to 192, at 4fcfb4a
163 /// Analyze a stream of documents.
164 ///
165 /// Lazy: nothing is parsed until the returned iterator is pulled,
166 /// and each pull runs one document through [`Engine::analyze_one`]
167 /// to completion. An `Err` input item passes through unchanged
168 /// without touching the pipeline. Collect into
169 /// [`domain::CorpusResult`] to partition successes from failures:
170 ///
171 /// ```no_run
172 /// # fn example(engine: &matra::Engine) -> matra::domain::Result<()> {
173 /// let result: matra::domain::CorpusResult =
174 /// engine.analyze(matra::Ingest::path("docs/")?).collect();
175 /// # Ok(())
176 /// # }
177 /// ```
178 ///
179 /// The returned iterator borrows the engine and is not `Send`:
180 /// [`NlpProvider`] is `Send` without `Sync`, so the stream cannot be
181 /// shared across threads.
182 pub fn analyze<I>(
183 &self,
184 input: I,
185 ) -> impl Iterator<Item = std::result::Result<domain::CorpusEntry, domain::DocumentError>>
186 where
187 I: IntoIterator<Item = Ingested>,
188 {
189 input
190 .into_iter()
191 .map(|item| item.and_then(|raw| self.analyze_one(raw)))
192 }
89 /// A stream from a path: one document for a file, zero or more for a
90 /// directory.
91 ///
92 /// The `Err` here is top-level only: the path does not exist, or the
93 /// directory cannot be listed. Everything per-file (unreadable,
94 /// oversized, a symlink) is deferred to iteration and yielded as an
95 /// `Err` item, so one bad file cannot abort a directory walk.
96 pub fn path(input: impl AsRef<Path>) -> domain::Result<Self> {
97 let input = input.as_ref();
98 let metadata = std::fs::symlink_metadata(input)?;
99 let pending = if metadata.file_type().is_dir() {
100 source::directory::DirectorySource
101 .candidate_paths(input)?
102 .into_iter()
103 .map(Pending::File)
104 .collect()
105 } else {
106 // A file, a symlink, or something stranger. FileSource's own
107 // guards (symlink refusal, size cap) run at pull time.
108 vec![Pending::File(input.to_path_buf())]
109 };
110 Ok(Self {
111 items: pending.into_iter(),
112 })
113 }
The two stages are public.
annotate returns a Document with structure and sentences and every metric slot None; compose fills the slots and cannot fail ● observedlib.rs:212-253.
analyze_one is exactly annotate then compose● observedlib.rs:194-210.
src/lib.rs lines 212 to 253, at 4fcfb4a
212 /// Decompose and parse one document into an unmeasured [`Document`]:
213 /// structure from the format's decomposer, sentences attached
214 /// per-paragraph, every metric slot still `None`.
215 ///
216 /// This is the only route from text to the parser, which is what
217 /// makes the input size cap a property of the pipeline rather than
218 /// of each entry point: no text over [`MAX_INPUT_BYTES`] reaches
219 /// [`NlpProvider::parse`].
220 ///
221 /// Structure materializes here (RFC-0008): derived facts whose
222 /// detectors live outside the domain are filled onto each parsed
223 /// sentence at this choke point. Today that is
224 /// [`domain::Sentence::hearst_pairs`], computed by
225 /// [`hearst::hypernymy_pairs`]; sentence-level facts whose
226 /// detectors live in the domain are computed by `Sentence::new`
227 /// inside the provider.
228 pub fn annotate(&self, raw: &domain::RawDocument) -> domain::Result<Document> {
229 check_input_size(&raw.text)?;
230 let decomposer = self
231 .decomposers
232 .get(&raw.format)
233 .ok_or_else(|| domain::Error::UnsupportedFormat(raw.format.clone()))?;
234 let mut doc = Document::new(decomposer.decompose(&raw.text));
235 for para in doc.paragraphs_mut() {
236 if para.in_blockquote {
237 continue;
238 }
239 para.sentences = self.nlp.parse(¶.text)?;
240 for sentence in &mut para.sentences {
241 sentence.hearst_pairs = hearst::hypernymy_pairs(&sentence.tokens);
242 }
243 }
244 Ok(doc)
245 }
246247 /// Run the metric suite over an annotated document. Total: metrics
248 /// read what is attached and skip what is not; there is no failure
249 /// path.
250 pub fn compose(&self, doc: &mut Document) {
251 let suite = metrics::default_suite();
252 metrics::run_suite(doc, &suite);
253 }
Text over 8 MiB is refused with Error::InputTooLarge { what: "input" } before any parse ● observeddomain.rs:12-28lib.rs:23-38.
Blockquote paragraphs are kept in the structure but never parsed or
measured.
src/domain.rs lines 12 to 28, at 4fcfb4a
12 /// The upper bound on text entering the pipeline, in bytes. A constant,
13 /// not a setting: no configuration, argument or environment variable
14 /// changes it.
15 ///
16 /// 8 MiB accommodates book-length English (a typical novel is ~1.5 MiB / 200k
17 /// words) with headroom for multilingual prose and structured documents. Beyond
18 /// this bound, the underlying NLP provider's intermediate memory grows past
19 /// safe limits on a typical workstation (UDPipe's per-token allocations cross
20 /// ~1 GiB resident at this input size).
21 ///
22 /// `Engine::annotate`, the only route from text to the parser, enforces it
23 /// and returns [`Error::InputTooLarge`] with `what = "input"` when it is
24 /// exceeded, so every `Engine` method inherits it. `FileSource` applies the
25 /// same limit to a file's size before reading it (`what = "file_source"`).
26 /// A provider's own `NlpProvider::parse` has no limit: calling it directly
27 /// bypasses this bound.
28 pub const MAX_INPUT_BYTES: usize = 8 * 1024 * 1024;
The Python Matra class wraps one Engine and routes every method through it ● observedlib.rs:457-496,
so the cap and the per-paragraph parse hold there too.
src/lib.rs lines 457 to 496, at 4fcfb4a
457 /// Holds the assembled pipeline. Create once, reuse across calls.
458 ///
459 /// Marked unsendable because NLP models may contain C state that is
460 /// not thread-safe. Python's GIL provides the necessary synchronization.
461 #[pyclass(unsendable)]
462 struct Matra {
463 engine: crate::Engine,
464 }
465466 impl Matra {
467 fn from_nlp(nlp: Box<dyn NlpProvider>) -> Self {
468 Self {
469 engine: crate::Engine::new(nlp, crate::standard_decomposers()),
470 }
471 }
472473 /// One in-memory document through the whole pipeline.
474 fn document(&self, text: &str, format: Format) -> Result<domain::Document, MatraError> {
475 let raw = RawDocument::new(text.to_string(), None, format);
476 self.engine
477 .analyze_one(raw)
478 .map(|entry| entry.analysis)
479 .map_err(|e| MatraError(e.error))
480 }
481482 /// Annotated structure for one in-memory plain-text document.
483 ///
484 /// Parse only: compose would fill a metric suite the callers of
485 /// this helper immediately discard.
486 fn annotated(&self, text: &str) -> Result<domain::Document, MatraError> {
487 let raw = RawDocument::new(text.to_string(), None, Format::PlainText);
488 self.engine.annotate(&raw).map_err(MatraError)
489 }
490491 /// Pipeline-routed sentences for the extractors: same size gate,
492 /// same decomposition, same blockquote skipping as `analyze`.
493 fn sentences(&self, text: &str) -> Result<Vec<domain::Sentence>, MatraError> {
494 Ok(self.annotated(text)?.sentences().cloned().collect())
495 }
496 }
The tree has four layers, and imports only point inward:
src/domain.rs: every type the library hands back. Depends on serde,
thiserror and std.
The ports: src/source/mod.rs, src/decompose/mod.rs,
src/nlp/mod.rs, src/embed/mod.rs. Each declares one trait and
imports only domain.
The adapters, one port each, beside their port. src/nlp/udpipe.rs is
the only file that names udpipe_rs, because the panic boundary lives
there. metrics/, extraction/ and hearst.rs are plain functions over
domain types and touch no port.
src/lib.rs, the composition root, the only file that knows every port
and adapter. src/config.rs sits beside it; the command line in
src/cli/ sits above it and uses only the public surface
(EPR-0004).
Adding a format is a table entry: a Decomposer adapter plus one arm in
the composition root's exhaustive match. Adding a metric is a function
appended to the suite. Neither adds an entry point.
Concrete iterator of Result<RawDocument, DocumentError>. Ingest::text never fails; Ingest::path lists a directory eagerly and reads each file when pulled, through FileSource.
abstract is reserved by RFC-0007 as
the empty tier between structure and purpose-fitted output, where rule
evaluation lands. It is a Rust keyword, so it names the tier and never code.
No module, type or function in src/ occupies it ● observedlib.rs:1-21.
src/lib.rs lines 1 to 21, at 4fcfb4a
1 #![doc = include_str!("../README.md")]
2 3 #[cfg(feature = "cli")]
4 pub mod cli;
5 pub mod config;
6 pub mod decompose;
7 pub mod domain;
8 pub mod embed;
9 pub mod extraction;
10 pub mod hearst;
11 pub mod metrics;
12 pub mod nlp;
13 pub mod source;
14 mod stopwords;
1516 use std::path::Path;
1718 use decompose::Decomposer;
19 use domain::{Document, MAX_INPUT_BYTES};
20 use nlp::NlpProvider;
21 use source::Source;
The format table is data. Decomposers is a Vec keyed on
Format; with replaces on a duplicate key; lookup is the partial step and
each decomposer stays total.
The composition root fills it through default_decomposer, a match over Format with no wildcard, so a new Format variant fails to compile until someone decides whether it has a decomposer ● observedlib.rs:325-358.
Pdf and Docx are reserved variants with none.
src/lib.rs lines 325 to 358, at 4fcfb4a
325 /// The decomposer this build ships for `format`, or `None` for formats
326 /// that are reserved but unimplemented.
327 ///
328 /// The match is deliberately exhaustive with no wildcard: adding a
329 /// `Format` variant fails compilation here, so a new format is a
330 /// conscious registration decision rather than a silent
331 /// `Error::UnsupportedFormat` at run time.
332 fn default_decomposer(format: &domain::Format) -> Option<Box<dyn Decomposer>> {
333 match format {
334 domain::Format::Markdown => Some(Box::new(decompose::markdown::MarkdownDecomposer)),
335 domain::Format::PlainText => Some(Box::new(decompose::plain::PlainTextDecomposer)),
336 domain::Format::Pdf | domain::Format::Docx => None,
337 }
338 }
339340 /// The decomposer table this build ships: markdown and plain text.
341 ///
342 /// Lives in the composition root because it is the only place that
343 /// names every adapter (boundary rule 7). Callers who want a different
344 /// table build their own with [`decompose::Decomposers::with`].
345 pub fn standard_decomposers() -> decompose::Decomposers {
346 let mut table = decompose::Decomposers::new();
347 for format in [
348 domain::Format::Markdown,
349 domain::Format::PlainText,
350 domain::Format::Pdf,
351 domain::Format::Docx,
352 ] {
353 if let Some(decomposer) = default_decomposer(&format) {
354 table = table.with(format, decomposer);
355 }
356 }
357 table
358 }
src/lib.rs declares every module, builds Engine, builds the
standard decomposer table, and holds the one function that pairs a
Document with an Embedder (embed_and_cluster). The
udpipe-gated constructors Engine::from_config,
Engine::from_config_with_notice and Engine::with_defaults wire the
standard pipeline (constructors). The PyO3 module lives in
the same file behind python.
MAX_INPUT_BYTES is a constant (8 MiB), not a setting.
Two places enforce it: Engine::annotate for text (what = "input"), and FileSource on a file's metadata size before reading (what = "file_source") ● observedlib.rs:212-253source/file.rs:9-58.
The domain documents the gap honestly: a caller who invokes NlpProvider::parse directly bypasses the bound ● observeddomain.rs:12-28.
src/lib.rs lines 212 to 253, at 4fcfb4a
212 /// Decompose and parse one document into an unmeasured [`Document`]:
213 /// structure from the format's decomposer, sentences attached
214 /// per-paragraph, every metric slot still `None`.
215 ///
216 /// This is the only route from text to the parser, which is what
217 /// makes the input size cap a property of the pipeline rather than
218 /// of each entry point: no text over [`MAX_INPUT_BYTES`] reaches
219 /// [`NlpProvider::parse`].
220 ///
221 /// Structure materializes here (RFC-0008): derived facts whose
222 /// detectors live outside the domain are filled onto each parsed
223 /// sentence at this choke point. Today that is
224 /// [`domain::Sentence::hearst_pairs`], computed by
225 /// [`hearst::hypernymy_pairs`]; sentence-level facts whose
226 /// detectors live in the domain are computed by `Sentence::new`
227 /// inside the provider.
228 pub fn annotate(&self, raw: &domain::RawDocument) -> domain::Result<Document> {
229 check_input_size(&raw.text)?;
230 let decomposer = self
231 .decomposers
232 .get(&raw.format)
233 .ok_or_else(|| domain::Error::UnsupportedFormat(raw.format.clone()))?;
234 let mut doc = Document::new(decomposer.decompose(&raw.text));
235 for para in doc.paragraphs_mut() {
236 if para.in_blockquote {
237 continue;
238 }
239 para.sentences = self.nlp.parse(¶.text)?;
240 for sentence in &mut para.sentences {
241 sentence.hearst_pairs = hearst::hypernymy_pairs(&sentence.tokens);
242 }
243 }
244 Ok(doc)
245 }
246247 /// Run the metric suite over an annotated document. Total: metrics
248 /// read what is attached and skip what is not; there is no failure
249 /// path.
250 pub fn compose(&self, doc: &mut Document) {
251 let suite = metrics::default_suite();
252 metrics::run_suite(doc, &suite);
253 }
9 /// Reads a single file.
10 ///
11 /// Two pre-read guards before touching the file contents:
12 /// 1. **Symlinks are rejected.** The read uses `symlink_metadata` (which
13 /// does not traverse) and refuses any path whose file type is a symlink.
14 /// This matches `DirectorySource`'s behavior and prevents an attacker
15 /// who controls a path passed to `FileSource` from redirecting the read
16 /// to an arbitrary file via a symlink.
17 /// 2. **Files larger than [`MAX_INPUT_BYTES`] are rejected.** The check
18 /// is on the metadata-reported size, before any read into memory, so
19 /// a 1 GB file does not OOM the host before the gate runs.
20 pub struct FileSource;
2122 impl Source for FileSource {
23 fn read(&self, input: &Path) -> domain::Result<Vec<RawDocument>> {
24 let metadata = std::fs::symlink_metadata(input)?;
25 let file_type = metadata.file_type();
2627 if file_type.is_symlink() {
28 return Err(Error::Io(std::io::Error::new(
29 std::io::ErrorKind::Unsupported,
30 format!("refusing to read symlink: {}", input.display()),
31 )));
32 }
3334 // Files only — directories and other non-regular entries are not our concern.
35 if !file_type.is_file() {
36 return Err(Error::Io(std::io::Error::new(
37 std::io::ErrorKind::InvalidInput,
38 format!("not a regular file: {}", input.display()),
39 )));
40 }
4142 let len = metadata.len();
43 if len > MAX_INPUT_BYTES as u64 {
44 return Err(Error::InputTooLarge {
45 limit: MAX_INPUT_BYTES,
46 actual: usize::try_from(len).unwrap_or(usize::MAX),
47 what: "file_source",
48 });
49 }
5051 let text = std::fs::read_to_string(input)?;
52 let format = Format::from_path(input);
53 Ok(vec![RawDocument {
54 text,
55 path: Some(input.to_path_buf()),
56 format,
57 }])
58 }
12 /// The upper bound on text entering the pipeline, in bytes. A constant,
13 /// not a setting: no configuration, argument or environment variable
14 /// changes it.
15 ///
16 /// 8 MiB accommodates book-length English (a typical novel is ~1.5 MiB / 200k
17 /// words) with headroom for multilingual prose and structured documents. Beyond
18 /// this bound, the underlying NLP provider's intermediate memory grows past
19 /// safe limits on a typical workstation (UDPipe's per-token allocations cross
20 /// ~1 GiB resident at this input size).
21 ///
22 /// `Engine::annotate`, the only route from text to the parser, enforces it
23 /// and returns [`Error::InputTooLarge`] with `what = "input"` when it is
24 /// exceeded, so every `Engine` method inherits it. `FileSource` applies the
25 /// same limit to a file's size before reading it (`what = "file_source"`).
26 /// A provider's own `NlpProvider::parse` has no limit: calling it directly
27 /// bypasses this bound.
28 pub const MAX_INPUT_BYTES: usize = 8 * 1024 * 1024;
L1 analyze(a.chain(b)) = analyze(a).chain(analyze(b))L2 analyze(empty()) = empty()L3 analyze(once(Ok(raw))) = once(analyze_one(raw))L4 analyze_one(r).analysis = { let mut d = annotate(&r)?; compose(&mut d); d }L5 |entries| + |errors| = |input|L6 Err input item => identical Err output, analyze_one not calledL7 no text over MAX_INPUT_BYTES reaches NlpProvider::parse
L1 to L3 say a single document is a collection of one.
L7 is provable rather than empirical because annotate is the only caller of the parser in the pipeline ◐ inferredlib.rs:212-253.
src/lib.rs lines 212 to 253, at 4fcfb4a
212 /// Decompose and parse one document into an unmeasured [`Document`]:
213 /// structure from the format's decomposer, sentences attached
214 /// per-paragraph, every metric slot still `None`.
215 ///
216 /// This is the only route from text to the parser, which is what
217 /// makes the input size cap a property of the pipeline rather than
218 /// of each entry point: no text over [`MAX_INPUT_BYTES`] reaches
219 /// [`NlpProvider::parse`].
220 ///
221 /// Structure materializes here (RFC-0008): derived facts whose
222 /// detectors live outside the domain are filled onto each parsed
223 /// sentence at this choke point. Today that is
224 /// [`domain::Sentence::hearst_pairs`], computed by
225 /// [`hearst::hypernymy_pairs`]; sentence-level facts whose
226 /// detectors live in the domain are computed by `Sentence::new`
227 /// inside the provider.
228 pub fn annotate(&self, raw: &domain::RawDocument) -> domain::Result<Document> {
229 check_input_size(&raw.text)?;
230 let decomposer = self
231 .decomposers
232 .get(&raw.format)
233 .ok_or_else(|| domain::Error::UnsupportedFormat(raw.format.clone()))?;
234 let mut doc = Document::new(decomposer.decompose(&raw.text));
235 for para in doc.paragraphs_mut() {
236 if para.in_blockquote {
237 continue;
238 }
239 para.sentences = self.nlp.parse(¶.text)?;
240 for sentence in &mut para.sentences {
241 sentence.hearst_pairs = hearst::hypernymy_pairs(&sentence.tokens);
242 }
243 }
244 Ok(doc)
245 }
246247 /// Run the metric suite over an annotated document. Total: metrics
248 /// read what is attached and skip what is not; there is no failure
249 /// path.
250 pub fn compose(&self, doc: &mut Document) {
251 let suite = metrics::default_suite();
252 metrics::run_suite(doc, &suite);
253 }
The eight rules are canonical in
site/content/reference/boundary-rules.md, with the
motivation for each. In summary: the domain depends only on serde,
thiserror, std (1); ports import only domain (2) and never another
port (3); only nlp/udpipe.rs names udpipe_rs (4); metrics/ and
extraction/ import only domain and stopwords (5);
cargo check --no-default-features compiles (6); the composition root alone
knows every adapter and port, and src/cli/ uses only the public surface
(7); no tracing in the domain or the ports (8).
Rule 6 is compiled by the rust and msrv jobs in CI ● observedci.yml:18-100.
Rules 1, 2, 3, 4, 5, the src/cli/ part of 7, and 8 are semgrep rules under .semgrep/, each tested against a fixture of the forms it claims, run by scripts/check-boundaries.sh from just check and the Boundary check job ● observedci.yml:122-153.
EP-0014 records what each
rule catches and what it leaves to review: whether a trait's shape leaks an
adapter, whether a function takes text where it should take structure, and
whether a file other than src/lib.rs wires adapters together.
.github/workflows/ci.yml lines 18 to 100, at 4fcfb4a
18 rust:
19 name: Rust ${{ matrix.os }} / ${{ matrix.features }}
20 runs-on: ${{ matrix.os }}
21 strategy:
22 fail-fast: false
23 matrix:
24 os: [ubuntu-latest, macos-latest]
25 features: [default, no-default-features, model2vec, cli]
26 include:
27 - features: default
28 flags: ""
29 - features: no-default-features
30 flags: "--no-default-features"
31 # The command line's own tests, and with them tests/skill.rs,
32 # which executes every incantation in skills/matra/ against
33 # cli::run (RFC-0012). The `cli` feature is not in the default
34 # set, so without this lane none of that compiles in CI. The
35 # blocks that need the UDPipe model are `#[ignore]`, so this
36 # lane downloads nothing.
37 - features: cli
38 flags: "--features cli"
39 # The adapter's tests, including the cross-target bit-identity
40 # assertion, which only means something because this lane runs
41 # on both x86_64 (ubuntu) and aarch64 (macos).
42 - features: model2vec
43 flags: "--no-default-features --features model2vec"
44 steps:
45 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
46 with:
47 persist-credentials: false
48 49 - name: Install Rust toolchain
50 uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master (no tagged release; pinned to known-good commit)
51 with:
52 toolchain: stable
53 components: clippy, rustfmt
54 55 - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
56 57 - name: cargo fmt --check
58 if: matrix.features == 'default' && matrix.os == 'ubuntu-latest'
59 run: cargo fmt --all -- --check
60 61 - name: cargo check
62 run: cargo check --all-targets ${{ matrix.flags }}
63 64 - name: cargo clippy
65 run: cargo clippy --all-targets ${{ matrix.flags }} -- -D warnings
66 67 - name: cargo test (lib + doc)
68 run: cargo test ${{ matrix.flags }}
69 70 msrv:
71 name: MSRV (Rust 1.88)
72 runs-on: ubuntu-latest
73 env:
74 # rust-toolchain.toml pins `stable`, and rustup lets that file override
75 # the toolchain the install step selects. RUSTUP_TOOLCHAIN outranks the
76 # file, so every cargo and rustc below runs on the MSRV. Keep this equal
77 # to `rust-version` in Cargo.toml.
78 RUSTUP_TOOLCHAIN: "1.88"
79 steps:
80 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
81 with:
82 persist-credentials: false
83 - uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master
84 with:
85 toolchain: "1.88"
86 - name: Verify the compiler is the MSRV
87 # A check reports what it examined: fail unless the compiler that
88 # will run is the one this job is named for.
89 run: |
90 version="$(rustc --version)"
91 echo "$version"
92 case "$version" in
93 "rustc 1.88."*) ;;
94 *) echo "::error::expected rustc 1.88.x, got: $version"; exit 1 ;;
95 esac
96 - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
97 - name: cargo check (MSRV, all features)
98 run: cargo check --all-targets --all-features
99 - name: cargo check (MSRV, no default features)
100 run: cargo check --all-targets --no-default-features
.github/workflows/ci.yml lines 122 to 153, at 4fcfb4a
122 # The two script gates below used to run only from `just check` and the
123 # opt-in pre-commit hook. Each tool is pinned: semgrep and every package it
124 # pulls in by version and hash in .github/requirements/semgrep.txt, which
125 # pip refuses to install from if any hash differs; ripgrep, lychee and bun
126 # as release assets checked against the SHA-256 their releases publish.
127 #
128 # The job name is what branch protection requires; keep it.
129 boundaries:
130 name: Boundary check
131 runs-on: ubuntu-latest
132 steps:
133 # The full checkout (with .git) matters: the rules scope themselves by
134 # path, semgrep resolves paths against the git work tree, and the check
135 # refuses to run without one.
136 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
137 with:
138 persist-credentials: false
139 - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
140 with:
141 python-version: "3.12"
142 - name: Install semgrep (pinned by hash)
143 run: |
144 set -euo pipefail
145 python -m venv "$RUNNER_TEMP/semgrep"
146 "$RUNNER_TEMP/semgrep/bin/python" -m pip install --quiet --require-hashes \
147 -r .github/requirements/semgrep.txt
148 echo "$RUNNER_TEMP/semgrep/bin" >> "$GITHUB_PATH"
149 # Rules 1, 2, 3, 4, 5, 7 (the CLI's part) and 8, and the library's
150 # error conventions. Each rule file is tested against its fixture
151 # first, then src/ is scanned, then the scan's coverage is checked.
152 - name: Boundary rules (semgrep)
153 run: bash scripts/check-boundaries.sh
The panic boundary that rule 4 protects is catch_parse_panic, which wraps Model::parse and turns a panic into Error::ParseFailed● observednlp/udpipe.rs:704-734.
src/nlp/udpipe.rs lines 704 to 734, at 4fcfb4a
704 /// Run a closure that may panic inside the `udpipe-rs` C/C++ boundary,
705 /// catching any panic and converting it into [`Error::ParseFailed`]. This
706 /// keeps a process-aborting C-side bug (Taleb #1: SPOF with no panic
707 /// boundary) from taking down the host. Without this wrapper, a panic
708 /// inside `Model::parse` aborts the host process; in Python it manifests
709 /// as interpreter death, in WASM as a trap.
710 fn catch_parse_panic<F, T>(f: F) -> crate::domain::Result<T>
711 where
712 F: FnOnce() -> crate::domain::Result<T>,
713 {
714 use std::panic::{AssertUnwindSafe, catch_unwind};
715 match catch_unwind(AssertUnwindSafe(f)) {
716 Ok(result) => result,
717 Err(payload) => {
718 let message = payload
719 .downcast_ref::<&'static str>()
720 .map(|s| (*s).to_string())
721 .or_else(|| payload.downcast_ref::<String>().cloned())
722 .unwrap_or_else(|| "udpipe panic (no message captured)".to_string());
723 Err(Error::ParseFailed(format!("udpipe panicked: {message}")))
724 }
725 }
726 }
727728 impl NlpProvider for Udpipe {
729 fn parse(&self, text: &str) -> crate::domain::Result<Vec<Sentence>> {
730 let words = catch_parse_panic(|| {
731 self.model
732 .parse(text)
733 .map_err(|e| Error::ParseFailed(e.to_string()))
734 })?;
udpipe is the default and model2vec, cli and python are opt-in; cli enables udpipe, and python enables cli● observedCargo.toml:92-97.
Disabling defaults removes the UDPipe adapter and leaves the domain,
the ports, the pipeline, the metrics and the extractors.
The five-verb vocabulary.RFC-0002's
ingest / decompose / parse / measure with extract as a peer was
superseded by RFC-0007: the verbs
named calling conventions, not transformations. Its kept trait names
remain.
The workspace split.RFC-0003
was never implemented and is superseded by
RFC-0004; there is one
[package] and no [workspace] in Cargo.toml.
"No function name mentions a format or a source kind" holds for the
Rust surface only.RFC-0007
states it for the whole surface.
The Python class still has analyze_markdown and analyze_path, each a thin call into the one Engine● observedlib.rs:534-548lib.rs:658.
The invariants still live in one place; the names do not.
Boundary enforcement by grep and review alone. Records before
2026-09-26 say rules 1, 2, 5 and 7 have no mechanical check and that 3, 4
and 8 are a ripgrep over literal imports.
EP-0014 replaced that
with the semgrep rules above.
Not fewer names.RFC-0007
bought one implementation per invariant, not a smaller namespace.
Callers hold an Engine. Someone owns the decomposer table and the
provider; free functions could not.
Work moves to consumption time. A directory that lists is not a
directory that reads; failures surface as items.
The result stream is not Send.It borrows the engine, and NlpProvider is Send without Sync● observedlib.rs:163-192.
One parse call per paragraph. Not measured:
a document of many short paragraphs pays the per-call overhead many times◐ inferred.
The correctness argument (FM1) outranks it until a measurement says
otherwise.
The cap is bypassable below the pipeline.Calling NlpProvider::parse directly skips it; the domain documents this rather than enforcing it ● observeddomain.rs:12-28.
src/lib.rs lines 163 to 192, at 4fcfb4a
163 /// Analyze a stream of documents.
164 ///
165 /// Lazy: nothing is parsed until the returned iterator is pulled,
166 /// and each pull runs one document through [`Engine::analyze_one`]
167 /// to completion. An `Err` input item passes through unchanged
168 /// without touching the pipeline. Collect into
169 /// [`domain::CorpusResult`] to partition successes from failures:
170 ///
171 /// ```no_run
172 /// # fn example(engine: &matra::Engine) -> matra::domain::Result<()> {
173 /// let result: matra::domain::CorpusResult =
174 /// engine.analyze(matra::Ingest::path("docs/")?).collect();
175 /// # Ok(())
176 /// # }
177 /// ```
178 ///
179 /// The returned iterator borrows the engine and is not `Send`:
180 /// [`NlpProvider`] is `Send` without `Sync`, so the stream cannot be
181 /// shared across threads.
182 pub fn analyze<I>(
183 &self,
184 input: I,
185 ) -> impl Iterator<Item = std::result::Result<domain::CorpusEntry, domain::DocumentError>>
186 where
187 I: IntoIterator<Item = Ingested>,
188 {
189 input
190 .into_iter()
191 .map(|item| item.and_then(|raw| self.analyze_one(raw)))
192 }
12 /// The upper bound on text entering the pipeline, in bytes. A constant,
13 /// not a setting: no configuration, argument or environment variable
14 /// changes it.
15 ///
16 /// 8 MiB accommodates book-length English (a typical novel is ~1.5 MiB / 200k
17 /// words) with headroom for multilingual prose and structured documents. Beyond
18 /// this bound, the underlying NLP provider's intermediate memory grows past
19 /// safe limits on a typical workstation (UDPipe's per-token allocations cross
20 /// ~1 GiB resident at this input size).
21 ///
22 /// `Engine::annotate`, the only route from text to the parser, enforces it
23 /// and returns [`Error::InputTooLarge`] with `what = "input"` when it is
24 /// exceeded, so every `Engine` method inherits it. `FileSource` applies the
25 /// same limit to a file's size before reading it (`what = "file_source"`).
26 /// A provider's own `NlpProvider::parse` has no limit: calling it directly
27 /// bypasses this bound.
28 pub const MAX_INPUT_BYTES: usize = 8 * 1024 * 1024;
These are the alternatives the legacy records weighed and rejected.
Six entry points with a parity test (the 0.1.0 pre-release surface):
rejected because the test catches drift after it happens and the
compiler checks none of the restatements
(EP-0008).
ingest / decompose / abstract / compose as four stages: rejected
because Decomposer::decompose and NlpProvider::parse share one shape
(string in, latent structure out) and differ by dependency and failure
mode, which is what ports are factored by; and because abstract cannot
name code in Rust (EP-0008).
A typestate split of Document (structure, annotated, measured):
declined because the Option slots also encode inapplicability, so the
split would not remove them, and law L4 answers "has compose run" more
cheaply (EP-0008, costs 5).
Join paragraphs, parse once, match sentences back: rejected after
FM1 (above).
A Cargo workspace with a matcher-bridge crate: rejected by
RFC-0004; rule evaluation is
to land inside matra, and no external NlpProvider implementors exist.
An async or reactor-driven pipeline: deferred.
RFC-0007 records that the lazy
stream is safe only because analyze_one runs to completion inside one
next(), so staying synchronous is load-bearing, not merely simpler.
Semgrep's Rust AST patterns for the boundary checks: rejected after
measurement showed they miss brace groups, deep paths and type positions
silently (EP-0014).
Not doing this would leave each caller and each binding to restate the cap,
the paragraph wiring and the format dispatch, which is the defect class the
current shape removed.
Ports and adapters (Alistair Cockburn's hexagonal architecture) is the
layering this follows: the domain at the centre, ports as the interfaces
it needs, adapters at the edge, one place that wires them.
Streaming document pipelines in NLP libraries, such as spaCy's
Language.pipe, take an iterable of texts and yield processed documents
lazily. Context, not a reason: matra's laws and its per-document error
items are its own.
For the owner to settle in this proposal's review. Each is a decision of its
own.
An adapter imports a sibling adapter.src/source/directory.rs imports FileSource from src/source/file.rs and reads each listed file through it ● observedsource/directory.rs:1-62grounds for decision 1.
The boundary rules do not name adapter-to-adapter imports, and
EP-0014 leaves "one
adapter importing another" to review.
src/source/directory.rs lines 1 to 62, at 4fcfb4a
1 //! Directory source adapter. Reads all files in a directory.
2 3 use std::path::{Path, PathBuf};
4 5 use crate::domain::{self, RawDocument};
6 7 use super::Source;
8 use super::file::FileSource;
910 /// Reads all regular files in a directory (non-recursive, symlinks skipped).
11 ///
12 /// Behavior:
13 /// - Entries that cannot be listed (`read_dir` error on a child) are skipped.
14 /// - Symlinks are skipped to avoid following attacker-controlled paths
15 /// into unexpected parts of the filesystem.
16 /// - Files are yielded sorted by path (lexicographic, byte-wise on
17 /// `PathBuf`) — consumers may rely on this ordering.
18 /// - [`Source::read`] (the trait method) returns only the successfully-read
19 /// documents; per-file failures are silently dropped. Callers that care
20 /// about which files failed should stream through `Ingest`, which
21 /// yields each failure as an item carrying its path.
22 pub struct DirectorySource;
2324 impl DirectorySource {
25 /// List candidate paths in the directory, sorted, after the symlink
26 /// filter and `FileSource::accepts` (a regular file). No extension
27 /// is filtered: every regular file is a candidate, and its format is
28 /// assigned from its extension when it is read. Listing is separate from reading so
29 /// the composition root can enumerate eagerly and read lazily.
30 pub(crate) fn candidate_paths(&self, input: &Path) -> domain::Result<Vec<PathBuf>> {
31 let file_source = FileSource;
32 let mut paths: Vec<_> = std::fs::read_dir(input)?
33 .filter_map(|e| e.ok())
34 .filter(|e| {
35 // Skip symlinks; `symlink_metadata` does not traverse.
36 e.path()
37 .symlink_metadata()
38 .map(|m| m.file_type().is_file())
39 .unwrap_or(false)
40 })
41 .map(|e| e.path())
42 .filter(|p| file_source.accepts(p))
43 .collect();
44 paths.sort();
45 Ok(paths)
46 }
47 }
4849 impl Source for DirectorySource {
50 /// Reads all documents in the directory, dropping per-file failures
51 /// silently. Callers that care about which files failed should
52 /// stream through `Ingest` instead.
53 fn read(&self, input: &Path) -> domain::Result<Vec<RawDocument>> {
54 let file_source = FileSource;
55 let docs = self
56 .candidate_paths(input)?
57 .into_iter()
58 .filter_map(|path| file_source.read(&path).ok())
59 .flatten()
60 .collect();
61 Ok(docs)
62 }
DirectorySource keeps reading each file through FileSource, and the boundary rules page records that an adapter may compose another adapter of the same port.
B No; the composition root does it
DirectorySource is given the Source it reads each file through, and src/lib.rs passes it FileSource, as the composition root already does its own wiring for the streaming route in Ingest::path.
Claude recommends A, Yes, within one port. ◐ inferred, a judgment
Both adapters implement one port, and the symlink refusal and the size cap live once, in FileSource; reading through it is what keeps them in one place. An import inside one port does not cross a layer, which is what the boundary rules exist to hold. Recording the case turns an unrecorded exception into a reviewed one. Undoing it later, by choice B, would change the public DirectorySource, so this is costly to reverse.○ assumedgrounds for decision 1
The strongest case against it
Rule 7 makes the composition root the only place adapters are wired together, and an import inside one port looks the same to a reviewer as one across ports, so an accepted exception rests on review alone. Ingest::path shows the composition root can do this wiring.
The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.
11 /// Decomposes raw text into structural sections with paragraphs.
12 pub trait Decomposer {
13 /// Decompose `text` into a section tree. Infallible: malformed input
14 /// is interpreted as best it can be (e.g., malformed markdown is
15 /// treated as plain text). The returned section tree has paragraphs
16 /// in document order with `in_blockquote` flags set correctly.
17 fn decompose(&self, text: &str) -> Vec<Section>;
18 }
13 /// Reads documents from a path. Paths only, not strings.
14 pub trait Source: Send {
15 /// Read zero or more documents from `input`. Adapters translate
16 /// external errors into [`domain::Error`] variants. Postconditions
17 /// are documented per adapter; see [`crate::source::file::FileSource`]
18 /// and [`crate::source::directory::DirectorySource`].
19 fn read(&self, input: &Path) -> domain::Result<Vec<domain::RawDocument>>;
20 /// Cheap pre-check: whether this adapter can read `input`. Must not
21 /// read file contents.
22 ///
23 /// The composition root does not consult it to pick an adapter:
24 /// `Ingest::path` chooses by the path's own metadata. Its one caller
25 /// in the library is `DirectorySource`, which keeps only the listed
26 /// entries that `FileSource::accepts`.
27 fn accepts(&self, input: &Path) -> bool;
28 }
11 /// Any NLP provider implements this. The pipeline depends on this trait,
12 /// not on any specific provider.
13 ///
14 /// # Contract
15 ///
16 /// Every [`domain::Sentence`] a provider returns satisfies the invariants
17 /// the domain documents on that type, because the tree walks and the
18 /// metrics read them without checking:
19 ///
20 /// - `tokens` are in ascending `id` order, numbered `1..=n` with no gaps,
21 /// where `n` is the token count.
22 /// - Every `head` is `0` or the `id` of a token in the same sentence.
23 /// - Exactly one token has `head == 0`: the root.
24 /// - Following `head` from any token reaches the root; the references
25 /// form no cycle.
26 /// - The sentence is built with [`domain::Sentence::new`], which derives
27 /// the structural fields from `tokens`. `hearst_pairs` is left empty;
28 /// `Engine::annotate` fills it.
29 ///
30 /// The trait cannot enforce any of this and nothing validates it on the
31 /// way in. A provider that breaks the cycle rule is reported rather than
32 /// trusted where it matters most: [`domain::Sentence::tree_depth`]
33 /// returns `usize::MAX` on a cycle, and [`domain::Sentence::subtree`]
34 /// terminates on one.
35 ///
36 /// A failure is an `Err`, typically [`domain::Error::ParseFailed`], never
37 /// a panic: a provider wrapping native code converts a panic at its own
38 /// boundary, as the UDPipe adapter does.
39 ///
40 /// # Size and blocking
41 ///
42 /// `parse` is blocking and may be slow on very large inputs. The trait
43 /// has no input size limit and no cancellation mechanism.
44 /// `Engine::annotate` applies [`domain::MAX_INPUT_BYTES`] before any text
45 /// reaches a provider; a caller invoking `parse` directly bypasses that
46 /// bound and should validate size itself.
47 pub trait NlpProvider: Send {
48 /// Parse text into sentences with POS tags and dependency labels,
49 /// satisfying the contract above.
50 fn parse(&self, text: &str) -> domain::Result<Vec<domain::Sentence>>;
51 }
18 /// Any embedding provider implements this. Consumers depend on this trait,
19 /// not on any specific backend.
20 ///
21 /// # Contract
22 ///
23 /// `embed` returns exactly one vector per input text, in input order, and
24 /// every returned vector has the same dimension. An implementation that
25 /// cannot honor this for some input returns an error rather than a short
26 /// or ragged result.
27 ///
28 /// Callers should be aware that `embed` is blocking and may allocate
29 /// proportionally to the batch. There is no built-in input size limit;
30 /// text that has passed through the pipeline is already bounded by
31 /// [`domain::MAX_INPUT_BYTES`], and applications embedding user-supplied
32 /// text directly should validate size before calling.
33 pub trait Embedder: Send {
34 /// Embed each text into a dense vector.
35 fn embed(&self, texts: &[&str]) -> domain::Result<Vec<domain::Embedding>>;
3637 /// A stable identifier for the model behind this embedder, carried
38 /// into every provenance-bearing result derived from its vectors.
39 /// For file-loaded models this is a digest over the artifacts; the
40 /// point is that scores stay attributable to one geometry, so two
41 /// embedders that can disagree must not share an identity.
42 fn identity(&self) -> &str;
43 }
20 /// Format-indexed table of decomposers. Lookup is the partial step;
21 /// each [`Decomposer`] stays total.
22 ///
23 /// The table is data, not code: which formats a build supports is a
24 /// question you ask a value, and `Error::UnsupportedFormat` means
25 /// exactly "no entry in this table". Population happens in the
26 /// composition root, the only place that names concrete adapters.
27 ///
28 /// Backed by a `Vec` keyed on [`Format`] equality. The registry holds a
29 /// handful of entries, so linear lookup is not a cost worth a `Hash`
30 /// bound on a `#[non_exhaustive]` enum.
31 #[derive(Default)]
32 pub struct Decomposers {
33 entries: Vec<(Format, Box<dyn Decomposer>)>,
34 }
3536 impl Decomposers {
37 /// An empty table: every format is unsupported.
38 pub fn new() -> Self {
39 Self {
40 entries: Vec::new(),
41 }
42 }
4344 /// Register `decomposer` for `format`, replacing any existing entry
45 /// for that format.
46 #[must_use]
47 pub fn with(mut self, format: Format, decomposer: Box<dyn Decomposer>) -> Self {
48 if let Some(slot) = self.entries.iter_mut().find(|(f, _)| *f == format) {
49 slot.1 = decomposer;
50 } else {
51 self.entries.push((format, decomposer));
52 }
53 self
54 }
5556 /// The decomposer registered for `format`, if any.
57 pub fn get(&self, format: &Format) -> Option<&dyn Decomposer> {
58 self.entries
59 .iter()
60 .find(|(f, _)| f == format)
61 .map(|(_, d)| d.as_ref())
62 }
63 }
Decomposer gains Send as a supertrait, as the other three ports have, so an Engine can move to another thread.
B Yes: record the asymmetry as intended
Decomposer stays as it is, and its documentation and the ports table say that Engine is not Send and why.
Claude recommends A, No: Decomposer requires Send. ◐ inferred, a judgment
The other three ports require Send, which pays off only if an Engine holding them can move between threads, and Decomposer is the one bound that stops it. The two decomposers matra ships are unit structs with no state to make them unsendable.
The strongest case against it
A new supertrait breaks any decomposer written outside matra that holds a non-Send value, and nothing in matra needs Send today. A bound added without a caller who needs it is a constraint chosen in advance.
The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.
○ 1 claim stands on no grounds yet. Confirm or strike each with a comment on it.