Enhancement proposal EPR-0001 The pipeline and its ports proposed

0 of 2 decisions settled 1 assumption to confirm pinned at 4fcfb4a pull request #133 tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track) started: 2026-10-04feature name: pipeline_and_ports decides: the owner, who alone sets the status to accepted

EPR-0001 0 of 2 settled 1 to confirm

Enhancement proposal EPR-0001 The pipeline and its ports proposed

pinned at 4fcfb4a pull request #133 tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track) started: 2026-10-04feature name: pipeline_and_ports decides: the owner, who alone sets the status to accepted

EPR-0001: The pipeline and its ports

enhancement proposal: proposed

on this record: read, decide, confirm
Started
2026-10-04
Pinned at
4fcfb4a, the commit every code citation below opens at
Decides
the owner, who alone sets the status to accepted
Proposal PR
#133
Tracking issue
none (a baseline proposal describes code that already ships, so there is no milestone to track)
Feature name
pipeline_and_ports

Summary

matra is one crate whose surface is one pipeline. A caller turns input into an Ingest stream and hands it to an Engine, which runs each document through two stages: annotate (decompose the text into sections and paragraphs, then parse each paragraph) and compose (run the metric suite). annotate is the only route from text to the parser, so the input size cap is a property of the pipeline rather than of each entry point. Four ports (Source, Decomposer, NlpProvider, Embedder) declare what the pipeline needs from the outside; adapters implement them; src/lib.rs is the composition root that wires them; eight boundary rules keep the dependency arrows pointing inward.

This is a baseline proposal. It describes the code at the pinned commit and carries forward the reasons that still hold from RFC-0002, RFC-0003, RFC-0004, RFC-0007, EP-0008 and EP-0014. A later proposal that changes any of it says so. Unmarked statements about the code are observed at the pinned commit through the link beside them; a claim that is inferred or assumed says so.

Motivation

Why this is the foundation, rather than any other shape, comes down to four reasons, each paid for by a defect or a near miss.

One implementation of each invariant. Before 0.1.0 the public Rust surface was six free functions, each a partial application of the same chain, and each restated the invariants itself. Two live defects came from that shape: the input size cap held on four Python methods and not on the other four, and analyze_from returned documents whose paragraph metrics were always None because the sentence set travelled twice with nothing enforcing agreement (EP-0008, "The two defects"). RFC-0007 replaced the six with one pipeline so each invariant has one home. Today the size check sits in check_input_size, called first thing in Engine::annotate, and the metric suite takes the document alone (Metric), so the sentence set exists once observed lib.rs:23-38 lib.rs:212-253 metrics/mod.rs:22-57.

Per-paragraph parsing. An earlier pipeline joined paragraphs, parsed once, and wired sentences back to paragraphs by prefix match. Two paragraphs sharing their first 30 characters had their sentences silently reassigned (the failure the project calls FM1). annotate parses each paragraph on its own, and a regression test, parse_per_paragraph_scopes_sentences_to_originating_paragraph, pins the case observed lib.rs:235-243 lib.rs:1475-1496.

Ports factored by dependency and failure mode. A port exists where the outside world differs in what it depends on and how it fails: reading bytes (Source), cutting text into structure (Decomposer, total), parsing (NlpProvider, native code and a panic boundary), embedding (Embedder, model output). RFC-0007 kept the trait names RFC-0002 chose for this reason: they are ports, not stages.

One crate. RFC-0004 kept matra a single crate because the test for splitting out a port crate is whether external implementors need to pin the contract apart from the main crate's version churn, and no such implementors exist. There is one [package] in Cargo.toml and no workspace observed Cargo.toml:1-14.

Guide-level explanation

For a caller

You hold an Engine and feed it documents. A string is a stream of one, a file is a stream of one, a directory is a stream of many; all three are an Ingest, one concrete type (Ingest).

use matra::domain::{CorpusResult, Format};
use matra::{Engine, Ingest};

let engine = Engine::with_defaults()?;

// One document in memory: a stream of one.
let one: CorpusResult = engine
    .analyze(Ingest::text("The report was filed.", Format::PlainText))
    .collect();

// A directory: each file is read when the stream is pulled, and a file
// that fails becomes an error item rather than ending the walk.
let many: CorpusResult = engine.analyze(Ingest::path("docs/")?).collect();
for failure in &many.errors { /* failure.path, failure.error */ }

analyze is lazy: nothing is read or parsed until the iterator is pulled, and each pull runs one document to completion observed lib.rs:163-192. So Ingest::path(dir)? returning Ok means the directory listed, not that every file read. Per-file failures arrive as DocumentError items carrying the path observed lib.rs:89-113, and collecting into CorpusResult partitions successes from failures.

The two stages are public. annotate returns a Document with structure and sentences and every metric slot None; compose fills the slots and cannot fail observed lib.rs:212-253. analyze_one is exactly annotate then compose observed lib.rs:194-210.

Text over 8 MiB is refused with Error::InputTooLarge { what: "input" } before any parse observed domain.rs:12-28 lib.rs:23-38. Blockquote paragraphs are kept in the structure but never parsed or measured.

The Python Matra class wraps one Engine and routes every method through it observed lib.rs:457-496, so the cap and the per-paragraph parse hold there too.

For a contributor

The tree has four layers, and imports only point inward:

  1. src/domain.rs: every type the library hands back. Depends on serde, thiserror and std.
  2. The ports: src/source/mod.rs, src/decompose/mod.rs, src/nlp/mod.rs, src/embed/mod.rs. Each declares one trait and imports only domain.
  3. The adapters, one port each, beside their port. src/nlp/udpipe.rs is the only file that names udpipe_rs, because the panic boundary lives there. metrics/, extraction/ and hearst.rs are plain functions over domain types and touch no port.
  4. src/lib.rs, the composition root, the only file that knows every port and adapter. src/config.rs sits beside it; the command line in src/cli/ sits above it and uses only the public surface (EPR-0004).

Adding a format is a table entry: a Decomposer adapter plus one arm in the composition root's exhaustive match. Adding a metric is a function appended to the suite. Neither adds an entry point.

Reference-level explanation

The pipeline and its ports, as they ship

Shipped: the pipeline at 4fcfb4a; each arrow is a call, and each port box names its adapters

CallerEngineanalyzeannotate, composeIngest(text, path)embed_and_clusterDecomposerMarkdown, PlainTextNlpProviderUdpipeSourceFileSource,DirectorySourceEmbedderModel2Vec
show the data
statedrawnelementidlabel
shippedshippedboxcallerCaller
shippedshippedboxengineEngine analyze annotate, compose
shippedshippedboxingestIngest (text, path)
shippedshippedboxclusterembed_and_cluster
shippedshippedboxdecomposerDecomposer Markdown, PlainText
shippedshippedboxnlpNlpProvider Udpipe
shippedshippedboxsourceSource FileSource, DirectorySource
shippedshippedboxembedderEmbedder Model2Vec
shippedshippedarrowcaller>enginecaller to engine
shippedshippedarrowcaller>clustercaller to cluster
shippedshippedarrowengine>ingestengine to ingest
shippedshippedarrowengine>decomposerengine to decomposer
shippedshippedarrowengine>nlpengine to nlp
shippedshippedarrowingest>sourceingest to source
shippedshippedarrowcluster>embeddercluster to embedder
sketchy: proposed, not decided crisp: shipped at 4fcfb4a

The stages

StageWhereWhat it does
ingestIngestConcrete iterator of Result<RawDocument, DocumentError>. Ingest::text never fails; Ingest::path lists a directory eagerly and reads each file when pulled, through FileSource.
decomposeinside annotateLooks up the document's Format in the Decomposers table; a missing entry is Error::UnsupportedFormat.
parseinside annotateOne NlpProvider::parse call per non-blockquote paragraph; the Hearst detector then fills each sentence's hearst_pairs.
composecomposeRuns default_suite(): readability, lexical density, document-level ratios, compression. Total.

abstract is reserved by RFC-0007 as the empty tier between structure and purpose-fitted output, where rule evaluation lands. It is a Rust keyword, so it names the tier and never code. No module, type or function in src/ occupies it observed lib.rs:1-21.

The ports

PortTraitContract, as written on the traitAdapters
sourceSource (Send)read a path into documents; accepts without reading contentsFileSource, DirectorySource
decomposeDecomposerdecompose is total; malformed input is read as best it can beMarkdownDecomposer, PlainTextDecomposer
nlpNlpProvider (Send)sentences satisfy the domain's tree invariants; failure is an Err, never a panic; no size limit of its ownUdpipe, behind udpipe
embedEmbedder (Send)one vector per text, uniform dimension; a stable identityModel2Vec, behind model2vec (EPR-0005)

The format table is data. Decomposers is a Vec keyed on Format; with replaces on a duplicate key; lookup is the partial step and each decomposer stays total. The composition root fills it through default_decomposer, a match over Format with no wildcard, so a new Format variant fails to compile until someone decides whether it has a decomposer observed lib.rs:325-358. Pdf and Docx are reserved variants with none.

The composition root

src/lib.rs declares every module, builds Engine, builds the standard decomposer table, and holds the one function that pairs a Document with an Embedder (embed_and_cluster). The udpipe-gated constructors Engine::from_config, Engine::from_config_with_notice and Engine::with_defaults wire the standard pipeline (constructors). The PyO3 module lives in the same file behind python.

The size gate

MAX_INPUT_BYTES is a constant (8 MiB), not a setting. Two places enforce it: Engine::annotate for text (what = "input"), and FileSource on a file's metadata size before reading (what = "file_source") observed lib.rs:212-253 source/file.rs:9-58. The domain documents the gap honestly: a caller who invokes NlpProvider::parse directly bypasses the bound observed domain.rs:12-28.

The laws

Seven equivalence laws from RFC-0007 pin the grains together, each a test in src/lib.rs (law_l1 through law_l7):

L1  analyze(a.chain(b))        = analyze(a).chain(analyze(b))
L2  analyze(empty())           = empty()
L3  analyze(once(Ok(raw)))     = once(analyze_one(raw))
L4  analyze_one(r).analysis    = { let mut d = annotate(&r)?; compose(&mut d); d }
L5  |entries| + |errors|       = |input|
L6  Err input item             => identical Err output, analyze_one not called
L7  no text over MAX_INPUT_BYTES reaches NlpProvider::parse

L1 to L3 say a single document is a collection of one. L7 is provable rather than empirical because annotate is the only caller of the parser in the pipeline inferred lib.rs:212-253.

The boundary rules and how they are checked

The eight rules are canonical in site/content/reference/boundary-rules.md, with the motivation for each. In summary: the domain depends only on serde, thiserror, std (1); ports import only domain (2) and never another port (3); only nlp/udpipe.rs names udpipe_rs (4); metrics/ and extraction/ import only domain and stopwords (5); cargo check --no-default-features compiles (6); the composition root alone knows every adapter and port, and src/cli/ uses only the public surface (7); no tracing in the domain or the ports (8).

Rule 6 is compiled by the rust and msrv jobs in CI observed ci.yml:18-100. Rules 1, 2, 3, 4, 5, the src/cli/ part of 7, and 8 are semgrep rules under .semgrep/, each tested against a fixture of the forms it claims, run by scripts/check-boundaries.sh from just check and the Boundary check job observed ci.yml:122-153. EP-0014 records what each rule catches and what it leaves to review: whether a trait's shape leaks an adapter, whether a function takes text where it should take structure, and whether a file other than src/lib.rs wires adapters together.

The panic boundary that rule 4 protects is catch_parse_panic, which wraps Model::parse and turns a panic into Error::ParseFailed observed nlp/udpipe.rs:704-734.

Features

udpipe is the default and model2vec, cli and python are opt-in; cli enables udpipe, and python enables cli observed Cargo.toml:92-97. Disabling defaults removes the UDPipe adapter and leaves the domain, the ports, the pipeline, the metrics and the extractors.

No longer in force

  • The five-verb vocabulary. RFC-0002's ingest / decompose / parse / measure with extract as a peer was superseded by RFC-0007: the verbs named calling conventions, not transformations. Its kept trait names remain.
  • The workspace split. RFC-0003 was never implemented and is superseded by RFC-0004; there is one [package] and no [workspace] in Cargo.toml.
  • "No function name mentions a format or a source kind" holds for the Rust surface only. RFC-0007 states it for the whole surface. The Python class still has analyze_markdown and analyze_path, each a thin call into the one Engine observed lib.rs:534-548 lib.rs:658. The invariants still live in one place; the names do not.
  • Boundary enforcement by grep and review alone. Records before 2026-09-26 say rules 1, 2, 5 and 7 have no mechanical check and that 3, 4 and 8 are a ripgrep over literal imports. EP-0014 replaced that with the semgrep rules above.

Drawbacks

  • Not fewer names. RFC-0007 bought one implementation per invariant, not a smaller namespace.
  • Callers hold an Engine. Someone owns the decomposer table and the provider; free functions could not.
  • Work moves to consumption time. A directory that lists is not a directory that reads; failures surface as items.
  • The result stream is not Send. It borrows the engine, and NlpProvider is Send without Sync observed lib.rs:163-192.
  • One parse call per paragraph. Not measured: a document of many short paragraphs pays the per-call overhead many times inferred. The correctness argument (FM1) outranks it until a measurement says otherwise.
  • The cap is bypassable below the pipeline. Calling NlpProvider::parse directly skips it; the domain documents this rather than enforcing it observed domain.rs:12-28.

Rationale and alternatives

These are the alternatives the legacy records weighed and rejected.

  • Six entry points with a parity test (the 0.1.0 pre-release surface): rejected because the test catches drift after it happens and the compiler checks none of the restatements (EP-0008).
  • ingest / decompose / abstract / compose as four stages: rejected because Decomposer::decompose and NlpProvider::parse share one shape (string in, latent structure out) and differ by dependency and failure mode, which is what ports are factored by; and because abstract cannot name code in Rust (EP-0008).
  • A typestate split of Document (structure, annotated, measured): declined because the Option slots also encode inapplicability, so the split would not remove them, and law L4 answers "has compose run" more cheaply (EP-0008, costs 5).
  • Join paragraphs, parse once, match sentences back: rejected after FM1 (above).
  • A Cargo workspace with a matcher-bridge crate: rejected by RFC-0004; rule evaluation is to land inside matra, and no external NlpProvider implementors exist.
  • An async or reactor-driven pipeline: deferred. RFC-0007 records that the lazy stream is safe only because analyze_one runs to completion inside one next(), so staying synchronous is load-bearing, not merely simpler.
  • Semgrep's Rust AST patterns for the boundary checks: rejected after measurement showed they miss brace groups, deep paths and type positions silently (EP-0014).

Not doing this would leave each caller and each binding to restate the cap, the paragraph wiring and the format dispatch, which is the defect class the current shape removed.

Prior art

  • Ports and adapters (Alistair Cockburn's hexagonal architecture) is the layering this follows: the domain at the centre, ports as the interfaces it needs, adapters at the edge, one place that wires them.
  • Streaming document pipelines in NLP libraries, such as spaCy's Language.pipe, take an iterable of texts and yield processed documents lazily. Context, not a reason: matra's laws and its per-document error items are its own.
  • RFC-0002, RFC-0004, RFC-0007, EP-0008 and EP-0014 hold the full history.

Unresolved questions

For the owner to settle in this proposal's review. Each is a decision of its own.

An adapter imports a sibling adapter. src/source/directory.rs imports FileSource from src/source/file.rs and reads each listed file through it observed source/directory.rs:1-62 grounds for decision 1. The boundary rules do not name adapter-to-adapter imports, and EP-0014 leaves "one adapter importing another" to review.

Decision · open · costly to reverse

1. May an adapter compose another adapter of its own port?

A Yes, within one port recommended

DirectorySource keeps reading each file through FileSource, and the boundary rules page records that an adapter may compose another adapter of the same port.

B No; the composition root does it

DirectorySource is given the Source it reads each file through, and src/lib.rs passes it FileSource, as the composition root already does its own wiring for the streaming route in Ingest::path.

Claude recommends A, Yes, within one port. inferred, a judgment

Both adapters implement one port, and the symlink refusal and the size cap live once, in FileSource; reading through it is what keeps them in one place. An import inside one port does not cross a layer, which is what the boundary rules exist to hold. Recording the case turns an unrecorded exception into a reviewed one. Undoing it later, by choice B, would change the public DirectorySource, so this is costly to reverse. assumed grounds for decision 1

The strongest case against it

Rule 7 makes the composition root the only place adapters are wired together, and an import inside one port looks the same to a reviewer as one across ports, so an accepted exception rests on review alone. Ingest::path shows the composition root can do this wiring.

The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.

Decomposer has no Send bound. Source, NlpProvider and Embedder require Send; Decomposer does not observed decompose/mod.rs:11-18 source/mod.rs:13-28 nlp/mod.rs:11-51 embed/mod.rs:18-43 grounds for decision 2. Decomposers holds Box<dyn Decomposer>, so Engine is not Send either: observed by compiling a Send assertion on Engine at the pinned commit, which fails on dyn Decomposer observed decompose/mod.rs:20-63 grounds for decision 2. Nothing in matra needs it today (the Python class is unsendable) observed lib.rs:457-496 grounds for decision 2, but the asymmetry is unrecorded.

Decision · open · costly to reverse

2. Is Engine meant to be single-threaded?

A No: Decomposer requires Send recommended

Decomposer gains Send as a supertrait, as the other three ports have, so an Engine can move to another thread.

B Yes: record the asymmetry as intended

Decomposer stays as it is, and its documentation and the ports table say that Engine is not Send and why.

Claude recommends A, No: Decomposer requires Send. inferred, a judgment

The other three ports require Send, which pays off only if an Engine holding them can move between threads, and Decomposer is the one bound that stops it. The two decomposers matra ships are unit structs with no state to make them unsendable.

The strongest case against it

A new supertrait breaks any decomposer written outside matra that holds a non-Send value, and nothing in matra needs Send today. A bound added without a caller who needs it is a constraint chosen in advance.

The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.

1 claim stands on no grounds yet. Confirm or strike each with a comment on it.

  1. Undoing it later, by choice B, would change the public DirectorySource, so this is costly to reverse. where: Unresolved questions bears on decision 1

Future possibilities

  • Rule evaluation in the abstract tier, as ROADMAP.md describes it, reading the fields the pipeline materializes.
  • PDF and DOCX decomposers as new table entries; the Format variants are already reserved.
  • A separate matra-nlp-api crate if one of RFC-0004's re-open conditions fires, chiefly a third-party NlpProvider crate.