Enhancement proposalEPR-0005Embeddings and semantic clusters○ proposed
no decisions○ 1 assumption to confirmpinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: embeddings_and_semantic_clustersdecides: the owner, who alone sets the status to accepted
EPR-0005no decisions○ 1 to confirm
Enhancement proposalEPR-0005Embeddings and semantic clusters○ proposed
pinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: embeddings_and_semantic_clustersdecides: the owner, who alone sets the status to accepted
matra can group a document's sentences that say the same thing in different
words. It does so through the Embedder port, whose one built-in adapter,
Model2Vec, loads a static embedding model in the model2vec format behind
the model2vec feature. Because an embedding is a model's opinion rather
than structure a caller can check against the text, nothing derived from
one ever becomes a field on Document or any type the deterministic
pipeline returns. Semantic results arrive from separate calls, as
SemanticClusters, which carry the identity of the model that produced
them and the threshold the caller chose.
This is a baseline proposal. It describes the code at the pinned commit and
carries forward the reasons that still hold from RFC-0010 and EP-0009, with
the provisioning that RFC-0011 and RFC-0015 added described in
EPR-0003. A later proposal that
changes any of it says so. Unmarked statements about the code are observed
at the pinned commit through the link beside them; a claim that is
inferred or assumed says so.
Lexical overlap cannot see a paraphrase: "the committee approved the
proposal without debate" and "the proposal was approved by the committee
with no discussion" share few words. Sentence embeddings can. But two
models give two geometries, and no caller can verify a cosine score against
the source bytes, which every other output of matra allows
(EP-0009, "The tier line is the
design"). The capability is admitted only on terms that keep that
difference visible:
A separate channel.RFC-0008's
rule that derivations cross as fields applies to derivations of the
parse. An embedding is not one, so it gets its own types and calls
(RFC-0010, decision 1).
Provenance in the result. Every result names the model whose geometry
produced it, read from the embedder itself so it cannot be the wrong one
(RFC-0010, M5 amendment).
A port, so the model is replaceable. A caller with embeddings from a
service or a library matra does not ship can supply them, in Rust or in
Python.
use matra::embed::model2vec::Model2Vec;let cfg = matra::config::Config::resolve()?;let model = Model2Vec::from_config(&cfg)?; // downloads the pinned model oncelet engine = matra::Engine::from_config(&cfg)?;let raw = matra::domain::RawDocument::new(text, None, matra::domain::Format::PlainText);let doc = engine.annotate(&raw)?;let clusters = matra::embed_and_cluster(&doc, &model, 0.8)?;for c in &clusters.clusters { // c.members: sentence indices in document order // c.edges: the pairs that cleared 0.8, with their scores}assert_eq!(clusters.model_hash, model.model_hash());
A caller who already has vectors calls
matra::extraction::semantic_clusters(&embeddings, threshold, model_hash)
directly.
model may be any object with embed(texts) -> list[list[float]] and identity() -> str; matra.types.Embedder is the protocol ● observedtypes.py:305.
A module-level semantic_clusters(embeddings, threshold, model_hash)
clusters vectors the caller already holds.
Clusters are connected components.If a restates b and b restates c, all three are one cluster even when a and c did not clear the threshold together ● observeddomain.rs:95-145.
Co-membership is not pairwise similarity; the edges say which pairs
actually cleared it.
A sentence with no edge is in no cluster.Singletons are excluded, so "in no cluster" is a meaningful count ● observeddomain.rs:95-145.
The threshold is yours. Published paraphrase cutoffs run from 0.67 to
0.9 with no consensus, so matra has no default; every call takes it as an
argument.
An edge is kept when its score is at or above it ● observedextraction/semantic.rs:1-155.
Scores belong to one model. Compare clusters only across results with
the same model_hash.
src/domain.rs lines 95 to 145, at 4fcfb4a
95 /// Semantic-similarity clusters over a document's sentences: Tier 2
96 /// output, standing alone, never attached to [`Document`] (RFC-0010).
97 ///
98 /// Clusters are connected components of the similarity graph whose edges
99 /// cleared `threshold`, so co-membership is transitive: two sentences can
100 /// share a cluster without sharing an edge, and co-membership must never
101 /// be read as pairwise similarity. The above-threshold edges travel in
102 /// each cluster precisely so a consumer can see which pairs actually
103 /// cleared the bar; a missing edge means no claim, not low similarity.
104 ///
105 /// A sentence with no above-threshold edge appears in no cluster:
106 /// singletons are excluded by construction, so "not in any cluster" is a
107 /// meaningful consumer count, not an artifact.
108 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
109 #[non_exhaustive]
110 pub struct SemanticClusters {
111 /// Identity of the model whose geometry produced the scores, as the
112 /// embed adapter reports it (a digest over the model artifacts).
113 pub model_hash: String,
114 /// The caller-supplied similarity cutoff the edges cleared. matra
115 /// knows no universal threshold; the literature's published values
116 /// span 0.67 to 0.9 with no consensus.
117 pub threshold: f32,
118 /// The clusters, ordered by smallest member index; each cluster's
119 /// members and edges are sorted, so output order is deterministic.
120 pub clusters: Vec<SemanticCluster>,
121 }
122123 /// One connected component of the above-threshold similarity graph.
124 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
125 #[non_exhaustive]
126 pub struct SemanticCluster {
127 /// Sentence indices (document order positions) in this component.
128 pub members: Vec<usize>,
129 /// Every pair whose cosine similarity cleared the threshold. Pairs of
130 /// members without an edge here did not clear it individually; they
131 /// share the cluster through the chain.
132 pub edges: Vec<SemanticEdge>,
133 }
134135 /// An above-threshold similarity between two sentences, `a < b`.
136 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
137 #[non_exhaustive]
138 pub struct SemanticEdge {
139 /// Lower sentence index.
140 pub a: usize,
141 /// Higher sentence index.
142 pub b: usize,
143 /// Cosine similarity, in the model's geometry.
144 pub score: f32,
145 }
95 /// Semantic-similarity clusters over a document's sentences: Tier 2
96 /// output, standing alone, never attached to [`Document`] (RFC-0010).
97 ///
98 /// Clusters are connected components of the similarity graph whose edges
99 /// cleared `threshold`, so co-membership is transitive: two sentences can
100 /// share a cluster without sharing an edge, and co-membership must never
101 /// be read as pairwise similarity. The above-threshold edges travel in
102 /// each cluster precisely so a consumer can see which pairs actually
103 /// cleared the bar; a missing edge means no claim, not low similarity.
104 ///
105 /// A sentence with no above-threshold edge appears in no cluster:
106 /// singletons are excluded by construction, so "not in any cluster" is a
107 /// meaningful consumer count, not an artifact.
108 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
109 #[non_exhaustive]
110 pub struct SemanticClusters {
111 /// Identity of the model whose geometry produced the scores, as the
112 /// embed adapter reports it (a digest over the model artifacts).
113 pub model_hash: String,
114 /// The caller-supplied similarity cutoff the edges cleared. matra
115 /// knows no universal threshold; the literature's published values
116 /// span 0.67 to 0.9 with no consensus.
117 pub threshold: f32,
118 /// The clusters, ordered by smallest member index; each cluster's
119 /// members and edges are sorted, so output order is deterministic.
120 pub clusters: Vec<SemanticCluster>,
121 }
122123 /// One connected component of the above-threshold similarity graph.
124 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
125 #[non_exhaustive]
126 pub struct SemanticCluster {
127 /// Sentence indices (document order positions) in this component.
128 pub members: Vec<usize>,
129 /// Every pair whose cosine similarity cleared the threshold. Pairs of
130 /// members without an edge here did not clear it individually; they
131 /// share the cluster through the chain.
132 pub edges: Vec<SemanticEdge>,
133 }
134135 /// An above-threshold similarity between two sentences, `a < b`.
136 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
137 #[non_exhaustive]
138 pub struct SemanticEdge {
139 /// Lower sentence index.
140 pub a: usize,
141 /// Higher sentence index.
142 pub b: usize,
143 /// Cosine similarity, in the model's geometry.
144 pub score: f32,
145 }
src/extraction/semantic.rs lines 1 to 155, at 4fcfb4a
1 //! Semantic-equivalence clustering over precomputed sentence embeddings.
2 //!
3 //! A pure function over domain values (rule 5): it never calls the embed
4 //! port. Whoever holds both the `Document` and an `Embedder` runs one,
5 //! then the other; the composition root owns that pairing. Everything
6 //! here is a total computation over the vectors it is handed.
7 8 use crate::domain::{Embedding, Error, Result, SemanticCluster, SemanticClusters, SemanticEdge};
9 10 // The O(n^2) cap: literally TextRank's bound, not a copy of it, so the
11 // two cannot drift apart.
12 use super::textrank::MAX_SENTENCES;
13 14 /// Cluster texts whose embedding cosine similarity clears `threshold`.
15 ///
16 /// Indices in the result are positions in `embeddings`; the caller knows
17 /// what each vector stands for (the composition root's
18 /// `embed_and_cluster` pairs them with a document's sentences and
19 /// validates the correspondence before calling here).
20 ///
21 /// Clusters are connected components of the above-threshold graph, with
22 /// the clearing edges reported alongside the members (see
23 /// [`SemanticClusters`] for what co-membership does and does not mean).
24 /// `model_hash` is the identity of the model that produced `embeddings`,
25 /// as the embed adapter reports it; it travels into the result so the
26 /// scores stay attributable to a geometry.
27 ///
28 /// A zero-magnitude embedding (an empty sentence embeds to zero) has no
29 /// defined cosine with anything, so it gets no edges and lands in no
30 /// cluster: no claim, rather than a fabricated zero score.
31 ///
32 /// # Errors
33 ///
34 /// [`Error::InvalidInput`] when the embeddings disagree on dimension,
35 /// contain a non-finite value, or when `threshold` is not finite.
36 /// [`Error::InputTooLarge`] (what = `"semantic_clusters"`) past the
37 /// O(n^2) cap.
38 pub fn semantic_clusters(
39 embeddings: &[Embedding],
40 threshold: f32,
41 model_hash: &str,
42 ) -> Result<SemanticClusters> {
43 if !threshold.is_finite() {
44 return Err(Error::InvalidInput(format!(
45 "threshold must be finite, got {threshold}"
46 )));
47 }
48 if embeddings.len() > MAX_SENTENCES {
49 return Err(Error::InputTooLarge {
50 limit: MAX_SENTENCES,
51 actual: embeddings.len(),
52 what: "semantic_clusters",
53 });
54 }
55 let dim = embeddings.first().map_or(0, |e| e.0.len());
56 for (i, e) in embeddings.iter().enumerate() {
57 if e.0.len() != dim {
58 return Err(Error::InvalidInput(format!(
59 "embedding {i} has dimension {}, expected {dim}",
60 e.0.len()
61 )));
62 }
63 // A zero vector is a real geometry (documented above); a NaN or
64 // infinity is not a geometry at all, and letting it through
65 // would silently park its sentence in the unclustered count.
66 if !e.0.iter().all(|v| v.is_finite()) {
67 return Err(Error::InvalidInput(format!(
68 "embedding {i} contains a non-finite value"
69 )));
70 }
71 }
72 73 let n = embeddings.len();
74 let norms: Vec<f32> = embeddings
75 .iter()
76 .map(|e| e.0.iter().map(|v| v * v).sum::<f32>().sqrt())
77 .collect();
78 79 // Above-threshold edges, and union-find over them. The visited
80 // structure is the component id table, so the walk is cycle-safe by
81 // construction rather than by depth ceiling.
82 let mut parent: Vec<usize> = (0..n).collect();
83 fn find(parent: &mut [usize], mut x: usize) -> usize {
84 while parent[x] != x {
85 parent[x] = parent[parent[x]];
86 x = parent[x];
87 }
88 x
89 }
90 let mut edges: Vec<SemanticEdge> = Vec::new();
91 for a in 0..n {
92 if norms[a] == 0.0 {
93 continue;
94 }
95 for b in (a + 1)..n {
96 if norms[b] == 0.0 {
97 continue;
98 }
99 let dot: f32 = embeddings[a]
100 .0
101 .iter()
102 .zip(&embeddings[b].0)
103 .map(|(x, y)| x * y)
104 .sum();
105 let score = dot / (norms[a] * norms[b]);
106 if score >= threshold {
107 edges.push(SemanticEdge { a, b, score });
108 let (ra, rb) = (find(&mut parent, a), find(&mut parent, b));
109 if ra != rb {
110 parent[rb] = ra;
111 }
112 }
113 }
114 }
115116 // Group members by component root; only sentences with an edge are
117 // in a component of size > 1, so singletons fall out naturally.
118 let mut clusters: Vec<SemanticCluster> = Vec::new();
119 let mut root_to_cluster: Vec<Option<usize>> = vec![None; n];
120 for i in 0..n {
121 let root = find(&mut parent, i);
122 if root == i {
123 continue; // handled when a member maps to it, if any edge exists
124 }
125 let slot = match root_to_cluster[root] {
126 Some(s) => s,
127 None => {
128 clusters.push(SemanticCluster {
129 members: vec![root],
130 edges: Vec::new(),
131 });
132 root_to_cluster[root] = Some(clusters.len() - 1);
133 clusters.len() - 1
134 }
135 };
136 clusters[slot].members.push(i);
137 }
138 for edge in edges {
139 let root = find(&mut parent, edge.a);
140 if let Some(slot) = root_to_cluster[root] {
141 clusters[slot].edges.push(edge);
142 }
143 }
144 for cluster in &mut clusters {
145 cluster.members.sort_unstable();
146 cluster.edges.sort_unstable_by_key(|e| (e.a, e.b));
147 }
148 clusters.sort_unstable_by_key(|c| c.members[0]);
149150 Ok(SemanticClusters {
151 model_hash: model_hash.to_string(),
152 threshold,
153 clusters,
154 })
155 }
Embedder lives in src/embed/mod.rs, imports only domain, and requires Send● observedembed/mod.rs:1-43.
embed returns one vector per input, in
order, all of one dimension, or an error; identity returns a stable
identifier for the geometry, and two embedders that can disagree must not
share one. The name follows the agent-noun pattern of Decomposer and
Source (RFC-0010, decision 2).
src/embed/mod.rs lines 1 to 43, at 4fcfb4a
1 //! Embedding port. Defines the boundary between matra and embedding
2 //! providers.
3 //!
4 //! This module contains ONLY the port trait. The [`domain::Embedding`]
5 //! carrier lives in domain.rs. Adapters live in submodules behind feature
6 //! flags.
7 //!
8 //! Embeddings are Tier 2 output: a model's opinion about meaning, not
9 //! structure verifiable against the source bytes. RFC-0010 records the
10 //! channel discipline: nothing derived from embeddings becomes a field on
11 //! the deterministic pipeline's types.
1213 #[cfg(feature = "model2vec")]
14 pub mod model2vec;
1516 use crate::domain;
1718 /// Any embedding provider implements this. Consumers depend on this trait,
19 /// not on any specific backend.
20 ///
21 /// # Contract
22 ///
23 /// `embed` returns exactly one vector per input text, in input order, and
24 /// every returned vector has the same dimension. An implementation that
25 /// cannot honor this for some input returns an error rather than a short
26 /// or ragged result.
27 ///
28 /// Callers should be aware that `embed` is blocking and may allocate
29 /// proportionally to the batch. There is no built-in input size limit;
30 /// text that has passed through the pipeline is already bounded by
31 /// [`domain::MAX_INPUT_BYTES`], and applications embedding user-supplied
32 /// text directly should validate size before calling.
33 pub trait Embedder: Send {
34 /// Embed each text into a dense vector.
35 fn embed(&self, texts: &[&str]) -> domain::Result<Vec<domain::Embedding>>;
3637 /// A stable identifier for the model behind this embedder, carried
38 /// into every provenance-bearing result derived from its vectors.
39 /// For file-loaded models this is a digest over the artifacts; the
40 /// point is that scores stay attributable to one geometry, so two
41 /// embedders that can disagree must not share an identity.
42 fn identity(&self) -> &str;
43 }
The carrier Embedding is pub struct Embedding(pub Vec<f32>) in domain.rs, because a port may name only domain types ● observeddomain.rs:78-93.
It serializes as a bare array.
It is the one public type with public fields that is not #[non_exhaustive], deliberately: on a tuple struct the attribute makes the constructor private to the crate, and an external Embedder must construct these values ● observeddomain.rs:78-93.
RFC-0010 records the departure so review does not "fix" it.
src/domain.rs lines 78 to 93, at 4fcfb4a
78 /// A dense vector representation of a text, produced by an implementation
79 /// of the embed port's `Embedder` trait.
80 ///
81 /// Model opinion, not verifiable structure: an embedding cannot be checked
82 /// against the source bytes, so it never appears as a field on [`Document`]
83 /// or any other type the deterministic pipeline returns (RFC-0010). Values
84 /// derived from embeddings carry their provenance (model hash, parameters)
85 /// in their own standalone types.
86 ///
87 /// Serde treats the newtype transparently, so the wire form is the bare
88 /// array. Deliberately not `#[non_exhaustive]`: on a tuple struct that
89 /// attribute makes the constructor crate-private, and external `Embedder`
90 /// implementors must construct these values; RFC-0010 records the
91 /// departure from the convention.
92 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
93 pub struct Embedding(pub Vec<f32>);
78 /// A dense vector representation of a text, produced by an implementation
79 /// of the embed port's `Embedder` trait.
80 ///
81 /// Model opinion, not verifiable structure: an embedding cannot be checked
82 /// against the source bytes, so it never appears as a field on [`Document`]
83 /// or any other type the deterministic pipeline returns (RFC-0010). Values
84 /// derived from embeddings carry their provenance (model hash, parameters)
85 /// in their own standalone types.
86 ///
87 /// Serde treats the newtype transparently, so the wire form is the bare
88 /// array. Deliberately not `#[non_exhaustive]`: on a tuple struct that
89 /// attribute makes the constructor crate-private, and external `Embedder`
90 /// implementors must construct these values; RFC-0010 records the
91 /// departure from the convention.
92 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
93 pub struct Embedding(pub Vec<f32>);
Model2Vec loads model.safetensors, tokenizer.json and
config.json, and embeds by tokenizing, dropping the unknown token,
gathering rows of the matrix, applying optional per-token weights, mean
pooling, and normalizing when the config says so.
Its identity is the SHA-256 over the three files in order ● observedembed/model2vec.rs:115-140embed/model2vec.rs:598-606.
src/embed/model2vec.rs is the only file that names safetensors and
tokenizers, checked by the rule 4 analog in .semgrep/
(EP-0014).
A panic in either crate's parsing is caught and returned as Error::ModelInvalid● observedembed/model2vec.rs:1186-1209.
tokenizers is built with default-features = false and unstable_wasm, which keeps the closure pure Rust ● observedCargo.toml:61-67.
src/embed/model2vec.rs lines 115 to 140, at 4fcfb4a
115 /// A loaded static embedding model in the model2vec artifact format.
116 ///
117 /// Construct with [`Model2Vec::from_dir`]. Each artifact's bytes are
118 /// read once, hashed, and parsed from that same buffer (the read-then-
119 /// consume discipline `read_and_verify` established for UDPipe; nothing
120 /// re-reads disk between hash and load). The digest over all three
121 /// files is the model identity that provenance-carrying result types
122 /// report.
123 pub struct Model2Vec {
124 /// Row-major `[vocab, dim]` embedding matrix, converted to f32.
125 matrix: Vec<f32>,
126 dim: usize,
127 /// Optional per-token-id scale factors (tensor `"weights"`).
128 weights: Option<Vec<f32>>,
129 /// Optional token-id remap (tensor `"mapping"`): row = mapping[id].
130 mapping: Option<Vec<usize>>,
131 tokenizer: Tokenizer,
132 /// Token id of the tokenizer's unknown token, dropped before pooling.
133 unk_id: Option<u32>,
134 /// Median byte length of vocabulary keys; drives character truncation.
135 median_token_length: usize,
136 /// `normalize` from config.json (default true): L2-normalize outputs.
137 normalize: bool,
138 /// SHA-256 over matrix, tokenizer, and config bytes, lowercase hex.
139 model_hash: String,
140 }
src/embed/model2vec.rs lines 1186 to 1209, at 4fcfb4a
1186 /// Run a closure that may panic inside the safetensors or tokenizers
1187 /// parsing paths, converting any panic to [`Error::ModelInvalid`]. Both
1188 /// crates are pure Rust, so a panic here is a Rust panic on malformed
1189 /// input, not a C abort; the boundary exists because library code must
1190 /// not panic (the UDPipe precedent, same shape).
1191 fn catch_embed_panic<F, T>(f: F) -> domain::Result<T>
1192 where
1193 F: FnOnce() -> domain::Result<T>,
1194 {
1195 use std::panic::{AssertUnwindSafe, catch_unwind};
1196 match catch_unwind(AssertUnwindSafe(f)) {
1197 Ok(result) => result,
1198 Err(payload) => {
1199 let message = payload
1200 .downcast_ref::<&'static str>()
1201 .map(|s| (*s).to_string())
1202 .or_else(|| payload.downcast_ref::<String>().cloned())
1203 .unwrap_or_else(|| "model2vec panic (no message captured)".to_string());
1204 Err(Error::ModelInvalid(format!(
1205 "model2vec panicked: {message}"
1206 )))
1207 }
1208 }
1209 }
61 safetensors = { version = "0.8.0", optional = true }
62 # default-features = false drops the C regex engine (onig) and the C++
63 # suffix-array path; unstable_wasm keeps the closure pure Rust so the
64 # wasm32 target stays open (RFC-0010). The swapped regex engine is not
65 # on the code path for WordPiece tokenizers, which is what the
66 # model2vec adapter loads.
67 tokenizers = { version = "0.23.2", default-features = false, features = ["unstable_wasm"], optional = true }
Static first, transformer later, for one decisive reason and three
supporting ones (RFC-0010, decision 3): a static embedding is a table gather
and a mean, with no kernel dispatch, so its vectors are bit-identical
across targets and across bindings, which lets conformance assert exact
vectors rather than tolerances.
The quality cost is about ten percent against a small transformer at a third of the size○ assumed;
the dependency closure stays two crates; and a candle BERT adapter can
arrive later behind the same port. Those quality and size figures are
RFC-0010's, from a survey outside this repository, and are assumed here
rather than re-measured.
Model2Vec::from_dir loads a caller's directory and never touches the
network. Model2Vec::potion_base_8m and Model2Vec::from_config provision
the pinned reference model, potion-base-8M (256 dimensions), as
EPR-0003 describes.
semantic_clusters is a plain function in extraction/, over domain values, so it never calls the port (boundary rule 5) and tests without a model ● observedextraction/semantic.rs:1-155.
It refuses a non-finite threshold, a non-finite value, or vectors that disagree on dimension (InvalidInput), and more than 2000 vectors (InputTooLarge, what = "semantic_clusters"), the same O(n²) bound TextRank uses, imported rather than copied ● observedextraction/semantic.rs:1-155extraction/mod.rs:12.
It computes every pairwise cosine, keeps the pairs at
or above the threshold, joins them with union-find (cycle-safe by
construction), and returns clusters with sorted members and edges, ordered
by their first member.
A zero vector, which is what an empty sentence embeds to, has no defined cosine and gets no edge ● observedextraction/semantic.rs:1-155.
src/extraction/semantic.rs lines 1 to 155, at 4fcfb4a
1 //! Semantic-equivalence clustering over precomputed sentence embeddings.
2 //!
3 //! A pure function over domain values (rule 5): it never calls the embed
4 //! port. Whoever holds both the `Document` and an `Embedder` runs one,
5 //! then the other; the composition root owns that pairing. Everything
6 //! here is a total computation over the vectors it is handed.
7 8 use crate::domain::{Embedding, Error, Result, SemanticCluster, SemanticClusters, SemanticEdge};
9 10 // The O(n^2) cap: literally TextRank's bound, not a copy of it, so the
11 // two cannot drift apart.
12 use super::textrank::MAX_SENTENCES;
13 14 /// Cluster texts whose embedding cosine similarity clears `threshold`.
15 ///
16 /// Indices in the result are positions in `embeddings`; the caller knows
17 /// what each vector stands for (the composition root's
18 /// `embed_and_cluster` pairs them with a document's sentences and
19 /// validates the correspondence before calling here).
20 ///
21 /// Clusters are connected components of the above-threshold graph, with
22 /// the clearing edges reported alongside the members (see
23 /// [`SemanticClusters`] for what co-membership does and does not mean).
24 /// `model_hash` is the identity of the model that produced `embeddings`,
25 /// as the embed adapter reports it; it travels into the result so the
26 /// scores stay attributable to a geometry.
27 ///
28 /// A zero-magnitude embedding (an empty sentence embeds to zero) has no
29 /// defined cosine with anything, so it gets no edges and lands in no
30 /// cluster: no claim, rather than a fabricated zero score.
31 ///
32 /// # Errors
33 ///
34 /// [`Error::InvalidInput`] when the embeddings disagree on dimension,
35 /// contain a non-finite value, or when `threshold` is not finite.
36 /// [`Error::InputTooLarge`] (what = `"semantic_clusters"`) past the
37 /// O(n^2) cap.
38 pub fn semantic_clusters(
39 embeddings: &[Embedding],
40 threshold: f32,
41 model_hash: &str,
42 ) -> Result<SemanticClusters> {
43 if !threshold.is_finite() {
44 return Err(Error::InvalidInput(format!(
45 "threshold must be finite, got {threshold}"
46 )));
47 }
48 if embeddings.len() > MAX_SENTENCES {
49 return Err(Error::InputTooLarge {
50 limit: MAX_SENTENCES,
51 actual: embeddings.len(),
52 what: "semantic_clusters",
53 });
54 }
55 let dim = embeddings.first().map_or(0, |e| e.0.len());
56 for (i, e) in embeddings.iter().enumerate() {
57 if e.0.len() != dim {
58 return Err(Error::InvalidInput(format!(
59 "embedding {i} has dimension {}, expected {dim}",
60 e.0.len()
61 )));
62 }
63 // A zero vector is a real geometry (documented above); a NaN or
64 // infinity is not a geometry at all, and letting it through
65 // would silently park its sentence in the unclustered count.
66 if !e.0.iter().all(|v| v.is_finite()) {
67 return Err(Error::InvalidInput(format!(
68 "embedding {i} contains a non-finite value"
69 )));
70 }
71 }
72 73 let n = embeddings.len();
74 let norms: Vec<f32> = embeddings
75 .iter()
76 .map(|e| e.0.iter().map(|v| v * v).sum::<f32>().sqrt())
77 .collect();
78 79 // Above-threshold edges, and union-find over them. The visited
80 // structure is the component id table, so the walk is cycle-safe by
81 // construction rather than by depth ceiling.
82 let mut parent: Vec<usize> = (0..n).collect();
83 fn find(parent: &mut [usize], mut x: usize) -> usize {
84 while parent[x] != x {
85 parent[x] = parent[parent[x]];
86 x = parent[x];
87 }
88 x
89 }
90 let mut edges: Vec<SemanticEdge> = Vec::new();
91 for a in 0..n {
92 if norms[a] == 0.0 {
93 continue;
94 }
95 for b in (a + 1)..n {
96 if norms[b] == 0.0 {
97 continue;
98 }
99 let dot: f32 = embeddings[a]
100 .0
101 .iter()
102 .zip(&embeddings[b].0)
103 .map(|(x, y)| x * y)
104 .sum();
105 let score = dot / (norms[a] * norms[b]);
106 if score >= threshold {
107 edges.push(SemanticEdge { a, b, score });
108 let (ra, rb) = (find(&mut parent, a), find(&mut parent, b));
109 if ra != rb {
110 parent[rb] = ra;
111 }
112 }
113 }
114 }
115116 // Group members by component root; only sentences with an edge are
117 // in a component of size > 1, so singletons fall out naturally.
118 let mut clusters: Vec<SemanticCluster> = Vec::new();
119 let mut root_to_cluster: Vec<Option<usize>> = vec![None; n];
120 for i in 0..n {
121 let root = find(&mut parent, i);
122 if root == i {
123 continue; // handled when a member maps to it, if any edge exists
124 }
125 let slot = match root_to_cluster[root] {
126 Some(s) => s,
127 None => {
128 clusters.push(SemanticCluster {
129 members: vec![root],
130 edges: Vec::new(),
131 });
132 root_to_cluster[root] = Some(clusters.len() - 1);
133 clusters.len() - 1
134 }
135 };
136 clusters[slot].members.push(i);
137 }
138 for edge in edges {
139 let root = find(&mut parent, edge.a);
140 if let Some(slot) = root_to_cluster[root] {
141 clusters[slot].edges.push(edge);
142 }
143 }
144 for cluster in &mut clusters {
145 cluster.members.sort_unstable();
146 cluster.edges.sort_unstable_by_key(|e| (e.a, e.b));
147 }
148 clusters.sort_unstable_by_key(|c| c.members[0]);
149150 Ok(SemanticClusters {
151 model_hash: model_hash.to_string(),
152 threshold,
153 clusters,
154 })
155 }
src/extraction/semantic.rs lines 1 to 155, at 4fcfb4a
1 //! Semantic-equivalence clustering over precomputed sentence embeddings.
2 //!
3 //! A pure function over domain values (rule 5): it never calls the embed
4 //! port. Whoever holds both the `Document` and an `Embedder` runs one,
5 //! then the other; the composition root owns that pairing. Everything
6 //! here is a total computation over the vectors it is handed.
7 8 use crate::domain::{Embedding, Error, Result, SemanticCluster, SemanticClusters, SemanticEdge};
9 10 // The O(n^2) cap: literally TextRank's bound, not a copy of it, so the
11 // two cannot drift apart.
12 use super::textrank::MAX_SENTENCES;
13 14 /// Cluster texts whose embedding cosine similarity clears `threshold`.
15 ///
16 /// Indices in the result are positions in `embeddings`; the caller knows
17 /// what each vector stands for (the composition root's
18 /// `embed_and_cluster` pairs them with a document's sentences and
19 /// validates the correspondence before calling here).
20 ///
21 /// Clusters are connected components of the above-threshold graph, with
22 /// the clearing edges reported alongside the members (see
23 /// [`SemanticClusters`] for what co-membership does and does not mean).
24 /// `model_hash` is the identity of the model that produced `embeddings`,
25 /// as the embed adapter reports it; it travels into the result so the
26 /// scores stay attributable to a geometry.
27 ///
28 /// A zero-magnitude embedding (an empty sentence embeds to zero) has no
29 /// defined cosine with anything, so it gets no edges and lands in no
30 /// cluster: no claim, rather than a fabricated zero score.
31 ///
32 /// # Errors
33 ///
34 /// [`Error::InvalidInput`] when the embeddings disagree on dimension,
35 /// contain a non-finite value, or when `threshold` is not finite.
36 /// [`Error::InputTooLarge`] (what = `"semantic_clusters"`) past the
37 /// O(n^2) cap.
38 pub fn semantic_clusters(
39 embeddings: &[Embedding],
40 threshold: f32,
41 model_hash: &str,
42 ) -> Result<SemanticClusters> {
43 if !threshold.is_finite() {
44 return Err(Error::InvalidInput(format!(
45 "threshold must be finite, got {threshold}"
46 )));
47 }
48 if embeddings.len() > MAX_SENTENCES {
49 return Err(Error::InputTooLarge {
50 limit: MAX_SENTENCES,
51 actual: embeddings.len(),
52 what: "semantic_clusters",
53 });
54 }
55 let dim = embeddings.first().map_or(0, |e| e.0.len());
56 for (i, e) in embeddings.iter().enumerate() {
57 if e.0.len() != dim {
58 return Err(Error::InvalidInput(format!(
59 "embedding {i} has dimension {}, expected {dim}",
60 e.0.len()
61 )));
62 }
63 // A zero vector is a real geometry (documented above); a NaN or
64 // infinity is not a geometry at all, and letting it through
65 // would silently park its sentence in the unclustered count.
66 if !e.0.iter().all(|v| v.is_finite()) {
67 return Err(Error::InvalidInput(format!(
68 "embedding {i} contains a non-finite value"
69 )));
70 }
71 }
72 73 let n = embeddings.len();
74 let norms: Vec<f32> = embeddings
75 .iter()
76 .map(|e| e.0.iter().map(|v| v * v).sum::<f32>().sqrt())
77 .collect();
78 79 // Above-threshold edges, and union-find over them. The visited
80 // structure is the component id table, so the walk is cycle-safe by
81 // construction rather than by depth ceiling.
82 let mut parent: Vec<usize> = (0..n).collect();
83 fn find(parent: &mut [usize], mut x: usize) -> usize {
84 while parent[x] != x {
85 parent[x] = parent[parent[x]];
86 x = parent[x];
87 }
88 x
89 }
90 let mut edges: Vec<SemanticEdge> = Vec::new();
91 for a in 0..n {
92 if norms[a] == 0.0 {
93 continue;
94 }
95 for b in (a + 1)..n {
96 if norms[b] == 0.0 {
97 continue;
98 }
99 let dot: f32 = embeddings[a]
100 .0
101 .iter()
102 .zip(&embeddings[b].0)
103 .map(|(x, y)| x * y)
104 .sum();
105 let score = dot / (norms[a] * norms[b]);
106 if score >= threshold {
107 edges.push(SemanticEdge { a, b, score });
108 let (ra, rb) = (find(&mut parent, a), find(&mut parent, b));
109 if ra != rb {
110 parent[rb] = ra;
111 }
112 }
113 }
114 }
115116 // Group members by component root; only sentences with an edge are
117 // in a component of size > 1, so singletons fall out naturally.
118 let mut clusters: Vec<SemanticCluster> = Vec::new();
119 let mut root_to_cluster: Vec<Option<usize>> = vec![None; n];
120 for i in 0..n {
121 let root = find(&mut parent, i);
122 if root == i {
123 continue; // handled when a member maps to it, if any edge exists
124 }
125 let slot = match root_to_cluster[root] {
126 Some(s) => s,
127 None => {
128 clusters.push(SemanticCluster {
129 members: vec![root],
130 edges: Vec::new(),
131 });
132 root_to_cluster[root] = Some(clusters.len() - 1);
133 clusters.len() - 1
134 }
135 };
136 clusters[slot].members.push(i);
137 }
138 for edge in edges {
139 let root = find(&mut parent, edge.a);
140 if let Some(slot) = root_to_cluster[root] {
141 clusters[slot].edges.push(edge);
142 }
143 }
144 for cluster in &mut clusters {
145 cluster.members.sort_unstable();
146 cluster.edges.sort_unstable_by_key(|e| (e.a, e.b));
147 }
148 clusters.sort_unstable_by_key(|c| c.members[0]);
149150 Ok(SemanticClusters {
151 model_hash: model_hash.to_string(),
152 threshold,
153 clusters,
154 })
155 }
src/extraction/semantic.rs lines 1 to 155, at 4fcfb4a
1 //! Semantic-equivalence clustering over precomputed sentence embeddings.
2 //!
3 //! A pure function over domain values (rule 5): it never calls the embed
4 //! port. Whoever holds both the `Document` and an `Embedder` runs one,
5 //! then the other; the composition root owns that pairing. Everything
6 //! here is a total computation over the vectors it is handed.
7 8 use crate::domain::{Embedding, Error, Result, SemanticCluster, SemanticClusters, SemanticEdge};
9 10 // The O(n^2) cap: literally TextRank's bound, not a copy of it, so the
11 // two cannot drift apart.
12 use super::textrank::MAX_SENTENCES;
13 14 /// Cluster texts whose embedding cosine similarity clears `threshold`.
15 ///
16 /// Indices in the result are positions in `embeddings`; the caller knows
17 /// what each vector stands for (the composition root's
18 /// `embed_and_cluster` pairs them with a document's sentences and
19 /// validates the correspondence before calling here).
20 ///
21 /// Clusters are connected components of the above-threshold graph, with
22 /// the clearing edges reported alongside the members (see
23 /// [`SemanticClusters`] for what co-membership does and does not mean).
24 /// `model_hash` is the identity of the model that produced `embeddings`,
25 /// as the embed adapter reports it; it travels into the result so the
26 /// scores stay attributable to a geometry.
27 ///
28 /// A zero-magnitude embedding (an empty sentence embeds to zero) has no
29 /// defined cosine with anything, so it gets no edges and lands in no
30 /// cluster: no claim, rather than a fabricated zero score.
31 ///
32 /// # Errors
33 ///
34 /// [`Error::InvalidInput`] when the embeddings disagree on dimension,
35 /// contain a non-finite value, or when `threshold` is not finite.
36 /// [`Error::InputTooLarge`] (what = `"semantic_clusters"`) past the
37 /// O(n^2) cap.
38 pub fn semantic_clusters(
39 embeddings: &[Embedding],
40 threshold: f32,
41 model_hash: &str,
42 ) -> Result<SemanticClusters> {
43 if !threshold.is_finite() {
44 return Err(Error::InvalidInput(format!(
45 "threshold must be finite, got {threshold}"
46 )));
47 }
48 if embeddings.len() > MAX_SENTENCES {
49 return Err(Error::InputTooLarge {
50 limit: MAX_SENTENCES,
51 actual: embeddings.len(),
52 what: "semantic_clusters",
53 });
54 }
55 let dim = embeddings.first().map_or(0, |e| e.0.len());
56 for (i, e) in embeddings.iter().enumerate() {
57 if e.0.len() != dim {
58 return Err(Error::InvalidInput(format!(
59 "embedding {i} has dimension {}, expected {dim}",
60 e.0.len()
61 )));
62 }
63 // A zero vector is a real geometry (documented above); a NaN or
64 // infinity is not a geometry at all, and letting it through
65 // would silently park its sentence in the unclustered count.
66 if !e.0.iter().all(|v| v.is_finite()) {
67 return Err(Error::InvalidInput(format!(
68 "embedding {i} contains a non-finite value"
69 )));
70 }
71 }
72 73 let n = embeddings.len();
74 let norms: Vec<f32> = embeddings
75 .iter()
76 .map(|e| e.0.iter().map(|v| v * v).sum::<f32>().sqrt())
77 .collect();
78 79 // Above-threshold edges, and union-find over them. The visited
80 // structure is the component id table, so the walk is cycle-safe by
81 // construction rather than by depth ceiling.
82 let mut parent: Vec<usize> = (0..n).collect();
83 fn find(parent: &mut [usize], mut x: usize) -> usize {
84 while parent[x] != x {
85 parent[x] = parent[parent[x]];
86 x = parent[x];
87 }
88 x
89 }
90 let mut edges: Vec<SemanticEdge> = Vec::new();
91 for a in 0..n {
92 if norms[a] == 0.0 {
93 continue;
94 }
95 for b in (a + 1)..n {
96 if norms[b] == 0.0 {
97 continue;
98 }
99 let dot: f32 = embeddings[a]
100 .0
101 .iter()
102 .zip(&embeddings[b].0)
103 .map(|(x, y)| x * y)
104 .sum();
105 let score = dot / (norms[a] * norms[b]);
106 if score >= threshold {
107 edges.push(SemanticEdge { a, b, score });
108 let (ra, rb) = (find(&mut parent, a), find(&mut parent, b));
109 if ra != rb {
110 parent[rb] = ra;
111 }
112 }
113 }
114 }
115116 // Group members by component root; only sentences with an edge are
117 // in a component of size > 1, so singletons fall out naturally.
118 let mut clusters: Vec<SemanticCluster> = Vec::new();
119 let mut root_to_cluster: Vec<Option<usize>> = vec![None; n];
120 for i in 0..n {
121 let root = find(&mut parent, i);
122 if root == i {
123 continue; // handled when a member maps to it, if any edge exists
124 }
125 let slot = match root_to_cluster[root] {
126 Some(s) => s,
127 None => {
128 clusters.push(SemanticCluster {
129 members: vec![root],
130 edges: Vec::new(),
131 });
132 root_to_cluster[root] = Some(clusters.len() - 1);
133 clusters.len() - 1
134 }
135 };
136 clusters[slot].members.push(i);
137 }
138 for edge in edges {
139 let root = find(&mut parent, edge.a);
140 if let Some(slot) = root_to_cluster[root] {
141 clusters[slot].edges.push(edge);
142 }
143 }
144 for cluster in &mut clusters {
145 cluster.members.sort_unstable();
146 cluster.edges.sort_unstable_by_key(|e| (e.a, e.b));
147 }
148 clusters.sort_unstable_by_key(|c| c.members[0]);
149150 Ok(SemanticClusters {
151 model_hash: model_hash.to_string(),
152 threshold,
153 clusters,
154 })
155 }
embed_and_cluster in the composition root holds both halves: it collects the document's sentence texts, refuses more than the cap before running the embedder, embeds, checks the embedder returned one vector per sentence, and clusters with embedder.identity() as the model_hash● observedlib.rs:360-399.
Sentence text has already passed the pipeline's size cap in
annotate, so no second byte cap applies.
src/lib.rs lines 360 to 399, at 4fcfb4a
360 /// Embed a document's sentences, then cluster them: the one place both
361 /// halves of the semantic pipeline meet (rule 7). The embedder's own
362 /// [`identity`](embed::Embedder::identity) travels into the result, so
363 /// the scores cannot be attributed to a model that did not produce them.
364 ///
365 /// Sentence text reaching the embedder already passed the pipeline's
366 /// size cap on its way through `annotate`; no second cap applies here.
367 ///
368 /// # Errors
369 ///
370 /// Whatever the embedder returns, plus [`domain::Error::InvalidInput`]
371 /// if the embedder violates its length contract, plus everything
372 /// [`extraction::semantic_clusters`] rejects.
373 pub fn embed_and_cluster(
374 doc: &Document,
375 embedder: &dyn embed::Embedder,
376 threshold: f32,
377 ) -> domain::Result<domain::SemanticClusters> {
378 let texts: Vec<&str> = doc.sentences().map(|s| s.text.as_str()).collect();
379 // The count cap fires before the expensive stage, not after it: an
380 // over-cap document must not run a full embedding pass just to be
381 // rejected by the clustering gate. The same check inside
382 // `extraction::semantic_clusters` still guards the vectors-in path.
383 if texts.len() > extraction::MAX_SEMANTIC_SENTENCES {
384 return Err(domain::Error::InputTooLarge {
385 limit: extraction::MAX_SEMANTIC_SENTENCES,
386 actual: texts.len(),
387 what: "semantic_clusters",
388 });
389 }
390 let embeddings = embedder.embed(&texts)?;
391 if embeddings.len() != texts.len() {
392 return Err(domain::Error::InvalidInput(format!(
393 "embedder returned {} vectors for {} sentences, violating its contract",
394 embeddings.len(),
395 texts.len()
396 )));
397 }
398 extraction::semantic_clusters(&embeddings, threshold, embedder.identity())
399 }
SemanticClusters carries model_hash, threshold and
clusters; each SemanticCluster carries members and
edges; each SemanticEdge carries a < b and score. All three are
#[non_exhaustive] and serialize as plain fields.
Matra.semantic_clusters uses the built-in adapter directly when given a Model2Vec, and otherwise wraps the object in PyEmbedder, which reads identity() once at construction (so one result cannot carry two identities), calls embed under the GIL, narrows each float to f32, and turns a Python exception or a malformed return into Error::InvalidInput, which raises ValueError● observedlib.rs:601-636lib.rs:890-981.
The wrap happens before the parse, so an object that cannot name its geometry costs one method call, not a document's parse ● observedlib.rs:601-636.
src/lib.rs lines 601 to 636, at 4fcfb4a
601 /// Parse plain text, embed its sentences with `model`, and
602 /// cluster them at `threshold`. Returns the serialized
603 /// `SemanticClusters` as a dict.
604 ///
605 /// `model` is any Python object with `embed` and `identity`,
606 /// `Model2Vec` included. Anything that is not the built-in
607 /// adapter is wrapped in [`PyEmbedder`], which needs only the
608 /// `embed` port, so a build without the `model2vec` feature
609 /// still accepts a caller's own embedder.
610 ///
611 /// The embedder is settled before the parse runs. Wrapping reads
612 /// `identity()`, and an object that cannot name its geometry is
613 /// refused there; doing that first costs one method call instead
614 /// of a whole document's parse.
615 fn semantic_clusters<'py>(
616 &self,
617 py: Python<'py>,
618 text: &str,
619 threshold: f32,
620 model: &Bound<'py, PyAny>,
621 ) -> PyResult<Bound<'py, PyAny>> {
622 // Fast path: the built-in adapter is used directly, so the
623 // embedding route makes no Python call at all.
624 #[cfg(feature = "model2vec")]
625 if let Ok(loaded) = model.cast::<Model2Vec>() {
626 let doc = self.annotated(text)?;
627 let result = crate::embed_and_cluster(&doc, &loaded.get().inner, threshold)
628 .map_err(MatraError)?;
629 return to_dict(py, &result);
630 }
631 let embedder = PyEmbedder::new(model)?;
632 let doc = self.annotated(text)?;
633 let result =
634 crate::embed_and_cluster(&doc, &embedder, threshold).map_err(MatraError)?;
635 to_dict(py, &result)
636 }
890 /// A Python object standing in for an [`Embedder`](crate::embed::Embedder).
891 ///
892 /// The port is a trait, so the extension point was already there;
893 /// this is the adapter that lets a Python object reach it. A caller
894 /// who has embeddings from a service, a cache, or a library matra
895 /// does not ship passes an object with `embed` and `identity` and
896 /// gets the same clustering the built-in adapter gets.
897 ///
898 /// [`Embedder`](crate::embed::Embedder) requires `Send`. The object
899 /// is held as `Py<PyAny>`, which is `Send` because it carries no GIL
900 /// attachment: it is a reference the interpreter owns, not a pointer
901 /// into interpreter state. Every access below re-attaches with
902 /// [`Python::attach`] before touching it, so moving this value
903 /// between threads never moves an attachment with it.
904 ///
905 /// The identity is read once, at construction. The trait's promise
906 /// is that an identity names one geometry, and a Python object could
907 /// return a different string on every call; reading it once keeps
908 /// the promise on the object's behalf rather than trusting it.
909 struct PyEmbedder {
910 obj: Py<PyAny>,
911 identity: String,
912 }
913914 impl PyEmbedder {
915 /// Wrap `obj`, reading its `identity()` now.
916 ///
917 /// # Errors
918 ///
919 /// `ValueError` if `identity()` is missing, raises, or returns
920 /// something other than a string.
921 fn new(obj: &Bound<'_, PyAny>) -> PyResult<Self> {
922 let raw = obj.call_method0("identity").map_err(|e| {
923 MatraError(domain::Error::InvalidInput(format!(
924 "calling identity() on the embedder failed: {e}"
925 )))
926 })?;
927 let identity = raw.extract::<String>().map_err(|e| {
928 MatraError(domain::Error::InvalidInput(format!(
929 "the embedder's identity() must return a str: {e}"
930 )))
931 })?;
932 Ok(Self {
933 obj: obj.clone().unbind(),
934 identity,
935 })
936 }
937 }
938939 impl crate::embed::Embedder for PyEmbedder {
940 fn embed(&self, texts: &[&str]) -> domain::Result<Vec<domain::Embedding>> {
941 Python::attach(|py| {
942 let returned = self
943 .obj
944 .bind(py)
945 .call_method1("embed", (texts.to_vec(),))
946 .map_err(|e| {
947 domain::Error::InvalidInput(format!("the embedder's embed() raised {e}"))
948 })?;
949 // Two stages rather than one `Vec<Vec<f64>>` extraction,
950 // so the message names which level of the shape was
951 // wrong and which vector went astray.
952 let rows = returned.extract::<Vec<Bound<'_, PyAny>>>().map_err(|e| {
953 domain::Error::InvalidInput(format!(
954 "the embedder's embed() must return a sequence of vectors: {e}"
955 ))
956 })?;
957 rows.iter()
958 .enumerate()
959 .map(|(i, row)| {
960 // Python floats are f64 and matra's carrier is
961 // f32, so every vector narrows here. The count
962 // and dimension contract is checked once, in
963 // `embed_and_cluster` and `semantic_clusters`,
964 // and is not restated here.
965 let values = row.extract::<Vec<f64>>().map_err(|e| {
966 domain::Error::InvalidInput(format!(
967 "the embedder's embed() returned a vector at index {i} that is not a sequence of numbers: {e}"
968 ))
969 })?;
970 Ok(domain::Embedding(
971 values.into_iter().map(|v| v as f32).collect(),
972 ))
973 })
974 .collect()
975 })
976 }
977978 fn identity(&self) -> &str {
979 &self.identity
980 }
981 }
601 /// Parse plain text, embed its sentences with `model`, and
602 /// cluster them at `threshold`. Returns the serialized
603 /// `SemanticClusters` as a dict.
604 ///
605 /// `model` is any Python object with `embed` and `identity`,
606 /// `Model2Vec` included. Anything that is not the built-in
607 /// adapter is wrapped in [`PyEmbedder`], which needs only the
608 /// `embed` port, so a build without the `model2vec` feature
609 /// still accepts a caller's own embedder.
610 ///
611 /// The embedder is settled before the parse runs. Wrapping reads
612 /// `identity()`, and an object that cannot name its geometry is
613 /// refused there; doing that first costs one method call instead
614 /// of a whole document's parse.
615 fn semantic_clusters<'py>(
616 &self,
617 py: Python<'py>,
618 text: &str,
619 threshold: f32,
620 model: &Bound<'py, PyAny>,
621 ) -> PyResult<Bound<'py, PyAny>> {
622 // Fast path: the built-in adapter is used directly, so the
623 // embedding route makes no Python call at all.
624 #[cfg(feature = "model2vec")]
625 if let Ok(loaded) = model.cast::<Model2Vec>() {
626 let doc = self.annotated(text)?;
627 let result = crate::embed_and_cluster(&doc, &loaded.get().inner, threshold)
628 .map_err(MatraError)?;
629 return to_dict(py, &result);
630 }
631 let embedder = PyEmbedder::new(model)?;
632 let doc = self.annotated(text)?;
633 let result =
634 crate::embed_and_cluster(&doc, &embedder, threshold).map_err(MatraError)?;
635 to_dict(py, &result)
636 }
"No network in the library" for the embedding model.EP-0009 and RFC-0010 decision
6 as first written. RFC-0011
amended it to "no unpinned network"; the reference model now downloads
from pinned URLs against a pinned digest
(EPR-0003).
A configured default threshold. EP-0009's surface sketch and
RFC-0011 gave Config a semantic threshold.
RFC-0020 deprecated
it because no call read it; every clustering call takes the threshold as
an argument.
semantic_clusters taking the sentence slice. EP-0009's first sketch
passed the sentences too; RFC-0010's M5 amendment removed them, since only
their count was read, and moved the length check to
embed_and_cluster.
A quality ceiling, chosen. Paraphrases a transformer would catch near
the threshold are missed (RFC-0010; the size of the gap is assumed from
its survey).
Embedding is public surface although most callers only see
SemanticClusters, because the port needs a domain carrier.
The format is a third party's. A change to the model2vec artifact
format would surface as a digest mismatch or a load failure, not
silently, but would need adapter work.
The exact reference-model check is not in CI. The bit-identity test
in CI uses a constructed model; drift specific to the real artifact would
show only when someone runs just conformance.
Quadratic in sentences. Clustering compares every pair, which is why
the 2000-sentence cap exists.
These are the alternatives RFC-0010 and EP-0009 weighed.
A candle BERT adapter first: viable and verified for wasm32, but
kernel dispatch costs bit-parity and the closure is a whole inference
stack. Designated as the second adapter, behind the same port.
Depending on the existing model2vec Rust crate: rejected for its
closure (command-line dependencies as library dependencies, version skew,
licence metadata that trips scanning). The adapter is matra's own.
ONNX Runtime (ort, fastembed): C FFI, which would close the wasm32
path for every downstream caller.
A feature named embeddings: rejected because a feature gates an
adapter and so carries the adapter's name, on the udpipe precedent; a
capability name would collide with the next adapter's flag.
Cliques instead of connected components: would split chained
restatement, which is the pattern callers look for. The cost of
components, transitive co-membership, is made visible by the edges.
Semantic results as fields on Document: rejected by decision 1,
above; it would put model output behind a surface whose every other value
a caller can verify.
Could this live in a caller's code? The clustering could, and the
vectors-in function makes that easy. matra ships it so the threshold, the
provenance and the component semantics are one implementation across Rust
and Python.
model2vec (Minish Lab) defines the static embedding format and publishes
potion-base-8M; its Python implementation is the parity reference.
Union-find connected components over a thresholded similarity graph is
the standard construction; matra's TextRank already builds a sentence
similarity graph of the same size.
A transformer adapter behind the same port, with a feature named for its
backend, when a caller shows the static model's ceiling costs them
clusters that matter.
A command-line form, which would need the threshold as an argument, since
configuration does not select it.
The same port used by a binding for TypeScript, which the wasm32 check
keeps open for the embedding half.