Semantic clusters
The value semantic clustering returns, the model it embeds with, and its bounds. Cluster sentences by meaning has the calls; What the clustering threshold does explains what a cluster means.
The value
clusters(&doc, &embedder, threshold) and Python's matra.clusters(doc, embedder, threshold) return a Clusters, a standalone value and never a field on Document. Its JSON Schema is spec/schema/v2/clusters.schema.json.
Clusters
provenance the document's, plus method "cosine" and parameters:
threshold the cutoff you supplied
embedder the embedder's identity: name, model, sha256, license
clusters each: member sentence indices
edges every pair that cleared the threshold
| Type | Field | Holds |
|---|---|---|
Clusters | provenance | The document's Provenance, with method and parameters. parameters.embedder is the Identity the embedder reports, the shape provenance.parser has: for Model2Vec, name model2vec, sha256 the digest over its three artifacts (the value its model_hash returns), and model potion-base-8M and license MIT when that digest is the pinned model's, None otherwise. parameters.threshold is the cosine cutoff the call was given |
clusters | The connected components, ordered by smallest member index | |
edges | Every pair whose cosine cleared the threshold, ordered by a, then b; both ends lie in one cluster | |
Cluster | members | Sentence indices in document order, sorted, at least two |
SimilarityEdge | a, b | The two sentence indices, a < b |
score | Their cosine similarity, an f64 |
A sentence with no pair above the threshold is in no cluster. Two members of one cluster need not share an edge: they can be joined through a chain of edges.
The threshold is recorded exactly as it was passed. A score is computed in the vectors' own f32 and widened to f64, so it carries more digits than the model resolved: compare scores with a tolerance, not for equality with a number you typed.
Vectors you already hold go through clusters with an Embedder that returns them, in Rust and in Python alike: the clustering always runs over a document's sentences, so its result always carries a provenance.
The model
The Model2Vec adapter, behind the model2vec feature, loads static embedding models in the model2vec artifact format: an embedding matrix (model.safetensors), a tokenizer.json, and a config.json in one directory. clusters and the Embedder port are always compiled; with your own Embedder implementation, clustering needs no feature at all.
| Constructor | Rust | Python | Reaches the network |
|---|---|---|---|
| The configured directory | Model2Vec::from_config(&cfg) | Model2Vec.from_config(), or Model2Vec.from_config(config_file) | when the pinned model is absent |
| The directory the environment names, or the default one; no config file read | Model2Vec::potion_base_8m(dir) with the directory you resolve | Model2Vec.potion_base_8m() | when the pinned model is absent |
A models directory you name, the model in its potion-base-8M subdirectory | Model2Vec::potion_base_8m(dir) | Model2Vec.potion_base_8m(dir) | when the pinned model is absent |
| Files you supply | Model2Vec::from_dir(dir) | Model2Vec.from_dir(dir) | never |
The configured directory is models.embedding (potion-base-8M as shipped) inside the configured model directory.
The reference model
potion-base-8M, 30.9 MB in three files, MIT per its model card. The SHA-256 over its three files, concatenated in the order above, is a constant compiled into the library:
81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4
It is the digest spec/tests/semantic/reference-model.json pins and the value model_hash reports once the model is loaded. For files loaded with from_dir, model_hash is identity rather than verification: it names the artifacts that produced a score, and says nothing about whether they are the ones you meant to fetch.
Provisioning
- The three artifacts are verified as a set, in memory, before anything is parsed. A mismatch downloads once more without writing anything; a second mismatch raises with the directory as the call found it.
- Downloading happens only into a directory holding none of the three file names. If all three are there and the digest does not match, or only some of them are, the call raises and names the directory, having downloaded nothing and deleted nothing.
- Each artifact download is capped at 64 MiB.
Inference
A static model is a lookup table: inference is a row gather, a mean, and a normalize, and the vectors are bit-identical on every platform and in every language binding. Every text is capped at 512 tokens, pre-truncated on bytes and then truncated on token ids, so a longer text is embedded from its opening only. An empty text embeds to the zero vector.
Bounds and failure
| Condition | Rust | Python |
|---|---|---|
| More than 2,000 sentences, checked before the embedding pass runs | InputTooLarge, what is "clusters" | matra.InputTooLargeError, a ValueError |
| Vectors that disagree on dimension, a non-finite value, a non-finite threshold | InvalidInput | matra.InvalidInputError, a ValueError |
| An embedder returning the wrong number of vectors | InvalidInput | matra.InvalidInputError, a ValueError |
The 2,000 cap counts the document's sentences. The contract violations mean the call site or the embedder is wrong, never the text. A zero-magnitude vector has no defined cosine with anything, so it gets no edge at any threshold and no cluster. Errors has the full table, the provisioning failures included.
This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 11 paragraphs, in the margin. the data