# Semantic clusters

The value semantic clustering returns, the model it embeds with, and its bounds. [Cluster sentences by meaning](../guides/semantic-clusters.md) has the calls; [What the clustering threshold does](../explanation/semantic-clusters.md) explains what a cluster means.

## The value

`clusters(&doc, &embedder, threshold)` and Python's `matra.clusters(doc, embedder, threshold)` return a `Clusters`, a standalone value and never a field on `Document`. Its JSON Schema is `spec/schema/v2/clusters.schema.json`.

```text
Clusters
  provenance   the document's, plus method "cosine" and parameters:
                 threshold   the cutoff you supplied
                 embedder    the embedder's identity: name, model, sha256, license
  clusters     each: member sentence indices
  edges        every pair that cleared the threshold
```

| Type | Field | Holds |
|---|---|---|
| `Clusters` | `provenance` | The document's `Provenance`, with `method` and `parameters`. `parameters.embedder` is the `Identity` the embedder reports, the shape `provenance.parser` has: for `Model2Vec`, `name` `model2vec`, `sha256` the digest over its three artifacts (the value its `model_hash` returns), and `model` `potion-base-8M` and `license` `MIT` when that digest is the pinned model's, `None` otherwise. `parameters.threshold` is the cosine cutoff the call was given |
| | `clusters` | The connected components, ordered by smallest member index |
| | `edges` | Every pair whose cosine cleared the threshold, ordered by `a`, then `b`; both ends lie in one cluster |
| `Cluster` | `members` | Sentence indices in document order, sorted, at least two |
| `SimilarityEdge` | `a`, `b` | The two sentence indices, `a < b` |
| | `score` | Their cosine similarity, an `f64` |

A sentence with no pair above the threshold is in no cluster. Two members of one cluster need not share an edge: they can be joined through a chain of edges.

The threshold is recorded exactly as it was passed. A score is computed in the vectors' own `f32` and widened to `f64`, so it carries more digits than the model resolved: compare scores with a tolerance, not for equality with a number you typed.

Vectors you already hold go through `clusters` with an `Embedder` that returns them, in Rust and in Python alike: the clustering always runs over a document's sentences, so its result always carries a provenance.

## The model

The `Model2Vec` adapter, behind the `model2vec` feature, loads static embedding models in the model2vec artifact format: an embedding matrix (`model.safetensors`), a `tokenizer.json`, and a `config.json` in one directory. `clusters` and the `Embedder` port are always compiled; with your own `Embedder` implementation, clustering needs no feature at all.

| Constructor | Rust | Python | Reaches the network |
|---|---|---|---|
| The configured directory | `Model2Vec::from_config(&cfg)` | `Model2Vec.from_config()`, or `Model2Vec.from_config(config_file)` | when the pinned model is absent |
| The directory the environment names, or the default one; no config file read | `Model2Vec::potion_base_8m(dir)` with the directory you resolve | `Model2Vec.potion_base_8m()` | when the pinned model is absent |
| A models directory you name, the model in its `potion-base-8M` subdirectory | `Model2Vec::potion_base_8m(dir)` | `Model2Vec.potion_base_8m(dir)` | when the pinned model is absent |
| Files you supply | `Model2Vec::from_dir(dir)` | `Model2Vec.from_dir(dir)` | never |

The configured directory is `models.embedding` (`potion-base-8M` as shipped) inside the configured model directory.

### The reference model

[potion-base-8M](https://huggingface.co/minishlab/potion-base-8M), 30.9 MB in three files, MIT per its model card. The SHA-256 over its three files, concatenated in the order above, is a constant compiled into the library:

```text
81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4
```

It is the digest `spec/tests/semantic/reference-model.json` pins and the value `model_hash` reports once the model is loaded. For files loaded with `from_dir`, `model_hash` is identity rather than verification: it names the artifacts that produced a score, and says nothing about whether they are the ones you meant to fetch.

### Provisioning

- The three artifacts are verified as a set, in memory, before anything is parsed. A mismatch downloads once more without writing anything; a second mismatch raises with the directory as the call found it.
- Downloading happens only into a directory holding none of the three file names. If all three are there and the digest does not match, or only some of them are, the call raises and names the directory, having downloaded nothing and deleted nothing.
- Each artifact download is capped at 64 MiB.

### Inference

A static model is a lookup table: inference is a row gather, a mean, and a normalize, and the vectors are bit-identical on every platform and in every language binding. Every text is capped at 512 tokens, pre-truncated on bytes and then truncated on token ids, so a longer text is embedded from its opening only. An empty text embeds to the zero vector.

## Bounds and failure

| Condition | Rust | Python |
|---|---|---|
| More than 2,000 sentences, checked before the embedding pass runs | `InputTooLarge`, `what` is `"clusters"` | `matra.InputTooLargeError`, a `ValueError` |
| Vectors that disagree on dimension, a non-finite value, a non-finite threshold | `InvalidInput` | `matra.InvalidInputError`, a `ValueError` |
| An embedder returning the wrong number of vectors | `InvalidInput` | `matra.InvalidInputError`, a `ValueError` |

The 2,000 cap counts the document's sentences. The contract violations mean the call site or the embedder is wrong, never the text. A zero-magnitude vector has no defined cosine with anything, so it gets no edge at any threshold and no cluster. [Errors](errors.md) has the full table, the provisioning failures included.
