{
  "generator": {
    "matra": "0.3.0",
    "udpipe_model": {
      "name": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957"
    }
  },
  "page": "reference/semantic-clusters.md",
  "paragraphs": [
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5333,
      "readability_grade": 8.3533,
      "sentences": 2,
      "text": "The value semantic clustering returns, the model it embeds with, and its bounds. Cluster sentences by meaning has the calls; What the clustering threshold does explains what a cluster means.",
      "words": 30
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.68,
      "readability_grade": 11.376,
      "sentences": 2,
      "text": "clusters(&doc, &embedder, threshold) and Python's matra.clusters(doc, embedder, threshold) return a Clusters, a standalone value and never a field on Document. Its JSON Schema is spec/schema/v2/clusters.schema.json.",
      "words": 32
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4516,
      "readability_grade": 5.3002,
      "sentences": 2,
      "text": "A sentence with no pair above the threshold is in no cluster. Two members of one cluster need not share an edge: they can be joined through a chain of edges.",
      "words": 31
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.4545,
      "readability_grade": 11.4945,
      "sentences": 2,
      "text": "The threshold is recorded exactly as it was passed. A score is computed in the vectors' own f32 and widened to f64, so it carries more digits than the model resolved: compare scores with a tolerance, not for equality with a number you typed.",
      "words": 45
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5588,
      "readability_grade": 17.1053,
      "sentences": 1,
      "text": "Vectors you already hold go through clusters with an Embedder that returns them, in Rust and in Python alike: the clustering always runs over a document's sentences, so its result always carries a provenance.",
      "words": 35
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.617,
      "readability_grade": 12.1781,
      "sentences": 3,
      "text": "The Model2Vec adapter, behind the model2vec feature, loads static embedding models in the model2vec artifact format: an embedding matrix (model.safetensors), a tokenizer.json, and a config.json in one directory. clusters and the Embedder port are always compiled; with your own Embedder implementation, clustering needs no feature at all.",
      "words": 47
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.6923,
      "readability_grade": 18.7142,
      "sentences": 1,
      "text": "The configured directory is models.embedding (potion-base-8M as shipped) inside the configured model directory.",
      "words": 16
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5517,
      "readability_grade": 9.1891,
      "sentences": 2,
      "text": "potion-base-8M, 30.9 MB in three files, MIT per its model card. The SHA-256 over its three files, concatenated in the order above, is a constant compiled into the library:",
      "words": 34
    },
    {
      "compression_ratio": null,
      "kind": "paragraph",
      "lexical_density": 0.5417,
      "readability_grade": 12.0375,
      "sentences": 2,
      "text": "It is the digest spec/tests/semantic/reference-model.json pins and the value model_hash reports once the model is loaded. For files loaded with from_dir, model_hash is identity rather than verification: it names the artifacts that produced a score, and says nothing about whether they are the ones you meant to fetch.",
      "words": 50
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.5294,
      "readability_grade": 10.4753,
      "sentences": 2,
      "text": "The three artifacts are verified as a set, in memory, before anything is parsed. A mismatch downloads once more without writing anything; a second mismatch raises with the directory as the call found it.",
      "words": 34
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.4651,
      "readability_grade": 11.181,
      "sentences": 2,
      "text": "Downloading happens only into a directory holding none of the three file names. If all three are there and the digest does not match, or only some of them are, the call raises and names the directory, having downloaded nothing and deleted nothing.",
      "words": 43
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.5,
      "readability_grade": null,
      "sentences": 1,
      "text": "Each artifact download is capped at 64 MiB.",
      "words": 8
    },
    {
      "compression_ratio": 0.5836,
      "kind": "paragraph",
      "lexical_density": 0.5313,
      "readability_grade": 11.905,
      "sentences": 3,
      "text": "A static model is a lookup table: inference is a row gather, a mean, and a normalize, and the vectors are bit-identical on every platform and in every language binding. Every text is capped at 512 tokens, pre-truncated on bytes and then truncated on token ids, so a longer text is embedded from its opening only. An empty text embeds to the zero vector.",
      "words": 64
    },
    {
      "compression_ratio": 0.6547,
      "kind": "paragraph",
      "lexical_density": 0.5686,
      "readability_grade": 7.8923,
      "sentences": 4,
      "text": "The 2,000 cap counts the document's sentences. The contract violations mean the call site or the embedder is wrong, never the text. A zero-magnitude vector has no defined cosine with anything, so it gets no edge at any threshold and no cluster. Errors has the full table, the provisioning failures included.",
      "words": 54
    }
  ]
}
