{
  "generator": {
    "matra": "0.3.0",
    "udpipe_model": {
      "name": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957"
    }
  },
  "page": "explanation/semantic-clusters.md",
  "paragraphs": [
    {
      "compression_ratio": 0.581,
      "kind": "paragraph",
      "lexical_density": 0.5714,
      "readability_grade": 7.1486,
      "sentences": 4,
      "text": "matra can group the sentences of a document that say the same thing in different words. It embeds each sentence as a vector, scores every pair by cosine similarity, and links each pair whose score clears a threshold you choose. The clusters are the connected groups those links make. Cluster sentences by meaning has the calls.",
      "words": 56
    },
    {
      "compression_ratio": 0.5673,
      "kind": "paragraph",
      "lexical_density": 0.4915,
      "readability_grade": 10.48,
      "sentences": 3,
      "text": "Everything else matra returns is structure read off the text, checkable against its bytes. Clusters are not: they depend on a model's representation of meaning. So they arrive as their own value from a separate call, never as a field on Document, and their provenance carries the identity of the model that produced them and the threshold you chose.",
      "words": 60
    },
    {
      "compression_ratio": 0.6077,
      "kind": "list_item",
      "lexical_density": 0.5472,
      "readability_grade": 7.3888,
      "sentences": 4,
      "text": "Co-membership is transitive, not pairwise. Clusters are connected components, so sentence A and sentence C can share a cluster because both resemble B, without resembling each other. The edges travel in the result so you can see which pairs actually cleared the bar. A missing edge is no claim, not a low score.",
      "words": 53
    },
    {
      "compression_ratio": null,
      "kind": "list_item",
      "lexical_density": 0.48,
      "readability_grade": 7.221,
      "sentences": 2,
      "text": "Singletons are always excluded. A sentence with no pair above the threshold appears in no cluster, so \"not in any cluster\" is a meaningful count.",
      "words": 25
    },
    {
      "compression_ratio": 0.6061,
      "kind": "list_item",
      "lexical_density": 0.4912,
      "readability_grade": 6.4525,
      "sentences": 3,
      "text": "The threshold is yours, and it does not travel. Published cutoffs for paraphrase detection span 0.67 to 0.9 with no consensus; the working value depends on the model, the domain and the text length. A cutoff calibrated on sentences is not the cutoff for whole documents, because a document vector is the mean over far more tokens.",
      "words": 57
    },
    {
      "compression_ratio": 0.5884,
      "kind": "paragraph",
      "lexical_density": 0.3929,
      "readability_grade": 3.7387,
      "sentences": 4,
      "text": "Ten short sentences, with the reference model. Before you move the threshold, predict: sentence 8 is about a different decision by the same committee. As the threshold falls from 0.85, at which value do you expect it to join sentences 1, 2 and 3, and through which of them? Then drag the threshold and find out.",
      "words": 56
    },
    {
      "compression_ratio": 0.5606,
      "kind": "paragraph",
      "lexical_density": 0.4746,
      "readability_grade": 3.212,
      "sentences": 2,
      "text": "It joins at 0.70, through sentence 1 alone: their pair scores 0.7239 and clears the bar, while its pairs with sentences 2 and 3 do not, yet it shares their cluster. A cluster can grow through a chain of pairs, on shared words (\"committee approved\") rather than shared meaning, and the clusters dissolve as the bar rises past 0.85.",
      "words": 59
    },
    {
      "compression_ratio": 0.5605,
      "kind": "paragraph",
      "lexical_density": 0.623,
      "readability_grade": 16.423,
      "sentences": 2,
      "text": "The reference model is a static embedding model, a lookup table rather than a transformer: inference is a row gather, a mean, and a normalize. That costs roughly ten percent of a small transformer's benchmark quality, and buys bit-identical vectors on every platform and in every language binding, which is what lets the conformance suite pin exact vectors rather than tolerances.",
      "words": 62
    }
  ]
}
