# Summarize a long document

The three sentences TextRank ranks highest in the opening of Darwin's chapter on the struggle for existence, 2,214 words.

## Input

[On the Origin of Species, the opening of chapter III, Struggle for Existence](https://www.gutenberg.org/ebooks/1228), Charles Darwin. Public domain (first published 1859).

The calls read it as `origin-struggle.txt` in the directory they run from:

```bash
curl -fLO https://mox-labs.github.io/matra/example-files/summarize/origin-struggle.txt
```
## The call

Rust:

```rust
// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, SummaryMethod};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = Engine::new()?;
    let text = std::fs::read_to_string("origin-struggle.txt")?;
    let doc = engine.analyze(Input::text(text, Format::PlainText))?;
    // Parse once; a summary is a function of the document.
    let summary = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
    println!("{}", serde_json::to_string_pretty(&summary)?);
    Ok(())
}
```

Python:

```python
# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
    doc = engine.analyze(f.read())  # read as plain text
# Parse once; a summary is a function of the document.
summary = matra.summarize(doc, n=3, method="textrank")
print(json.dumps(summary, indent=2, ensure_ascii=False))
```

The command line, which prints the output below inside an envelope:

```bash
matra summarize --method textrank -n 3 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt
```

```jsonc
{
  "format_version": 2,
  "command": "summarize",
  "input": "origin-struggle.txt",
  "result": /* the output below */,
  "error": null
}
```
## Output

Trimmed: long strings shortened.

```jsonc
{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "plain_text",
    "source": {
      "path": null,
      "bytes": 12808,
      "sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
    },
    "method": "textrank",
    "parameters": {
      "n": 3
    }
  },
  "sentences": [
    {
      "index": 32,
      "span": {
        "byte_start": 6000,
        "byte_end": 6328
      },
      "text": "Every being, which during its natural lifetime produces several eggs or…",
      "score": 0.029215311359903445
    },
    {
      "index": 33,
      "span": {
        "byte_start": 6329,
        "byte_end": 6591
      },
      "text": "Hence, as more individuals are produced than can possibly survive, ther…",
      "score": 0.03081881474196398
    },
    {
      "index": 58,
      "span": {
        "byte_start": 12049,
        "byte_end": 12444
      },
      "text": "In looking at Nature, it is most necessary to keep the foregoing consid…",
      "score": 0.030089482622730077
    }
  ]
}
```

The whole output: https://mox-labs.github.io/matra/example-files/summarize/output.json, and what the CLI prints: https://mox-labs.github.io/matra/example-files/summarize/cli.json.
Every sentence's score in document order, with the three the summary keeps marked:

<figure-textrank input="origin-struggle" />

## What to notice

- The summary carries the text's `provenance` with what made it: `method` is `textrank` and `parameters` is `{"n": 3}`. Each of its `sentences` has `index`, the sentence's place among the document's sentences counting from 0, its `span` in the input, its `text` and its `score`. The three come back in document order (indices 32, 33 and 58), not by score: the highest score, 0.0308, is the second. The figure numbers sentences from 1, so there they are 33, 34 and 59.
- All three calls hand matra the text rather than the file, so the provenance records no `path`: the Python method takes text, and the three must record the same input.
- The summary is extractive: the sentences are as Darwin wrote them, and nothing is rephrased.
- matra has a second summarizer, TF-IDF, and it is the default: `matra::summarize` in Rust, and the command line's `summary.method` as shipped. This example names TextRank in all three calls (`SummaryMethod::TextRank`, `method="textrank"`, `--method textrank`), so they agree whatever a configuration file says.
