Summarize a long document

on this page
grade9.2density0.58compressionnone

The three sentences TextRank ranks highest in the opening of Darwin's chapter on the struggle for existence, 2,214 words.

Input

On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859). 2,214 words.

The calls read it as origin-struggle.txt in the directory you run them from. Save it there:

curl -fLO https://mox-labs.github.io/matra/example-files/summarize/origin-struggle.txt

or download origin-struggle.txt.

The text, 10 paragraphs

Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.

Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.

We will now discuss in a little more detail the struggle for existence. In my future work this subject shall be treated, as it well deserves, at much greater length. The elder De Candolle and Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. In regard to plants, no one has treated this subject with more spirit and ability than W. Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.

I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.

A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.

There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.

But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.

In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.

The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.

In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.

The call

Rust

// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, SummaryMethod};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = Engine::new()?;
    let text = std::fs::read_to_string("origin-struggle.txt")?;
    let doc = engine.analyze(Input::text(text, Format::PlainText))?;
    // Parse once; a summary is a function of the document.
    let summary = matra::summarize_with(&doc, 3, SummaryMethod::TextRank)?;
    println!("{}", serde_json::to_string_pretty(&summary)?);
    Ok(())
}

Python

# pip install 'matra>=0.3'
import json

import matra

engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
    doc = engine.analyze(f.read())  # read as plain text
# Parse once; a summary is a function of the document.
summary = matra.summarize(doc, n=3, method="textrank")
print(json.dumps(summary, indent=2, ensure_ascii=False))

CLI

$ matra summarize --method textrank -n 3 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt

It prints the output below, inside an envelope that names the command and the input:

{
  "format_version": 2,
  "command": "summarize",
  "input": "origin-struggle.txt",
  "result": /* the output below */,
  "error": null
}

Output

Trimmed: long strings shortened.

{
  "provenance": {
    "matra": "0.3.0",
    "schema": "2.0",
    "parser": {
      "name": "udpipe",
      "model": "english-ewt-ud-2.5-191206",
      "sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
      "license": "CC BY-NC-SA 4.0 (non-commercial)"
    },
    "format": "plain_text",
    "source": {
      "path": null,
      "bytes": 12808,
      "sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
    },
    "method": "textrank",
    "parameters": {
      "n": 3
    }
  },
  "sentences": [
    {
      "index": 32,
      "span": {
        "byte_start": 6000,
        "byte_end": 6328
      },
      "text": "Every being, which during its natural lifetime produces several eggs or…",
      "score": 0.029215311359903445
    },
    {
      "index": 33,
      "span": {
        "byte_start": 6329,
        "byte_end": 6591
      },
      "text": "Hence, as more individuals are produced than can possibly survive, ther…",
      "score": 0.03081881474196398
    },
    {
      "index": 58,
      "span": {
        "byte_start": 12049,
        "byte_end": 12444
      },
      "text": "In looking at Nature, it is most necessary to keep the foregoing consid…",
      "score": 0.030089482622730077
    }
  ]
}

The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.

grade10.4density0.69compressionnone

Every sentence's score in document order, with the three the summary keeps marked:

TextRank: the summary and the scores it came from

TextRank score of each sentence, in document order0.0000.0100.0200.0300.040333459sentence 1sentence 61document order; bands alternate by paragraph
  1. 33Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product.
  2. 34Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life.
  3. 59In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals.
show the data
#ScoreSummarySentence
110.0155Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection.
210.0145It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed.
310.0182It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted.
410.0180But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature.
510.0148How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected?
610.0155We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.
720.0123Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species?
820.0089How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise?
920.0168All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life.
1020.0240Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring.
1120.0153The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive.
1220.0102I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection.
1320.0192We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature.
1420.0113But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.
1530.0114We will now discuss in a little more detail the struggle for existence.
1630.0060In my future work this subject shall be treated, as it well deserves, at much greater length.
1730.0025The elder De Candolle and
1830.0055Lyell have largely and philosophically shown that all organic beings are exposed to severe competition.
1930.0103In regard to plants, no one has treated this subject with more spirit and ability than W.
2030.0046Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge.
2130.0121Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind.
2230.0123Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood.
2330.0237We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.
2440.0202I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny.
2540.0152Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live.
2640.0161But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture.
2740.0237A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground.
2840.0121The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die.
2940.0153But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other.
3040.0216As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants.
3140.0120In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.
3250.0197A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase.
3350.0292yesEvery being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product.
3450.0308yesHence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life.
3550.0129It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage.
3650.0154Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.
3760.0142There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair.
3860.0098Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny.
3960.0173Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants.
4060.0152The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.
4170.0199But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons.
4270.0154Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible.
4370.0153So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years.
4470.0093Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery.
4570.0198In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree.
4670.0112The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed.
4770.0109In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.
4880.0213In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair.
4980.0245Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life.
5080.0205Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.
5190.0179The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large.
5290.0115The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world.
5390.0173One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district.
5490.0182A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number.
5590.0237But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one.
5690.0247If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct.
5790.0180It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place.
5890.0225So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.
59100.0301yesIn looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals.
60100.0183Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount.
61100.0120The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.
  • in the summary (3 sentences, matra's default)
  • not picked

TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.

Input: On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: origin-struggle/textrank.json.

What to notice

  • The summary carries the text's provenance with what made it: method is textrank and parameters is {"n": 3}. Each of its sentences has index, the sentence's place among the document's sentences counting from 0, its span in the input, its text and its score. The three come back in document order (indices 32, 33 and 58), not by score: the highest score, 0.0308, is the second. The figure numbers sentences from 1, so there they are 33, 34 and 59.
  • All three calls hand matra the text rather than the file, so the provenance records no path: the Python method takes text, and the three must record the same input.
  • The summary is extractive: the sentences are as Darwin wrote them, and nothing is rephrased.
  • matra has a second summarizer, TF-IDF, and it is the default: matra::summarize in Rust, and the command line's summary.method as shipped. This example names TextRank in all three calls (SummaryMethod::TextRank, method="textrank", --method textrank), so they agree whatever a configuration file says.

This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 2 paragraphs, in the margin. the data