Extract a document's keyphrases
on this page
The ten phrases RAKE ranks highest in the same Darwin text as the summary example.
Input
The calls read it as origin-struggle.txt in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/keyphrases/origin-struggle.txt or download origin-struggle.txt.
The text, 10 paragraphs
Before entering on the subject of this chapter, I must make a few preliminary remarks, to show how the struggle for existence bears on Natural Selection. It has been seen in the last chapter that amongst organic beings in a state of nature there is some individual variability; indeed I am not aware that this has ever been disputed. It is immaterial for us whether a multitude of doubtful forms be called species or sub-species or varieties; what rank, for instance, the two or three hundred doubtful forms of British plants are entitled to hold, if the existence of any well-marked varieties be admitted. But the mere existence of individual variability and of some few well-marked varieties, though necessary as the foundation for the work, helps us but little in understanding how species arise in nature. How have all those exquisite adaptations of one part of the organisation to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected? We see these beautiful co-adaptations most plainly in the woodpecker and missletoe; and only a little less plainly in the humblest parasite which clings to the hairs of a quadruped or feathers of a bird; in the structure of the beetle which dives through the water; in the plumed seed which is wafted by the gentlest breeze; in short, we see beautiful adaptations everywhere and in every part of the organic world.
Again, it may be asked, how is it that varieties, which I have called incipient species, become ultimately converted into good and distinct species, which in most cases obviously differ from each other far more than do the varieties of the same species? How do those groups of species, which constitute what are called distinct genera, and which differ from each other more than do the species of the same genus, arise? All these results, as we shall more fully see in the next chapter, follow inevitably from the struggle for life. Owing to this struggle for life, any variation, however slight and from whatever cause proceeding, if it be in any degree profitable to an individual of any species, in its infinitely complex relations to other organic beings and to external nature, will tend to the preservation of that individual, and will generally be inherited by its offspring. The offspring, also, will thus have a better chance of surviving, for, of the many individuals of any species which are periodically born, but a small number can survive. I have called this principle, by which each slight variation, if useful, is preserved, by the term of Natural Selection, in order to mark its relation to man’s power of selection. We have seen that man by selection can certainly produce great results, and can adapt organic beings to his own uses, through the accumulation of slight but useful variations, given to him by the hand of Nature. But Natural Selection, as we shall hereafter see, is a power incessantly ready for action, and is as immeasurably superior to man’s feeble efforts, as the works of Nature are to those of Art.
We will now discuss in a little more detail the struggle for existence. In my future work this subject shall be treated, as it well deserves, at much greater length. The elder De Candolle and Lyell have largely and philosophically shown that all organic beings are exposed to severe competition. In regard to plants, no one has treated this subject with more spirit and ability than W. Herbert, Dean of Manchester, evidently the result of his great horticultural knowledge. Nothing is easier than to admit in words the truth of the universal struggle for life, or more difficult—at least I have found it so—than constantly to bear this conclusion in mind. Yet unless it be thoroughly engrained in the mind, I am convinced that the whole economy of nature, with every fact on distribution, rarity, abundance, extinction, and variation, will be dimly seen or quite misunderstood. We behold the face of nature bright with gladness, we often see superabundance of food; we do not see, or we forget, that the birds which are idly singing round us mostly live on insects or seeds, and are thus constantly destroying life; or we forget how largely these songsters, or their eggs, or their nestlings, are destroyed by birds and beasts of prey; we do not always bear in mind, that though food may be now superabundant, it is not so at all seasons of each recurring year.
I should premise that I use the term Struggle for Existence in a large and metaphorical sense, including dependence of one being on another, and including (which is more important) not only the life of the individual, but success in leaving progeny. Two canine animals in a time of dearth, may be truly said to struggle with each other which shall get food and live. But a plant on the edge of a desert is said to struggle for life against the drought, though more properly it should be said to be dependent on the moisture. A plant which annually produces a thousand seeds, of which on an average only one comes to maturity, may be more truly said to struggle with the plants of the same and other kinds which already clothe the ground. The missletoe is dependent on the apple and a few other trees, but can only in a far-fetched sense be said to struggle with these trees, for if too many of these parasites grow on the same tree, it will languish and die. But several seedling missletoes, growing close together on the same branch, may more truly be said to struggle with each other. As the missletoe is disseminated by birds, its existence depends on birds; and it may metaphorically be said to struggle with other fruit-bearing plants, in order to tempt birds to devour and thus disseminate its seeds rather than those of other plants. In these several senses, which pass into each other, I use for convenience sake the general term of struggle for existence.
A struggle for existence inevitably follows from the high rate at which all organic beings tend to increase. Every being, which during its natural lifetime produces several eggs or seeds, must suffer destruction during some period of its life, and during some season or occasional year, otherwise, on the principle of geometrical increase, its numbers would quickly become so inordinately great that no country could support the product. Hence, as more individuals are produced than can possibly survive, there must in every case be a struggle for existence, either one individual with another of the same species, or with the individuals of distinct species, or with the physical conditions of life. It is the doctrine of Malthus applied with manifold force to the whole animal and vegetable kingdoms; for in this case there can be no artificial increase of food, and no prudential restraint from marriage. Although some species may be now increasing, more or less rapidly, in numbers, all cannot do so, for the world would not hold them.
There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed, the earth would soon be covered by the progeny of a single pair. Even slow-breeding man has doubled in twenty-five years, and at this rate, in a few thousand years, there would literally not be standing room for his progeny. Linnæus has calculated that if an annual plant produced only two seeds—and there is no plant so unproductive as this—and their seedlings next year produced two, and so on, then in twenty years there would be a million plants. The elephant is reckoned to be the slowest breeder of all known animals, and I have taken some pains to estimate its probable minimum rate of natural increase: it will be under the mark to assume that it breeds when thirty years old, and goes on breeding till ninety years old, bringing forth three pair of young in this interval; if this be so, at the end of the fifth century there would be alive fifteen million elephants, descended from the first pair.
But we have better evidence on this subject than mere theoretical calculations, namely, the numerous recorded cases of the astonishingly rapid increase of various animals in a state of nature, when circumstances have been favourable to them during two or three following seasons. Still more striking is the evidence from our domestic animals of many kinds which have run wild in several parts of the world: if the statements of the rate of increase of slow-breeding cattle and horses in South America, and latterly in Australia, had not been well authenticated, they would have been quite incredible. So it is with plants: cases could be given of introduced plants which have become common throughout whole islands in a period of less than ten years. Several of the plants now most numerous over the wide plains of La Plata, clothing square leagues of surface almost to the exclusion of all other plants, have been introduced from Europe; and there are plants which now range in India, as I hear from Dr. Falconer, from Cape Comorin to the Himalaya, which have been imported from America since its discovery. In such cases, and endless instances could be given, no one supposes that the fertility of these animals or plants has been suddenly and temporarily increased in any sensible degree. The obvious explanation is that the conditions of life have been very favourable, and that there has consequently been less destruction of the old and young, and that nearly all the young have been enabled to breed. In such cases the geometrical ratio of increase, the result of which never fails to be surprising, simply explains the extraordinarily rapid increase and wide diffusion of naturalised productions in their new homes.
In a state of nature almost every plant produces seed, and amongst animals there are very few which do not annually pair. Hence we may confidently assert, that all plants and animals are tending to increase at a geometrical ratio, that all would most rapidly stock every station in which they could any how exist, and that the geometrical tendency to increase must be checked by destruction at some period of life. Our familiarity with the larger domestic animals tends, I think, to mislead us: we see no great destruction falling on them, and we forget that thousands are annually slaughtered for food, and that in a state of nature an equal number would have somehow to be disposed of.
The only difference between organisms which annually produce eggs or seeds by the thousand, and those which produce extremely few, is, that the slow-breeders would require a few more years to people, under favourable conditions, a whole district, let it be ever so large. The condor lays a couple of eggs and the ostrich a score, and yet in the same country the condor may be the more numerous of the two: the Fulmar petrel lays but one egg, yet it is believed to be the most numerous bird in the world. One fly deposits hundreds of eggs, and another, like the hippobosca, a single one; but this difference does not determine how many individuals of the two species can be supported in a district. A large number of eggs is of some importance to those species, which depend on a rapidly fluctuating amount of food, for it allows them rapidly to increase in number. But the real importance of a large number of eggs or seeds is to make up for much destruction at some period of life; and this period in the great majority of cases is an early one. If an animal can in any way protect its own eggs or young, a small number may be produced, and yet the average stock be fully kept up; but if many eggs or young are destroyed, many must be produced, or the species will become extinct. It would suffice to keep up the full number of a tree, which lived on an average for a thousand years, if a single seed were produced once in a thousand years, supposing that this seed were never destroyed, and could be ensured to germinate in a fitting place. So that in all cases, the average number of any animal or plant depends only indirectly on the number of its eggs or seeds.
In looking at Nature, it is most necessary to keep the foregoing considerations always in mind—never to forget that every single organic being around us may be said to be striving to the utmost to increase in numbers; that each lives by a struggle at some period of its life; that heavy destruction inevitably falls either on the young or old, during each generation or at recurrent intervals. Lighten any check, mitigate the destruction ever so little, and the number of the species will almost instantaneously increase to any amount. The face of Nature may be compared to a yielding surface, with ten thousand sharp wedges packed close together and driven inwards by incessant blows, sometimes one wedge being struck, and then another with greater force.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::domain::Format;
use matra::{Engine, Input, KeyphraseMethod};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let engine = Engine::new()?;
let text = std::fs::read_to_string("origin-struggle.txt")?;
let doc = engine.analyze(Input::text(text, Format::PlainText))?;
// Parse once; keyphrases are a function of the document.
let phrases = matra::keyphrases_with(&doc, 10, KeyphraseMethod::Rake)?;
println!("{}", serde_json::to_string_pretty(&phrases)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine()
with open("origin-struggle.txt", encoding="utf-8") as f:
doc = engine.analyze(f.read()) # read as plain text
# Parse once; keyphrases are a function of the document.
phrases = matra.keyphrases(doc, n=10, method="rake")
print(json.dumps(phrases, indent=2, ensure_ascii=False)) CLI
$ matra keyphrases --method rake -n 10 --json --stdin-filename origin-struggle.txt - < origin-struggle.txt It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "keyphrases",
"input": "origin-struggle.txt",
"result": /* the output below */,
"error": null
}Output
Trimmed: lists cut to the first 5.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "plain_text",
"source": {
"path": null,
"bytes": 12808,
"sha256": "858cb32e93b9a95293e00afae7bd4049e8c0fda79abca22ea01b456f2a18a580"
},
"method": "rake",
"parameters": {
"n": 10
}
},
"phrases": [
{
"phrase": "larger domestic animal tend",
"score": 12.636363636363637,
"occurrences": [
{
"byte_start": 10132,
"byte_end": 10161
}
]
},
{
"phrase": "elder de candolle",
"score": 9,
"occurrences": [
{
"byte_start": 3228,
"byte_end": 3245
}
]
},
{
"phrase": "fulmar petrel lay",
"score": 9,
"occurrences": [
{
"byte_start": 10791,
"byte_end": 10809
}
]
},
{
"phrase": "mere theoretical calculation",
"score": 8.5,
"occurrences": [
{
"byte_start": 8016,
"byte_end": 8045
}
]
},
{
"phrase": "great horticultural knowledge",
"score": 8.2,
"occurrences": [
{
"byte_start": 3501,
"byte_end": 3530
}
]
}
// … 5 more
]
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
RAKE against YAKE on the same text, as ranks:
RAKE and YAKE on the same text
show the data
| Phrase | RAKE rank | RAKE score | YAKE rank | YAKE score |
|---|---|---|---|---|
| immaterial whether multitude | not ranked | 1 | 45079.3 | |
| larger domestic animal tend | 1 | 12.636 | not ranked | |
| elder de candolle | 2 | 9.000 | 1659 | 0.548 |
| fulmar petrel lay | 2 | 9.000 | 2421 | 0.241 |
| sub-specy varietie rank | not ranked | 2 | 26693.1 | |
| enter subject | not ranked | 3 | 8987.0 | |
| enter subject chapter | not ranked | 4 | 8544.3 | |
| mere theoretical calculation | 4 | 8.500 | 2245 | 0.352 |
| enter | not ranked | 5 | 2233.5 | |
| great horticultural knowledge | 5 | 8.200 | 1919 | 0.459 |
| probable minimum rate | 6 | 7.600 | 2307 | 0.323 |
| whether multitude doubtful | not ranked | 6 | 1735.7 | |
| astonishingly rapid increase | 7 | 7.357 | 2176 | 0.377 |
| cling hair quadruped | not ranked | 7 | 1679.6 | |
| great destruction fal | 8 | 6.914 | 2379 | 0.281 |
| hair quadruped feather | not ranked | 8 | 1615.5 | |
| slow -breeding man | 9 | 6.667 | 1810 | 0.496 |
| structure beetle dive | not ranked | 9 | 1383.8 | |
| beetle dive water | not ranked | 10 | 1334.5 | |
| numerous record case | 10 | 5.972 | 2361 | 0.292 |
| domestic animal | 11 | 4.636 | 1547 | 0.590 |
| immaterial whether | not ranked | 11 | 1313.1 | |
| rapid increase | 12 | 4.357 | 1612 | 0.561 |
| whether multitude | not ranked | 12 | 1214.9 |
- in RAKE's top 12
- in YAKE's top 12
- only one method produces the phrase
Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 321 RAKE and 2430 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.
Input: On the Origin of Species, the opening of chapter III, Struggle for Existence, Charles Darwin. Public domain (first published 1859).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: origin-struggle/keyphrases.json.
What to notice
- The result carries the text's
provenancewithmethodrakeandparameters{"n": 10}. Each of itsphraseshasphrase,scoreandoccurrences, the span of every place in the input it was found, highest score first, and a higher score means more relevant. RAKE's score sums each word's co-occurrence degree over its frequency. - Phrases are built from lemmas, so the parser's lemma errors show through.
great destruction falis "great destruction falling", withfallingread asfal, andslow -breeding manis "slow-breeding man" split at the hyphen. RAKE takes the words between stopwords and punctuation, which is whylarger domestic animal tendruns "animals tends" together. - Two phrases tie at 9.0. Equal scores come back in the order the phrases first occur, the same on every run, so a cut that falls inside a tie is decided by position in the text, not by merit. This example asks for ten, where the tenth scores 5.972 and no phrase outside the list ties it, so no tie decides the list.
- The figure shows how little RAKE and YAKE agree on a text like this one. Neither method is right; they rank by different evidence (Methodology).
This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 2 paragraphs, in the margin. the data