What matra gives you
on this page
Every value the pipeline returns, by tier. The tables say which language surfaces carry each value: fields cross to Python, methods do not.
1. Structure
Produced by Engine::analyze, as the document is read and parsed.
Token
The ten CoNLL-U columns plus one derived flag. All cross to Python.
| Field | Holds |
|---|---|
id | 1-based position in the sentence |
text | the word as written |
lemma | dictionary form (approved becomes approve) |
pos | universal part of speech (NOUN, VERB, ADJ) |
xpos | treebank-specific tag, finer grained than pos |
features | morphology (tense, number, person), as a map from feature to value |
head | id of the token this one depends on, 0 for the root |
dep | the dependency relation to that head |
deps | secondary dependencies, always _ from the UDPipe adapter, whose binding does not surface them |
misc | annotation |
is_punct | punctuation flag |
Token::feat(key) reads one morphological feature. Rust only.
Sentence
| Field | Holds |
|---|---|
text | sentence text, rebuilt by the provider from its tokens |
tokens | Vec<Token>, id-sorted |
word_count | non-punctuation tokens |
tree_depth | nesting depth, None on a malformed cycle |
passive | whether the sentence carries nsubj:pass, nsubjpass, or aux:pass |
negations | Vec<Negation>: cue id, cue lemma, head id |
modals | Vec<Modal>: auxiliary id, lemma, head id |
clausal_complements | Vec<ClausalComplement>: verb id and lemma, ccomp head id, subject when present |
root_adverbials | Vec<RootAdverbial>: adverbial id and lemma |
hearst_pairs | Vec<HearstPair>: pattern tag plus hypernym and hyponym HearstSpan values |
root_mood | Option<RootMood>: the root clause's Mood feature and the root's id |
All eleven cross to Python. The methods below are Rust only.
| Method | Returns |
|---|---|
root_token() | the token everything else hangs off |
head_of(id) | the governor of a token |
children_of(id) | its direct dependents |
subtree(id) | the whole clause under it |
content_tokens() | non-punctuation tokens |
clausal_complements_in(lexicon) | the clausal complements whose verb lemma is in a list you supply |
root_adverbials_in(lexicon) | the root adverbials whose lemma is in a list you supply |
Paragraph, Section, Document
| Type | Fields (cross to Python) | Methods (Rust only) |
|---|---|---|
Paragraph | span, text, kind, sentences, metrics | word_count(), sentence_count() |
Section | heading, level, paragraphs | none |
Document | provenance, sections, metrics | paragraphs(), sentences(), tokens() |
provenance records the matra version, the schema version ("2.0"), the parser with its model, digest and licence, the format, and the input's path, and the length and SHA-256 of the analysed text.
2. Metrics
Produced by Engine::analyze, as its last step, into ParagraphMetrics on each paragraph and DocumentMetrics on the document. Every ratio is Option<f64>, and None means not computed, which is distinct from a computed zero. Methodology gives each formula and the exact condition for None.
| Field | On | Measures |
|---|---|---|
readability_grade | ParagraphMetrics | Flesch-Kincaid grade level, from sentence and syllable length |
lexical_density | ParagraphMetrics | content words as a share of all words |
compression_ratio | ParagraphMetrics | brotli compressed size over raw size, a repetition proxy |
total_words, total_sentences, paragraph_count | DocumentMetrics | the counts, as integers |
mean_sentence_length, sentence_length_std | DocumentMetrics | words per sentence, and its sample standard deviation |
vocabulary_ttr | DocumentMetrics | distinct lemmas over total words, and it falls as text grows, so documents of different lengths are not comparable on it |
nominalization_ratio | DocumentMetrics | nouns ending in a nominalizing suffix (-tion, -ment, -ness, -ity, -ence, -ance) over all words, a surface proxy for nouns formed from verbs (decide becoming decision) |
passive_ratio | DocumentMetrics | share of sentences carrying a passive construction |
Every measure is per document. Aggregating across documents is the caller's decision: the counts sum, and vocabulary_ttr does not.
3. Summarization
Free functions over a Document, needing no engine and no parser. Both return a Summary: the document's provenance with the method and n recorded, and sentences, each a SummarySentence (index, span, text, score), in document order, capped at 2,000 sentences.
| Function | Ranks by |
|---|---|
summarize(doc, n), summarize_with(doc, n, SummaryMethod::TfIdf) | mean TF-IDF of the sentence's lemmatized terms, each sentence its own document for IDF (tfidf, the default) |
summarize_with(doc, n, SummaryMethod::TextRank) | PageRank over a similarity graph of shared content lemmas normalized by log length (textrank) |
Python spells them matra.summarize(doc, n=3) and matra.summarize(doc, n=3, method="textrank"), over the Document an engine returned.
TextRank's summary of The Federalist No. 10 at the default length, and every sentence's score it was chosen from:
TextRank: the summary and the scores it came from
- 6Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority.
- 47When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens.
- 48To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed.
show the data
| # | ยถ | Score | Summary | Sentence |
|---|---|---|---|---|
| 1 | 1 | 0.0086 | AMONG the numerous advantages promised by a well constructed Union, none deserves to be more accurately developed than its tendency to break and control the violence of faction. | |
| 2 | 1 | 0.0097 | The friend of popular governments never finds himself so much alarmed for their character and fate, as when he contemplates their propensity to this dangerous vice. | |
| 3 | 1 | 0.0058 | He will not fail, therefore, to set a due value on any plan which, without violating the principles to which he is attached, provides a proper cure for it. | |
| 4 | 1 | 0.0112 | The instability, injustice, and confusion introduced into the public councils, have, in truth, been the mortal diseases under which popular governments have everywhere perished; as they continue to be the favorite and fruitful topics from which the adversaries to liberty derive their most specious declamations. | |
| 5 | 1 | 0.0064 | The valuable improvements made by the American constitutions on the popular models, both ancient and modern, cannot certainly be too much admired; but it would be an unwarrantable partiality, to contend that they have as effectually obviated the danger on this side, as was wished and expected. | |
| 6 | 1 | 0.0221 | yes | Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority. |
| 7 | 1 | 0.0050 | However anxiously we may wish that these complaints had no foundation, the evidence, of known facts will not permit us to deny that they are in some degree true. | |
| 8 | 1 | 0.0163 | It will be found, indeed, on a candid review of our situation, that some of the distresses under which we labor have been erroneously charged on the operation of our governments; but it will be found, at the same time, that other causes will not alone account for many of our heaviest misfortunes; and, particularly, for that prevailing and increasing distrust of public engagements, and alarm for private rights, which are echoed from one end of the continent to the other. | |
| 9 | 1 | 0.0099 | These must be chiefly, if not wholly, effects of the unsteadiness and injustice with which a factious spirit has tainted our public administrations. | |
| 10 | 2 | 0.0211 | By a faction, I understand a number of citizens, whether amounting to a majority or a minority of the whole, who are united and actuated by some common impulse of passion, or of interest, adversed to the rights of other citizens, or to the permanent and aggregate interests of the community. | |
| 11 | 3 | 0.0145 | There are two methods of curing the mischiefs of faction: the one, by removing its causes; the other, by controlling its effects. | |
| 12 | 4 | 0.0191 | There are again two methods of removing the causes of faction: the one, by destroying the liberty which is essential to its existence; the other, by giving to every citizen the same opinions, the same passions, and the same interests. | |
| 13 | 5 | 0.0050 | It could never be more truly said than of the first remedy, that it was worse than the disease. | |
| 14 | 5 | 0.0068 | Liberty is to faction what air is to fire, an aliment without which it instantly expires. | |
| 15 | 5 | 0.0096 | But it could not be less folly to abolish liberty, which is essential to political life, because it nourishes faction, than it would be to wish the annihilation of air, which is essential to animal life, because it imparts to fire its destructive agency. | |
| 16 | 6 | 0.0036 | The second expedient is as impracticable as the first would be unwise. | |
| 17 | 6 | 0.0095 | As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed. | |
| 18 | 6 | 0.0090 | As long as the connection subsists between his reason and his self-love, his opinions and his passions will have a reciprocal influence on each other; and the former will be objects to which the latter will attach themselves. | |
| 19 | 6 | 0.0127 | The diversity in the faculties of men, from which the rights of property originate, is not less an insuperable obstacle to a uniformity of interests. | |
| 20 | 6 | 0.0100 | The protection of these faculties is the first object of government. | |
| 21 | 6 | 0.0182 | From the protection of different and unequal faculties of acquiring property, the possession of different degrees and kinds of property immediately results; and from the influence of these on the sentiments and views of the respective proprietors, ensues a division of the society into different interests and parties. | |
| 22 | 7 | 0.0143 | The latent causes of faction are thus sown in the nature of man; and we see them everywhere brought into different degrees of activity, according to the different circumstances of civil society. | |
| 23 | 7 | 0.0210 | A zeal for different opinions concerning religion, concerning government, and many other points, as well of speculation as of practice; an attachment to different leaders ambitiously contending for pre-eminence and power; or to persons of other descriptions whose fortunes have been interesting to the human passions, have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to co-operate for their common good. | |
| 24 | 7 | 0.0062 | So strong is this propensity of mankind to fall into mutual animosities, that where no substantial occasion presents itself, the most frivolous and fanciful distinctions have been sufficient to kindle their unfriendly passions and excite their most violent conflicts. | |
| 25 | 7 | 0.0051 | But the most common and durable source of factions has been the various and unequal distribution of property. | |
| 26 | 7 | 0.0093 | Those who hold and those who are without property have ever formed distinct interests in society. | |
| 27 | 7 | 0.0028 | Those who are creditors, and those who are debtors, fall under a like discrimination. | |
| 28 | 7 | 0.0112 | A landed interest, a manufacturing interest, a mercantile interest, a moneyed interest, with many lesser interests, grow up of necessity in civilized nations, and divide them into different classes, actuated by different sentiments and views. | |
| 29 | 7 | 0.0193 | The regulation of these various and interfering interests forms the principal task of modern legislation, and involves the spirit of party and faction in the necessary and ordinary operations of the government. | |
| 30 | 8 | 0.0089 | No man is allowed to be a judge in his own cause, because his interest would certainly bias his judgment, and, not improbably, corrupt his integrity. | |
| 31 | 8 | 0.0174 | With equal, nay with greater reason, a body of men are unfit to be both judges and parties at the same time; yet what are many of the most important acts of legislation, but so many judicial determinations, not indeed concerning the rights of single persons, but concerning the rights of large bodies of citizens? | |
| 32 | 8 | 0.0085 | And what are the different classes of legislators but advocates and parties to the causes which they determine? | |
| 33 | 8 | 0.0031 | Is a law proposed concerning private debts? | |
| 34 | 8 | 0.0101 | It is a question to which the creditors are parties on one side and the debtors on the other. | |
| 35 | 8 | 0.0025 | Justice ought to hold the balance between them. | |
| 36 | 8 | 0.0134 | Yet the parties are, and must be, themselves the judges; and the most numerous party, or, in other words, the most powerful faction must be expected to prevail. | |
| 37 | 8 | 0.0033 | Shall domestic manufactures be encouraged, and in what degree, by restrictions on foreign manufactures? | |
| 38 | 8 | 0.0082 | are questions which would be differently decided by the landed and the manufacturing classes, and probably by neither with a sole regard to justice and the public good. | |
| 39 | 8 | 0.0108 | The apportionment of taxes on the various descriptions of property is an act which seems to require the most exact impartiality; yet there is, perhaps, no legislative act in which greater opportunity and temptation are given to a predominant party to trample on the rules of justice. | |
| 40 | 8 | 0.0050 | Every shilling with which they overburden the inferior number, is a shilling saved to their own pockets. | |
| 41 | 9 | 0.0121 | It is in vain to say that enlightened statesmen will be able to adjust these clashing interests, and render them all subservient to the public good. | |
| 42 | 9 | 0.0024 | Enlightened statesmen will not always be at the helm. | |
| 43 | 9 | 0.0202 | Nor, in many cases, can such an adjustment be made at all without taking into view indirect and remote considerations, which will rarely prevail over the immediate interest which one party may find in disregarding the rights of another or the good of the whole. | |
| 44 | 10 | 0.0110 | The inference to which we are brought is, that the CAUSES of faction cannot be removed, and that relief is only to be sought in the means of controlling its EFFECTS. | |
| 45 | 11 | 0.0119 | If a faction consists of less than a majority, relief is supplied by the republican principle, which enables the majority to defeat its sinister views by regular vote. | |
| 46 | 11 | 0.0070 | It may clog the administration, it may convulse the society; but it will be unable to execute and mask its violence under the forms of the Constitution. | |
| 47 | 11 | 0.0274 | yes | When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens. |
| 48 | 11 | 0.0224 | yes | To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed. |
| 49 | 11 | 0.0093 | Let me add that it is the great desideratum by which this form of government can be rescued from the opprobrium under which it has so long labored, and be recommended to the esteem and adoption of mankind. | |
| 50 | 12 | 0.0040 | By what means is this object attainable? | |
| 51 | 12 | 0.0052 | Evidently by one of two only. | |
| 52 | 12 | 0.0201 | Either the existence of the same passion or interest in a majority at the same time must be prevented, or the majority, having such coexistent passion or interest, must be rendered, by their number and local situation, unable to concert and carry into effect schemes of oppression. | |
| 53 | 12 | 0.0045 | If the impulse and the opportunity be suffered to coincide, we well know that neither moral nor religious motives can be relied on as an adequate control. | |
| 54 | 12 | 0.0087 | They are not found to be such on the injustice and violence of individuals, and lose their efficacy in proportion to the number combined together, that is, in proportion as their efficacy becomes needful. | |
| 55 | 13 | 0.0198 | From this view of the subject it may be concluded that a pure democracy, by which I mean a society consisting of a small number of citizens, who assemble and administer the government in person, can admit of no cure for the mischiefs of faction. | |
| 56 | 13 | 0.0219 | A common passion or interest will, in almost every case, be felt by a majority of the whole; a communication and concert result from the form of government itself; and there is nothing to check the inducements to sacrifice the weaker party or an obnoxious individual. | |
| 57 | 13 | 0.0087 | Hence it is that such democracies have ever been spectacles of turbulence and contention; have ever been found incompatible with personal security or the rights of property; and have in general been as short in their lives as they have been violent in their deaths. | |
| 58 | 13 | 0.0118 | Theoretic politicians, who have patronized this species of government, have erroneously supposed that by reducing mankind to a perfect equality in their political rights, they would, at the same time, be perfectly equalized and assimilated in their possessions, their opinions, and their passions. | |
| 59 | 14 | 0.0122 | A republic, by which I mean a government in which the scheme of representation takes place, opens a different prospect, and promises the cure for which we are seeking. | |
| 60 | 14 | 0.0085 | Let us examine the points in which it varies from pure democracy, and we shall comprehend both the nature of the cure and the efficacy which it must derive from the Union. | |
| 61 | 15 | 0.0210 | The two great points of difference between a democracy and a republic are: first, the delegation of the government, in the latter, to a small number of citizens elected by the rest; secondly, the greater number of citizens, and greater sphere of country, over which the latter may be extended. | |
| 62 | 16 | 0.0192 | The effect of the first difference is, on the one hand, to refine and enlarge the public views, by passing them through the medium of a chosen body of citizens, whose wisdom may best discern the true interest of their country, and whose patriotism and love of justice will be least likely to sacrifice it to temporary or partial considerations. | |
| 63 | 16 | 0.0088 | Under such a regulation, it may well happen that the public voice, pronounced by the representatives of the people, will be more consonant to the public good than if pronounced by the people themselves, convened for the purpose. | |
| 64 | 16 | 0.0041 | On the other hand, the effect may be inverted. | |
| 65 | 16 | 0.0112 | Men of factious tempers, of local prejudices, or of sinister designs, may, by intrigue, by corruption, or by other means, first obtain the suffrages, and then betray the interests, of the people. | |
| 66 | 16 | 0.0108 | The question resulting is, whether small or extensive republics are more favorable to the election of proper guardians of the public weal; and it is clearly decided in favor of the latter by two obvious considerations: | |
| 67 | 17 | 0.0143 | In the first place, it is to be remarked that, however small the republic may be, the representatives must be raised to a certain number, in order to guard against the cabals of a few; and that, however large it may be, they must be limited to a certain number, in order to guard against the confusion of a multitude. | |
| 68 | 17 | 0.0165 | Hence, the number of representatives in the two cases not being in proportion to that of the two constituents, and being proportionally greater in the small republic, it follows that, if the proportion of fit characters be not less in the large than in the small republic, the former will present a greater option, and consequently a greater probability of a fit choice. | |
| 69 | 18 | 0.0163 | In the next place, as each representative will be chosen by a greater number of citizens in the large than in the small republic, it will be more difficult for unworthy candidates to practice with success the vicious arts by which elections are too often carried; and the suffrages of the people being more free, will be more likely to centre in men who possess the most attractive merit and the most diffusive and established characters. | |
| 70 | 19 | 0.0071 | It must be confessed that in this, as in most other cases, there is a mean, on both sides of which inconveniences will be found to lie. | |
| 71 | 19 | 0.0154 | By enlarging too much the number of electors, you render the representatives too little acquainted with all their local circumstances and lesser interests; as by reducing it too much, you render him unduly attached to these, and too little fit to comprehend and pursue great and national objects. | |
| 72 | 19 | 0.0110 | The federal Constitution forms a happy combination in this respect; the great and aggregate interests being referred to the national, the local and particular to the State legislatures. | |
| 73 | 20 | 0.0188 | The other point of difference is, the greater number of citizens and extent of territory which may be brought within the compass of republican than of democratic government; and it is this circumstance principally which renders factious combinations less to be dreaded in the former than in the latter. | |
| 74 | 20 | 0.0198 | The smaller the society, the fewer probably will be the distinct parties and interests composing it; the fewer the distinct parties and interests, the more frequently will a majority be found of the same party; and the smaller the number of individuals composing a majority, and the smaller the compass within which they are placed, the more easily will they concert and execute their plans of oppression. | |
| 75 | 20 | 0.0207 | Extend the sphere, and you take in a greater variety of parties and interests; you make it less probable that a majority of the whole will have a common motive to invade the rights of other citizens; or if such a common motive exists, it will be more difficult for all who feel it to discover their own strength, and to act in unison with each other. | |
| 76 | 20 | 0.0080 | Besides other impediments, it may be remarked that, where there is a consciousness of unjust or dishonorable purposes, communication is always checked by distrust in proportion to the number whose concurrence is necessary. | |
| 77 | 21 | 0.0156 | Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic,--is enjoyed by the Union over the States composing it. | |
| 78 | 21 | 0.0105 | Does the advantage consist in the substitution of representatives whose enlightened views and virtuous sentiments render them superior to local prejudices and schemes of injustice? | |
| 79 | 21 | 0.0051 | It will not be denied that the representation of the Union will be most likely to possess these requisite endowments. | |
| 80 | 21 | 0.0132 | Does it consist in the greater security afforded by a greater variety of parties, against the event of any one party being able to outnumber and oppress the rest? | |
| 81 | 21 | 0.0114 | In an equal degree does the increased variety of parties comprised within the Union, increase this security. | |
| 82 | 21 | 0.0080 | Does it, in fine, consist in the greater obstacles opposed to the concert and accomplishment of the secret wishes of an unjust and interested majority? | |
| 83 | 21 | 0.0055 | Here, again, the extent of the Union gives it the most palpable advantage. | |
| 84 | 22 | 0.0055 | The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States. | |
| 85 | 22 | 0.0109 | A religious sect may degenerate into a political faction in a part of the Confederacy; but the variety of sects dispersed over the entire face of it must secure the national councils against any danger from that source. | |
| 86 | 22 | 0.0110 | A rage for paper money, for an abolition of debts, for an equal division of property, or for any other improper or wicked project, will be less apt to pervade the whole body of the Union than a particular member of it; in the same proportion as such a malady is more likely to taint a particular county or district, than an entire State. | |
| 87 | 23 | 0.0100 | In the extent and proper structure of the Union, therefore, we behold a republican remedy for the diseases most incident to republican government. | |
| 88 | 23 | 0.0067 | And according to the degree of pleasure and pride we feel in being republicans, ought to be our zeal in cherishing the spirit and supporting the character of Federalists. |
- in the summary (3 sentences, matra's default)
- not picked
TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.
Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: federalist-10/textrank.json.
4. Keyphrases
Free functions over a Document. Both return a Keyphrases: the document's provenance with the method and n recorded, and phrases, each a Keyphrase (phrase, score, occurrences, the span of every place it was found), highest score first, capped at 200,000 tokens.
| Function | Ranks by |
|---|---|
keyphrases(doc, n), keyphrases_with(doc, n, KeyphraseMethod::Rake) | co-occurrence degree over frequency, on NOUN, ADJ, and PROPN runs between stop words (rake, the default) |
keyphrases_with(doc, n, KeyphraseMethod::Yake) | per-term position, frequency, and context diversity, assembled into 1-word to 3-word candidates, score inverted so higher is more relevant (yake) |
Python spells them matra.keyphrases(doc, n=10) and matra.keyphrases(doc, n=10, method="yake").
The two rank by different properties of the text, and on The Federalist No. 10 their top twelve lists share no phrase:
RAKE and YAKE on the same text
show the data
| Phrase | RAKE rank | RAKE score | YAKE rank | YAKE score |
|---|---|---|---|---|
| fate contemplate propensity | not ranked | 1 | 32553.6 | |
| substantial occasion present | 1 | 9.000 | 2694 | 0.451 |
| many judicial determination | 2 | 7.833 | 2642 | 0.468 |
| tendency break | not ranked | 2 | 22722.0 | |
| develop tendency break | not ranked | 3 | 22609.3 | |
| federal constitution form | 3 | 6.429 | 2983 | 0.335 |
| many lesser interest | 4 | 5.875 | 2273 | 0.576 |
| tendency break control | not ranked | 4 | 16940.2 | |
| fate contemplate | not ranked | 5 | 4523.1 | |
| lesser interest | 5 | 4.042 | 1722 | 0.727 |
| adequate control | 6 | 4.000 | 2426 | 0.520 |
| alone account | 6 | 4.000 | 1486 | 0.823 |
| american constitution | 6 | 4.000 | 1793 | 0.703 |
| attractive merit | 6 | 4.000 | 2849 | 0.404 |
| candid review | 6 | 4.000 | 1465 | 0.840 |
| character fate contemplate | not ranked | 6 | 3038.8 | |
| civilized nation | 6 | 4.000 | 2136 | 0.615 |
| dangerous vice | 6 | 4.000 | 1294 | 0.967 |
| complaint foundation evidence | not ranked | 7 | 1730.8 | |
| contemplate propensity | not ranked | 8 | 463.1 | |
| contemplate propensity dangerous | not ranked | 9 | 455.5 | |
| break control violence | not ranked | 10 | 362.2 | |
| annihilation air essential | not ranked | 11 | 218.2 | |
| tendency | 272 | 1.000 | 12 | 158.4 |
- in RAKE's top 12
- in YAKE's top 12
- only one method produces the phrase
Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 466 RAKE and 3111 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.
Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787).
Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206
(SHA-256 784bd0fa85e3). Data: federalist-10/keyphrases.json.
5. Semantic clusters
Not a field on any type above. The port and clusters are always compiled; the Model2Vec adapter is behind the model2vec feature.
| Item | Is |
|---|---|
Embedder | the port: embed(&[&str]) returning one Embedding per text, plus identity() |
Model2Vec | the adapter, reading model.safetensors, tokenizer.json, and config.json from a directory. from_dir(dir) loads what is there; potion_base_8m(dir), with dir a models directory, and from_config(cfg) fetch the pinned reference model into its potion-base-8M subdirectory when it is absent |
clusters(doc, embedder, threshold) | embeds a document's sentences with the embedder you pass, then clusters them; vectors you already hold arrive through an Embedder that returns them |
Clusters | the document's provenance with the method (cosine), the threshold and the embedder's identity recorded, clusters, and edges |
Cluster, SimilarityEdge | a cluster's members (sentence indices), and a pair a, b whose score cleared the threshold |
Capped at 2,000 sentences. Python spells them matra.clusters(doc, embedder, threshold=0.85) and Model2Vec (including Model2Vec.potion_base_8m); matra.clusters takes a Model2Vec or any object with embed and identity.
The pipeline surface
| Value | Constructors | Carries |
|---|---|---|
Input | text(string, format), bytes(data, format), file(path), blocks(blocks) | one document: text, bytes, a file, or headings and paragraphs you supply |
Inputs | dir(path) | the files in a directory, as an iterator of Input |
Engine | new(), from_config(cfg), from_config_with_notice(cfg, notice) | analyze one input, analyze_all of many |
Config | resolve(), from_file(path), from_sources(...), with_model_dir(dir) | where models live and what the defaults are, plus the ValueSource each value came from |
Engine::new() is the no-setup constructor: locate the model directory from the environment, then build the standard pipeline, fetching the pinned English model if it is not on disk. It reads no configuration file; Engine::from_config(&Config::resolve()?) reads one.
engine.analyze_all(inputs) returns a lazy iterator of per-document results, one per input in order, and one bad file does not abort the stream. Python's Engine.analyze_all(dir=path) is the same lazy walk, yielding a Document or a DocumentError dict per file.
Traceability
Every heading, paragraph, sentence and token carries a span into the analysed text, in UTF-8 bytes, and skipped lists what the reader saw and did not analyse (a table, a code block, front matter), each with its span. Every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span, so nothing is dropped without a trace. The text fields are normalized: Paragraph.text is what the parser reads, a markdown paragraph's inline markup removed, and Sentence.text is rebuilt by the provider, the UDPipe adapter joining the token forms with a space unless a token carries SpaceAfter=No. The span, not the text, says where a value came from. Each paragraph is parsed on its own and its sentences are attached to it, so the chain from a token up through its sentence, paragraph, and section holds by construction rather than by text matching.
Where to go next
Domain model is the full type graph. Methodology is every formula. Errors is every failure.
This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 20 paragraphs, in the margin. the data