What matra gives you

on this page
grade6.9density0.65compressionnone

Every value the pipeline returns, by tier. The tables say which language surfaces carry each value: fields cross to Python, methods do not.

1. Structure

grade10.7density0.50compressionnone

Produced by Engine::analyze, as the document is read and parsed.

Token

grade3.5density0.75compressionnone

The ten CoNLL-U columns plus one derived flag. All cross to Python.

FieldHolds
id1-based position in the sentence
textthe word as written
lemmadictionary form (approved becomes approve)
posuniversal part of speech (NOUN, VERB, ADJ)
xpostreebank-specific tag, finer grained than pos
featuresmorphology (tense, number, person), as a map from feature to value
headid of the token this one depends on, 0 for the root
depthe dependency relation to that head
depssecondary dependencies, always _ from the UDPipe adapter, whose binding does not surface them
miscannotation
is_punctpunctuation flag
gradenonedensity0.86compressionnone

Token::feat(key) reads one morphological feature. Rust only.

Sentence

FieldHolds
textsentence text, rebuilt by the provider from its tokens
tokensVec<Token>, id-sorted
word_countnon-punctuation tokens
tree_depthnesting depth, None on a malformed cycle
passivewhether the sentence carries nsubj:pass, nsubjpass, or aux:pass
negationsVec<Negation>: cue id, cue lemma, head id
modalsVec<Modal>: auxiliary id, lemma, head id
clausal_complementsVec<ClausalComplement>: verb id and lemma, ccomp head id, subject when present
root_adverbialsVec<RootAdverbial>: adverbial id and lemma
hearst_pairsVec<HearstPair>: pattern tag plus hypernym and hyponym HearstSpan values
root_moodOption<RootMood>: the root clause's Mood feature and the root's id
grade4.8density0.45compressionnone

All eleven cross to Python. The methods below are Rust only.

MethodReturns
root_token()the token everything else hangs off
head_of(id)the governor of a token
children_of(id)its direct dependents
subtree(id)the whole clause under it
content_tokens()non-punctuation tokens
clausal_complements_in(lexicon)the clausal complements whose verb lemma is in a list you supply
root_adverbials_in(lexicon)the root adverbials whose lemma is in a list you supply

Paragraph, Section, Document

TypeFields (cross to Python)Methods (Rust only)
Paragraphspan, text, kind, sentences, metricsword_count(), sentence_count()
Sectionheading, level, paragraphsnone
Documentprovenance, sections, metricsparagraphs(), sentences(), tokens()
grade8.0density0.53compressionnone

provenance records the matra version, the schema version ("2.0"), the parser with its model, digest and licence, the format, and the input's path, and the length and SHA-256 of the analysed text.

2. Metrics

grade12.5density0.53compressionnone

Produced by Engine::analyze, as its last step, into ParagraphMetrics on each paragraph and DocumentMetrics on the document. Every ratio is Option<f64>, and None means not computed, which is distinct from a computed zero. Methodology gives each formula and the exact condition for None.

FieldOnMeasures
readability_gradeParagraphMetricsFlesch-Kincaid grade level, from sentence and syllable length
lexical_densityParagraphMetricscontent words as a share of all words
compression_ratioParagraphMetricsbrotli compressed size over raw size, a repetition proxy
total_words, total_sentences, paragraph_countDocumentMetricsthe counts, as integers
mean_sentence_length, sentence_length_stdDocumentMetricswords per sentence, and its sample standard deviation
vocabulary_ttrDocumentMetricsdistinct lemmas over total words, and it falls as text grows, so documents of different lengths are not comparable on it
nominalization_ratioDocumentMetricsnouns ending in a nominalizing suffix (-tion, -ment, -ness, -ity, -ence, -ance) over all words, a surface proxy for nouns formed from verbs (decide becoming decision)
passive_ratioDocumentMetricsshare of sentences carrying a passive construction
grade11.1density0.63compressionnone

Every measure is per document. Aggregating across documents is the caller's decision: the counts sum, and vocabulary_ttr does not.

3. Summarization

grade12.6density0.57compressionnone

Free functions over a Document, needing no engine and no parser. Both return a Summary: the document's provenance with the method and n recorded, and sentences, each a SummarySentence (index, span, text, score), in document order, capped at 2,000 sentences.

FunctionRanks by
summarize(doc, n), summarize_with(doc, n, SummaryMethod::TfIdf)mean TF-IDF of the sentence's lemmatized terms, each sentence its own document for IDF (tfidf, the default)
summarize_with(doc, n, SummaryMethod::TextRank)PageRank over a similarity graph of shared content lemmas normalized by log length (textrank)
grade15.5density0.67compressionnone

Python spells them matra.summarize(doc, n=3) and matra.summarize(doc, n=3, method="textrank"), over the Document an engine returned.

grade10.5density0.47compressionnone

TextRank's summary of The Federalist No. 10 at the default length, and every sentence's score it was chosen from:

TextRank: the summary and the scores it came from

TextRank score of each sentence, in document order0.0000.0100.0200.03064748sentence 1sentence 88document order; bands alternate by paragraph
  1. 6Complaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority.
  2. 47When a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens.
  3. 48To secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed.
show the data
#ยถScoreSummarySentence
110.0086AMONG the numerous advantages promised by a well constructed Union, none deserves to be more accurately developed than its tendency to break and control the violence of faction.
210.0097The friend of popular governments never finds himself so much alarmed for their character and fate, as when he contemplates their propensity to this dangerous vice.
310.0058He will not fail, therefore, to set a due value on any plan which, without violating the principles to which he is attached, provides a proper cure for it.
410.0112The instability, injustice, and confusion introduced into the public councils, have, in truth, been the mortal diseases under which popular governments have everywhere perished; as they continue to be the favorite and fruitful topics from which the adversaries to liberty derive their most specious declamations.
510.0064The valuable improvements made by the American constitutions on the popular models, both ancient and modern, cannot certainly be too much admired; but it would be an unwarrantable partiality, to contend that they have as effectually obviated the danger on this side, as was wished and expected.
610.0221yesComplaints are everywhere heard from our most considerate and virtuous citizens, equally the friends of public and private faith, and of public and personal liberty, that our governments are too unstable, that the public good is disregarded in the conflicts of rival parties, and that measures are too often decided, not according to the rules of justice and the rights of the minor party, but by the superior force of an interested and overbearing majority.
710.0050However anxiously we may wish that these complaints had no foundation, the evidence, of known facts will not permit us to deny that they are in some degree true.
810.0163It will be found, indeed, on a candid review of our situation, that some of the distresses under which we labor have been erroneously charged on the operation of our governments; but it will be found, at the same time, that other causes will not alone account for many of our heaviest misfortunes; and, particularly, for that prevailing and increasing distrust of public engagements, and alarm for private rights, which are echoed from one end of the continent to the other.
910.0099These must be chiefly, if not wholly, effects of the unsteadiness and injustice with which a factious spirit has tainted our public administrations.
1020.0211By a faction, I understand a number of citizens, whether amounting to a majority or a minority of the whole, who are united and actuated by some common impulse of passion, or of interest, adversed to the rights of other citizens, or to the permanent and aggregate interests of the community.
1130.0145There are two methods of curing the mischiefs of faction: the one, by removing its causes; the other, by controlling its effects.
1240.0191There are again two methods of removing the causes of faction: the one, by destroying the liberty which is essential to its existence; the other, by giving to every citizen the same opinions, the same passions, and the same interests.
1350.0050It could never be more truly said than of the first remedy, that it was worse than the disease.
1450.0068Liberty is to faction what air is to fire, an aliment without which it instantly expires.
1550.0096But it could not be less folly to abolish liberty, which is essential to political life, because it nourishes faction, than it would be to wish the annihilation of air, which is essential to animal life, because it imparts to fire its destructive agency.
1660.0036The second expedient is as impracticable as the first would be unwise.
1760.0095As long as the reason of man continues fallible, and he is at liberty to exercise it, different opinions will be formed.
1860.0090As long as the connection subsists between his reason and his self-love, his opinions and his passions will have a reciprocal influence on each other; and the former will be objects to which the latter will attach themselves.
1960.0127The diversity in the faculties of men, from which the rights of property originate, is not less an insuperable obstacle to a uniformity of interests.
2060.0100The protection of these faculties is the first object of government.
2160.0182From the protection of different and unequal faculties of acquiring property, the possession of different degrees and kinds of property immediately results; and from the influence of these on the sentiments and views of the respective proprietors, ensues a division of the society into different interests and parties.
2270.0143The latent causes of faction are thus sown in the nature of man; and we see them everywhere brought into different degrees of activity, according to the different circumstances of civil society.
2370.0210A zeal for different opinions concerning religion, concerning government, and many other points, as well of speculation as of practice; an attachment to different leaders ambitiously contending for pre-eminence and power; or to persons of other descriptions whose fortunes have been interesting to the human passions, have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to co-operate for their common good.
2470.0062So strong is this propensity of mankind to fall into mutual animosities, that where no substantial occasion presents itself, the most frivolous and fanciful distinctions have been sufficient to kindle their unfriendly passions and excite their most violent conflicts.
2570.0051But the most common and durable source of factions has been the various and unequal distribution of property.
2670.0093Those who hold and those who are without property have ever formed distinct interests in society.
2770.0028Those who are creditors, and those who are debtors, fall under a like discrimination.
2870.0112A landed interest, a manufacturing interest, a mercantile interest, a moneyed interest, with many lesser interests, grow up of necessity in civilized nations, and divide them into different classes, actuated by different sentiments and views.
2970.0193The regulation of these various and interfering interests forms the principal task of modern legislation, and involves the spirit of party and faction in the necessary and ordinary operations of the government.
3080.0089No man is allowed to be a judge in his own cause, because his interest would certainly bias his judgment, and, not improbably, corrupt his integrity.
3180.0174With equal, nay with greater reason, a body of men are unfit to be both judges and parties at the same time; yet what are many of the most important acts of legislation, but so many judicial determinations, not indeed concerning the rights of single persons, but concerning the rights of large bodies of citizens?
3280.0085And what are the different classes of legislators but advocates and parties to the causes which they determine?
3380.0031Is a law proposed concerning private debts?
3480.0101It is a question to which the creditors are parties on one side and the debtors on the other.
3580.0025Justice ought to hold the balance between them.
3680.0134Yet the parties are, and must be, themselves the judges; and the most numerous party, or, in other words, the most powerful faction must be expected to prevail.
3780.0033Shall domestic manufactures be encouraged, and in what degree, by restrictions on foreign manufactures?
3880.0082are questions which would be differently decided by the landed and the manufacturing classes, and probably by neither with a sole regard to justice and the public good.
3980.0108The apportionment of taxes on the various descriptions of property is an act which seems to require the most exact impartiality; yet there is, perhaps, no legislative act in which greater opportunity and temptation are given to a predominant party to trample on the rules of justice.
4080.0050Every shilling with which they overburden the inferior number, is a shilling saved to their own pockets.
4190.0121It is in vain to say that enlightened statesmen will be able to adjust these clashing interests, and render them all subservient to the public good.
4290.0024Enlightened statesmen will not always be at the helm.
4390.0202Nor, in many cases, can such an adjustment be made at all without taking into view indirect and remote considerations, which will rarely prevail over the immediate interest which one party may find in disregarding the rights of another or the good of the whole.
44100.0110The inference to which we are brought is, that the CAUSES of faction cannot be removed, and that relief is only to be sought in the means of controlling its EFFECTS.
45110.0119If a faction consists of less than a majority, relief is supplied by the republican principle, which enables the majority to defeat its sinister views by regular vote.
46110.0070It may clog the administration, it may convulse the society; but it will be unable to execute and mask its violence under the forms of the Constitution.
47110.0274yesWhen a majority is included in a faction, the form of popular government, on the other hand, enables it to sacrifice to its ruling passion or interest both the public good and the rights of other citizens.
48110.0224yesTo secure the public good and private rights against the danger of such a faction, and at the same time to preserve the spirit and the form of popular government, is then the great object to which our inquiries are directed.
49110.0093Let me add that it is the great desideratum by which this form of government can be rescued from the opprobrium under which it has so long labored, and be recommended to the esteem and adoption of mankind.
50120.0040By what means is this object attainable?
51120.0052Evidently by one of two only.
52120.0201Either the existence of the same passion or interest in a majority at the same time must be prevented, or the majority, having such coexistent passion or interest, must be rendered, by their number and local situation, unable to concert and carry into effect schemes of oppression.
53120.0045If the impulse and the opportunity be suffered to coincide, we well know that neither moral nor religious motives can be relied on as an adequate control.
54120.0087They are not found to be such on the injustice and violence of individuals, and lose their efficacy in proportion to the number combined together, that is, in proportion as their efficacy becomes needful.
55130.0198From this view of the subject it may be concluded that a pure democracy, by which I mean a society consisting of a small number of citizens, who assemble and administer the government in person, can admit of no cure for the mischiefs of faction.
56130.0219A common passion or interest will, in almost every case, be felt by a majority of the whole; a communication and concert result from the form of government itself; and there is nothing to check the inducements to sacrifice the weaker party or an obnoxious individual.
57130.0087Hence it is that such democracies have ever been spectacles of turbulence and contention; have ever been found incompatible with personal security or the rights of property; and have in general been as short in their lives as they have been violent in their deaths.
58130.0118Theoretic politicians, who have patronized this species of government, have erroneously supposed that by reducing mankind to a perfect equality in their political rights, they would, at the same time, be perfectly equalized and assimilated in their possessions, their opinions, and their passions.
59140.0122A republic, by which I mean a government in which the scheme of representation takes place, opens a different prospect, and promises the cure for which we are seeking.
60140.0085Let us examine the points in which it varies from pure democracy, and we shall comprehend both the nature of the cure and the efficacy which it must derive from the Union.
61150.0210The two great points of difference between a democracy and a republic are: first, the delegation of the government, in the latter, to a small number of citizens elected by the rest; secondly, the greater number of citizens, and greater sphere of country, over which the latter may be extended.
62160.0192The effect of the first difference is, on the one hand, to refine and enlarge the public views, by passing them through the medium of a chosen body of citizens, whose wisdom may best discern the true interest of their country, and whose patriotism and love of justice will be least likely to sacrifice it to temporary or partial considerations.
63160.0088Under such a regulation, it may well happen that the public voice, pronounced by the representatives of the people, will be more consonant to the public good than if pronounced by the people themselves, convened for the purpose.
64160.0041On the other hand, the effect may be inverted.
65160.0112Men of factious tempers, of local prejudices, or of sinister designs, may, by intrigue, by corruption, or by other means, first obtain the suffrages, and then betray the interests, of the people.
66160.0108The question resulting is, whether small or extensive republics are more favorable to the election of proper guardians of the public weal; and it is clearly decided in favor of the latter by two obvious considerations:
67170.0143In the first place, it is to be remarked that, however small the republic may be, the representatives must be raised to a certain number, in order to guard against the cabals of a few; and that, however large it may be, they must be limited to a certain number, in order to guard against the confusion of a multitude.
68170.0165Hence, the number of representatives in the two cases not being in proportion to that of the two constituents, and being proportionally greater in the small republic, it follows that, if the proportion of fit characters be not less in the large than in the small republic, the former will present a greater option, and consequently a greater probability of a fit choice.
69180.0163In the next place, as each representative will be chosen by a greater number of citizens in the large than in the small republic, it will be more difficult for unworthy candidates to practice with success the vicious arts by which elections are too often carried; and the suffrages of the people being more free, will be more likely to centre in men who possess the most attractive merit and the most diffusive and established characters.
70190.0071It must be confessed that in this, as in most other cases, there is a mean, on both sides of which inconveniences will be found to lie.
71190.0154By enlarging too much the number of electors, you render the representatives too little acquainted with all their local circumstances and lesser interests; as by reducing it too much, you render him unduly attached to these, and too little fit to comprehend and pursue great and national objects.
72190.0110The federal Constitution forms a happy combination in this respect; the great and aggregate interests being referred to the national, the local and particular to the State legislatures.
73200.0188The other point of difference is, the greater number of citizens and extent of territory which may be brought within the compass of republican than of democratic government; and it is this circumstance principally which renders factious combinations less to be dreaded in the former than in the latter.
74200.0198The smaller the society, the fewer probably will be the distinct parties and interests composing it; the fewer the distinct parties and interests, the more frequently will a majority be found of the same party; and the smaller the number of individuals composing a majority, and the smaller the compass within which they are placed, the more easily will they concert and execute their plans of oppression.
75200.0207Extend the sphere, and you take in a greater variety of parties and interests; you make it less probable that a majority of the whole will have a common motive to invade the rights of other citizens; or if such a common motive exists, it will be more difficult for all who feel it to discover their own strength, and to act in unison with each other.
76200.0080Besides other impediments, it may be remarked that, where there is a consciousness of unjust or dishonorable purposes, communication is always checked by distrust in proportion to the number whose concurrence is necessary.
77210.0156Hence, it clearly appears, that the same advantage which a republic has over a democracy, in controlling the effects of faction, is enjoyed by a large over a small republic,--is enjoyed by the Union over the States composing it.
78210.0105Does the advantage consist in the substitution of representatives whose enlightened views and virtuous sentiments render them superior to local prejudices and schemes of injustice?
79210.0051It will not be denied that the representation of the Union will be most likely to possess these requisite endowments.
80210.0132Does it consist in the greater security afforded by a greater variety of parties, against the event of any one party being able to outnumber and oppress the rest?
81210.0114In an equal degree does the increased variety of parties comprised within the Union, increase this security.
82210.0080Does it, in fine, consist in the greater obstacles opposed to the concert and accomplishment of the secret wishes of an unjust and interested majority?
83210.0055Here, again, the extent of the Union gives it the most palpable advantage.
84220.0055The influence of factious leaders may kindle a flame within their particular States, but will be unable to spread a general conflagration through the other States.
85220.0109A religious sect may degenerate into a political faction in a part of the Confederacy; but the variety of sects dispersed over the entire face of it must secure the national councils against any danger from that source.
86220.0110A rage for paper money, for an abolition of debts, for an equal division of property, or for any other improper or wicked project, will be less apt to pervade the whole body of the Union than a particular member of it; in the same proportion as such a malady is more likely to taint a particular county or district, than an entire State.
87230.0100In the extent and proper structure of the Union, therefore, we behold a republican remedy for the diseases most incident to republican government.
88230.0067And according to the degree of pleasure and pride we feel in being republicans, ought to be our zeal in cherishing the spirit and supporting the character of Federalists.
  • in the summary (3 sentences, matra's default)
  • not picked

TextRank scores each sentence by how much it shares with the others, and the summary is the top 3 returned in document order, as listed above. The scores come from a similarity graph between sentences that matra builds and does not return, so the figure shows the scores and not the graph.

Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: federalist-10/textrank.json.

4. Keyphrases

grade10.8density0.59compressionnone

Free functions over a Document. Both return a Keyphrases: the document's provenance with the method and n recorded, and phrases, each a Keyphrase (phrase, score, occurrences, the span of every place it was found), highest score first, capped at 200,000 tokens.

FunctionRanks by
keyphrases(doc, n), keyphrases_with(doc, n, KeyphraseMethod::Rake)co-occurrence degree over frequency, on NOUN, ADJ, and PROPN runs between stop words (rake, the default)
keyphrases_with(doc, n, KeyphraseMethod::Yake)per-term position, frequency, and context diversity, assembled into 1-word to 3-word candidates, score inverted so higher is more relevant (yake)
grade14.4density0.78compressionnone

Python spells them matra.keyphrases(doc, n=10) and matra.keyphrases(doc, n=10, method="yake").

grade8.0density0.50compressionnone

The two rank by different properties of the text, and on The Federalist No. 10 their top twelve lists share no phrase:

RAKE and YAKE on the same text

RAKE rank against YAKE rank for the top phrases of eachRAKE rankYAKE rank1101001000substantial occasion present1many judicial determination2federal constitution form3many lesser interest4lesser interest5adequate control6alone account6american constitution6attractive merit6candid review6civilized nation6dangerous vice61fate contemplate propensity2tendency break3develop tendency break4tendency break control5fate contemplate6character fate contemplate7complaint foundation evidence8contemplate propensity9contemplate propensity dangerous10break control violence11annihilation air essential12tendency
show the data
PhraseRAKE rankRAKE scoreYAKE rankYAKE score
fate contemplate propensitynot ranked132553.6
substantial occasion present19.00026940.451
many judicial determination27.83326420.468
tendency breaknot ranked222722.0
develop tendency breaknot ranked322609.3
federal constitution form36.42929830.335
many lesser interest45.87522730.576
tendency break controlnot ranked416940.2
fate contemplatenot ranked54523.1
lesser interest54.04217220.727
adequate control64.00024260.520
alone account64.00014860.823
american constitution64.00017930.703
attractive merit64.00028490.404
candid review64.00014650.840
character fate contemplatenot ranked63038.8
civilized nation64.00021360.615
dangerous vice64.00012940.967
complaint foundation evidencenot ranked71730.8
contemplate propensitynot ranked8463.1
contemplate propensity dangerousnot ranked9455.5
break control violencenot ranked10362.2
annihilation air essentialnot ranked11218.2
tendency2721.00012158.4
  • in RAKE's top 12
  • in YAKE's top 12
  • only one method produces the phrase

Both of matra's scores rank higher as more relevant: RAKE sums each word's co-occurrence degree over its frequency, and matra's YAKE reports the reciprocal of the published YAKE score, which runs lower as more relevant. The two scores share no scale, so the figure compares ranks, on a log scale, out of 466 RAKE and 3111 YAKE candidates. Equal scores share a rank, and a tie at the cut of 12 is taken in alphabetical order. A phrase a method does not produce at all is a hollow dot on the side that ranks it, with no line.

Input: The Federalist No. 10, the essay body, James Madison (as Publius). Public domain (first published 1787). Produced by matra 0.3.0 with the UDPipe model english-ewt-ud-2.5-191206 (SHA-256 784bd0fa85e3). Data: federalist-10/keyphrases.json.

5. Semantic clusters

grade7.5density0.55compressionnone

Not a field on any type above. The port and clusters are always compiled; the Model2Vec adapter is behind the model2vec feature.

ItemIs
Embedderthe port: embed(&[&str]) returning one Embedding per text, plus identity()
Model2Vecthe adapter, reading model.safetensors, tokenizer.json, and config.json from a directory. from_dir(dir) loads what is there; potion_base_8m(dir), with dir a models directory, and from_config(cfg) fetch the pinned reference model into its potion-base-8M subdirectory when it is absent
clusters(doc, embedder, threshold)embeds a document's sentences with the embedder you pass, then clusters them; vectors you already hold arrive through an Embedder that returns them
Clustersthe document's provenance with the method (cosine), the threshold and the embedder's identity recorded, clusters, and edges
Cluster, SimilarityEdgea cluster's members (sentence indices), and a pair a, b whose score cleared the threshold
grade12.5density0.68compressionnone

Capped at 2,000 sentences. Python spells them matra.clusters(doc, embedder, threshold=0.85) and Model2Vec (including Model2Vec.potion_base_8m); matra.clusters takes a Model2Vec or any object with embed and identity.

The pipeline surface

ValueConstructorsCarries
Inputtext(string, format), bytes(data, format), file(path), blocks(blocks)one document: text, bytes, a file, or headings and paragraphs you supply
Inputsdir(path)the files in a directory, as an iterator of Input
Enginenew(), from_config(cfg), from_config_with_notice(cfg, notice)analyze one input, analyze_all of many
Configresolve(), from_file(path), from_sources(...), with_model_dir(dir)where models live and what the defaults are, plus the ValueSource each value came from
grade12.4density0.58compressionnone

Engine::new() is the no-setup constructor: locate the model directory from the environment, then build the standard pipeline, fetching the pinned English model if it is not on disk. It reads no configuration file; Engine::from_config(&Config::resolve()?) reads one.

grade10.8density0.66compressionnone

engine.analyze_all(inputs) returns a lazy iterator of per-document results, one per input in order, and one bad file does not abort the stream. Python's Engine.analyze_all(dir=path) is the same lazy walk, yielding a Document or a DocumentError dict per file.

Traceability

grade11.0density0.55compression0.48

Every heading, paragraph, sentence and token carries a span into the analysed text, in UTF-8 bytes, and skipped lists what the reader saw and did not analyse (a table, a code block, front matter), each with its span. Every byte of a markdown or plain-text input that is not whitespace lies in exactly one heading, paragraph or skipped span, so nothing is dropped without a trace. The text fields are normalized: Paragraph.text is what the parser reads, a markdown paragraph's inline markup removed, and Sentence.text is rebuilt by the provider, the UDPipe adapter joining the token forms with a space unless a token carries SpaceAfter=No. The span, not the text, says where a value came from. Each paragraph is parsed on its own and its sentences are attached to it, so the chain from a token up through its sentence, paragraph, and section holds by construction rather than by text matching.

Where to go next

grade9.2density0.73compressionnone

Domain model is the full type graph. Methodology is every formula. Errors is every failure.

This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 20 paragraphs, in the margin. the data