Read a Markdown document's structure
on this page
grade8.4density0.60compressionnone
A short Markdown document with two headings and a quote, read into sections and paragraphs.
Input
The calls read it as field-notes.md in the directory you run them from. Save it there:
curl -fLO https://mox-labs.github.io/matra/example-files/markdown-structure/field-notes.md # Field notes The survey team reached the northern ridge on the third day. Snow had closed the lower pass, so the route ran along the eastern face instead. > Radio contact was lost for six hours. ## Findings Three of the five markers were intact and matched their recorded positions. The fourth had been moved about ten metres downhill by a rockfall, and its base plate was cracked. The fifth could not be found, although the team searched the slope below it for most of an afternoon. A replacement marker will be set on the next visit, once the pass is open.
The call
Rust
// cargo add matra@0.3 serde_json
use matra::{Engine, Input};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Loads the English model, downloading it on first use.
let engine = Engine::new()?;
let doc = engine.analyze(Input::file("field-notes.md"))?;
println!("{}", serde_json::to_string_pretty(&doc)?);
Ok(())
} Python
# pip install 'matra>=0.3'
import json
import matra
engine = matra.Engine() # downloads the English model on first use
doc = engine.analyze(file="field-notes.md") # a str alone would be text, not a path
print(json.dumps(doc.to_dict(), indent=2, ensure_ascii=False)) CLI
$ matra analyze --json field-notes.md It prints the output below, inside an envelope that names the command and the input:
{
"format_version": 2,
"command": "analyze",
"input": "field-notes.md",
"result": /* the output below */,
"error": null
}Output
Trimmed: sentences shown as a count, long strings shortened.
{
"provenance": {
"matra": "0.3.0",
"schema": "2.0",
"parser": {
"name": "udpipe",
"model": "english-ewt-ud-2.5-191206",
"sha256": "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957",
"license": "CC BY-NC-SA 4.0 (non-commercial)"
},
"format": "markdown",
"source": {
"path": "field-notes.md",
"bytes": 564,
"sha256": "bbf7f1c703c0bf2b998b1a8fc61e1fe0f0e6aeb670b169fd5e2da372c62fe55f"
}
},
"sections": [
{
"heading": "Field notes",
"level": 1,
"span": {
"byte_start": 0,
"byte_end": 13
},
"paragraphs": [
{
"span": {
"byte_start": 15,
"byte_end": 156
},
"text": "The survey team reached the northern ridge on the third day. Snow had c…",
"kind": "paragraph",
"sentences": [ /* 2 items */ ],
"metrics": {
"readability_grade": 4.91076923076923,
"lexical_density": 0.6538461538461539,
"compression_ratio": null
}
},
{
"span": {
"byte_start": 158,
"byte_end": 197
},
"text": "Radio contact was lost for six hours.",
"kind": "quote",
"sentences": [],
"metrics": {
"readability_grade": null,
"lexical_density": null,
"compression_ratio": null
}
}
]
},
{
"heading": "Findings",
"level": 2,
"span": {
"byte_start": 199,
"byte_end": 210
},
"paragraphs": [
{
"span": {
"byte_start": 212,
"byte_end": 563
},
"text": "Three of the five markers were intact and matched their recorded positi…",
"kind": "paragraph",
"sentences": [ /* 4 items */ ],
"metrics": {
"readability_grade": 6.8750000000000036,
"lexical_density": 0.484375,
"compression_ratio": 0.5897435897435898
}
}
]
}
],
"metrics": {
"total_words": 91,
"total_sentences": 6,
"paragraph_count": 3,
"mean_sentence_length": 15.166666666666666,
"sentence_length_std": 3.3714487489307423,
"vocabulary_ttr": 0.6923076923076923,
"nominalization_ratio": 0.01098901098901099,
"passive_ratio": 0.5
},
"skipped": []
} The whole output: output.json, and what the CLI prints, cli.json. The examples gate runs all three calls and compares what each prints with these files.
grade7.8density0.50compressionnone
Programming model steps this same document through each stage of the pipeline, to show what each stage adds.
What to notice
sectionsfollow the headings: each hasheading, the heading's text,level, the number of#marks, andspan, where the heading lies in the file,#marks included.- The quote is kept as a paragraph whose
kindisquote, and it is never parsed: itssentenceslist is empty and all three of its measures arenull. Itsspancovers the>markers its text leaves out. - The Markdown reader is chosen by the
.mdextension in all three calls. Read as plain text, the same file is one section with no heading, and the heading lines and the quote are parsed as sentences,#marks and all.
This page, measured by matra 0.3.0 with english-ewt-ud-2.5-191206: 2 paragraphs, in the margin. the data