Enhancement proposalEPR-0003Distribution and provisioning○ proposed
● 0 of 2 decisions settled○ 3 assumptions to confirmpinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: distribution_and_provisioningdecides: the owner, who alone sets the status to accepted
EPR-0003● 0 of 2 settled○ 3 to confirm
Enhancement proposalEPR-0003Distribution and provisioning○ proposed
pinned at 4fcfb4apull request #133tracking issue: none (a baseline proposal describes code that already ships, so there is no milestone to track)started: 2026-10-04feature name: distribution_and_provisioningdecides: the owner, who alone sets the status to accepted
matra ships as one crate on crates.io and one package on PyPI, under one
name, one version and one attribution. PyPI gets four prebuilt wheels per
release, all against CPython's stable ABI from 3.12, plus a source
distribution. The models matra parses and embeds with are not shipped: each
is pinned in the source by URL, size and SHA-256, downloaded on first use
into a directory matra resolves, verified in memory, and loaded from the
bytes that were verified. A failure to fetch is reported as I/O, never as an
invalid model. A release runs from one workflow that a person approves for
each registry, and every artifact carries build provenance a caller can
verify.
This is a baseline proposal. It describes the code at the pinned commit and
carries forward the reasons that still hold from RFC-0005, RFC-0010
(decision 6), RFC-0011, RFC-0013, RFC-0014 and RFC-0015. A later proposal
that changes any of it says so. Unmarked statements about the code are
observed at the pinned commit through the link beside them; a claim that is
inferred or assumed says so.
A caller meets this foundation before any other: pip install matra,
cargo add matra, and the first run that has no model yet. Each decision
here was made after that first meeting went wrong.
Wheels for the platforms people actually run. Two clean-room installs
before 0.2.0 found that the wheels missed Linux on arm64, required glibc
2.34, and were tied to CPython 3.12, so most first installs fell to a source
build, which then failed for want of a C++ compiler
(RFC-0014). The stable ABI and
a manylinux container fixed all three at once.
A model is pinned, not named. matra downloads exactly one artifact set
per model, the one whose digest is a constant in its source. That is what
makes a download acceptable in a library at all
(RFC-0010 decision 6, as
amended by RFC-0011): what matra
refuses is unpinned network access, fetching whatever a name resolves to
today.
A failed fetch must say what failed. A clean-room pass measured a first
run that wrote nothing for up to 35 seconds, had no timeout at all, left a
15 MB orphan on Ctrl-C, and reported DNS, TLS and full-disk failures as
"invalid model" (RFC-0015).
The provisioning code today answers each of those.
Publishing is irreversible. Every downstream build inherits matra's
supply-chain posture, so publishing runs from a least-privilege workflow
with no stored registry token and waits for a person
(RFC-0005).
pip install matra or uvx matra gets a prebuilt wheel on Linux x86_64
and aarch64 (glibc 2.17 and later), and macOS x86_64 and arm64, for every
GIL-enabled CPython from 3.12 on.
The wheel carries the library, the Python bindings and the command line, and has no runtime dependencies ● observedpyproject.toml:5-42.
cargo add matra gets the library with UDPipe; cargo install matra --features cli gets the binary.
Building from source, by either route, needs Rust 1.88 or later and a C++ compiler, because UDPipe is C++ ● observedCargo.toml:1-14grounds for decision 1.
Windows has no wheel.
pyproject.toml lines 5 to 42, at 4fcfb4a
5 [project]
6 name = "matra"
7 version = "0.2.1"
8 description = "NLP library: UDPipe structured parse, base text metrics, summarization, keyphrase extraction."
9 readme = "README.md"
10 license = { text = "MIT" }
11 authors = [{ name = "mox labs", email = "yza.v@moxlabs.org" }]
12 requires-python = ">=3.12"
13 # No runtime dependencies. The command line is the Rust binary reached
14 # through the extension module, the way ruff and uv ship theirs.
15 dependencies = []
1617 [project.optional-dependencies]
18 typecheck = [
19 "mypy>=1.10",
20 ]
21 test = [
22 "pytest>=8",
23 "pytest-cov>=5",
24 ]
25 lint = [
26 "ruff>=0.6",
27 ]
28 dev = [
29 "matra[typecheck,test,lint]",
30 ]
3132 [project.scripts]
33 matra = "matra.cli:main"
3435 [tool.maturin]
36 features = ["python", "udpipe", "model2vec"]
37 python-source = "python"
38 module-name = "matra._core"
39 include = [
40 { path = "python/matra/py.typed", format = "wheel" },
41 { path = "python/matra/_core.pyi", format = "wheel" },
42 ]
The first call that needs the parsing model downloads 16 MB from LINDAT
into the model directory (by default ~/.local/share/matra/models;
EPR-0004 gives the resolution
order). The command line says so on stderr, naming the file, its size, the
directory and the model's licence;
the library says nothing itself and hands a caller who wants to say something a ProvisionNotice● observeddomain.rs:34-72:
src/domain.rs lines 34 to 72, at 4fcfb4a
34 /// What a constructor is about to download, reported before the transfer
35 /// begins.
36 ///
37 /// A first run fetches a model over a link whose speed matra cannot know.
38 /// Measured cold starts ran from 3 to 35 seconds with nothing on screen,
39 /// which is indistinguishable from a hung process, and the natural
40 /// response to a hung process makes it worse. The library writes to no
41 /// terminal, so it hands the facts to whoever does: the UDPipe
42 /// constructors that provision carry a `_with_notice` form that calls a
43 /// closure once per fetch, and only when the artifact is not already on
44 /// disk.
45 ///
46 /// That is `Udpipe::english_with_notice`,
47 /// `Udpipe::from_config_with_notice` and `Engine::from_config_with_notice`.
48 /// `Model2Vec::potion_base_8m` fetches 30.2 MB across three artifacts
49 /// and has no notice form, so a semantic first run is still silent.
50 ///
51 /// Facts rather than a sentence. The caller decides the wording, the
52 /// stream, and whether to say anything at all.
53 #[derive(Debug, Clone, PartialEq, Eq)]
54 #[non_exhaustive]
55 pub struct ProvisionNotice {
56 /// The artifact's filename, as it will appear in `destination`.
57 pub artifact: String,
58 /// The artifact's pinned size in bytes. Known before the fetch
59 /// because the size is a constant in the source, checked against
60 /// what arrives.
61 pub bytes: u64,
62 /// The directory the artifact is being fetched into.
63 pub destination: PathBuf,
64 /// The artifact's license, named for a reader, for example
65 /// `CC BY-NC-SA 4.0 (non-commercial)`. A model is a separate work
66 /// from matra and carries its own terms, which need not be matra's
67 /// MIT. Pinned in the source beside the URL, so a change of model
68 /// changes both.
69 pub license: String,
70 /// Where the full text of `license` is published.
71 pub license_url: String,
72 }
let engine = matra::Engine::from_config_with_notice(&cfg, |n| { eprintln!("fetching {} ({} bytes) into {}", n.artifact, n.bytes, n.destination.display());})?;
The embedding model, 30 MB across three files from Hugging Face, is fetched the same way the first time Model2Vec::potion_base_8m or Model2Vec::from_config runs, and has no notice form ● observeddomain.rs:34-72.
src/domain.rs lines 34 to 72, at 4fcfb4a
34 /// What a constructor is about to download, reported before the transfer
35 /// begins.
36 ///
37 /// A first run fetches a model over a link whose speed matra cannot know.
38 /// Measured cold starts ran from 3 to 35 seconds with nothing on screen,
39 /// which is indistinguishable from a hung process, and the natural
40 /// response to a hung process makes it worse. The library writes to no
41 /// terminal, so it hands the facts to whoever does: the UDPipe
42 /// constructors that provision carry a `_with_notice` form that calls a
43 /// closure once per fetch, and only when the artifact is not already on
44 /// disk.
45 ///
46 /// That is `Udpipe::english_with_notice`,
47 /// `Udpipe::from_config_with_notice` and `Engine::from_config_with_notice`.
48 /// `Model2Vec::potion_base_8m` fetches 30.2 MB across three artifacts
49 /// and has no notice form, so a semantic first run is still silent.
50 ///
51 /// Facts rather than a sentence. The caller decides the wording, the
52 /// stream, and whether to say anything at all.
53 #[derive(Debug, Clone, PartialEq, Eq)]
54 #[non_exhaustive]
55 pub struct ProvisionNotice {
56 /// The artifact's filename, as it will appear in `destination`.
57 pub artifact: String,
58 /// The artifact's pinned size in bytes. Known before the fetch
59 /// because the size is a constant in the source, checked against
60 /// what arrives.
61 pub bytes: u64,
62 /// The directory the artifact is being fetched into.
63 pub destination: PathBuf,
64 /// The artifact's license, named for a reader, for example
65 /// `CC BY-NC-SA 4.0 (non-commercial)`. A model is a separate work
66 /// from matra and carries its own terms, which need not be matra's
67 /// MIT. Pinned in the source beside the URL, so a change of model
68 /// changes both.
69 pub license: String,
70 /// Where the full text of `license` is published.
71 pub license_url: String,
72 }
Both models are separate works with their own licences.
The parsing model is CC BY-NC-SA 4.0, which does not permit commercial use; the notice carries that licence and its URL ● observednlp/udpipe.rs:18-108.
The embedding model is MIT per its model card○ assumedgrounds for decision 2,
which the documentation states and
the code does not carry ● observedembed/model2vec.rs:40-113.
src/nlp/udpipe.rs lines 18 to 108, at 4fcfb4a
18 /// Expected SHA-256 of the English UD-EWT 2.5 (release 191206) UDPipe model.
19 /// Refresh with `scripts/fetch-model-hash.sh` when updating the model version.
20 const ENGLISH_MODEL_SHA256: &str =
21 "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957";
22 23 /// Expected size in bytes, checked before hashing as a fast-fail guard.
24 const ENGLISH_MODEL_SIZE: u64 = 16_309_608;
25 26 /// Direct download URL for the pinned model, fetched by [`fetch_capped`].
27 /// LINDAT migrated the bitstream endpoint from
28 /// `/repository/xmlui/bitstream/...` (now a 200 HTML preview) to
29 /// `/repository/server/api/core/bitstreams/...` (the actual binary), so
30 /// the URL is pinned here in its working form rather than derived from a
31 /// model name.
32 const ENGLISH_MODEL_URL: &str = "https://lindat.mff.cuni.cz/repository/server/api/core/bitstreams/handle/11234/1-3131/english-ewt-ud-2.5-191206.udpipe?sequence=17&isAllowed=y";
33 34 /// The pinned model's license, as its LINDAT record
35 /// (<https://hdl.handle.net/11234/1-3131>) states it: Creative Commons
36 /// Attribution-NonCommercial-ShareAlike 4.0 International. matra's code
37 /// is MIT; this file is a separate work with its own terms, and the
38 /// download notice carries them so the command line can say so. Beside
39 /// the URL because a new model means checking both.
40 const ENGLISH_MODEL_LICENSE: &str = "CC BY-NC-SA 4.0 (non-commercial)";
41 42 /// Where the full text of [`ENGLISH_MODEL_LICENSE`] is published.
43 const ENGLISH_MODEL_LICENSE_URL: &str = "https://creativecommons.org/licenses/by-nc-sa/4.0/";
44 45 /// The pinned English model as one record, so the provisioning path
46 /// takes the pin whole rather than six loose arguments.
47 const ENGLISH_MODEL: Pin<'static> = Pin {
48 filename: ENGLISH_MODEL_FILENAME,
49 size: ENGLISH_MODEL_SIZE,
50 sha256: ENGLISH_MODEL_SHA256,
51 url: ENGLISH_MODEL_URL,
52 license: ENGLISH_MODEL_LICENSE,
53 license_url: ENGLISH_MODEL_LICENSE_URL,
54 };
55 56 /// A pinned artifact: what it is called on disk, what it must measure,
57 /// where it comes from, and the terms it is published under. A test pins
58 /// a fixture of its own through the same shape.
59 #[derive(Debug, Clone, Copy)]
60 struct Pin<'a> {
61 filename: &'a str,
62 size: u64,
63 sha256: &'a str,
64 url: &'a str,
65 license: &'a str,
66 license_url: &'a str,
67 }
68 69 /// Ceiling on the downloaded model. The pinned artifact is 16.3 MB, so
70 /// this is four times the real thing: headroom for a later model version
71 /// and small enough that a redirect to something enormous costs a
72 /// bounded read rather than the machine's memory. Past it,
73 /// [`Error::InputTooLarge`] with `what` set to `"udpipe_download"`. The
74 /// same number the embedding adapter uses, because it is the same
75 /// question.
76 const MAX_MODEL_BYTES: usize = 64 * 1024 * 1024;
77 78 /// Ceiling on the fetch, DNS lookup through the last byte of the body.
79 /// The pinned model is 16.3 MB, which needs this whole budget only on a
80 /// link under 0.5 Mbit/s, so the margin is generous for a slow
81 /// connection and still finite: a stalled or black-holed transfer fails
82 /// instead of holding the caller for as long as the socket stays open.
83 /// Before this existed, 90 seconds against an unreachable host produced
84 /// zero bytes with the process still running, unbounded.
85 const FETCH_TIMEOUT: Duration = Duration::from_secs(300);
86 87 /// Ceiling on establishing the connection, socket and TLS handshake
88 /// included. Separate from [`FETCH_TIMEOUT`] so an unreachable host
89 /// fails in seconds rather than consuming the whole transfer budget.
90 const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
91 92 /// Age past which a temporary download directory cannot belong to a
93 /// live provisioning call, so it is a leftover and this call reclaims
94 /// it.
95 ///
96 /// Twice [`FETCH_TIMEOUT`]. A fetch cannot outlive that budget, and the
97 /// directory is created after the bytes are already in memory, so its
98 /// modification time is within milliseconds of the write that follows.
99 /// Anything older than ten minutes is therefore an orphan from a killed
100 /// process, not a peer. The margin is what protects a concurrent cold
101 /// start: three processes racing on one empty model directory each see
102 /// the others' directories as minutes-fresh and leave them alone.
103 const STALE_TEMP_AGE: Duration = Duration::from_secs(600);
104105 /// Prefix of the temporary directory a download lands in. Shared by the
106 /// creation and the reclaim, so the sweep cannot look for a name the
107 /// writer does not use.
108 const TEMP_DIR_PREFIX: &str = ".tmp.download.";
src/embed/model2vec.rs lines 40 to 113, at 4fcfb4a
40 /// The three artifact filenames, in the order the identity digest covers
41 /// them. Every path in this file is built from this array, so the digest
42 /// and the loader can never disagree about which file is which.
43 const ARTIFACT_FILES: [&str; 3] = ["model.safetensors", "tokenizer.json", "config.json"];
44 45 /// SHA-256 over the three artifact files of the pinned potion-base-8M
46 /// release, concatenated in [`ARTIFACT_FILES`] order. The same value
47 /// `spec/tests/semantic/reference-model.json` carries as `artifact_hash`,
48 /// and the same value [`Model2Vec::model_hash`] reports once loaded.
49 ///
50 /// Refresh this and [`POTION_BASE_8M_URLS`] together with
51 /// `scripts/fetch-embedding-hash.sh <revision>`.
52 #[cfg(not(target_arch = "wasm32"))]
53 const POTION_BASE_8M_SHA256: &str =
54 "81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4";
55 56 /// Download URLs for the pinned release, one per [`ARTIFACT_FILES`]
57 /// entry in the same order. The path segment after `resolve/` is a git
58 /// commit on the Hugging Face repository rather than a branch, so the
59 /// URL names one immutable tree; `main` would move under the pin. The
60 /// digest is what actually decides whether the bytes are trusted, and
61 /// the revision is what makes a mismatch a bug report rather than an
62 /// upstream edit.
63 ///
64 /// Refresh with `scripts/fetch-embedding-hash.sh <revision>`.
65 #[cfg(not(target_arch = "wasm32"))]
66 const POTION_BASE_8M_URLS: [&str; 3] = [
67 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/model.safetensors",
68 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/tokenizer.json",
69 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/config.json",
70 ];
71 72 /// Ceiling on any one downloaded artifact. The largest of the three in
73 /// the pinned release is `model.safetensors` at 30,236,760 bytes, so this
74 /// is a shade over twice the real thing: headroom for a later revision,
75 /// and small enough that a redirect to something enormous costs a bounded
76 /// read rather than the machine's memory. Past it,
77 /// [`Error::InputTooLarge`] with `what` set to `"embedding_download"`.
78 #[cfg(not(target_arch = "wasm32"))]
79 const MAX_ARTIFACT_BYTES: usize = 64 * 1024 * 1024;
80 81 /// Ceiling on one artifact fetch, DNS lookup through the last byte of
82 /// the body. The largest artifact in the pinned release is about 30 MB,
83 /// which needs the whole of this budget only on a link under 1 Mbit/s,
84 /// so the margin is generous for a slow connection and still finite: a
85 /// stalled or black-holed transfer fails instead of holding the caller
86 /// for as long as the socket stays open.
87 #[cfg(not(target_arch = "wasm32"))]
88 const FETCH_TIMEOUT: Duration = Duration::from_secs(300);
89 90 /// Ceiling on establishing the connection, socket and TLS handshake
91 /// included. Separate from [`FETCH_TIMEOUT`] so an unreachable host
92 /// fails in seconds rather than consuming the whole transfer budget.
93 #[cfg(not(target_arch = "wasm32"))]
94 const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
95 96 /// Prefix of the temporary an artifact lands through. Shared by the
97 /// creation and the reclaim, so the sweep cannot look for a name the
98 /// writer does not use.
99 #[cfg(not(target_arch = "wasm32"))]
100 const TEMP_FILE_PREFIX: &str = ".tmp.";
101102 /// Age past which a temporary cannot belong to a live provisioning call,
103 /// so it is a leftover and the next call reclaims it.
104 ///
105 /// Twice [`FETCH_TIMEOUT`], the same rule and the same number
106 /// `nlp/udpipe.rs` uses. All three artifacts are fetched and verified
107 /// before any of them is written, so a temporary's modification time is
108 /// within milliseconds of the write that follows it, and anything older
109 /// than ten minutes is an orphan from a killed process rather than a
110 /// peer. The margin is what protects a concurrent cold start: racing
111 /// processes see each other's temporaries as seconds old.
112 #[cfg(not(target_arch = "wasm32"))]
113 const STALE_TEMP_AGE: Duration = Duration::from_secs(600);
Error::Io, ErrorKind::NotConnected, naming the URL
io
Connect or transfer timeout
Error::Io, ErrorKind::TimedOut
io
TLS certificate rejected
Error::Io, with a sentence saying why installing a CA does not help
io
Non-2xx status
Error::Io
io
Response over 64 MiB
Error::InputTooLarge, what = udpipe_download or embedding_download
input_too_large
Model directory cannot be created or written
Error::Io, naming the operation and the path
io
Bytes arrived and failed the digest twice, or verified and did not load
Error::ModelInvalid
model_invalid
The rule behind the table:
model_invalid is about bytes that arrived ● observednlp/udpipe.rs:469-489
(RFC-0015). Behind a proxy
that re-signs TLS, matra cannot be made to trust the proxy; the way through
is to fetch the pinned file by hand and place it in the model directory,
which is exactly as trustworthy because the digest decides.
src/nlp/udpipe.rs lines 469 to 489, at 4fcfb4a
469 /// Map a `ureq` failure to [`Error::Io`].
470 ///
471 /// A download that never arrives is a transport failure, not an invalid
472 /// model: [`Error::ModelInvalid`] is reserved for bytes that did arrive
473 /// and then failed the digest or the loader. A non-2xx status belongs on
474 /// this side too, because what came back was a server's answer about the
475 /// request rather than a model. The `io::ErrorKind` is preserved where
476 /// `ureq` knows it, so a caller can tell a timeout from an unreachable
477 /// host without reading the message. RFC-0015 records the classification
478 /// and what it replaced.
479 fn transport_failure(url: &str, error: &ureq::Error) -> Error {
480 use std::io::ErrorKind;
481482 let kind = match error {
483 ureq::Error::Io(e) => e.kind(),
484 ureq::Error::Timeout(_) => ErrorKind::TimedOut,
485 ureq::Error::ConnectionFailed | ureq::Error::HostNotFound => ErrorKind::NotConnected,
486 _ => ErrorKind::Other,
487 };
488 Error::Io(std::io::Error::new(kind, download_message(url, error)))
489 }
122 impl Udpipe {
123 /// Load from a file path. Fails fast if the model is invalid.
124 pub fn from_path(path: impl AsRef<Path>) -> crate::domain::Result<Self> {
125 let path = path.as_ref();
126 if !path.exists() {
127 return Err(Error::ModelNotFound(path.to_path_buf()));
128 }
129 let model = Model::load(path).map_err(|e| Error::ModelInvalid(e.to_string()))?;
130 Ok(Self { model })
131 }
132133 /// Load from bytes (e.g. embedded via include_bytes!).
134 pub fn from_bytes(data: &[u8]) -> crate::domain::Result<Self> {
135 let model =
136 Model::load_from_memory(data).map_err(|e| Error::ModelInvalid(e.to_string()))?;
137 Ok(Self { model })
138 }
src/embed/model2vec.rs lines 143 to 174, at 4fcfb4a
143 /// Load a model from a directory holding `model.safetensors`,
144 /// `tokenizer.json`, and `config.json`.
145 ///
146 /// No network is touched, ever, whatever the directory holds:
147 /// this constructor loads what the caller supplied and nothing else.
148 /// [`Model2Vec::potion_base_8m`] is the one that provisions.
149 ///
150 /// Paths are trusted as given, matching the `Udpipe::from_path`
151 /// precedent for caller-supplied model locations: no size cap (the
152 /// artifacts are the model, and a huge one fails in safetensors
153 /// validation, not in an OOM loop) and no symlink rejection.
154 ///
155 /// # Errors
156 ///
157 /// [`Error::ModelNotFound`] if a file is absent, [`Error::Io`] if one
158 /// cannot be read, [`Error::ModelInvalid`] if the artifact does not
159 /// parse, uses an embedding dtype other than f32, or panics the
160 /// loader internally (the panic is caught at this boundary).
161 pub fn from_dir(dir: impl AsRef<Path>) -> domain::Result<Self> {
162 let dir = dir.as_ref();
163 let [model_path, tokenizer_path, config_path] = ARTIFACT_FILES.map(|f| dir.join(f));
164 for p in [&model_path, &tokenizer_path, &config_path] {
165 if !p.exists() {
166 return Err(Error::ModelNotFound(p.clone()));
167 }
168 }
169 let model_bytes = fs::read(&model_path).map_err(|e| io_at("read", &model_path, &e))?;
170 let tokenizer_bytes =
171 fs::read(&tokenizer_path).map_err(|e| io_at("read", &tokenizer_path, &e))?;
172 let config_bytes = fs::read(&config_path).map_err(|e| io_at("read", &config_path, &e))?;
173 catch_embed_panic(|| Self::from_bytes(&model_bytes, &tokenizer_bytes, &config_bytes))
174 }
The same organization is the copyright holder in LICENSE, the entity
author in CITATION.cff, and the author in .claude-plugin/plugin.json,
because registry metadata is immutable per version and attribution that
disagrees across files cannot be corrected in place
(RFC-0013).
The crate excludes blueprints/, site/ and the tests that read site/, and keeps skills/matra/, which the binary embeds ● observedCargo.toml:15-37.
Cargo.toml lines 15 to 37, at 4fcfb4a
15 # An exclude list, so anything not named here ships. `blueprints/` is the
16 # design record (RFCs and EPs) and is for contributors, not crate users. `skills/matra/**` must
17 # keep shipping: it is the agent-facing skill, and the crate is where it is
18 # distributed from. Adding it here would remove public surface, not weight.
19 # `site/` is the docsite, its pages under `site/content/`.
20 # `tests/cited_figures.rs` and `tests/error_tables.rs` read `site/content/`,
21 # which does not ship, so either would fail every `cargo test` run from the
22 # published tarball. A new test that reads `site/` belongs on this list too,
23 # and so do `examples/docsite_figures.rs`, which generates the docsite's
24 # figure data from `site/inputs/`, and `examples/docsite_examples.rs`, which
25 # compiles the docsite's worked examples from `site/examples/`.
26 exclude = [
27 "scratch/*",
28 ".claude/*",
29 "target/*",
30 "site/*",
31 "blueprints/*",
32 "*.udpipe",
33 "tests/cited_figures.rs",
34 "tests/error_tables.rs",
35 "examples/docsite_figures.rs",
36 "examples/docsite_examples.rs",
37 ]
build-wheels builds four wheels: Linux x86_64 on ubuntu-latest and Linux aarch64 on ubuntu-24.04-arm, both inside ghcr.io/pyo3/maturin pinned by index digest; macOS arm64 natively and macOS x86_64 cross-compiled, both on macos-14● observedrelease.yml:311-351.
pyo3 is built with abi3-py312● observedCargo.toml:53-57.
The workflow fails a release if a wheel is not tagged cp312-abi3 or a Linux wheel is not manylinux2014, and installs each native wheel on CPython 3.13 with --only-binary :all:● observedrelease.yml:402-441.
CI asserts the abi3 tag on every push ● observedci.yml:359-387.
.github/workflows/release.yml lines 311 to 351, at 4fcfb4a
311 # No third-party build action: the org Actions policy validates every
312 # `uses` ref at workflow start, and keeping the build to `actions/*`
313 # plus shell removes that variable. The Linux half is built inside
314 # ghcr.io/pyo3/maturin instead, which is the same manylinux2014 base
315 # PyO3/maturin-action would have used, reached with plain `docker run`
316 # and pinned by index digest the way the policy pins action SHAs.
317 #
318 # Building on the runner rather than in that image is what produced a
319 # manylinux_2_34 wheel for 0.1.x, which pip rejects on Debian 11,
320 # Ubuntu 20.04, RHEL 8 and Amazon Linux 2. The container drops the
321 # floor to glibc 2.17. See RFC-0014.
322 #
323 # Nothing watches the image digest: dependabot covers cargo, actions
324 # and pip, and cannot see a digest written inline in a `run:` step.
325 # Bumping it is a person's job, on the release checklist.
326 build-wheels:
327 name: Wheel ${{ matrix.platform.name }}
328 needs: [verify, tag]
329 runs-on: ${{ matrix.platform.runner }}
330 permissions:
331 contents: read
332 # A starved runner must fail loudly rather than squat on the
333 # concurrency group and block the next dispatch.
334 timeout-minutes: 45
335 strategy:
336 fail-fast: false
337 matrix:
338 platform:
339 # ubuntu-24.04-arm is a GitHub-hosted arm64 runner, free for
340 # public repositories. It exists so the Linux aarch64 wheel is
341 # built natively rather than under QEMU, which would push the
342 # job past its timeout.
343 #
344 # macos-x86_64 cross-compiles from the arm64 runner: GitHub
345 # is retiring the Intel macos-13 runners and jobs targeting
346 # them starve in the queue. Cross-built wheels cannot smoke
347 # test on the arm host, so those steps are native-only.
348 - { name: linux-x86_64, runner: ubuntu-latest, manylinux: true, target: "" }
349 - { name: linux-aarch64, runner: ubuntu-24.04-arm, manylinux: true, target: "" }
350 - { name: macos-x86_64, runner: macos-14, manylinux: false, target: x86_64-apple-darwin }
351 - { name: macos-aarch64, runner: macos-14, manylinux: false, target: "" }
53 # abi3-py312 builds against the stable ABI, so one wheel per platform
54 # covers CPython 3.12 and every later GIL-enabled 3.x. Without it the wheel
55 # is tagged cp312 and pip on 3.13 or 3.14 falls through to a source build.
56 # Free-threaded builds take a different ABI tag (RFC-0014).
57 pyo3 = { version = "0.29", features = ["extension-module", "abi3-py312"], optional = true }
.github/workflows/release.yml lines 402 to 441, at 4fcfb4a
402 # The abi3 contract, checked rather than assumed: one wheel per
403 # platform, tagged cp312-abi3, and Linux wheels no higher than the
404 # manylinux2014 floor. A dropped `abi3-py312` feature or a build
405 # that escaped the container fails here, not on a user's machine.
406 - name: The wheel is abi3, and Linux wheels are manylinux2014
407 shell: bash
408 run: |
409 set -euo pipefail
410 count=$(find dist -maxdepth 1 -name '*.whl' | wc -l | tr -d ' ')
411 [ "$count" = "1" ] || { echo "::error::expected 1 wheel, got $count"; exit 1; }
412 name=$(basename dist/*.whl)
413 echo "built $name"
414 case "$name" in
415 *-cp312-abi3-*) ;;
416 *) echo "::error::$name is not an abi3 wheel"; exit 1 ;;
417 esac
418 if [ "${{ matrix.platform.manylinux }}" = "true" ]; then
419 case "$name" in
420 *manylinux2014_*) ;;
421 *) echo "::error::$name is not manylinux2014"; exit 1 ;;
422 esac
423 fi
424 # Native builds only: a cross-built wheel cannot run on the host.
425 # The interpreter here is deliberately newer than the one the
426 # wheel was built against, so this proves the abi3 wheel installs
427 # without a source build on a later CPython. --only-binary :all:
428 # makes pip fail rather than quietly fall back to building.
429 - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
430 if: matrix.platform.target == ''
431 with:
432 python-version: "3.13"
433 - name: Smoke test on a newer CPython than the wheel was built for
434 if: matrix.platform.target == ''
435 run: |
436 set -euo pipefail
437 python -m venv /tmp/smoke
438 # The package has no runtime dependencies, so --find-links
439 # plus --no-index is enough on its own.
440 /tmp/smoke/bin/pip install --no-index --only-binary :all: --find-links dist matra
441 /tmp/smoke/bin/python -c "import sys, matra; assert sys.version_info[:2] == (3, 13), sys.version_info; print(matra.Matra)"
.github/workflows/ci.yml lines 359 to 387, at 4fcfb4a
359 # The published wheels are abi3 so one build serves 3.12 and every
360 # later GIL-enabled CPython (RFC-0014). Free-threaded builds take a
361 # different ABI tag and fall to the sdist. Losing the pyo3
362 # `abi3-py312` feature
363 # would silently narrow that back to cp312 and send 3.13/3.14
364 # users to a source build, so the tag is asserted here on every
365 # push rather than discovered at release time.
366 #
367 # The count guard is not decoration. `basename dist/matra-*.whl`
368 # with two matches becomes `basename A B`, where B is read as a
369 # suffix to strip rather than a second file, so the tag case below
370 # would inspect only the first wheel and pass with a non-abi3 one
371 # sitting beside it. release.yml guards the same contract the
372 # same way.
373 - name: The wheel is abi3
374 shell: bash
375 run: |
376 set -euo pipefail
377 # find, not ls: shellcheck's SC2012 flags parsing ls, and with no
378 # match find yields 0 and the guard below says so, where ls fails
379 # with its own message instead.
380 count=$(find dist -maxdepth 1 -name 'matra-*.whl' | wc -l | tr -d ' ')
381 [ "$count" = "1" ] || { echo "::error::expected 1 wheel, got $count"; exit 1; }
382 name=$(basename dist/matra-*.whl)
383 echo "built $name"
384 case "$name" in
385 *-cp312-abi3-*) ;;
386 *) echo "::error::$name is not an abi3 wheel"; exit 1 ;;
387 esac
RUSTUP_TOOLCHAIN=stable in the container run is load-bearing: it keeps rustup from re-syncing the channel inside the image, a rename across the overlay boundary that fails with os error 18● observedrelease.yml:360-384
(RFC-0014).
.github/workflows/release.yml lines 360 to 384, at 4fcfb4a
360 # RUSTUP_TOOLCHAIN is not optional. rust-toolchain.toml asks for
361 # a component the image does not carry, so rustup re-syncs the
362 # channel inside the container, and the rename it does to swap a
363 # component crosses the overlay boundary between the image layer
364 # and the container, which fails with "Invalid cross-device link
365 # (os error 18)". RUSTUP_TOOLCHAIN sits above rust-toolchain.toml
366 # in rustup's override precedence, so this builds on the toolchain
367 # the image already ships and never re-syncs at all. Remove it and
368 # every Linux wheel fails.
369 #
370 # The other way out is RUSTUP_PERMIT_COPY_RENAME=1, which rustup
371 # documents as unstable, says sacrifices some transaction
372 # protections, and says may be removed at any point. RFC-0014
373 # records why this branch was taken instead.
374 - name: Build wheel in the manylinux container (Linux)
375 if: matrix.platform.manylinux
376 run: |
377 set -euo pipefail
378 docker run --rm \
379 -v "$PWD:/io" -w /io \
380 -e RUSTUP_TOOLCHAIN=stable \
381 -e CARGO_TARGET_DIR=/build-target \
382 ghcr.io/pyo3/maturin:v1.14.1@sha256:2665227312dd1eab1c29c70a001dc8aac53155a2d048bede3b2df7f1691c8e38 \
383 build --release --out dist
384 sudo chown -R "$(id -u):$(id -g)" dist
The two adapters implement one discipline, each in its own file, as
RFC-0015 decided: a shared
module would add a third file to the wiring to save a few dozen lines.
fetch into memory, verify, write through a temporary and one rename, load the verified bytes (provision)
fetch all three into memory, verify the set, write as one transaction, load the verified bytes (provision)
Retry
one more fetch on a digest mismatch; a transport failure is not retried (fetch_verified)
the same
A file already there that is not the pin
replaced only when verified bytes arrive; never deleted
the directory is refused and nothing is written or removed
Leftover temporaries
reclaimed when older than 600 s, twice the fetch budget (reclaim)
the same rule
Notice
_with_notice forms on Udpipe and Engine
none
The asymmetry on a file already present is deliberate. The UDPipe filename
names one release, so a file under it that is not that release is matra's
own stale cache. The three Model2Vec filenames belong to the artifact
format, so a directory holding them may be a caller's own model, and the
provisioner refuses rather than overwrite it.
The digest check and the load use the same bytes, so there is no window between verifying and loading in which a file could be swapped ● observednlp/udpipe.rs:665-694.
Every filesystem failure names the operation and the path ● observednlp/udpipe.rs:544-554.
Transport failures are classified by transport_failure, and a rejected certificate gets the explanation in download_message● observednlp/udpipe.rs:469-489nlp/udpipe.rs:491-515.
The classification for every binding is stated in spec/tests/corpus/items.json● observeditems.json:64-81.
src/nlp/udpipe.rs lines 665 to 694, at 4fcfb4a
665 /// Read a file and verify its SHA-256 in a single read.
666 ///
667 /// Returns:
668 /// - `Ok(Some(bytes))` when size and hash both match — the returned bytes
669 /// are the *exact* bytes that were hashed; the caller can pass them
670 /// directly to a from-memory loader without a second disk read.
671 /// - `Ok(None)` when size mismatches (fast-fail, no read of contents)
672 /// or hash mismatches.
673 /// - `Err` when the file cannot be read.
674 ///
675 /// This shape closes the TOCTOU window the previous `verify_file` had:
676 /// previously the caller hashed the file then re-read it via
677 /// `Model::load(path)`, giving an attacker with directory write access
678 /// a window to swap the file between verify and load.
679 fn read_and_verify(
680 path: &Path,
681 expected_size: u64,
682 expected_hash: &str,
683 ) -> crate::domain::Result<Option<Vec<u8>>> {
684 let meta = std::fs::metadata(path).map_err(|e| io_at("read", path, &e))?;
685 if meta.len() != expected_size {
686 return Ok(None);
687 }
688 let bytes = std::fs::read(path).map_err(|e| io_at("read", path, &e))?;
689 if verified(&bytes, expected_size, expected_hash) {
690 Ok(Some(bytes))
691 } else {
692 Ok(None)
693 }
694 }
544 /// A filesystem failure that names the operation and the path.
545 ///
546 /// `io error: Permission denied (os error 13)` was the whole message a
547 /// user got when the model directory could not be created, with the
548 /// directory sitting in a variable one line away.
549 fn io_at(operation: &str, path: &Path, error: &std::io::Error) -> Error {
550 Error::Io(std::io::Error::new(
551 error.kind(),
552 format!("cannot {operation} {}: {error}", path.display()),
553 ))
554 }
469 /// Map a `ureq` failure to [`Error::Io`].
470 ///
471 /// A download that never arrives is a transport failure, not an invalid
472 /// model: [`Error::ModelInvalid`] is reserved for bytes that did arrive
473 /// and then failed the digest or the loader. A non-2xx status belongs on
474 /// this side too, because what came back was a server's answer about the
475 /// request rather than a model. The `io::ErrorKind` is preserved where
476 /// `ureq` knows it, so a caller can tell a timeout from an unreachable
477 /// host without reading the message. RFC-0015 records the classification
478 /// and what it replaced.
479 fn transport_failure(url: &str, error: &ureq::Error) -> Error {
480 use std::io::ErrorKind;
481482 let kind = match error {
483 ureq::Error::Io(e) => e.kind(),
484 ureq::Error::Timeout(_) => ErrorKind::TimedOut,
485 ureq::Error::ConnectionFailed | ureq::Error::HostNotFound => ErrorKind::NotConnected,
486 _ => ErrorKind::Other,
487 };
488 Error::Io(std::io::Error::new(kind, download_message(url, error)))
489 }
491 /// What the user reads when a download fails.
492 ///
493 /// A rejected certificate gets a sentence instead of a `Debug` rendering
494 /// of a `rustls` enum. `invalid peer certificate: Other(OtherError(
495 /// CaUsedAsEndEntity))` is what someone behind a TLS-intercepting
496 /// corporate proxy saw, and it says nothing about why matra cannot be
497 /// made to trust their proxy: matra verifies against root certificates
498 /// compiled into the binary and never reads the system trust store,
499 /// which is why it needs no `ca-certificates` package and also why
500 /// adding one to the system store changes nothing. The raw failure is
501 /// kept at the end, because a bug report needs it.
502 fn download_message(url: &str, error: &ureq::Error) -> String {
503 let detail = error.to_string();
504 if !is_certificate_rejection(&detail, error) {
505 return format!("download {url}: {detail}");
506 }
507 format!(
508 "download {url}: the TLS certificate offered for {} was rejected. matra verifies \
509 TLS against root certificates compiled into it and never reads the system trust \
510 store, so a proxy that re-signs TLS cannot be trusted by installing its CA. \
511 Fetch {} by hand and put it in the model directory instead. Underlying failure: {detail}",
512 host_of(url),
513 ENGLISH_MODEL_FILENAME,
514 )
515 }
spec/tests/corpus/items.json lines 64 to 81, at 4fcfb4a
64 "provisioning": {
65 "description": "Which kind a model provisioning failure reports (RFC-0015). A download that never arrived is a transport failure rather than an invalid model, so it is `io` with the URL in the message; a filesystem failure creating or writing the model directory is `io` with the operation and the path in the message; only bytes that arrived and then failed the pinned digest are `model_invalid`. Before 0.2.0 all three were `model_invalid`, so a consumer that branched on it for a DNS failure has to move to `io`. The filesystem row is the one a runner can produce with no network, so it is the one asserted here; the other two state the contract the adapters are written to.",
66 "kinds": {
67 "transport": "io",
68 "filesystem": "io",
69 "digest_mismatch": "model_invalid"
70 },
71 "unwritable_model_dir": {
72 "description": "A model directory whose parent is a regular file. `create_dir_all` fails before anything reaches the network, so this needs neither a model nor a link.",
73 "expect": {
74 "kind": "io",
75 "message_contains": [
76 "create the model directory"
77 ]
78 }
79 }
80 }
81 }
On wasm32 the download half of the embedding adapter is not compiled: every fetch item is behind cfg(not(target_arch = "wasm32")), and ureq is declared under that target predicate ● observedCargo.toml:76-84.
Cargo.toml lines 76 to 84, at 4fcfb4a
76 # The pinned-download client. Both provisioning adapters fetch through it
77 # directly, so both size caps, both timeout budgets and both transport
78 # error classifications live in matra rather than in a dependency
79 # (RFC-0015). Declared under a target predicate so `model2vec` on wasm32
80 # keeps the pure-Rust, no-network closure RFC-0010 asks for: the download
81 # half of the embedding adapter is `model2vec` plus
82 # `not(target_arch = "wasm32")`.
83 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
84 ureq = { version = "3.3", optional = true }
release.yml is the one release path, dispatched by hand from main with a version ● observedrelease.yml:1-66;
it verifies, creates an annotated tag, builds the crate, the four wheels and
the sdist, attests them, publishes, and smoke-tests what the registries
serve. What a caller can rely on:
.github/workflows/release.yml lines 1 to 66, at 4fcfb4a
1 name: Release
2 3 # One release path for both registries. This file replaces publish.yml
4 # (crates.io) and publish-pypi.yml (PyPI), which were two tag-triggered
5 # workflows racing on the same tag and re-deriving the same version
6 # checks in two places, where they had already drifted apart.
7 #
8 # The order is deliberate: verify, tag, build, attest, publish, release,
9 # then prove the registries serve what we built. Nothing is published
10 # that was not first verified and attested, and nothing is attested that
11 # was not built here.
12 #
13 # WHY DISPATCH AND NOT A TAG PUSH
14 #
15 # A release is started by hand from `main`, not by pushing a tag. The
16 # `crates-io` environment's deployment policy is branch-based, so a
17 # tag-triggered run failed before it ran a single step on the 0.2.0
18 # release:
19 #
20 # Tag "v0.2.0" is not allowed to deploy to crates-io due to
21 # environment protection rules.
22 #
23 # A dispatch from `main` satisfies a branch policy. The workflow creates
24 # the tag itself, after the checks pass. A tag trigger can come back if
25 # a release-PR tool ever needs one; it would need a `v*` tag deployment
26 # policy on both environments to work at all.
27 #
28 # MANUAL PREREQUISITES (one-time, outside this file, in three UIs)
29 #
30 # 1. GitHub, Settings -> Environments -> `crates-io`:
31 # - Required reviewers: the maintainer set. This is the
32 # per-publish approval gate for crates.io. Without it the
33 # workflow publishes unattended.
34 # - Deployment branches: allow `main`. If a tag trigger is ever
35 # restored, add a `v*` tag policy as well; a branch policy
36 # alone rejects a tag ref outright, which is the 0.2.0 failure
37 # above.
38 # 2. GitHub, Settings -> Environments -> `pypi`: the same two
39 # settings. As of the 0.2.0 release this environment carried no
40 # protection rules at all, so PyPI published with no human in the
41 # loop while crates.io waited for a click. Both registries are
42 # irreversible; both get a reviewer.
43 # 3. crates.io/crates/matra/settings -> Trusted Publishing, and
44 # pypi.org -> matra -> Publishing: both bind the OIDC subject to a
45 # workflow FILENAME. Both must now name `release.yml`. Until they
46 # do, the OIDC exchange fails and nothing publishes. Environment
47 # names (`crates-io`, `pypi`) are unchanged.
48 #
49 # Action refs are 40-character commit SHAs with a version comment, per
50 # RFC-0005. The wheel matrix, the manylinux container and the abi3
51 # contract are RFC-0014.
5253 on:
54 workflow_dispatch:
55 inputs:
56 version:
57 description: "Version to release, without the leading v (e.g. 0.2.1). Must equal the version in Cargo.toml on main."
58 required: true
59 type: string
60 skip_tag:
61 description: "Retry against the tag that already exists instead of creating it. Use only to re-run a failed publish."
62 required: false
63 default: false
64 type: boolean
6566 permissions: read-all
A person approved each upload.publish-crates runs in the crates-io environment and publish-pypi in pypi● observedrelease.yml:522-575release.yml:577-628;
each of the two environments waits for a reviewer before it publishes○ assumed.
The reviewer setting itself lives in the repository settings, not the
file;
the workflow header documents it as a prerequisite ● observedrelease.yml:1-66.
Assumed configured, not verifiable from the tree.
Provenance is verifiable.actions/attest-build-provenance attests the crate, the wheels and the sdist before publishing; the crate job checks the bytes it uploads equal the bytes attested ● observedrelease.yml:475-520release.yml:522-575.
Verify with
gh attestation verify matra-X.Y.Z.crate --repo mox-labs/matra.
PyPI uploads carry PEP 740 attestations ● observedrelease.yml:577-628.
Actions are pinned by commit SHA in the release workflow, which,
like CI, CodeQL and Scorecard,
defaults to permissions: read-all and grants more per job ● observedrelease.yml:1-66.
CI runs cargo-deny over the dependency tree and
cargo-semver-checks over the public API on every push
(ci.yml).
Does not hold at the pinned commit:the cargo-semver-checks job runs on pull requests only, and a push to main skips it ● observedci.yml:287-316.
The sentence before it is left as written for the owner to rule on.
.github/workflows/release.yml lines 522 to 575, at 4fcfb4a
522 publish-crates:
523 name: Publish to crates.io (gated by the crates-io environment)
524 needs: [attest]
525 environment: crates-io # <-- human-approval gate enforced here
526 runs-on: ubuntu-latest
527 timeout-minutes: 30
528 permissions:
529 contents: read
530 id-token: write # for Trusted Publishing OIDC
531 steps:
532 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
533 with:
534 ref: ${{ env.RELEASE_REF }}
535 persist-credentials: false
536 - uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master
537 with:
538 toolchain: stable
539 - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
540 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
541 with:
542 name: crate
543 # Outside the checkout. Inside it, the downloaded file is an
544 # untracked change and `cargo package` refuses a dirty tree,
545 # which failed the 0.2.1 publish before it reached crates.io.
546 path: ${{ runner.temp }}/attested
547 # cargo has no "publish this .crate file" command, so `cargo
548 # publish` repackages from source. That leaves a gap between the
549 # artifact that was attested and the bytes that reach crates.io,
550 # and the gap is closable rather than arguable: `cargo package`
551 # normalizes file mtimes, so the same source at the same commit
552 # produces a byte-identical tarball. Verified locally by packaging
553 # twice with every source file touched in between and getting the
554 # same sha256. If this ever fails, the two jobs disagree about
555 # something (cargo version, features, a generated file) and the
556 # attestation would not describe what shipped.
557 - name: What ships is what was attested
558 run: |
559 set -euo pipefail
560 cargo package --locked
561 built="target/package/matra-$RELEASE_VERSION.crate"
562 attested="$RUNNER_TEMP/attested/matra-$RELEASE_VERSION.crate"
563 a=$(sha256sum "$built" | cut -d' ' -f1)
564 b=$(sha256sum "$attested" | cut -d' ' -f1)
565 echo "packaged here: $a"
566 echo "attested: $b"
567 [ "$a" = "$b" ] || {
568 echo "::error::the crate packaged in this job differs from the attested artifact"; exit 1; }
569 - name: Mint a crates.io scoped token via Trusted Publishing (OIDC)
570 id: auth
571 uses: rust-lang/crates-io-auth-action@c6f97d42243bad5fab37ca0427f495c86d5b1a18 # v1
572 - name: cargo publish
573 env:
574 CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
575 run: cargo publish --locked
.github/workflows/release.yml lines 577 to 628, at 4fcfb4a
577 publish-pypi:
578 name: Publish to PyPI (gated by the pypi environment)
579 needs: [attest]
580 environment: pypi
581 runs-on: ubuntu-latest
582 timeout-minutes: 20
583 permissions:
584 contents: read
585 id-token: write # Trusted Publishing OIDC, and PEP 740 attestations
586 steps:
587 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
588 with:
589 pattern: wheels-*
590 merge-multiple: true
591 path: dist
592 # dist holds the five artifacts this run built and nothing else.
593 # The upload takes a directory, so an extra file in it is an extra
594 # file on PyPI, and a missing one is a platform that falls to a
595 # source build for the life of the release.
596 - name: dist is exactly the five artifacts of this version
597 env:
598 VERSION: ${{ inputs.version }}
599 run: |
600 set -euo pipefail
601 ls -l dist
602 total=$(find dist -maxdepth 1 -type f | wc -l | tr -d ' ')
603 [ "$total" = "5" ] || { echo "::error::expected 5 files in dist, got $total"; exit 1; }
604 for f in dist/*; do
605 case "$(basename "$f")" in
606 matra-"$VERSION"-*.whl|matra-"$VERSION".tar.gz) ;;
607 *) echo "::error::$(basename "$f") is not an artifact of $VERSION"; exit 1 ;;
608 esac
609 done
610 # The old workflow did the Trusted Publishing exchange by hand with
611 # curl and twine, because this action was a Docker action whose
612 # image was tagged by release tag and the org's mandatory SHA pin
613 # broke the image pull. It is a composite action from v1.12.0
614 # onward (checked in action.yml at both SHAs), so the pin works
615 # now, and the action emits PEP 740 attestations under Trusted
616 # Publishing, which twine did not. That was what the hand-rolled
617 # exchange cost.
618 - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
619 with:
620 packages-dir: dist
621 # A skip_tag retry rebuilds and re-reaches this step after PyPI
622 # already holds the version (0.2.1 published to PyPI while
623 # crates.io failed). Skipping files that exist keeps a retry of
624 # one registry from failing on the other. A first run cannot hit
625 # this: the tag job refuses a version whose tag exists.
626 skip-existing: true
627 attestations: true
628 print-hash: true
.github/workflows/release.yml lines 1 to 66, at 4fcfb4a
1 name: Release
2 3 # One release path for both registries. This file replaces publish.yml
4 # (crates.io) and publish-pypi.yml (PyPI), which were two tag-triggered
5 # workflows racing on the same tag and re-deriving the same version
6 # checks in two places, where they had already drifted apart.
7 #
8 # The order is deliberate: verify, tag, build, attest, publish, release,
9 # then prove the registries serve what we built. Nothing is published
10 # that was not first verified and attested, and nothing is attested that
11 # was not built here.
12 #
13 # WHY DISPATCH AND NOT A TAG PUSH
14 #
15 # A release is started by hand from `main`, not by pushing a tag. The
16 # `crates-io` environment's deployment policy is branch-based, so a
17 # tag-triggered run failed before it ran a single step on the 0.2.0
18 # release:
19 #
20 # Tag "v0.2.0" is not allowed to deploy to crates-io due to
21 # environment protection rules.
22 #
23 # A dispatch from `main` satisfies a branch policy. The workflow creates
24 # the tag itself, after the checks pass. A tag trigger can come back if
25 # a release-PR tool ever needs one; it would need a `v*` tag deployment
26 # policy on both environments to work at all.
27 #
28 # MANUAL PREREQUISITES (one-time, outside this file, in three UIs)
29 #
30 # 1. GitHub, Settings -> Environments -> `crates-io`:
31 # - Required reviewers: the maintainer set. This is the
32 # per-publish approval gate for crates.io. Without it the
33 # workflow publishes unattended.
34 # - Deployment branches: allow `main`. If a tag trigger is ever
35 # restored, add a `v*` tag policy as well; a branch policy
36 # alone rejects a tag ref outright, which is the 0.2.0 failure
37 # above.
38 # 2. GitHub, Settings -> Environments -> `pypi`: the same two
39 # settings. As of the 0.2.0 release this environment carried no
40 # protection rules at all, so PyPI published with no human in the
41 # loop while crates.io waited for a click. Both registries are
42 # irreversible; both get a reviewer.
43 # 3. crates.io/crates/matra/settings -> Trusted Publishing, and
44 # pypi.org -> matra -> Publishing: both bind the OIDC subject to a
45 # workflow FILENAME. Both must now name `release.yml`. Until they
46 # do, the OIDC exchange fails and nothing publishes. Environment
47 # names (`crates-io`, `pypi`) are unchanged.
48 #
49 # Action refs are 40-character commit SHAs with a version comment, per
50 # RFC-0005. The wheel matrix, the manylinux container and the abi3
51 # contract are RFC-0014.
5253 on:
54 workflow_dispatch:
55 inputs:
56 version:
57 description: "Version to release, without the leading v (e.g. 0.2.1). Must equal the version in Cargo.toml on main."
58 required: true
59 type: string
60 skip_tag:
61 description: "Retry against the tag that already exists instead of creating it. Use only to re-run a failed publish."
62 required: false
63 default: false
64 type: boolean
6566 permissions: read-all
.github/workflows/release.yml lines 522 to 575, at 4fcfb4a
522 publish-crates:
523 name: Publish to crates.io (gated by the crates-io environment)
524 needs: [attest]
525 environment: crates-io # <-- human-approval gate enforced here
526 runs-on: ubuntu-latest
527 timeout-minutes: 30
528 permissions:
529 contents: read
530 id-token: write # for Trusted Publishing OIDC
531 steps:
532 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
533 with:
534 ref: ${{ env.RELEASE_REF }}
535 persist-credentials: false
536 - uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master
537 with:
538 toolchain: stable
539 - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
540 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
541 with:
542 name: crate
543 # Outside the checkout. Inside it, the downloaded file is an
544 # untracked change and `cargo package` refuses a dirty tree,
545 # which failed the 0.2.1 publish before it reached crates.io.
546 path: ${{ runner.temp }}/attested
547 # cargo has no "publish this .crate file" command, so `cargo
548 # publish` repackages from source. That leaves a gap between the
549 # artifact that was attested and the bytes that reach crates.io,
550 # and the gap is closable rather than arguable: `cargo package`
551 # normalizes file mtimes, so the same source at the same commit
552 # produces a byte-identical tarball. Verified locally by packaging
553 # twice with every source file touched in between and getting the
554 # same sha256. If this ever fails, the two jobs disagree about
555 # something (cargo version, features, a generated file) and the
556 # attestation would not describe what shipped.
557 - name: What ships is what was attested
558 run: |
559 set -euo pipefail
560 cargo package --locked
561 built="target/package/matra-$RELEASE_VERSION.crate"
562 attested="$RUNNER_TEMP/attested/matra-$RELEASE_VERSION.crate"
563 a=$(sha256sum "$built" | cut -d' ' -f1)
564 b=$(sha256sum "$attested" | cut -d' ' -f1)
565 echo "packaged here: $a"
566 echo "attested: $b"
567 [ "$a" = "$b" ] || {
568 echo "::error::the crate packaged in this job differs from the attested artifact"; exit 1; }
569 - name: Mint a crates.io scoped token via Trusted Publishing (OIDC)
570 id: auth
571 uses: rust-lang/crates-io-auth-action@c6f97d42243bad5fab37ca0427f495c86d5b1a18 # v1
572 - name: cargo publish
573 env:
574 CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
575 run: cargo publish --locked
.github/workflows/release.yml lines 577 to 628, at 4fcfb4a
577 publish-pypi:
578 name: Publish to PyPI (gated by the pypi environment)
579 needs: [attest]
580 environment: pypi
581 runs-on: ubuntu-latest
582 timeout-minutes: 20
583 permissions:
584 contents: read
585 id-token: write # Trusted Publishing OIDC, and PEP 740 attestations
586 steps:
587 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
588 with:
589 pattern: wheels-*
590 merge-multiple: true
591 path: dist
592 # dist holds the five artifacts this run built and nothing else.
593 # The upload takes a directory, so an extra file in it is an extra
594 # file on PyPI, and a missing one is a platform that falls to a
595 # source build for the life of the release.
596 - name: dist is exactly the five artifacts of this version
597 env:
598 VERSION: ${{ inputs.version }}
599 run: |
600 set -euo pipefail
601 ls -l dist
602 total=$(find dist -maxdepth 1 -type f | wc -l | tr -d ' ')
603 [ "$total" = "5" ] || { echo "::error::expected 5 files in dist, got $total"; exit 1; }
604 for f in dist/*; do
605 case "$(basename "$f")" in
606 matra-"$VERSION"-*.whl|matra-"$VERSION".tar.gz) ;;
607 *) echo "::error::$(basename "$f") is not an artifact of $VERSION"; exit 1 ;;
608 esac
609 done
610 # The old workflow did the Trusted Publishing exchange by hand with
611 # curl and twine, because this action was a Docker action whose
612 # image was tagged by release tag and the org's mandatory SHA pin
613 # broke the image pull. It is a composite action from v1.12.0
614 # onward (checked in action.yml at both SHAs), so the pin works
615 # now, and the action emits PEP 740 attestations under Trusted
616 # Publishing, which twine did not. That was what the hand-rolled
617 # exchange cost.
618 - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
619 with:
620 packages-dir: dist
621 # A skip_tag retry rebuilds and re-reaches this step after PyPI
622 # already holds the version (0.2.1 published to PyPI while
623 # crates.io failed). Skipping files that exist keeps a retry of
624 # one registry from failing on the other. A first run cannot hit
625 # this: the tag job refuses a version whose tag exists.
626 skip-existing: true
627 attestations: true
628 print-hash: true
.github/workflows/release.yml lines 522 to 575, at 4fcfb4a
522 publish-crates:
523 name: Publish to crates.io (gated by the crates-io environment)
524 needs: [attest]
525 environment: crates-io # <-- human-approval gate enforced here
526 runs-on: ubuntu-latest
527 timeout-minutes: 30
528 permissions:
529 contents: read
530 id-token: write # for Trusted Publishing OIDC
531 steps:
532 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
533 with:
534 ref: ${{ env.RELEASE_REF }}
535 persist-credentials: false
536 - uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master
537 with:
538 toolchain: stable
539 - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
540 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
541 with:
542 name: crate
543 # Outside the checkout. Inside it, the downloaded file is an
544 # untracked change and `cargo package` refuses a dirty tree,
545 # which failed the 0.2.1 publish before it reached crates.io.
546 path: ${{ runner.temp }}/attested
547 # cargo has no "publish this .crate file" command, so `cargo
548 # publish` repackages from source. That leaves a gap between the
549 # artifact that was attested and the bytes that reach crates.io,
550 # and the gap is closable rather than arguable: `cargo package`
551 # normalizes file mtimes, so the same source at the same commit
552 # produces a byte-identical tarball. Verified locally by packaging
553 # twice with every source file touched in between and getting the
554 # same sha256. If this ever fails, the two jobs disagree about
555 # something (cargo version, features, a generated file) and the
556 # attestation would not describe what shipped.
557 - name: What ships is what was attested
558 run: |
559 set -euo pipefail
560 cargo package --locked
561 built="target/package/matra-$RELEASE_VERSION.crate"
562 attested="$RUNNER_TEMP/attested/matra-$RELEASE_VERSION.crate"
563 a=$(sha256sum "$built" | cut -d' ' -f1)
564 b=$(sha256sum "$attested" | cut -d' ' -f1)
565 echo "packaged here: $a"
566 echo "attested: $b"
567 [ "$a" = "$b" ] || {
568 echo "::error::the crate packaged in this job differs from the attested artifact"; exit 1; }
569 - name: Mint a crates.io scoped token via Trusted Publishing (OIDC)
570 id: auth
571 uses: rust-lang/crates-io-auth-action@c6f97d42243bad5fab37ca0427f495c86d5b1a18 # v1
572 - name: cargo publish
573 env:
574 CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
575 run: cargo publish --locked
.github/workflows/release.yml lines 577 to 628, at 4fcfb4a
577 publish-pypi:
578 name: Publish to PyPI (gated by the pypi environment)
579 needs: [attest]
580 environment: pypi
581 runs-on: ubuntu-latest
582 timeout-minutes: 20
583 permissions:
584 contents: read
585 id-token: write # Trusted Publishing OIDC, and PEP 740 attestations
586 steps:
587 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
588 with:
589 pattern: wheels-*
590 merge-multiple: true
591 path: dist
592 # dist holds the five artifacts this run built and nothing else.
593 # The upload takes a directory, so an extra file in it is an extra
594 # file on PyPI, and a missing one is a platform that falls to a
595 # source build for the life of the release.
596 - name: dist is exactly the five artifacts of this version
597 env:
598 VERSION: ${{ inputs.version }}
599 run: |
600 set -euo pipefail
601 ls -l dist
602 total=$(find dist -maxdepth 1 -type f | wc -l | tr -d ' ')
603 [ "$total" = "5" ] || { echo "::error::expected 5 files in dist, got $total"; exit 1; }
604 for f in dist/*; do
605 case "$(basename "$f")" in
606 matra-"$VERSION"-*.whl|matra-"$VERSION".tar.gz) ;;
607 *) echo "::error::$(basename "$f") is not an artifact of $VERSION"; exit 1 ;;
608 esac
609 done
610 # The old workflow did the Trusted Publishing exchange by hand with
611 # curl and twine, because this action was a Docker action whose
612 # image was tagged by release tag and the org's mandatory SHA pin
613 # broke the image pull. It is a composite action from v1.12.0
614 # onward (checked in action.yml at both SHAs), so the pin works
615 # now, and the action emits PEP 740 attestations under Trusted
616 # Publishing, which twine did not. That was what the hand-rolled
617 # exchange cost.
618 - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
619 with:
620 packages-dir: dist
621 # A skip_tag retry rebuilds and re-reaches this step after PyPI
622 # already holds the version (0.2.1 published to PyPI while
623 # crates.io failed). Skipping files that exist keeps a retry of
624 # one registry from failing on the other. A first run cannot hit
625 # this: the tag job refuses a version whose tag exists.
626 skip-existing: true
627 attestations: true
628 print-hash: true
.github/workflows/release.yml lines 1 to 66, at 4fcfb4a
1 name: Release
2 3 # One release path for both registries. This file replaces publish.yml
4 # (crates.io) and publish-pypi.yml (PyPI), which were two tag-triggered
5 # workflows racing on the same tag and re-deriving the same version
6 # checks in two places, where they had already drifted apart.
7 #
8 # The order is deliberate: verify, tag, build, attest, publish, release,
9 # then prove the registries serve what we built. Nothing is published
10 # that was not first verified and attested, and nothing is attested that
11 # was not built here.
12 #
13 # WHY DISPATCH AND NOT A TAG PUSH
14 #
15 # A release is started by hand from `main`, not by pushing a tag. The
16 # `crates-io` environment's deployment policy is branch-based, so a
17 # tag-triggered run failed before it ran a single step on the 0.2.0
18 # release:
19 #
20 # Tag "v0.2.0" is not allowed to deploy to crates-io due to
21 # environment protection rules.
22 #
23 # A dispatch from `main` satisfies a branch policy. The workflow creates
24 # the tag itself, after the checks pass. A tag trigger can come back if
25 # a release-PR tool ever needs one; it would need a `v*` tag deployment
26 # policy on both environments to work at all.
27 #
28 # MANUAL PREREQUISITES (one-time, outside this file, in three UIs)
29 #
30 # 1. GitHub, Settings -> Environments -> `crates-io`:
31 # - Required reviewers: the maintainer set. This is the
32 # per-publish approval gate for crates.io. Without it the
33 # workflow publishes unattended.
34 # - Deployment branches: allow `main`. If a tag trigger is ever
35 # restored, add a `v*` tag policy as well; a branch policy
36 # alone rejects a tag ref outright, which is the 0.2.0 failure
37 # above.
38 # 2. GitHub, Settings -> Environments -> `pypi`: the same two
39 # settings. As of the 0.2.0 release this environment carried no
40 # protection rules at all, so PyPI published with no human in the
41 # loop while crates.io waited for a click. Both registries are
42 # irreversible; both get a reviewer.
43 # 3. crates.io/crates/matra/settings -> Trusted Publishing, and
44 # pypi.org -> matra -> Publishing: both bind the OIDC subject to a
45 # workflow FILENAME. Both must now name `release.yml`. Until they
46 # do, the OIDC exchange fails and nothing publishes. Environment
47 # names (`crates-io`, `pypi`) are unchanged.
48 #
49 # Action refs are 40-character commit SHAs with a version comment, per
50 # RFC-0005. The wheel matrix, the manylinux container and the abi3
51 # contract are RFC-0014.
5253 on:
54 workflow_dispatch:
55 inputs:
56 version:
57 description: "Version to release, without the leading v (e.g. 0.2.1). Must equal the version in Cargo.toml on main."
58 required: true
59 type: string
60 skip_tag:
61 description: "Retry against the tag that already exists instead of creating it. Use only to re-run a failed publish."
62 required: false
63 default: false
64 type: boolean
6566 permissions: read-all
SLSA Build Level 3 through slsa-github-generator.RFC-0005 item 7.
Releases attest at SLSA Build Level 2 with
actions/attest-build-provenance, because the generator must be
referenced by tag and the organization's policy requires SHA pins
(attest).
Signed tags. RFC-0005 item 8.
Release tags are annotated and created through the API, which cannot carry a maintainer's signature; the workflow asserts annotation and reports whether the tag is signed ● observedrelease.yml:257-277.
Tag-triggered publishing from publish.yml and publish-pypi.yml.
RFC-0005 and RFC-0014 name both. They were merged into release.yml,
dispatched from main because the crates-io environment's branch
policy rejected a tag ref (header).
Rust 1.85 as the from-source floor. RFC-0014 states it.
The floor is 1.88, checked by the msrv job ● observedCargo.toml:1-14ci.yml:70-100.
"No network" for the embedding model. RFC-0010 decision 6 as first
written, and EP-0009's "the caller supplies model files". Amended by
RFC-0011: a pinned download is allowed, and Model2Vec::potion_base_8m
provisions.
The UDPipe fetch through udpipe_rs::download_model_from_url.
Replaced by matra's own fetch, which is what made the timeouts, the size
cap and the classification possible (RFC-0015).
book/book.toml as a place attribution lives. RFC-0013 lists it; the
mdBook site it configured has been replaced, and the file does not exist.
.github/workflows/release.yml lines 257 to 277, at 4fcfb4a
257 # Annotated is asserted; signed is reported. Today's tags are
258 # unsigned, and a tag created through the API cannot carry the
259 # maintainer's signature. Requiring one is a separate decision
260 # (RFC-0005 item 8), so this prints the state rather than failing.
261 - name: The tag is annotated, and say whether it is signed
262 env:
263 GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
264 run: |
265 set -euo pipefail
266 kind="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/tags/$RELEASE_TAG" --jq '.object.type')"
267 sha="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/tags/$RELEASE_TAG" --jq '.object.sha')"
268 if [ "$kind" != "tag" ]; then
269 echo "::error::$RELEASE_TAG is a lightweight tag (it points straight at a $kind). Release tags are annotated."
270 exit 1
271 fi
272 verified="$(gh api "repos/$GITHUB_REPOSITORY/git/tags/$sha" --jq '.verification.verified')"
273 if [ "$verified" = "true" ]; then
274 echo "$RELEASE_TAG is annotated and signed"
275 else
276 echo "::notice::$RELEASE_TAG is annotated and NOT signed. Signed release tags are RFC-0005 item 8, not yet in force."
277 fi
The stable ABI is a ceiling. Any pyo3 feature outside it is
unavailable, and leaving it would multiply the wheel matrix by the
CPython versions (RFC-0014).
Free-threaded CPython falls to the sdist. A free-threaded interpreter
accepts no abi3 tag, and
pyo3's free-threaded stable ABI begins at 3.15○ assumed
(RFC-0014; not re-verified against the current pyo3).
Nothing watches the maturin image digest.Dependabot cannot see a digest written inline in a run: step, so bumping it is a person's job on the release checklist ● observedrelease.yml:311-351.
A proxy that re-signs TLS blocks the download.matra verifies against root certificates compiled into it and never reads the system trust store ● observednlp/udpipe.rs:491-515;
the documented way through is placing the file by hand.
The embedding model's first run is silent, and
the code carries neither its size in a notice nor its licence ● observeddomain.rs:34-72embed/model2vec.rs:40-113.
The parsing model's licence is non-commercial. matra is MIT; its
default parsing model is not, and a commercial user must read and decide.
.github/workflows/release.yml lines 311 to 351, at 4fcfb4a
311 # No third-party build action: the org Actions policy validates every
312 # `uses` ref at workflow start, and keeping the build to `actions/*`
313 # plus shell removes that variable. The Linux half is built inside
314 # ghcr.io/pyo3/maturin instead, which is the same manylinux2014 base
315 # PyO3/maturin-action would have used, reached with plain `docker run`
316 # and pinned by index digest the way the policy pins action SHAs.
317 #
318 # Building on the runner rather than in that image is what produced a
319 # manylinux_2_34 wheel for 0.1.x, which pip rejects on Debian 11,
320 # Ubuntu 20.04, RHEL 8 and Amazon Linux 2. The container drops the
321 # floor to glibc 2.17. See RFC-0014.
322 #
323 # Nothing watches the image digest: dependabot covers cargo, actions
324 # and pip, and cannot see a digest written inline in a `run:` step.
325 # Bumping it is a person's job, on the release checklist.
326 build-wheels:
327 name: Wheel ${{ matrix.platform.name }}
328 needs: [verify, tag]
329 runs-on: ${{ matrix.platform.runner }}
330 permissions:
331 contents: read
332 # A starved runner must fail loudly rather than squat on the
333 # concurrency group and block the next dispatch.
334 timeout-minutes: 45
335 strategy:
336 fail-fast: false
337 matrix:
338 platform:
339 # ubuntu-24.04-arm is a GitHub-hosted arm64 runner, free for
340 # public repositories. It exists so the Linux aarch64 wheel is
341 # built natively rather than under QEMU, which would push the
342 # job past its timeout.
343 #
344 # macos-x86_64 cross-compiles from the arm64 runner: GitHub
345 # is retiring the Intel macos-13 runners and jobs targeting
346 # them starve in the queue. Cross-built wheels cannot smoke
347 # test on the arm host, so those steps are native-only.
348 - { name: linux-x86_64, runner: ubuntu-latest, manylinux: true, target: "" }
349 - { name: linux-aarch64, runner: ubuntu-24.04-arm, manylinux: true, target: "" }
350 - { name: macos-x86_64, runner: macos-14, manylinux: false, target: x86_64-apple-darwin }
351 - { name: macos-aarch64, runner: macos-14, manylinux: false, target: "" }
491 /// What the user reads when a download fails.
492 ///
493 /// A rejected certificate gets a sentence instead of a `Debug` rendering
494 /// of a `rustls` enum. `invalid peer certificate: Other(OtherError(
495 /// CaUsedAsEndEntity))` is what someone behind a TLS-intercepting
496 /// corporate proxy saw, and it says nothing about why matra cannot be
497 /// made to trust their proxy: matra verifies against root certificates
498 /// compiled into the binary and never reads the system trust store,
499 /// which is why it needs no `ca-certificates` package and also why
500 /// adding one to the system store changes nothing. The raw failure is
501 /// kept at the end, because a bug report needs it.
502 fn download_message(url: &str, error: &ureq::Error) -> String {
503 let detail = error.to_string();
504 if !is_certificate_rejection(&detail, error) {
505 return format!("download {url}: {detail}");
506 }
507 format!(
508 "download {url}: the TLS certificate offered for {} was rejected. matra verifies \
509 TLS against root certificates compiled into it and never reads the system trust \
510 store, so a proxy that re-signs TLS cannot be trusted by installing its CA. \
511 Fetch {} by hand and put it in the model directory instead. Underlying failure: {detail}",
512 host_of(url),
513 ENGLISH_MODEL_FILENAME,
514 )
515 }
34 /// What a constructor is about to download, reported before the transfer
35 /// begins.
36 ///
37 /// A first run fetches a model over a link whose speed matra cannot know.
38 /// Measured cold starts ran from 3 to 35 seconds with nothing on screen,
39 /// which is indistinguishable from a hung process, and the natural
40 /// response to a hung process makes it worse. The library writes to no
41 /// terminal, so it hands the facts to whoever does: the UDPipe
42 /// constructors that provision carry a `_with_notice` form that calls a
43 /// closure once per fetch, and only when the artifact is not already on
44 /// disk.
45 ///
46 /// That is `Udpipe::english_with_notice`,
47 /// `Udpipe::from_config_with_notice` and `Engine::from_config_with_notice`.
48 /// `Model2Vec::potion_base_8m` fetches 30.2 MB across three artifacts
49 /// and has no notice form, so a semantic first run is still silent.
50 ///
51 /// Facts rather than a sentence. The caller decides the wording, the
52 /// stream, and whether to say anything at all.
53 #[derive(Debug, Clone, PartialEq, Eq)]
54 #[non_exhaustive]
55 pub struct ProvisionNotice {
56 /// The artifact's filename, as it will appear in `destination`.
57 pub artifact: String,
58 /// The artifact's pinned size in bytes. Known before the fetch
59 /// because the size is a constant in the source, checked against
60 /// what arrives.
61 pub bytes: u64,
62 /// The directory the artifact is being fetched into.
63 pub destination: PathBuf,
64 /// The artifact's license, named for a reader, for example
65 /// `CC BY-NC-SA 4.0 (non-commercial)`. A model is a separate work
66 /// from matra and carries its own terms, which need not be matra's
67 /// MIT. Pinned in the source beside the URL, so a change of model
68 /// changes both.
69 pub license: String,
70 /// Where the full text of `license` is published.
71 pub license_url: String,
72 }
src/embed/model2vec.rs lines 40 to 113, at 4fcfb4a
40 /// The three artifact filenames, in the order the identity digest covers
41 /// them. Every path in this file is built from this array, so the digest
42 /// and the loader can never disagree about which file is which.
43 const ARTIFACT_FILES: [&str; 3] = ["model.safetensors", "tokenizer.json", "config.json"];
44 45 /// SHA-256 over the three artifact files of the pinned potion-base-8M
46 /// release, concatenated in [`ARTIFACT_FILES`] order. The same value
47 /// `spec/tests/semantic/reference-model.json` carries as `artifact_hash`,
48 /// and the same value [`Model2Vec::model_hash`] reports once loaded.
49 ///
50 /// Refresh this and [`POTION_BASE_8M_URLS`] together with
51 /// `scripts/fetch-embedding-hash.sh <revision>`.
52 #[cfg(not(target_arch = "wasm32"))]
53 const POTION_BASE_8M_SHA256: &str =
54 "81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4";
55 56 /// Download URLs for the pinned release, one per [`ARTIFACT_FILES`]
57 /// entry in the same order. The path segment after `resolve/` is a git
58 /// commit on the Hugging Face repository rather than a branch, so the
59 /// URL names one immutable tree; `main` would move under the pin. The
60 /// digest is what actually decides whether the bytes are trusted, and
61 /// the revision is what makes a mismatch a bug report rather than an
62 /// upstream edit.
63 ///
64 /// Refresh with `scripts/fetch-embedding-hash.sh <revision>`.
65 #[cfg(not(target_arch = "wasm32"))]
66 const POTION_BASE_8M_URLS: [&str; 3] = [
67 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/model.safetensors",
68 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/tokenizer.json",
69 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/config.json",
70 ];
71 72 /// Ceiling on any one downloaded artifact. The largest of the three in
73 /// the pinned release is `model.safetensors` at 30,236,760 bytes, so this
74 /// is a shade over twice the real thing: headroom for a later revision,
75 /// and small enough that a redirect to something enormous costs a bounded
76 /// read rather than the machine's memory. Past it,
77 /// [`Error::InputTooLarge`] with `what` set to `"embedding_download"`.
78 #[cfg(not(target_arch = "wasm32"))]
79 const MAX_ARTIFACT_BYTES: usize = 64 * 1024 * 1024;
80 81 /// Ceiling on one artifact fetch, DNS lookup through the last byte of
82 /// the body. The largest artifact in the pinned release is about 30 MB,
83 /// which needs the whole of this budget only on a link under 1 Mbit/s,
84 /// so the margin is generous for a slow connection and still finite: a
85 /// stalled or black-holed transfer fails instead of holding the caller
86 /// for as long as the socket stays open.
87 #[cfg(not(target_arch = "wasm32"))]
88 const FETCH_TIMEOUT: Duration = Duration::from_secs(300);
89 90 /// Ceiling on establishing the connection, socket and TLS handshake
91 /// included. Separate from [`FETCH_TIMEOUT`] so an unreachable host
92 /// fails in seconds rather than consuming the whole transfer budget.
93 #[cfg(not(target_arch = "wasm32"))]
94 const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
95 96 /// Prefix of the temporary an artifact lands through. Shared by the
97 /// creation and the reclaim, so the sweep cannot look for a name the
98 /// writer does not use.
99 #[cfg(not(target_arch = "wasm32"))]
100 const TEMP_FILE_PREFIX: &str = ".tmp.";
101102 /// Age past which a temporary cannot belong to a live provisioning call,
103 /// so it is a leftover and the next call reclaims it.
104 ///
105 /// Twice [`FETCH_TIMEOUT`], the same rule and the same number
106 /// `nlp/udpipe.rs` uses. All three artifacts are fetched and verified
107 /// before any of them is written, so a temporary's modification time is
108 /// within milliseconds of the write that follows it, and anything older
109 /// than ten minutes is an orphan from a killed process rather than a
110 /// peer. The margin is what protects a concurrent cold start: racing
111 /// processes see each other's temporaries as seconds old.
112 #[cfg(not(target_arch = "wasm32"))]
113 const STALE_TEMP_AGE: Duration = Duration::from_secs(600);
Document the old wheel matrix honestly instead of fixing it: rejected
in RFC-0014; it truthfully
documents a bad first install.
One wheel per CPython version: twelve jobs growing with each CPython
release, and an interpreter released after a publish still falls to the
sdist. Rejected in RFC-0014.
cibuildwheel: a build orchestrator with QEMU for arm64, for a matrix
four explicit jobs cover. Rejected in RFC-0014.
RUSTUP_PERMIT_COPY_RENAME=1: works, but rustup documents it as
unstable and removable, and it lets the calendar choose the compiler.
Rejected in RFC-0014.
Keep the upstream UDPipe fetch and patch around it: cannot add a
timeout or a size cap to a fetch matra does not perform. Rejected in
RFC-0015.
A new Error variant for transport failures: grows the kind
vocabulary with every place I/O happens, for a distinction
io::ErrorKind already carries. Rejected in RFC-0015.
An escape hatch to the system trust store: widens what can authorize a
model download, when hand placement costs one curl and is exactly as
trustworthy. Rejected in RFC-0015 pending a threat model.
Attribution to an individual, or to both an individual and the
organization: contradicts the repository and homepage, or reintroduces
the disagreement in a form that looks intentional. Rejected in
RFC-0013.
For the owner to settle in this proposal's review. Each is a decision of its
own.
Prebuilt command-line binaries. RFC-0005 deferred binary releases "if
matra ever ships a matra Rust CLI binary".
It now does, behind the cli feature ● observedCargo.toml:47-50grounds for decision 1;
a Rust user gets it with cargo install, and a Python user through the
wheel's launcher, but no release publishes a standalone binary.
cargo install matra --features cli and the wheel's launcher remain the ways to get the command line. The trigger is recorded as fired, and a standalone binary as not yet wanted.
B Yes, from release.yml
Each release attaches a prebuilt binary per platform, built and attested beside the wheels.
Claude recommends A, Not yet; the two routes stay. ◐ inferred, a judgment
Each new artifact joins the build, the attestation and the smoke test of a release path that cannot be undone, and nothing at the pinned commit shows a caller who can run neither cargo install nor the wheel's launcher.
The strongest case against it
The trigger RFC-0005 wrote has fired. Building from source needs a C++ compiler, the failure the wheel matrix was rebuilt to avoid, so a user with no Python cannot run the command line without one.
The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.
18 /// Expected SHA-256 of the English UD-EWT 2.5 (release 191206) UDPipe model.
19 /// Refresh with `scripts/fetch-model-hash.sh` when updating the model version.
20 const ENGLISH_MODEL_SHA256: &str =
21 "784bd0fa85e3d831fd02a55290d0acfd05c953159dc38cc33d52e1b28add9957";
22 23 /// Expected size in bytes, checked before hashing as a fast-fail guard.
24 const ENGLISH_MODEL_SIZE: u64 = 16_309_608;
25 26 /// Direct download URL for the pinned model, fetched by [`fetch_capped`].
27 /// LINDAT migrated the bitstream endpoint from
28 /// `/repository/xmlui/bitstream/...` (now a 200 HTML preview) to
29 /// `/repository/server/api/core/bitstreams/...` (the actual binary), so
30 /// the URL is pinned here in its working form rather than derived from a
31 /// model name.
32 const ENGLISH_MODEL_URL: &str = "https://lindat.mff.cuni.cz/repository/server/api/core/bitstreams/handle/11234/1-3131/english-ewt-ud-2.5-191206.udpipe?sequence=17&isAllowed=y";
33 34 /// The pinned model's license, as its LINDAT record
35 /// (<https://hdl.handle.net/11234/1-3131>) states it: Creative Commons
36 /// Attribution-NonCommercial-ShareAlike 4.0 International. matra's code
37 /// is MIT; this file is a separate work with its own terms, and the
38 /// download notice carries them so the command line can say so. Beside
39 /// the URL because a new model means checking both.
40 const ENGLISH_MODEL_LICENSE: &str = "CC BY-NC-SA 4.0 (non-commercial)";
41 42 /// Where the full text of [`ENGLISH_MODEL_LICENSE`] is published.
43 const ENGLISH_MODEL_LICENSE_URL: &str = "https://creativecommons.org/licenses/by-nc-sa/4.0/";
44 45 /// The pinned English model as one record, so the provisioning path
46 /// takes the pin whole rather than six loose arguments.
47 const ENGLISH_MODEL: Pin<'static> = Pin {
48 filename: ENGLISH_MODEL_FILENAME,
49 size: ENGLISH_MODEL_SIZE,
50 sha256: ENGLISH_MODEL_SHA256,
51 url: ENGLISH_MODEL_URL,
52 license: ENGLISH_MODEL_LICENSE,
53 license_url: ENGLISH_MODEL_LICENSE_URL,
54 };
55 56 /// A pinned artifact: what it is called on disk, what it must measure,
57 /// where it comes from, and the terms it is published under. A test pins
58 /// a fixture of its own through the same shape.
59 #[derive(Debug, Clone, Copy)]
60 struct Pin<'a> {
61 filename: &'a str,
62 size: u64,
63 sha256: &'a str,
64 url: &'a str,
65 license: &'a str,
66 license_url: &'a str,
67 }
68 69 /// Ceiling on the downloaded model. The pinned artifact is 16.3 MB, so
70 /// this is four times the real thing: headroom for a later model version
71 /// and small enough that a redirect to something enormous costs a
72 /// bounded read rather than the machine's memory. Past it,
73 /// [`Error::InputTooLarge`] with `what` set to `"udpipe_download"`. The
74 /// same number the embedding adapter uses, because it is the same
75 /// question.
76 const MAX_MODEL_BYTES: usize = 64 * 1024 * 1024;
77 78 /// Ceiling on the fetch, DNS lookup through the last byte of the body.
79 /// The pinned model is 16.3 MB, which needs this whole budget only on a
80 /// link under 0.5 Mbit/s, so the margin is generous for a slow
81 /// connection and still finite: a stalled or black-holed transfer fails
82 /// instead of holding the caller for as long as the socket stays open.
83 /// Before this existed, 90 seconds against an unreachable host produced
84 /// zero bytes with the process still running, unbounded.
85 const FETCH_TIMEOUT: Duration = Duration::from_secs(300);
86 87 /// Ceiling on establishing the connection, socket and TLS handshake
88 /// included. Separate from [`FETCH_TIMEOUT`] so an unreachable host
89 /// fails in seconds rather than consuming the whole transfer budget.
90 const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
91 92 /// Age past which a temporary download directory cannot belong to a
93 /// live provisioning call, so it is a leftover and this call reclaims
94 /// it.
95 ///
96 /// Twice [`FETCH_TIMEOUT`]. A fetch cannot outlive that budget, and the
97 /// directory is created after the bytes are already in memory, so its
98 /// modification time is within milliseconds of the write that follows.
99 /// Anything older than ten minutes is therefore an orphan from a killed
100 /// process, not a peer. The margin is what protects a concurrent cold
101 /// start: three processes racing on one empty model directory each see
102 /// the others' directories as minutes-fresh and leave them alone.
103 const STALE_TEMP_AGE: Duration = Duration::from_secs(600);
104105 /// Prefix of the temporary directory a download lands in. Shared by the
106 /// creation and the reclaim, so the sweep cannot look for a name the
107 /// writer does not use.
108 const TEMP_DIR_PREFIX: &str = ".tmp.download.";
src/embed/model2vec.rs lines 40 to 113, at 4fcfb4a
40 /// The three artifact filenames, in the order the identity digest covers
41 /// them. Every path in this file is built from this array, so the digest
42 /// and the loader can never disagree about which file is which.
43 const ARTIFACT_FILES: [&str; 3] = ["model.safetensors", "tokenizer.json", "config.json"];
44 45 /// SHA-256 over the three artifact files of the pinned potion-base-8M
46 /// release, concatenated in [`ARTIFACT_FILES`] order. The same value
47 /// `spec/tests/semantic/reference-model.json` carries as `artifact_hash`,
48 /// and the same value [`Model2Vec::model_hash`] reports once loaded.
49 ///
50 /// Refresh this and [`POTION_BASE_8M_URLS`] together with
51 /// `scripts/fetch-embedding-hash.sh <revision>`.
52 #[cfg(not(target_arch = "wasm32"))]
53 const POTION_BASE_8M_SHA256: &str =
54 "81c3592150873b1c5a8c4262850f795bff4fd568fbde80ac69889d087f16a0b4";
55 56 /// Download URLs for the pinned release, one per [`ARTIFACT_FILES`]
57 /// entry in the same order. The path segment after `resolve/` is a git
58 /// commit on the Hugging Face repository rather than a branch, so the
59 /// URL names one immutable tree; `main` would move under the pin. The
60 /// digest is what actually decides whether the bytes are trusted, and
61 /// the revision is what makes a mismatch a bug report rather than an
62 /// upstream edit.
63 ///
64 /// Refresh with `scripts/fetch-embedding-hash.sh <revision>`.
65 #[cfg(not(target_arch = "wasm32"))]
66 const POTION_BASE_8M_URLS: [&str; 3] = [
67 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/model.safetensors",
68 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/tokenizer.json",
69 "https://huggingface.co/minishlab/potion-base-8M/resolve/bf8b056651a2c21b8d2565580b8569da283cab23/config.json",
70 ];
71 72 /// Ceiling on any one downloaded artifact. The largest of the three in
73 /// the pinned release is `model.safetensors` at 30,236,760 bytes, so this
74 /// is a shade over twice the real thing: headroom for a later revision,
75 /// and small enough that a redirect to something enormous costs a bounded
76 /// read rather than the machine's memory. Past it,
77 /// [`Error::InputTooLarge`] with `what` set to `"embedding_download"`.
78 #[cfg(not(target_arch = "wasm32"))]
79 const MAX_ARTIFACT_BYTES: usize = 64 * 1024 * 1024;
80 81 /// Ceiling on one artifact fetch, DNS lookup through the last byte of
82 /// the body. The largest artifact in the pinned release is about 30 MB,
83 /// which needs the whole of this budget only on a link under 1 Mbit/s,
84 /// so the margin is generous for a slow connection and still finite: a
85 /// stalled or black-holed transfer fails instead of holding the caller
86 /// for as long as the socket stays open.
87 #[cfg(not(target_arch = "wasm32"))]
88 const FETCH_TIMEOUT: Duration = Duration::from_secs(300);
89 90 /// Ceiling on establishing the connection, socket and TLS handshake
91 /// included. Separate from [`FETCH_TIMEOUT`] so an unreachable host
92 /// fails in seconds rather than consuming the whole transfer budget.
93 #[cfg(not(target_arch = "wasm32"))]
94 const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
95 96 /// Prefix of the temporary an artifact lands through. Shared by the
97 /// creation and the reclaim, so the sweep cannot look for a name the
98 /// writer does not use.
99 #[cfg(not(target_arch = "wasm32"))]
100 const TEMP_FILE_PREFIX: &str = ".tmp.";
101102 /// Age past which a temporary cannot belong to a live provisioning call,
103 /// so it is a leftover and the next call reclaims it.
104 ///
105 /// Twice [`FETCH_TIMEOUT`], the same rule and the same number
106 /// `nlp/udpipe.rs` uses. All three artifacts are fetched and verified
107 /// before any of them is written, so a temporary's modification time is
108 /// within milliseconds of the write that follows it, and anything older
109 /// than ten minutes is an orphan from a killed process rather than a
110 /// peer. The margin is what protects a concurrent cold start: racing
111 /// processes see each other's temporaries as seconds old.
112 #[cfg(not(target_arch = "wasm32"))]
113 const STALE_TEMP_AGE: Duration = Duration::from_secs(600);
34 /// What a constructor is about to download, reported before the transfer
35 /// begins.
36 ///
37 /// A first run fetches a model over a link whose speed matra cannot know.
38 /// Measured cold starts ran from 3 to 35 seconds with nothing on screen,
39 /// which is indistinguishable from a hung process, and the natural
40 /// response to a hung process makes it worse. The library writes to no
41 /// terminal, so it hands the facts to whoever does: the UDPipe
42 /// constructors that provision carry a `_with_notice` form that calls a
43 /// closure once per fetch, and only when the artifact is not already on
44 /// disk.
45 ///
46 /// That is `Udpipe::english_with_notice`,
47 /// `Udpipe::from_config_with_notice` and `Engine::from_config_with_notice`.
48 /// `Model2Vec::potion_base_8m` fetches 30.2 MB across three artifacts
49 /// and has no notice form, so a semantic first run is still silent.
50 ///
51 /// Facts rather than a sentence. The caller decides the wording, the
52 /// stream, and whether to say anything at all.
53 #[derive(Debug, Clone, PartialEq, Eq)]
54 #[non_exhaustive]
55 pub struct ProvisionNotice {
56 /// The artifact's filename, as it will appear in `destination`.
57 pub artifact: String,
58 /// The artifact's pinned size in bytes. Known before the fetch
59 /// because the size is a constant in the source, checked against
60 /// what arrives.
61 pub bytes: u64,
62 /// The directory the artifact is being fetched into.
63 pub destination: PathBuf,
64 /// The artifact's license, named for a reader, for example
65 /// `CC BY-NC-SA 4.0 (non-commercial)`. A model is a separate work
66 /// from matra and carries its own terms, which need not be matra's
67 /// MIT. Pinned in the source beside the URL, so a change of model
68 /// changes both.
69 pub license: String,
70 /// Where the full text of `license` is published.
71 pub license_url: String,
72 }
The licence and its URL sit beside the three pinned URLs, and the provisioning Model2Vec constructors gain a notice form that reports the size, the destination and the licence before a fetch.
B No; the documentation is enough
The licence stays in the documentation, and the first run stays silent.
Claude recommends A, Yes, as the parsing pin does. ◐ inferred, a judgment
The parsing pin keeps its licence beside its URL because a new model means checking both, a reason that does not depend on which licence it is. A first run of 30 MB with nothing on screen is the state the notice was built to end for the parsing model.
The strongest case against it
The embedding model's licence places no restriction a caller has to decide on, unlike the parsing model's, and a notice form adds constructors to the public surface of an adapter a caller can replace with their own.
The owner's decision: open. Answer with accept, accept with a reservation, object or redirect, in a comment on this block or on the pull request; Claude then records your ruling here, as a named step in a pull request. A comment is never read as a ruling. Silence is not assent.
What the owner should confirm or strike, collected from the claims above
that stand on no grounds yet:
○ 3 claims stand on no grounds yet. Confirm or strike each with a comment on it.