Lab

The Lab is where matra's evals and experiments will live: measurements of how well matra does what it claims, and trials of what it might do next, each with the data and the method that produced it. Nothing is here yet.

The Lab sits apart from the documentation on purpose. The pages under Docs describe what ships, and the records under Blueprints say what was decided and why. An experiment is neither: it can fail, and a result it reports is evidence for a decision rather than the decision itself. When an experiment's result changes what matra does, the change goes through the process in Blueprints, and the experiment is cited there.

Each experiment will be set as a card: the hypothesis and its prediction, recorded with the day and the commit before the run; the result in words, with its interval stated as a sentence; every run as a dot, never a bare mean; the method; what the result does not show; and the inputs, licences, matra version and model digests that produced it. The card's syntax and the research behind it are in the docsite's README.