Reproducibility

A result is only worth something if it can be had again: by a colleague, by a reviewer, or by you in a year. noodlelab records every run (see noodlelab.core.provenance), and gives tools on top of that record.

Reproduce a run

noodlelab reproduce runs/20250102-093000-a1b2c3d4e5f6   # a run folder
noodlelab reproduce bundle.zip                            # a bundle downloaded from the editor
noodlelab reproduce bundle.zip --clean-env                # in a fresh environment (needs uv)

It runs the graph again exactly as recorded: the same settings, the same run seed, and nothing reused from caches or checkpoints. Then it compares every result with the recorded one (--rtol, --atol; the defaults are 1e-9 and 0). Exit code 0 means everything is the same, 1 that something differs, 2 that it could not run. When results differ it says why they may differ: node code that changed, input files that changed, other package versions, another Python or platform, another git commit, or uncommitted changes when the run was made.

--clean-env builds a new virtual environment with the recorded Python and package versions, and reproduces the run there. What is tested is then the record, not the machine it was made on. Agents have the same tool over MCP (reproduce, and diff_runs to compare two runs).

Reproducing a run: does it still give the same answer?

A result is only as good as the chance that someone else, or you in a year, gets it again. Most published research code does not even run for others, and when it runs it often gives other numbers, because a package changed, an input file was replaced, or a seed was different. Checking that by hand is a chore nobody does, so this makes it one command:

noodlelab reproduce runs/20250102-093000-a1b2c3d4e5f6
noodlelab reproduce bundle.zip --clean-env

It takes a run folder, its provenance.json or a reproducibility bundle (see bundle), runs the graph exactly as recorded (same settings, same run seed, nothing reused from caches or checkpoints), and compares every result with the recorded ones within a tolerance (see baseline). When results differ it says why they may differ, from the two provenance records (see rundiff): node code that changed, input files that changed, other package versions, another platform, uncommitted changes.

--clean-env goes further: it builds a fresh virtual environment with the recorded Python and package versions (with uv) and reproduces the run in it, so what is tested is the record, not the machine it was made on.

A reproduction is not a variant of the analysis (see variants): it runs one that was already run.

Compare two runs

noodlelab diff runs/<this run> runs/<an earlier run>   # folders or their provenance.json
noodlelab diff --json runs/<this run> runs/<an earlier run>

It says what changed from the earlier run to this one. For two records of a script (noodlelab.verify), that is the inputs and results by name, the requirements, each check’s verdict and how its margin moved, the script and its git commit, and the files read and written. For two graph runs, it is the settings and links, node code, input files and, when both runs kept their values, the results. Python and the package versions are compared for both. Exit code 0 means nothing differs (within --rtol and --atol), 1 that something does, and 2 that a record could not be read, or that one is a script’s and the other a graph run’s. In Python, nv.load(run).diff(earlier) gives the same.

What changed between two runs, from their provenance records.

For a graph run, noodlelab.core.baseline compares the results; this compares what produced them: settings and links, node code, input files, and the software environment. Nodes match by id, or else by title and type, as in noodlelab.core.baseline.compare().

A script’s record (noodlelab.verify.record()) holds its inputs, results and checks themselves, so diff_records() compares those too, by name. diff_runs() takes either kind, and is what noodlelab diff prints.

Run seeds

Random nodes (Monte Carlo, sampling designs, model training) take a seed with a fixed default, so a graph gives the same numbers every time. To see whether a result depends on the random numbers you happened to draw, run with other ones:

noodlelab run analysis.graph.json --seed 7

One seed for a whole run.

Random nodes (Monte Carlo, sampling designs, model training…) each take a seed input with a fixed default, so a graph gives the same numbers every time it runs. That settles “is it repeatable?”, but not “does the result depend on the random numbers I happened to draw?”: to answer that, run again with other random numbers, all of them.

A run seed does that (noodlelab run --seed 7): every seed input the graph leaves at its default gets its own seed, derived from the run seed and the node’s id, so two nodes never share a random stream and the same run seed always gives the same seeds. Seeds set in the graph are kept: the graph’s author chose them. Without a run seed nothing changes.

The run seed is recorded with the run (see provenance), and the seeds it gave, so noodlelab reproduce runs with the same numbers.

Analysis variants

Every change to a graph’s analysis is logged, so the paths tried on the way to a result are not forgotten.

Analysis variants: every version of an analysis that was run.

An analysis rarely goes straight from question to answer. On the way, a filter is tried, an outlier dropped, a model swapped, a threshold nudged, a random seed changed, and each of those runs is a forking path: a choice that could have gone another way. The result finally reported is one of them. The more of them there were, and the less the reader knows about them, the less a significant-looking result means (Gelman and Loken, “The garden of forking paths”, 2013).

So every run of a graph is compared with the one before it, and when the analysis changed, a line is added to .noodlelab/variants/<graph>.jsonl in the workspace: a fingerprint of every node (its type, settings, links and source code; see fingerprints()), the run seed, and what changed since the previous variant, by node title. Running the same analysis again adds nothing; runs are not variants. The report’s analysis history (Add Analysis History) and noodlelab verify (rule NL017) read it back, and so does a plan lock (see locks), which compares the analysis with the one planned.

Unlike tracked values, variants stay when their runs are deleted: deleting a run does not undo having tried it. Report nodes are left out: rewording the report does not change the analysis.

Plans and sign-offs

noodlelab plan lock analysis.graph.json --hypothesis "the coating lowers wear by 10 %" --tag
noodlelab plan status analysis.graph.json
noodlelab plan amend analysis.graph.json --reason "sensor 3 failed; its data is dropped"
noodlelab signoff analysis.graph.json --role checker --rev B
noodlelab signoff analysis.graph.json --list

The report nodes Add Analysis History and Add Sign-off put the plan, its deviations, the versions of the analysis that were run and the signatures into the PDF.

Locks on an analysis: a plan fixed before the results, sign-offs after them.

Both answer the question a reader of a result cannot answer alone: was the analysis decided before its outcome was known, and has it changed since someone checked it?

A plan lock (preregistration) is taken before the data is in, or at least before the results are looked at:

noodlelab plan lock analysis.graph.json --hypothesis "the coating lowers wear by 10 %"

It records a snapshot of the analysis (see snapshot()): every node’s fingerprint (type, settings, links, code; see variants), the requirements with their limits, and a hash of every data file the graph reads, or "absent" for a file not there yet, which shows the data was not in when the plan was made. Later, noodlelab verify compares the analysis with the plan: a node changed, a threshold moved or a data file replaced since the lock is a deviation (rules NL012 to NL014). Deviations are allowed, but they must be said: noodlelab plan amend --reason "..." records them with the reason, and they are then reported, not failed. In studies of preregistered research, most papers deviated from their plans and many did not say so; this makes the saying cheap and the not saying visible.

A sign-off is the engineering form of the same idea: a preparer, a checker and an approver put their names to a run of the analysis:

noodlelab signoff analysis.graph.json --role checker --rev B

The sign-off holds the same snapshot. Any later change to the analysis, its requirements or its data voids it (rule NL018), so a signed calculation cannot quietly drift from what was checked. Only a person can sign as checker or approver: the MCP tools refuse to (see noodlelab.agent.tools).

Both are plain JSON next to the graph (<graph>.plan.json and <graph>.signoff.json), meant to be committed with it: git then holds the proof of when each was made.

Results that are not numbers

Results that are not numbers.

A calculation that divides by zero, takes the log of a negative number or overflows does not stop: numpy returns nan or inf and everything downstream inherits it, often ending as a plot with a gap or a requirement that “failed” for no visible reason. The executor looks at what every node returns and warns at the node where such a value first appears (see noodlelab.core.context.warn()); the run goes on.

Not every nan is a mistake, so the rules follow what nan means where (and a node says that a result does not apply with None, not nan):

• a single number (a float, a quantity, the value of an uncertain number) that is not finite is always reported;

• inside arrays nan usually marks missing data (a gap in a series, a raster’s nodata), so only inf, or an array that is nan throughout, is reported;

• in tables inf is as often meant (the open ends of a set of classes, a bin edge), so only a numeric column that is nan throughout is reported;

• an uncertainty of nan means “not known” (see noodlelab.core.uncertainty) and is never reported.

This must stay cheap: it runs on every result, so it looks at numeric dtypes with vectorised numpy and never walks Python containers.