# Reproducibility

A result is only worth something if it can be had again: by a colleague, by a
reviewer, or by you in a year. noodlelab records every run (see
{mod}`noodlelab.core.provenance`), and gives tools on top of that record.

## Reproduce a run

```bash
noodlelab reproduce runs/20250102-093000-a1b2c3d4e5f6   # a run folder
noodlelab reproduce bundle.zip                            # a bundle downloaded from the editor
noodlelab reproduce bundle.zip --clean-env                # in a fresh environment (needs uv)
```

It runs the graph again exactly as recorded: the same settings, the same run
seed, and nothing reused from caches or checkpoints. Then it compares every
result with the recorded one (`--rtol`, `--atol`; the defaults are 1e-9 and 0).
Exit code 0 means everything is the same, 1 that something differs, 2 that it
could not run. When results differ it says why they may differ: node code that
changed, input files that changed, other package versions, another Python or
platform, another git commit, or uncommitted changes when the run was made.

`--clean-env` builds a new virtual environment with the recorded Python and
package versions, and reproduces the run there. What is tested is then the
record, not the machine it was made on. Agents have the same tool over MCP
(`reproduce`, and `diff_runs` to compare two runs).

```{eval-rst}
.. automodule:: noodlelab.core.reproduce
   :no-members:
```

## Compare two runs

```bash
noodlelab diff runs/<this run> runs/<an earlier run>   # folders or their provenance.json
noodlelab diff --json runs/<this run> runs/<an earlier run>
```

It says what changed from the earlier run to this one. For two records of a
script (`noodlelab.verify`), that is the inputs and results by name, the
requirements, each check's verdict and how its margin moved, the script and
its git commit, and the files read and written. For two graph runs, it is the
settings and links, node code, input files and, when both runs kept their
values, the results. Python and the package versions are compared for both.
Exit code 0 means nothing differs (within `--rtol` and `--atol`), 1 that
something does, and 2 that a record could not be read, or that one is a
script's and the other a graph run's. In Python, `nv.load(run).diff(earlier)`
gives the same.

```{eval-rst}
.. automodule:: noodlelab.core.rundiff
   :no-members:
```

## Run seeds

Random nodes (Monte Carlo, sampling designs, model training) take a `seed` with
a fixed default, so a graph gives the same numbers every time. To see whether a
result depends on the random numbers you happened to draw, run with other ones:

```bash
noodlelab run analysis.graph.json --seed 7
```

```{eval-rst}
.. automodule:: noodlelab.core.seeds
   :no-members:
```

## Analysis variants

Every change to a graph's analysis is logged, so the paths tried on the way to
a result are not forgotten.

```{eval-rst}
.. automodule:: noodlelab.core.variants
   :no-members:
```

## Plans and sign-offs

```bash
noodlelab plan lock analysis.graph.json --hypothesis "the coating lowers wear by 10 %" --tag
noodlelab plan status analysis.graph.json
noodlelab plan amend analysis.graph.json --reason "sensor 3 failed; its data is dropped"
noodlelab signoff analysis.graph.json --role checker --rev B
noodlelab signoff analysis.graph.json --list
```

The report nodes *Add Analysis History* and *Add Sign-off* put the plan, its
deviations, the versions of the analysis that were run and the signatures into
the PDF.

```{eval-rst}
.. automodule:: noodlelab.core.locks
   :no-members:
```

## Results that are not numbers

```{eval-rst}
.. automodule:: noodlelab.core.finite
   :no-members:
```