# Running graphs on a Slurm cluster

A noodlelab server can send runs to a [Slurm](https://slurm.schedmd.com/) cluster instead of running them in its own process. The editor works the same way either way: progress, node previews, logs, checks and the run's reports show up as they do for a local run.

This is optional. Nothing changes until an admin turns it on.

## How it works

A cluster run is a whole graph run, submitted as one batch job:

1. When someone runs a graph *on the cluster*, the server writes two files into `<workspace>/.noodlelab/jobs/<run id>/`:
   - `graph.json`: the graph as it is in the editor, saved or not;
   - `job.sh`: a batch script that runs it with `noodlelab run`.
2. The server submits the script with `sbatch --parsable`.
3. While the job runs, `noodlelab run` appends every run event to `events.jsonl` in the same folder. The server follows that file and forwards the events to the editor.
4. The job writes its run folder (`runs/<time>-<run id>/`, with reports and `provenance.json`) and its checkpoints into the workspace, as a local run would. The run appears in the editor's run list, and the next run reuses its checkpoints, whether it runs locally or on the cluster.
5. The server asks `squeue` (then `sacct`) about the job every *poll seconds*. The editor shows *Slurm job 1234 · pending* until the job starts. If the job ends without finishing the run (a time limit, out of memory, a node failure), the run fails with the end of the job's `slurm-<id>.out`.
6. **Stop** writes a `cancel` file, which the job checks between nodes. If the job is still pending, or hasn't stopped after 30 seconds, the server runs `scancel` on it.

```
editor ──run──▶ server ──sbatch job.sh──▶ Slurm ──▶ compute node: noodlelab run graph.json
   ▲               │                                              │
   └── events ◀────┴──── tails .noodlelab/jobs/<id>/events.jsonl ◀─┘  (shared file system)
```

## Requirements

- **A shared file system.** The workspace folders must be at the same path on the server and on the compute nodes: NFS, Lustre, GPFS or BeeGFS. So must the checkpoint folder, if *Checkpoint dir* is set in Settings. The server's home folder (`NOODLELAB_HOME`, which holds `config.json`) should be shared too, so jobs use the same settings and storage mounts. Without it, jobs fall back to the defaults.
- **noodlelab on the compute nodes.** Use the same version and install tier as the server. The *Setup* lines in the cluster settings run before the graph and can load it, for example:

  ```bash
  module load python/3.12
  source /shared/apps/noodlelab/bin/activate
  ```

  Alternatively, set *Noodlelab command* to the full path of the executable, for example `/shared/apps/noodlelab/bin/noodlelab`.
- **A submit host.** The server runs `sbatch`, `squeue`, `sacct` and `scancel` itself, as the user the server runs as. If the server is not a submit host, set *Command prefix* to reach one, for example `ssh -o BatchMode=yes login01` with a key-based login. The workspace paths must then be the same on the login node.
- **Accounting.** Every job is submitted as the server's user. The person who ran it is recorded in the job's provenance (`user`) and shown in the admin portal. To charge different groups, give each workspace its own *account* or *partition*.

## Setting it up

1. Open **Admin → Cluster**.
2. Fill in the partition, account, QoS, time limit, CPUs, memory and GPUs that jobs should request, and any extra `#SBATCH` options, one per line.
3. Fill in the *Setup* lines and the *Noodlelab command*.
4. Press **Test connection**. It runs `sbatch --version` and `sinfo` with the settings as they are on screen.
5. Turn on **Enabled** and save. Turn on **Default** as well if runs should go to the cluster unless someone picks *On the server*.
6. Optionally, under **Workspaces** on the same page, keep a workspace off the cluster, or give it its own partition or account.

In the editor, a **Run on** menu appears next to **Run** in every workspace that may use the cluster, for people who may run graphs there. The choice applies to the next run.

## Job script

A job script looks like this:

```bash
#!/bin/bash
#SBATCH --job-name=noodlelab-beam-study
#SBATCH --output=/lab/projects/beams/.noodlelab/jobs/8f10e83b-215/slurm-%j.out
#SBATCH --chdir=/lab/projects/beams
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=4
#SBATCH --partition=compute
#SBATCH --mem=8G

# written by noodlelab: runs one graph and reports its events to the server
module load python/3.12
exec noodlelab run /lab/projects/beams/.noodlelab/jobs/8f10e83b-215/graph.json \
  --workspace /lab/projects/beams --home /srv/noodlelab/home --run-id 8f10e83b-215 \
  --events .../events.jsonl --cancel-file .../cancel --info '{"user":"Ana",...}'
```

`noodlelab run` has the same options everywhere, so you can test a job by hand on a compute node with `srun --pty bash`, then the `noodlelab run …` line.

The job folders stay in the workspace after a run, so you can read the logs. Delete `.noodlelab/jobs/` whenever you like.

## Limits

- A whole graph is one job, and its nodes run one after another inside it. Give the job the CPUs, memory or GPUs that its heaviest node needs. Spreading one graph's nodes, or a sweep's passes, over several jobs is not supported yet.
- Live probes, the quick previews shown while editing, still run on the server.
- Demo servers (`NOODLELAB_DEMO`) never use the cluster.