Running graphs on a Slurm cluster

A noodlelab server can send runs to a Slurm cluster instead of running them in its own process. The editor works the same way either way: progress, node previews, logs, checks and the run’s reports show up as they do for a local run.

This is optional. Nothing changes until an admin turns it on.

How it works

A cluster run is a whole graph run, submitted as one batch job:

  1. When someone runs a graph on the cluster, the server writes two files into <workspace>/.noodlelab/jobs/<run id>/:

    • graph.json: the graph as it is in the editor, saved or not;

    • job.sh: a batch script that runs it with noodlelab run.

  2. The server submits the script with sbatch --parsable.

  3. While the job runs, noodlelab run appends every run event to events.jsonl in the same folder. The server follows that file and forwards the events to the editor.

  4. The job writes its run folder (runs/<time>-<run id>/, with reports and provenance.json) and its checkpoints into the workspace, as a local run would. The run appears in the editor’s run list, and the next run reuses its checkpoints, whether it runs locally or on the cluster.

  5. The server asks squeue (then sacct) about the job every poll seconds. The editor shows Slurm job 1234 · pending until the job starts. If the job ends without finishing the run (a time limit, out of memory, a node failure), the run fails with the end of the job’s slurm-<id>.out.

  6. Stop writes a cancel file, which the job checks between nodes. If the job is still pending, or hasn’t stopped after 30 seconds, the server runs scancel on it.

editor ──run──▶ server ──sbatch job.sh──▶ Slurm ──▶ compute node: noodlelab run graph.json
   ▲               │                                              │
   └── events ◀────┴──── tails .noodlelab/jobs/<id>/events.jsonl ◀─┘  (shared file system)

Requirements

  • A shared file system. The workspace folders must be at the same path on the server and on the compute nodes: NFS, Lustre, GPFS or BeeGFS. So must the checkpoint folder, if Checkpoint dir is set in Settings. The server’s home folder (NOODLELAB_HOME, which holds config.json) should be shared too, so jobs use the same settings and storage mounts. Without it, jobs fall back to the defaults.

  • noodlelab on the compute nodes. Use the same version and install tier as the server. The Setup lines in the cluster settings run before the graph and can load it, for example:

    module load python/3.12
source /shared/apps/noodlelab/bin/activate

    Alternatively, set Noodlelab command to the full path of the executable, for example /shared/apps/noodlelab/bin/noodlelab.

  • A submit host. The server runs sbatch, squeue, sacct and scancel itself, as the user the server runs as. If the server is not a submit host, set Command prefix to reach one, for example ssh -o BatchMode=yes login01 with a key-based login. The workspace paths must then be the same on the login node.

  • Accounting. Every job is submitted as the server’s user. The person who ran it is recorded in the job’s provenance (user) and shown in the admin portal. To charge different groups, give each workspace its own account or partition.

Setting it up

  1. Open Admin → Cluster.

  2. Fill in the partition, account, QoS, time limit, CPUs, memory and GPUs that jobs should request, and any extra #SBATCH options, one per line.

  3. Fill in the Setup lines and the Noodlelab command.

  4. Press Test connection. It runs sbatch --version and sinfo with the settings as they are on screen.

  5. Turn on Enabled and save. Turn on Default as well if runs should go to the cluster unless someone picks On the server.

  6. Optionally, under Workspaces on the same page, keep a workspace off the cluster, or give it its own partition or account.

In the editor, a Run on menu appears next to Run in every workspace that may use the cluster, for people who may run graphs there. The choice applies to the next run.

Job script

A job script looks like this:

#!/bin/bash
#SBATCH --job-name=noodlelab-beam-study
#SBATCH --output=/lab/projects/beams/.noodlelab/jobs/8f10e83b-215/slurm-%j.out
#SBATCH --chdir=/lab/projects/beams
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=4
#SBATCH --partition=compute
#SBATCH --mem=8G

# written by noodlelab: runs one graph and reports its events to the server
module load python/3.12
exec noodlelab run /lab/projects/beams/.noodlelab/jobs/8f10e83b-215/graph.json \
  --workspace /lab/projects/beams --home /srv/noodlelab/home --run-id 8f10e83b-215 \
  --events .../events.jsonl --cancel-file .../cancel --info '{"user":"Ana",...}'

noodlelab run has the same options everywhere, so you can test a job by hand on a compute node with srun --pty bash, then the noodlelab run … line.

The job folders stay in the workspace after a run, so you can read the logs. Delete .noodlelab/jobs/ whenever you like.

Limits

  • A whole graph is one job, and its nodes run one after another inside it. Give the job the CPUs, memory or GPUs that its heaviest node needs. Spreading one graph’s nodes, or a sweep’s passes, over several jobs is not supported yet.

  • Live probes, the quick previews shown while editing, still run on the server.

  • Demo servers (NOODLELAB_DEMO) never use the cluster.