Remote storage¶

Data rarely lives next to the graphs. A server can give remote storage a short name, a mount, so that graphs read lab-s3://2026/run1.csv instead of a bucket URL with credentials in it. People see the mount’s name and browse it in file pickers; they never see the URL or the secrets.

Adding a mount¶

Open Admin → Storage → Add storage and fill in:

Field	What it is
Name	The short name used in file references, `name://path`: lowercase letters, digits and `-`.
Label	What the file picker shows (optional).
URL	The root of the storage, as any fsspec URL.
Storage options	A JSON object passed to the backend: credentials, endpoint, region. Secrets are never shown again once saved.
Writable	Whether nodes may save files there. Mounts are read-only unless this is on.
Workspaces	The workspaces that can use it. None ticked means all of them.

Test connection lists the mount’s root with the settings as they are on screen, before anything is saved.

Some examples:

Storage	URL	Storage options
S3, MinIO, Ceph, R2	`s3://bucket/prefix`	`{"key": "...", "secret": "...", "endpoint_url": "https://minio.example.org"}`
Google Cloud Storage	`gs://bucket/prefix`	`{"token": "/secrets/gcs.json"}`
Azure Blob	`az://container/prefix`	`{"account_name": "...", "account_key": "..."}`
SFTP	`sftp://host/home/data`	`{"username": "...", "password": "..."}`
A share mounted on the server	`file:///mnt/nas/lab`	`{}`

Installing the backends¶

noodlelab needs only fsspec itself (the remote extra, included in full). Each kind of storage needs its fsspec backend installed where runs happen:

pip install "noodlelab[full,s3]"      # AWS S3 and compatible stores (s3fs)
pip install "noodlelab[full,gcs]"     # Google Cloud Storage (gcsfs)
pip install "noodlelab[full,azure]"   # Azure Blob and Data Lake (adlfs)
pip install "noodlelab[full,sftp]"    # SFTP (sshfs)

Any other fsspec implementation works too (smb://, http(s)://, your own): install it, and use its URL.

Mounts from the environment¶

For containers that get their credentials from the environment, set NOODLELAB_MOUNTS to a JSON list of mounts with the same fields:

NOODLELAB_MOUNTS='[{"name": "lab-s3", "url": "s3://lab-data/raw",
  "options": {"key": "...", "secret": "..."}, "workspaces": []}]'

These show up in the admin portal marked from NOODLELAB_MOUNTS and can only be changed there. Mounts added in the portal are saved in the server’s config.json (in NOODLELAB_HOME), credentials included, so keep that folder readable only by the server’s user.

Using files in nodes¶

A node takes a file with the FileRef type, a string that names it:

data/run1.csv               relative: inside the workspace
/mnt/nas/run1.csv           absolute: on the server (local mode only)
lab-s3://2026/run1.csv      on the mount named lab-s3

Absolute paths and paths that leave the workspace (../other) work only on your own machine, without logins. On a shared server (--auth users) and the demo, graphs reach files inside their workspace and, on a shared server, on the mounts an admin configured: add a folder on the server as a mount with the URL local:///mnt/nas/lab (or file:///mnt/nas/lab) and refer to it as nas://run1.csv.

from noodlelab import FileRef, node


@node(category="Tables")
def load_csv(path: FileRef) -> pd.DataFrame:
    with path.open() as f:  # works for every backend
        return pd.read_csv(f)

path.local_path() gives a real local file for libraries that insist on one, downloaded into a cache for remote mounts. The editor checks that the file exists, previews re-run when it changes, and cached results are keyed by the file’s size and modification time, so nodes that read files stay cacheable. A run’s provenance fingerprints every file it read: size, modification time and, for local files up to a size limit, a SHA-256 of the content.

Outside the server (scripts, exported graphs), a FileRef also accepts plain fsspec URLs such as s3://bucket/key. On a Slurm cluster, jobs use the mounts of the server’s config.json when NOODLELAB_HOME is on the shared file system (see Slurm clusters).