# Namespace Plugins

## Overview

A namespace plugin can be enabled to provide private, temporary file system space for each job.

When enabled on the cluster, a filesystem namespace will be created for each job with a unique, private instance of /tmp and /dev/shm for the job to use. These directories can be changed with the Dirs= option in the plugin-specific configuration file. The contents of these directories will be removed at job termination. Additionally, the namespace/linux plugin can be configured to create new PID and user namespaces.

## Installation

These plugins are built and installed as part of the default build, no extra installation steps are required.

## Setup

Slurm must be configured to load the namespace plugin by adding PrologFlags=contain and setting NamespaceType to the desired plugin in slurm.conf. Additional configuration must be done in the plugin-specific configuration file file, which should be placed in the same directory as slurm.conf.

If using pam_slurm_adopt, note the join_container option that controls whether an adopted SSH session will try to join the job's namespace.

### namespace/linux plugin

The namespace/linux plugin (added 25.11) uses the configuration file namespace.yaml. This plugin can be configured to create user and PID namespaces in addition to a temporary filesystem namespace. Namespaces can be configured for all nodes or for a subset of nodes.

By default the plugin creates private bind mounts for /tmp and /dev/shm. The dirs option can be used to change the list of directories. For more control, the dir_confs option provides per-directory configuration including per-directory backing storage, mount options, tmpfs support, and path substitutions.

As an example, if all nodes will be configured the same way, you could put the following in your namespace.yaml:

    defaults:
      auto_base_path: true
      base_path: "/var/nvme/storage"

Note the following important details with this plugin:

- This plugin requires cgroup/v2 to operate correctly.
- When using user namespaces, bpf token support (added in kernel 6.9) is required to use ConstrainDevices in cgroup.conf.

### namespace/tmpfs plugin

The namespace/tmpfs plugin (formerly job_container/tmpfs) uses the configuration file job_container.conf. Namespaces can be configured for all nodes, or for a subset of nodes. As an example, if all nodes will be configured the same way, you could put the following in your job_container.conf:

    AutoBasePath=true
    BasePath=/var/nvme/storage

## Initial Testing

An easy way to verify that the container is working is to run a job and ensure that the /tmp directory is empty (since it normally has some other files) and is owned by the user that submitted the job.

    tim@slurm-ctld:~$ srun ls -al /tmp
    total 8
    drwx------  2 tim    root 4096 Feb 10 17:14 .
    drwxr-xr-x 21 root   root 4096 Nov 15 08:46 ..

While a job is running, root should be able to confirm that `/$BasePath/$JobID/_tmp` exists and is empty. This directory is bind mounted into the job. `/$BasePath/$JobID` should be owned by root, and is not intended to be accessible to the user.

You can also verify the mount location is correct:

    parallels@linux_vb:~$ echo $SLURM_JOB_ID
    7
    parallels@linux_vb:~$ findmnt -o+PROPAGATION | grep /tmp
    └─/tmp  /dev/sda1[/storage/7/.7] ext4  rw,relatime,errors=remount-ro,data=ordered   private

In the example above, base_path points to /storage, so job ID 7 mounts /tmp on /storage/7/.7. When a user from inside a job tries to look up mounts, they can see that their /tmp is mounted, but they are prevented from mistakenly accessing the backing directory directly.

Additionally, when the Linux plugin is in use, you can confirm that a PID namespace is in effect by running a job and running `ps`. The only visible PIDs should be related to the job and PID 1 will be named `slurmstepd: [${job_id}.namespace]`.

## SPANK

This plugin interfaces with the SPANK API, and automatically joins the job's namespace in the following functions:

- spank_task_init_privileged()
- spank_task_init()

Mount namespace construction also happens before the job's SPANK environment is set up. Hence all SPANK-related job steps will view only the private /tmp the plugin creates.

In addition to the job itself, the TaskProlog will also be executed inside the container.