Build a Six-Node Slurm HPC Cluster with OpenFactory

A senior HPC infrastructure job description can compress a surprising amount of engineering into four bullets: build custom images, operate Kubernetes and a scheduler, move distributed workloads, and tune compute, storage, and scheduling. A useful build exercise should make those nouns interact instead of placing their packages on one VM.

This OpenFactory recipe creates six Debian 13 machines with separate controller, compute, login, storage, and observability roles. Five machines form a kubeadm cluster. SchedMD Slinky then places a real Slurm controller, LoginSet, REST API, and two-node CPU partition on that cluster. The sixth machine exports NFS and never joins Kubernetes.

The build passes only when a real two-node srun, a real batch job, shared storage, Slurm controller restart persistence, and monitoring placement all behave as specified. There are no HTTP stand-ins for sbatch, generated green metrics, or scheduler-shaped text files.

What the build produces

The result is a six-VM Debian 13 lab on one private subnet. Five VMs form Kubernetes; two of those become Slurm compute nodes. A separate storage VM serves controller state and shared home directories, while the observability VM carries the main Prometheus, Grafana, Alertmanager, operator, and kube-state-metrics pods.

The minimum guest allocation is 24 GB of RAM and seven vCPUs. The storage node also receives one generated 20 GB virtio disk with serial OPENFACTORY_HPC_DATA. Its first-boot service refuses to format anything when that serial is missing, duplicated, attached to the root disk, or already carries an unexpected filesystem.

How first boot assembles the cluster

All six images can boot together, but their services advance through explicit gates. Storage identifies and exports its dedicated disk. The controller initializes kubeadm and Flannel. Four workers join with one short-lived token. Only after five nodes are Ready does the controller revoke that token and install Slinky plus monitoring.

OpenFactory puts a secret-free intent object in the plan: controller, workers, Kubernetes version, pod CIDR, service CIDR, CNI, and a 900-second maximum token lifetime. The backend generates one token for the build group and injects the role-specific first-boot services. It does not ask the model to invent a token or leave one in recipe JSON.

The VM subnet 10.103.0.0/24, pod subnet 10.244.0.0/16, and service subnet 10.112.0.0/16 must not overlap. This catches a mundane but expensive class of cluster failure before any image is built.

The bootstrap security boundary

The convenience boundary is narrow and visible: one random kubeadm token per group, a maximum lifetime of 15 minutes, unsafe CA discovery permitted only on the RFC1918 lab subnet, redaction from user-facing responses, and mandatory deletion after all five Kubernetes members report Ready.

Kubeadm documents unsafe CA discovery as a weaker bootstrap mode because a stolen token can help an attacker impersonate the control plane. That trade permits parallel first boot without preloading a CA hash into every image, but it only belongs on the isolated lab network.

The controller uses the normal kubeadm token lifecycle. The acceptance gate checks the negative condition too: after the five-node Ready count is exact, kubeadm token list must contain no live bootstrap token. Production images need CA-pinned discovery and a protected join-material delivery path.

Where Slurm and storage run

Slinky turns Kubernetes resources into the Slurm service boundary. The operator, controller, and REST API stay on controller; LoginSet stays on login-ops; a DaemonSet NodeSet lands only on the two compute hosts. Static NFS volumes preserve controller state and give both compute nodes the same shared home directory.

SchedMD's Slinky installation guide describes the operator and Slurm chart split. Its NodeSet operations guide covers the compute-node resource model. Here the default CPU partition must contain exactly the two DaemonSet members, not every Kubernetes worker.

Slurm controller state and shared /home use static NFS PersistentVolumes and claims. That design lets a replacement slurmctld pod recover known jobs and lets a batch job write once for both compute nodes. Kubernetes documents that PersistentVolumes separate storage implementation from consumption, which is the exact separation this failure test exercises.

Commands on the login host are wrappers around the actual Slinky LoginSet. sinfo, squeue, srun, sbatch, scontrol, and scancel execute inside that service. The Slurm 26.05 command manuals remain the behavior contract.

What the acceptance gate proves

The enabled group scenario checks behavior across machine boundaries, not package presence. It waits up to 30 minutes where cluster startup needs it, fails on empty command output, uses strict shell settings, and requires exact counts for Kubernetes members, Slurm nodes, NFS exports, tool versions, and monitoring placement.

A second scenario records fio, iperf3, kubeadm duration, and scheduler latency commands, but it is disabled and non-gating. A single-host virtual lab is useful for comparing builds; it cannot establish production storage or network service levels.

What is pinned in the images

Version ownership sits in feature generators, not ad hoc package hooks written by the planner. Kubernetes, Flannel, Slinky, Slurm, monitoring, Helm, and Terraform have fixed versions. Chart archives, installers, and configuration enter the images with checksum checks, while the running cluster still needs egress for its pinned OCI images.

The storage generator is equally strict about physical identity. A size or bus match is not enough; the disk serial must be exact. That turns a destructive format step into a fail-closed operation tied to the topology declaration.

The exact build prompt

This prompt is the executable specification used by the article's Deploy button and the OpenFactory regression campaign. It fixes every address, role, placement rule, version, storage guard, test, warning, and exclusion. Editing away an awkward constraint may make planning easier, but it also changes what the result can honestly prove.

Build a real, disposable six-node HPC lab named `slurm-hpc`.

This fixture must exercise OpenFactory's actual multi-machine build and libvirt
test path. Do not create Python compatibility listeners, fake `/sbatch` HTTP
endpoints, mock metrics, or placeholder Slurm configuration. The running system
must be Kubernetes v1.36.3 bootstrapped with kubeadm, Flannel v0.28.8, SchedMD
Slinky Slurm Operator chart v1.2.0 / Slurm 26.05, and
`kube-prometheus-stack` 86.0.0.

This is intentionally a zero-touch lab image set. It uses one backend-generated
kubeadm token with a 15-minute maximum lifetime and unsafe CA discovery only on
the private RFC1918 lab network. The controller must revoke the token after all
five Kubernetes nodes are Ready. Make the warnings say plainly that these are
lab images, not secure production runtime bundles.

## Six-node topology

Use one network:

```json
{
  "name": "hpc-lab",
  "cidr": "10.103.0.0/24",
  "gateway": "10.103.0.1",
  "dns_servers": ["1.1.1.1", "8.8.8.8"],
  "is_default_route": true
}
```

Every node is a `linux-build` on `debian-trixie`, has one static interface on
`hpc-lab`, and has 4 GB minimum RAM. Use this exact inventory:

| node_id | IP | role | min CPU | Kubernetes/Slinky placement |
|---|---:|---|---:|---|
| `controller` | `10.103.0.10` | `controller` | 2 | control plane, Slinky operator/controller/restapi |
| `compute-1` | `10.103.0.21` | `compute` | 1 | worker, CPU DaemonSet NodeSet |
| `compute-2` | `10.103.0.22` | `compute` | 1 | worker, CPU DaemonSet NodeSet |
| `login-ops` | `10.103.0.30` | `login` | 1 | worker, LoginSet, kubectl/Helm/Terraform |
| `storage` | `10.103.0.40` | `storage` | 1 | outside Kubernetes, NFS server |
| `observability` | `10.103.0.50` | `observability` | 1 | worker, Prometheus/Grafana placement |

Use canvas-pixel coordinates with no overlap: controller `(320, 100)`,
compute-1 `(120, 340)`, compute-2 `(520, 340)`, login-ops `(120, 580)`,
storage `(520, 580)`, observability `(320, 820)`. Connect the controller to
all four Kubernetes workers, connect storage to controller, both computes, and
login-ops, and connect observability to controller.

The `storage` topology node must declare exactly this generated data disk:

```json
{
  "id": "hpc-data",
  "kind": "disk",
  "size_gb": 20,
  "bus": "virtio",
  "serial": "OPENFACTORY_HPC_DATA"
}
```

The storage image may format only the runtime disk whose serial is exactly
`OPENFACTORY_HPC_DATA`; it must refuse zero matches, multiple matches, the root
disk, and an unexpected existing filesystem.

## Kubernetes intent

Include this secret-free object at `plan.kubernetes`. Never include a token:

```json
{
  "mode": "kubeadm-lab",
  "version": "v1.36.3",
  "controller": "controller",
  "workers": ["compute-1", "compute-2", "login-ops", "observability"],
  "pod_cidr": "10.244.0.0/16",
  "service_cidr": "10.112.0.0/16",
  "cni": "flannel",
  "bootstrap_token_ttl_seconds": 900
}
```

The VM, pod, and service CIDRs must remain non-overlapping. The storage node is
not a Kubernetes member, so the live acceptance count is five Ready nodes.

## Recipe contract

Every recipe must use `base_image: "debian-trixie"`, set
`hardware.min_memory_gb: 4`, set the CPU minimum from the table, and keep only
role-appropriate features/packages. Every node has a normal `ops` user with
sudo and SSH for test access. Do not put secrets in the prompt beyond the
fixture's ordinary lab login password.

Features by role:

- `controller`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm`, `helm`
- `compute-1`, `compute-2`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm`
- `login-ops`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm`, `helm`,
  `terraform`
- `storage`: `headless`, `ssh`, `hpc-nfs`
- `observability`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm`

Do not list `helm`, `terraform`, `kubeadm`, `kubectl`, or `kubelet` as
additional apt packages and do not add ad hoc upstream repository hooks. The
declared OpenFactory features own those pinned installations.

The backend injects each Kubernetes member's
`openfactory-kubeadm-lab` service from `plan.kubernetes`. Do not handcraft that
service or a bootstrap token.

Every Kubernetes member must declare an enabled service named
`openfactory-slinky-slurm` with this shared config and its own correct `role`
(`controller`, `compute`, `login`, or `observability`):

```json
{
  "controller_node_id": "controller",
  "compute_node_ids": ["compute-1", "compute-2"],
  "login_node_id": "login-ops",
  "observability_node_id": "observability",
  "nfs_server": "10.103.0.40",
  "slinky_version": "1.2.0",
  "slurm_version": "26.05",
  "prometheus_stack_version": "86.0.0"
}
```

The storage recipe must declare an enabled service named
`openfactory-hpc-nfs`:

```json
{
  "disk_serial": "OPENFACTORY_HPC_DATA",
  "export_network": "10.103.0.0/24"
}
```

Do not add startup scripts that duplicate feature-generator behavior.

## Required running behavior

Slinky accounting stays disabled. Use NFS-backed static PV/PVC storage for
Slurm controller state and NFS-backed shared `/home` in LoginSet and NodeSet.
The controller, operator, and restapi run on `controller`; the LoginSet runs on
`login-ops`; a DaemonSet NodeSet runs only on `compute-1` and `compute-2`; and
the default CPU partition contains exactly those two Slurm nodes. Enable
controller metrics and its ServiceMonitor. Put the main Prometheus, Grafana,
Alertmanager, operator, and kube-state-metrics pods on `observability`.

The login-ops host commands `sinfo`, `squeue`, `srun`, `sbatch`, `scontrol`,
and `scancel` must execute the real command inside the Slinky LoginSet. They
must not emulate scheduler responses.

Runtime egress is allowed for the pinned Slinky container images. The Slinky
and monitoring charts, Kubernetes/Flannel configuration, and Helm/Terraform CLI
installers must be baked into the images with fixed versions and checksums.

## Gating acceptance scenario

Create one enabled group scenario named `real-hpc-acceptance`. Use
`command_succeeds` or `command_output` assertions with
`params.timeout_seconds` up to 1800. The scenario must verify:

1. On `controller`, exactly five Kubernetes nodes report Ready, kubeadm reports
   Kubernetes v1.36.3, Flannel pods are Ready, and `kubeadm token list` has no
   live bootstrap token.
2. On `login-ops`, `sinfo` reports exactly two Slurm nodes in the CPU partition.
3. On `login-ops`, `srun -N2 -n2 hostname` produces two distinct hostnames.
4. On `login-ops`, a real `sbatch --wait` job writes
   `/home/openfactory-hpc-acceptance.txt`, then `srun -N2 -n2 cat` reads the
   same content from both compute nodes.
5. From `controller`, submit a running job through the real LoginSet, delete the
   slurmctld pod, wait for its replacement, and verify the job remains known.
   This proves controller state survived a pod restart on the NFS PVC.
6. On `controller`, the Slinky controller ServiceMonitor exists; Prometheus and
   Grafana workloads are Ready in `monitoring`; their main pods are assigned to
   `observability`.
7. On `login-ops`, `helm version --short` contains `v4.2.3` and
   `terraform version` contains `v1.15.8`.
8. On `storage`, the mounted filesystem comes from the disk with serial
   `OPENFACTORY_HPC_DATA`, all three exports are active, and NFS is serving.

Avoid shell checks that pass on empty output. Every command must use
`set -euo pipefail`, explicit numeric/string assertions, and bounded waits.

## Non-gating measurements

Add a second scenario named `hpc-lab-measurements` with `enabled: false`. It
documents commands for NFS `fio`, VM-to-VM `iperf3`, kubeadm bootstrap duration,
and Slurm queue/scheduler latency. These are reported measurements, not pass/fail
SLOs for the first CPU lab.

## Warnings and scope

Preserve all of these warnings:

- Lab-only bootstrap uses unsafe CA discovery on a private network and is not a
  secure production runtime bundle.
- Production identity, multi-user UID/GID management, TLS, secrets rotation,
  network policy, backup/restore, and hardened RBAC are not included.
- SlurmDBD/accounting, fair-share/QoS/account hierarchies, HTCondor, and Ray are
  not included.
- GPU/GRES and RDMA/InfiniBand are render-tested extension hooks only; this
  acceptance run is CPU-only and does not validate hardware drivers or devices.
- Cloud image publishing and an on-prem-to-cloud migration workflow are not
  included in this six-VM libvirt lab.
- Runtime egress is required for pinned OCI images; chart archives are baked
  and checksum-verified at image build time.

The raw prompt is available as plain text for diffing and code review.

How to run it

Start from the Deploy button above, inspect the generated six-node plan and its lab warnings, then build the role images as one group. OpenFactory provisions the private libvirt topology and runs the enabled acceptance scenario. Treat any missing node, skipped assertion, or softened warning as a failed build.

1. Check that the planner preserved all six node IDs, static addresses, role-specific features, and the secret-free plan.kubernetes object.
2. Confirm the resource warning before allocating at least 24 GB of guest RAM, seven guest vCPUs, six system disks, and the 20 GB NFS data disk.
3. Build the group. Each image receives only its role's feature generators; the backend supplies the random kubeadm token outside the prompt and plan.
4. Boot all six VMs on the private network. Do not expose the bootstrap subnet or reuse these images as a production bundle.
5. Require every assertion in real-hpc-acceptance to pass. Use the disabled measurements only as observations for this host, not as performance promises.

What this proves against the role

The lab turns several resume nouns into inspectable evidence: role-specific Linux images, kubeadm orchestration, operator-managed Slurm, distributed jobs, persistent controller state, shared storage, placement-aware monitoring, and CI assertions. It also marks cloud migration as unimplemented, which is more useful than implying that an installed Terraform binary constitutes a migration.

For an interview or design review, the interesting discussion is in the failure choices: why storage stays outside Kubernetes, why a scheduler node count is exact, why controller persistence is tested by deletion, why a bootstrap token needs a negative assertion, and why empty shell output is never a pass.

What the lab deliberately leaves out

The recipe stops before the controls that define a production HPC service. There is no SlurmDBD or accounting, fair-share policy, QoS, account hierarchy, production identity, TLS lifecycle, secrets rotation, hardened RBAC, backup, network policy, cloud publication, HTCondor, Ray, validated GPU stack, or validated RDMA fabric.

• NFS is appropriate for this state and shared-home test, but it is not a benchmarked parallel filesystem for research data.
• The control plane and Slurm controller are single instances. Their state survives a pod restart, not a controller VM or storage failure.
• GPU/GRES and RDMA/InfiniBand values can be rendered while disabled. No device plugin, kernel driver, firmware, or hardware path is accepted by this CPU run.
• Helm and Terraform prove pinned operator tooling. They do not prove a cloud account, provider module, remote state design, or migrated workload.
• The lab has one ordinary operations user. It does not solve multi-user UID/GID consistency, OIDC, LDAP, MFA, tenant isolation, or job-data authorization.

From lab contract to production design

A production design should keep the acceptance style and replace the shortcuts: use authenticated CA-pinned node enrollment, real identity and account policy, highly available control services, backed-up accounting, fit-for-purpose data storage, segmented networks, signed image promotion, hardware qualification, and rehearsed cloud or recovery moves.

1. Replace unsafe discovery with CA-pinned enrollment and deliver join material through a protected, audited channel.
2. Add OIDC or directory identity, consistent UID/GID mapping, Slurm accounts, SlurmDBD, fair-share, QoS, reservation policy, and audit retention.
3. Design controller, database, DNS, time, and storage failure domains. Test node loss and storage recovery instead of only pod replacement.
4. Select scratch, project, archive, and object tiers from measured workload behavior. Add backup, restore, integrity, quota, and lifecycle tests.
5. Qualify the actual CPU, GPU, firmware, NUMA, RDMA, and fabric stack with the workload images and scheduler configuration that will ship.
6. Build cloud-specific Terraform modules, immutable image promotion, data-transfer controls, cost limits, observability, and a migration rehearsal with rollback.

That is the useful division of labor. The lab makes the base mechanisms repeatable and gives every green claim a command behind it. Production engineering then adds the controls, capacity data, and failure evidence that a real research or trading workload demands.