Build a real, disposable six-node HPC lab named `slurm-hpc`. This fixture must exercise OpenFactory's actual multi-machine build and libvirt test path. Do not create Python compatibility listeners, fake `/sbatch` HTTP endpoints, mock metrics, or placeholder Slurm configuration. The running system must be Kubernetes v1.36.3 bootstrapped with kubeadm, Flannel v0.28.8, SchedMD Slinky Slurm Operator chart v1.2.0 / Slurm 26.05, and `kube-prometheus-stack` 86.0.0. This is intentionally a zero-touch lab image set. It uses one backend-generated kubeadm token with a 15-minute maximum lifetime and unsafe CA discovery only on the private RFC1918 lab network. The controller must revoke the token after all five Kubernetes nodes are Ready. Make the warnings say plainly that these are lab images, not secure production runtime bundles. ## Six-node topology Use one network: ```json { "name": "hpc-lab", "cidr": "10.103.0.0/24", "gateway": "10.103.0.1", "dns_servers": ["1.1.1.1", "8.8.8.8"], "is_default_route": true } ``` Every node is a `linux-build` on `debian-trixie`, has one static interface on `hpc-lab`, and has 4 GB minimum RAM. Use this exact inventory: | node_id | IP | role | min CPU | Kubernetes/Slinky placement | |---|---:|---|---:|---| | `controller` | `10.103.0.10` | `controller` | 2 | control plane, Slinky operator/controller/restapi | | `compute-1` | `10.103.0.21` | `compute` | 1 | worker, CPU DaemonSet NodeSet | | `compute-2` | `10.103.0.22` | `compute` | 1 | worker, CPU DaemonSet NodeSet | | `login-ops` | `10.103.0.30` | `login` | 1 | worker, LoginSet, kubectl/Helm/Terraform | | `storage` | `10.103.0.40` | `storage` | 1 | outside Kubernetes, NFS server | | `observability` | `10.103.0.50` | `observability` | 1 | worker, Prometheus/Grafana placement | Use canvas-pixel coordinates with no overlap: controller `(320, 100)`, compute-1 `(120, 340)`, compute-2 `(520, 340)`, login-ops `(120, 580)`, storage `(520, 580)`, observability `(320, 820)`. Connect the controller to all four Kubernetes workers, connect storage to controller, both computes, and login-ops, and connect observability to controller. The `storage` topology node must declare exactly this generated data disk: ```json { "id": "hpc-data", "kind": "disk", "size_gb": 20, "bus": "virtio", "serial": "OPENFACTORY_HPC_DATA" } ``` The storage image may format only the runtime disk whose serial is exactly `OPENFACTORY_HPC_DATA`; it must refuse zero matches, multiple matches, the root disk, and an unexpected existing filesystem. ## Kubernetes intent Include this secret-free object at `plan.kubernetes`. Never include a token: ```json { "mode": "kubeadm-lab", "version": "v1.36.3", "controller": "controller", "workers": ["compute-1", "compute-2", "login-ops", "observability"], "pod_cidr": "10.244.0.0/16", "service_cidr": "10.112.0.0/16", "cni": "flannel", "bootstrap_token_ttl_seconds": 900 } ``` The VM, pod, and service CIDRs must remain non-overlapping. The storage node is not a Kubernetes member, so the live acceptance count is five Ready nodes. ## Recipe contract Every recipe must use `base_image: "debian-trixie"`, set `hardware.min_memory_gb: 4`, set the CPU minimum from the table, and keep only role-appropriate features/packages. Every node has a normal `ops` user with sudo and SSH for test access. Do not put secrets in the prompt beyond the fixture's ordinary lab login password. Features by role: - `controller`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm`, `helm` - `compute-1`, `compute-2`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm` - `login-ops`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm`, `helm`, `terraform` - `storage`: `headless`, `ssh`, `hpc-nfs` - `observability`: `headless`, `ssh`, `kubeadm-lab`, `slinky-slurm` Do not list `helm`, `terraform`, `kubeadm`, `kubectl`, or `kubelet` as additional apt packages and do not add ad hoc upstream repository hooks. The declared OpenFactory features own those pinned installations. The backend injects each Kubernetes member's `openfactory-kubeadm-lab` service from `plan.kubernetes`. Do not handcraft that service or a bootstrap token. Every Kubernetes member must declare an enabled service named `openfactory-slinky-slurm` with this shared config and its own correct `role` (`controller`, `compute`, `login`, or `observability`): ```json { "controller_node_id": "controller", "compute_node_ids": ["compute-1", "compute-2"], "login_node_id": "login-ops", "observability_node_id": "observability", "nfs_server": "10.103.0.40", "slinky_version": "1.2.0", "slurm_version": "26.05", "prometheus_stack_version": "86.0.0" } ``` The storage recipe must declare an enabled service named `openfactory-hpc-nfs`: ```json { "disk_serial": "OPENFACTORY_HPC_DATA", "export_network": "10.103.0.0/24" } ``` Do not add startup scripts that duplicate feature-generator behavior. ## Required running behavior Slinky accounting stays disabled. Use NFS-backed static PV/PVC storage for Slurm controller state and NFS-backed shared `/home` in LoginSet and NodeSet. The controller, operator, and restapi run on `controller`; the LoginSet runs on `login-ops`; a DaemonSet NodeSet runs only on `compute-1` and `compute-2`; and the default CPU partition contains exactly those two Slurm nodes. Enable controller metrics and its ServiceMonitor. Put the main Prometheus, Grafana, Alertmanager, operator, and kube-state-metrics pods on `observability`. The login-ops host commands `sinfo`, `squeue`, `srun`, `sbatch`, `scontrol`, and `scancel` must execute the real command inside the Slinky LoginSet. They must not emulate scheduler responses. Runtime egress is allowed for the pinned Slinky container images. The Slinky and monitoring charts, Kubernetes/Flannel configuration, and Helm/Terraform CLI installers must be baked into the images with fixed versions and checksums. ## Gating acceptance scenario Create one enabled group scenario named `real-hpc-acceptance`. Use `command_succeeds` or `command_output` assertions with `params.timeout_seconds` up to 1800. The scenario must verify: 1. On `controller`, exactly five Kubernetes nodes report Ready, kubeadm reports Kubernetes v1.36.3, Flannel pods are Ready, and `kubeadm token list` has no live bootstrap token. 2. On `login-ops`, `sinfo` reports exactly two Slurm nodes in the CPU partition. 3. On `login-ops`, `srun -N2 -n2 hostname` produces two distinct hostnames. 4. On `login-ops`, a real `sbatch --wait` job writes `/home/openfactory-hpc-acceptance.txt`, then `srun -N2 -n2 cat` reads the same content from both compute nodes. 5. From `controller`, submit a running job through the real LoginSet, delete the slurmctld pod, wait for its replacement, and verify the job remains known. This proves controller state survived a pod restart on the NFS PVC. 6. On `controller`, the Slinky controller ServiceMonitor exists; Prometheus and Grafana workloads are Ready in `monitoring`; their main pods are assigned to `observability`. 7. On `login-ops`, `helm version --short` contains `v4.2.3` and `terraform version` contains `v1.15.8`. 8. On `storage`, the mounted filesystem comes from the disk with serial `OPENFACTORY_HPC_DATA`, all three exports are active, and NFS is serving. Avoid shell checks that pass on empty output. Every command must use `set -euo pipefail`, explicit numeric/string assertions, and bounded waits. ## Non-gating measurements Add a second scenario named `hpc-lab-measurements` with `enabled: false`. It documents commands for NFS `fio`, VM-to-VM `iperf3`, kubeadm bootstrap duration, and Slurm queue/scheduler latency. These are reported measurements, not pass/fail SLOs for the first CPU lab. ## Warnings and scope Preserve all of these warnings: - Lab-only bootstrap uses unsafe CA discovery on a private network and is not a secure production runtime bundle. - Production identity, multi-user UID/GID management, TLS, secrets rotation, network policy, backup/restore, and hardened RBAC are not included. - SlurmDBD/accounting, fair-share/QoS/account hierarchies, HTCondor, and Ray are not included. - GPU/GRES and RDMA/InfiniBand are render-tested extension hooks only; this acceptance run is CPU-only and does not validate hardware drivers or devices. - Cloud image publishing and an on-prem-to-cloud migration workflow are not included in this six-VM libvirt lab. - Runtime egress is required for pinned OCI images; chart archives are baked and checksum-verified at image build time.