Container
Containers in general
Cgroup
check supported cgroup subsystem:
cat /proc/cgroupscheck process in which cgroup:
cat /proc/777/cgroupcgroup on ubuntu 18.04, seems not so built in on ubuntu, need to add systemd service by hands if you wanna auto start with boot….
cpu set on numa architecture. When using cpuset.cpus, cpuset.mems must also be specified to make it work. The value for mems, is the node number of the cpu binding mems.
my experience: say if you want to apply cgroup policy on existing proc, especially these service, then you need restart these service to make the policy work
syntax of cgrules,
@group.cpu subsystem parameters and default value, by some test, the default value is around 1000 for cpu.share.
Possible issue: confilict between external cgroup on users and slurm cgroup. Syndrome: task submit failure on cgroup enabled node.
slurmstepd-master: error: Failed to invoke task plugins: task_p_pre_launch error. More specifically, the error is happen due to external cpuset subsystem from cgroup. My guess is that the cpu core binding external to slurm is unknown to slurmctld, and the assigned cpu cores would be failed by sbatch. Possible solution: 1 easy way (which I took), use cpu instead of cpuset subsystem to limit user cpu usage in cgroup. 2 hard way (which I guess itt should work), in principle, we can assign cpu binding in slurm.conf to avoid assigenment failure. Note this is not a big issue in general, since it is rare when slurmd (which usually only on compute nodes) and cgroup (which usually only configured on login nodes) are both exist in the same machine.
Singularity
a runtime container suitable for HPC
Docker vs sigularity vs shifter in HPC: 1, 2, 3. No sudo dameon, less namespace separation, automount of home, share the same network by default, support MPI, env vars pass to container, user stay the same, more like a local process instead of a VM. And it is much easier than docker in general.
Process - Singularity - Docker -VM: the order of resource separation. Actually singularity is more like AppImage.
Singularity cached images layer are in ~/.singularity. Namely, they cannot be shared across different user by default. But it seems to be tuned by en vars $SINGULARITY_CACHEDIR.
basic commands
singularity exec container.img cat /etc/os-release.––containflag for a better separation, no pass of certain env vars, say local python packages.––cleanenv.––containallfor more strict namespace separation.--bind /host:/consingularity inspect -l --json ubuntu.imgsingularity shell centos7.img, ifwritableis add, then img may be changed.sigularity pull docker://, not only pull but also transform it to sif format img.running as background service using
sigularity instancecommands, see doc. (In this workflow, singularity is more like docker.)
build image
.sif is the standard format for singularity 3+. For image building, you must have root access, which cannot be done on HPC. But instead one can choose remote builder system here.
direct way by operating in the container
sudo singularity build --sandbox ubuntu/ library://ubuntu
sudo singularity exec --writable ubuntu touch /foo
singularity build new-sif sandbox
definition files (recommended)
Below is a demo. Detailed syntax on def files here.
sudo singularity build lolcow.sif lolcow.def
singularity hub
common images
clearlinux family?
Integrated with Kubernetes
k8s
For current status, seems no open source solution that can integerate k8s and slurm well in HPC setup. You can have slurm over container or k8s over mpi, but you have to choose one work load manager.
Detailed list and benchmark comparison on resource managers: arXiv
Possible routes:
Helm
Last updated
Was this helpful?