Container
Last updated
Was this helpful?
Last updated
Was this helpful?
check supported cgroup subsystem: cat /proc/cgroups
check process in which cgroup: cat /proc/777/cgroup
, seems not so built in on ubuntu, need to add systemd service by hands if you wanna auto start with boot….
. When using cpuset.cpus, cpuset.mems must also be specified to make it work. The value for mems, is the node number of the cpu binding mems.
my experience: say if you want to apply cgroup policy on existing proc, especially these service, then you need restart these service to make the policy work
, @group.
, by some test, the default value is around 1000 for cpu.share.
Possible issue: confilict between external cgroup on users and slurm cgroup. Syndrome: task submit failure on cgroup enabled node. slurmstepd-master: error: Failed to invoke task plugins: task_p_pre_launch error. More specifically, the error is happen due to external cpuset subsystem from cgroup. My guess is that the cpu core binding external to slurm is unknown to slurmctld, and the assigned cpu cores would be failed by sbatch. Possible solution: 1 easy way (which I took), use cpu instead of cpuset subsystem to limit user cpu usage in cgroup. 2 hard way (which I guess itt should work), in principle, we can assign cpu binding in slurm.conf to avoid assigenment failure. Note this is not a big issue in general, since it is rare when slurmd (which usually only on compute nodes) and cgroup (which usually only configured on login nodes) are both exist in the same machine.
a runtime container suitable for HPC
Process - Singularity - Docker -VM: the order of resource separation. Actually singularity is more like AppImage.
Singularity cached images layer are in ~/.singularity. Namely, they cannot be shared across different user by default. But it seems to be tuned by en vars $SINGULARITY_CACHEDIR.
singularity exec container.img cat /etc/os-release. ––contain flag for a better separation, no pass of certain env vars, say local python packages. ––cleanenv. ––containall for more strict namespace separation. --bind /host:/con
singularity inspect -l --json ubuntu.img
singularity shell centos7.img, if writable is add, then img may be changed.
sigularity pull docker://, not only pull but also transform it to sif format img.
sudo singularity build --sandbox ubuntu/ library://ubuntu
sudo singularity exec --writable ubuntu touch /foo
singularity build new-sif sandbox
sudo singularity build lolcow.sif lolcow.def
clearlinux family?
For current status, seems no open source solution that can integerate k8s and slurm well in HPC setup. You can have slurm over container or k8s over mpi, but you have to choose one work load manager.
Possible routes:
Docker vs sigularity vs shifter in HPC: , , . No sudo dameon, less namespace separation, automount of home, share the same network by default, support MPI, env vars pass to container, user stay the same, more like a local process instead of a VM. And it is much easier than docker in general.
running as background service using sigularity instance commands, see . (In this workflow, singularity is more like docker.)
.sif is the standard format for singularity 3+. For image building, you must have root access, which cannot be done on HPC. But instead one can choose remote builder system .
Below is a demo. Detailed syntax on def files .
reference:
Detailed list and benchmark comparison on resource managers: