Click here to view the Singularity presentation - Juno user group meeting on March 21, 2019
Singularity (Version 2.2) is installed as a RPM on the lilac cluster and does not need any additional modules to be loaded. Singularity containers are each stored in a single file, unlike Docker containers, for mobility and reproducibility.
Websites :
http://singularity.lbl.gov/#home
https://github.com/singularityware/singularity
https://groups.google.com/a/lbl.gov/forum/#!forum/singularity
Singularity workflow
Getting started:
Please note: HPC group will give guidance to create and use singularity container but will not manage user containers. If you decide to use singularity it is your responsibility to build and manage your containers. That includes managing your linux environment for development.
Create new empty container:
(These steps are to be performed on system where you have root privileges)
[root@otto2 ~]# singularity create testnew.img Creating a new image with a maximum size of 768MiB... Executing image create helper Formatting image with ext3 file system Done.
To create a empty container of desired size use :
[root@otto2 ~]# singularity create --size 8192 newtest.img
Bootstrap/Install container
In order to bootstrap a container create a .def file
Bootstrap yum example : BootStrap: yum OSVersion: 7 MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/ Include: yum %setup echo "Looking in directory '$SINGULARITY_ROOTFS' for /bin/sh" if [ ! -x "$SINGULARITY_ROOTFS/bin/sh" ]; then echo "Hrmm, this container does not have /bin/sh installed..." exit 1 fi exit 0
Singularity and Docker :
Singularity can use the Docker remote API to import containers
[root@otto2 dev]# singularity create --size 8192 tensorflow.img Creating a new image with a maximum size of 8192MiB... Executing image create helper Formatting image with ext3 file system Done.
[root@otto2 dev]# singularity import tensorflow.img docker://tensorflow/tensorflow:latest tensorflow/tensorflow:latest Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ... Done.
You can also use a spec file to bootstrap from docker
Bootstrap docker example:
[root@otto2 neeraj]# cat ubuntu.def BootStrap: docker From: centos:latest IncludeCmd: yes CMD ["/bin/bash"]
Docker2singularity
This is the best solution if you are working on a Windows or Linux desktop with no access to a linux system.
Requirement : Docker (native or for mac or for windows)
Usage:
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /Users/neeraj/Desktop/singularity:/output --privileged -t --rm singularityware/docker2singularity ubuntu:14.04
Common Errors:
“client is newer than server” error
If you are getting the following error:
docker: Error response from daemon: client is newer than server
You need to use the docker info command to check your docker version and use it to grab the correct corresponding version of docker2singularity.
docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v D:\host\path\where\to\ouptut\singularity\image:/output \ --privileged -t --rm \ singularityware/docker2singularity:1.11 \ Ubuntu:14.04 Supported versions: 1.10, 1.11, 1.12, and 1.13
If you are using a older version of docker then you need to upgrade.
“WARNING: Non existant bind point (directory) in container: ‘/shared_fs’”
If you are receiving the above error that means your singularity image needs needs custom mount point inside the container which can be established as below
docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v D:\host\path\where\to\ouptut\singularity\image:/output \ --privileged -t --rm \ singularityware/docker2singularity \ -m "/shared_fs /custom_mountpoint2" \ ubuntu:14.04
Binding Directories :
Singularity allows you to bind external directories from within your container. Singularity by default will bind your current working directory and your home directory (/home/$user). You can bind additional directories
- Create the corresponding bind points from within your containers
You can use the runtime option of using –bind to bind directories
[neeraj@lilac ~]$ singularity shell -w --bind /usr/bin:/usr/bin --bind /usr/lib64/nvidia:/usr/lib64/nvidia test.img
Or you can use the env variable like below
[neeraj@lilac ~]$ export SINGULARITY_BINDPATH="/usr/bin:/usr/bin”
Running jobs on singularity
To shell into singularity container use singularity shell -w container.img
[neeraj@lilac singularity]$ singularity shell -w relion.img Singularity: Invoking an interactive shell within container...
Singularity exec
This is similar to the docker exec command. This just forwards your command to run from within the container.
[neeraj@lilac singularity]$ singularity exec test.img python --version Python 3.6.0 :: Continuum Analytics, Inc.
Singularity run
If you want to configure the container as a executable you can use the runscript like below in your .def file
Bootstrap: docker From: tensorflow/tensorflow:latest %runscript exec /usr/bin/python "$@" %post echo "Post install stuffs!" [neeraj@lilac singularity]$ singularity run test.img --version Python 3.6.0 :: Continuum Analytics, Inc.