Commit edfba338 authored by Jakub Yaghob's avatar Jakub Yaghob
Browse files

working on initial README

parent 97c9ddb4
...@@ -8,3 +8,112 @@ Additionally, the Charliecloud is available on all worker nodes. ...@@ -8,3 +8,112 @@ Additionally, the Charliecloud is available on all worker nodes.
> Do not execute any of your code on the front-end nodes directly, use worker nodes instead! > Do not execute any of your code on the front-end nodes directly, use worker nodes instead!
All unknown terms (front-end node, worker node) will be explained later. Front-end nodes do not have installed any development software. All unknown terms (front-end node, worker node) will be explained later. Front-end nodes do not have installed any development software.
## SLURM Crash Course
All informations about SLURM can be found on its
[SLURM documentation](https://slurm.schedmd.com/documentation.html)
page or on [SLURM tutorials](https://slurm.schedmd.com/tutorials.html) page.
Anyway, we have provided a short description of SLURM and of our clusters.
### Terminology
A **cluster** is a bunch of **nodes**.
Nodes are grouped together to partitions.
Partitions may overlap, ie. one node can be in more **partitions**.
**Feature** is a string describing a feature of a node, e.g. avx2 for a node capable of executing AVX2 instructions.
Each node has two sets of features: current features and available features.
Usually, they are same.
But in some cases, the node is capable of changing current features set on demand.
SLURM manages **resources** and schedules jobs according to available resources.
The most important resources are CPUs and memory, but it is able to manage other generic resources (GRES), like GPUs, etc.
Moreover, the time is resource as well.
A **user** is identified by his/her login.
**Account** is a billing entity (well, we won't charge you for using our clusters).
Each user must have assigned an account. Moreover, user can be assigned to more accounts and use them depending on what he/she is doing.
Accounts can only be allowed access to some partitions.
A user can launch a **job**.
Job has a state, has some reserved and assigned resources, and returns an error code after completition.
Job consistes from **steps** (usually 1 step).
Each step is executed by a bunch of **tasks** (usually 1 task), where resources are equally assigned to the tasks of a step.
Jobs are inserted to a **scheduling queue**, where you can find them.
Partition has a priority.
A job submitted to a partition with higher **priority** can suspend an another job submitted to a partition with lower priority.
### Important commands
There are many commands (see
[SLURM man pages](https://slurm.schedmd.com/man_index.html) or [SLURM command summary](https://slurm.schedmd.com/pdfs/summary.pdf)
). The most important commands are:
- **srun** - reserves and assigns resources and runs an interactive job
- **sbatch** - reserves and assigns resources and submit a batch as a job
- **salloc** - only reserves and does not assign resources, use srun or sbatch for assigning resources subsequently
- **sinfo** - get info about cluster state
- **scancel** - cancels a job
- **squeue** - view info about scheduling queue
- **sbcast** - distributes a file to the nodes allocated to a job
- **sstat** - display info about a job
Job submission commands (srun, sbatch, salloc) have a common set of important options:
| Shrt | Long | Description |
| ---- | -------------------- |------------ |
| `-A` | `--acount=` | Charge resources used by the job to the specified account. If not sepcified, user's default account is charged |
| `-B` | `--extra-node-info=` | Select only nodes with at least specified number of sockets, cores per socket, and threads per core. This option does not specify the resource allocation, its just constraint. |
| `-C` | `--constraint=` | Select only nodes with matching features |
| `-c` | `--cpus-per-task=` | Number of CPUs per 1 task |
| `-e` | `--error=` | Standard error stream is redirected to the specified file |
| | `--gpus=` | Specifies the number of GPUs required for the job in the form `[type:]count`. It is a shortcut for --gres=gpu:type:count. |
| | `--gres=` | Specifies comma delimited list of GRES. Each entry on the list is in form `name[[:type]:count]` |
| `-i` | `--input=` | Standard input stream is redirected from the specified file |
| `-J` | `--job-name=` | Job name |
| `-L` | `--licenses=` | Specifies comma delimited list of licenses allocated to the job. Each entry on the list is in form `name[:count]` |
| `-m` | `--distribution=` | Select distribution method for tasks and resources. For more info see documentation |
| | `--mem=` | Specify the real memory required per node |
| | `--mem-per-cpu=` | Specify the memory required per allocated CPU |
| | `--mem-bind=` | Specify the memory binding. For more info see documentation |
| `-N` | `--nodes=` | A minimum of allocated nodes. Default is 1 |
| `-n` | `--ntasks=` | Number of tasks. Default is 1 |
| `-o` | `--output=` | Standard output stream is redirected to the specified file |
| `-p` | `--partition=` | Request a specific partition for the resource allocation. If not specified, default partition os chosen |
| `-t` | `--time=` | Set a limit on the total run time of a job. If not specified, default time for a selected partition is used. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours". |
### Description of available clusters
As mentioned above, we have two clusters **parlab** and **gpulab**.
Access to the cluster is always through the front-end server using SSH on port 42222.
Front-end servers have the same name as the cluster, i.e. **parlab.ms.mff.cuni.cz** and **gpulab.ms.mff.cuni.cz**.
#### Course users
Student logins will be created before the first assignment.
The login is in the form *s_sislogin*, where *sislogin* is a student's login to our student infomation system (SIS).
The generated password will be sent to student's official e-mail.
#### Other users
The login will be created on demand by the cluster administrator.
#### Others
Both clusters use one common LDAP, so you will need to change the password only once.
Each course tought on our clusters has its account.
Any research group or project have their account as well.
Logins are assigned to the corresponding account depending on visiting relevant courses or working in a research group.
Both clusters have access to the same disk array using NFS.
You may find your home mounted on `/mnt/home`.
Moreover, research projects can have an additional space mounted on `/mnt/research`.
You may use an environment variable **TMPDIR** set to a private local temporary directory for a job.
It is created on every node allocated to the job before the job starts and it is completely removed after the job finishes.
The temporary directory can by used as a scratchpad.
Moreover, it resides on a local SSD RAID, therefore it is faster access data here then accessing data on a remote NFS disk.
On the other hand, **the space is limited, be careful!**
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment