Running Jobs on Hazel with SLURM

Author

Affiliation

Seth Weaver

Published

July 23, 2026

In this section, we will learn how to submit jobs to compute nodes on Hazel! This workshop picks up where the Tour of Hazel left off. By the end, you will be able to:

Understand why job submission is important — learn why jobs must be submitted to a scheduler instead of run directly.
Write and submit a job script — create a SLURM job script, submit it, and read its output.
Manage your jobs — monitor, cancel, and diagnose jobs using SLURM’s built-in tools.

Use this guided walkthrough. When you see a command block that starts with a $, that is a command you should type. Lines below that command (without a $) are the expected output.

1 The Big Picture: Why SLURM?

When you first log in to Hazel, you land on a login node — a shared gateway that all users connect to. The login node is meant for light tasks: editing files, moving data, writing scripts. It is not for running computationally intensive work.

Hazel’s real computing power lives in its compute nodes — servers with many CPUs, large amounts of memory, and (on some nodes) GPUs. You cannot log in to a compute node directly. Instead, you submit a job to SLURM, the software that manages all the work happening on Hazel.

Think of SLURM like a dispatcher at a shipping facility:

You describe what resources your job needs (CPUs, memory, time)
SLURM finds a compute node where those resources are available
SLURM runs your script on that node and saves the output back to your files

Do not run jobs on the login node

Running computationally intensive work on the login node slows it down for every other user. SLURM exists precisely so that heavy work happens on compute nodes. Administrators monitor for this and may kill processes that are misusing the login node.

2 Exercise 1: Using SBATCH directives

A SLURM job script is a regular bash script with a special block of comments at the very top. Lines beginning with #SBATCH are not ignored by the shell — SLURM reads them before your script runs and uses them as instructions for what resources to reserve.

2.1 Writing the test script

Navigate to your scratch directory and create a new script:

$ cd /shares/$GROUP/$USER
$ nano hello_slurm.sh

Paste the following inside the editor:

#!/bin/bash

echo "Job started on $(hostname) at $(date)"
echo "Hello from SLURM, $USER!"
echo "Job finished at $(date)"

2.2 Add SBATCH directives

Now add to the scripts some SBATCH directives that control how the job is run. Try controlling these parameters:

Name your job “slurm_test”
Give your job 2Gb of memory
Have your job run for a maximum of 5 minutes
Have the job output go to a file called “job_name_job_id.out”

Save and exit (Ctrl+O, Enter, Ctrl+X).

Now run your job with sbatch hello_slurm.sh. Do you see the proper output?

$ sbatch hello_slurm.sh
submitted batch job XXXXXXX

2.3 Filled-in job script

Your job script with the SBATCH directives added should have looked something like this:

#!/bin/bash

#SBATCH --job-name=slurm_test
#SBATCH --mem=2G
#SBATCH --time=00:05:00
#SBATCH --output=%x_%j.out

echo "Job started on $(hostname) at $(date)"
echo "Hello from SLURM, $USER!"
echo "Job finished at $(date)"

Here is what exactly each #SBATCH line does:

Directive	What it does
`--job-name`	A human-readable label for your job — shows up in `squeue`
`--mem`	Total memory to reserve (e.g. `2G` = 1 gigabyte)
`--time`	Maximum wall-clock time the job may run (`HH:MM:SS`)
`--output`	File where standard output (printed text) is saved. `%x` is replaced by the job name (above) and `%j` is replaced by the job ID.

%k and %j placeholders

In --output and --error, %j and %x automatically replaced by the job’s name and numeric ID when the job runs. This prevents output from different runs from overwriting each other, and makes it easy to match output files back to a specific submission.

Time limits matter

Adding a time parameter helps SLURM schedule your job efficiently. However, If your job is still running when its requested time expires, SLURM kills it — even mid-computation! Always estimate how long your job will take and add some buffer.

3 Exercise 2: Monitoring job progress

A useful aspect of using SLURM is that you can monitor job progress. This will tell you if your jobs are pending, currently running, or completed as well as if jobs completed successfully or failed.

To practice monitoring our submitted job, we will add an extra line of code to our job script so that it doesn’t complete so quickly. Add this line: sleep 1m. This will have the job script wait and do nothing for 1 minute. The job script should now look like this:

#!/bin/bash

#SBATCH --job-name=slurm_test
#SBATCH --mem=2G
#SBATCH --time=00:05:00
#SBATCH --output=%x_%j.out

echo "Job started on $(hostname) at $(date)"
sleep 1m  # added this line, tells the script to pause for 1 minute
echo "Hello from SLURM, $USER!"
echo "Job finished at $(date)"

3.1 Checking job status

Submit the script as a job with sbatch hello_slurm.sh. Once submitted, check on its progress with the squeue -u $USER command. You should see something similar to the following:

$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            423135   compute slurm_te sdweave2  R       0:07      1 c207n02

The key columns are:

Column	Meaning
`JOBID`	Your job’s unique ID
`PARTITION`	Which partition it is running on
`NAME`	The job name from `--job-name`
`ST`	Job state (see below)
`TIME`	How long the job has been running
`NODELIST(REASON)`	Which node it is running on, or why it is waiting

The ST (state) column uses short codes:

Code	State	Meaning
`PD`	Pending	Waiting for a node to become available
`R`	Running	Currently executing on a compute node
`CG`	Completing	Wrapping up, about to finish
`F`	Failed	The job exited with an error
`CA`	Cancelled	Cancelled by you or an administrator

Why is my job pending?

A job in PD state is waiting its turn. The REASON column tells you why. Common reasons: Resources (nodes are fully busy), Priority (other jobs are ahead of yours in the queue), or QOSMaxJobsPerUserLimit (you have hit a limit on how many jobs you can run simultaneously).

3.2 Checking job completion

Once the job finishes, it disappears from squeue. To check if the job completed successfully, or if it failed, you can use the sacct -u $USER command. This will list completed jobs as well as pending and running jobs. You should see something like this:

$ sacct -u $USER
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
423135       slurm_test    compute    brc_cpu          1  COMPLETED      0:0
423135.batch      batch               brc_cpu          1  COMPLETED      0:0
423135.exte+     extern               brc_cpu          1  COMPLETED      0:0

You’ll notice that there are three lines for just the single job ID. The .batch and .exte+ are subprocesses that slurm runs under the hood. The jobID without any extension is the one that is the most relevant. You can see the JobName and the State (Completed). An exit code of 0:0 indicates that everything was successful.

4 Exercise 3: Monitoring performance

This exercise builds directly onto Exercise 2. We want to monitor completed or failed jobs for how much resources they actually used, compared to how much we requested.

In your scratch directory, open up a new file called performance_test.sh.

$ cd /shares/$GROUP/$USER
$ nano performance_test.sh

Paste in the following script:

#!/bin/bash
#SBATCH --job-name=performance_test
#SBATCH --output=%x_%j.out
#SBATCH --time=00:02:00
#SBATCH --mem=100M

# --- Load required software ---
module load python

# --- Execute ---
echo "Job started on $(hostname)"
echo "Requested memory: $SLURM_MEM_PER_NODE MB"  # Note: This variable will always contain the requested amount of memory from your SBATCH Directive.

# Execute this tiny python script, which initializes a data array
python3 -c "
print('Allocating a large array to simulate processing a dataset...')
data = bytearray(500_000_000) 
print('Allocation succeeded. Array size:', len(data), 'bytes')
print('Processing complete.')
"

# --- Finish ---
echo "Job finished"

Submit the job with sbatch performance_test.sh.

Check the job completion status with sacct -u $USER. What do you see? Likely, you will see something like this:

$ sacct -u $USER
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
439884       performan+    compute    brc_cpu          1 OUT_OF_ME+    0:125
439884.batch      batch               brc_cpu          1 OUT_OF_ME+    0:125
439884.exte+     extern               brc_cpu          1  COMPLETED      0:0

You can see that the state of the job is OUT_OF_ME+, short for out OUT_OF_MEMORY. How would you find out more information about the job performance?

ANSWER: Look in the output file for the job with cat performance_test_439884.out. You should see indications a couple of places that the job ran out of memory. The 100Mb of memory that we gave the job wasn’t enough.

To fix this error, edit the job script to give the job 1Gb of memory.

The edit that you should have made:

#SBATCH --mem=1G

Now re-run the job with sbatch performance_test.sh. Once it completes, use the strategies that we just learned to check its successful completion and performance.

5 On your own time: Extra SLURM information

5.1 Cancelling a Job

If you submitted a job by mistake, or it is stuck and not making progress, cancel it with scancel followed by the job ID:

$ scancel 1234567

To cancel all of your jobs at once:

$ scancel -u $USER

Cancellation is immediate

scancel kills the job right away with no warning. Any output that was not yet written to a file is lost. There is no undo.

5.2 Wall-clock time formats

The standard time format is HH:MM:SS. For longer jobs, you can also use D-HH:MM:SS:

#SBATCH --time=00:30:00    # 30 minutes
#SBATCH --time=04:00:00    # 4 hours
#SBATCH --time=1-00:00:00  # 1 day

FLAG: Time limits per partition

What are the maximum wall-clock times for each partition on Hazel? Jobs that request more time than a partition allows will be rejected at submission time. For example: short = 24 hours, long = 7 days — but please confirm and fill in the real values.

5.3 Checking Job Efficiency with seff

After a job finishes, you can see how efficiently it used the resources it requested with seff:

$ seff 1234567
Job ID: 1234567
Cluster: hazel
User/Group: sdweave2/brc
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 1
CPU Utilized: 00:00:03
CPU Efficiency: 92.00% of 00:00:03 core-walltime
Job Wall-clock time: 00:00:03
Memory Utilized: 48.00 MB
Memory Efficiency: 4.69% of 1.00 GB

The two key numbers are CPU Efficiency and Memory Efficiency. In the example above, memory efficiency is only 4.69% — the job requested 1 GB but used only 48 MB. For future runs, requesting --mem=100M would be more appropriate and may help the job start sooner.

5.4 Interactive Jobs

So far, every job you have submitted is batch — you write the script, submit it, and check the output file later. Sometimes, especially while debugging or testing, you want to work interactively on a compute node: typing commands and seeing results immediately, just like on the login node.

Use srun with the --pty flag to open an interactive shell on a compute node. The --partition=compute_partners and --qos=short are required arguments to and put you on a standard CPU partition:

$ srun --pty --partition=compute_partners --qos=short bash

Important

Running the above command with no other arguments will give you default memory allocation. To request additional memory, cores, and time you can pass SBATCH directives to as flags. Note that bash has to be at the end of the command. Example: srun --pty --partition=compute_partners --qos=short --time=00:30:00 --mem=10G --cpus_per_task=4 bash

After a moment (once SLURM allocates a node for you), your prompt changes:

[sdweave2@c207n08 ~]$

Notice the hostname changed from the login node to a compute node (c207n08). You can now run commands directly on the compute node — load modules, run tools, explore data. When you are done, type exit to release the node and return to the login node.

Interactive jobs hold resources the entire time you are connected

A running interactive session keeps a compute node reserved for you even if you are not actively using it. Always exit when you are finished so those resources become available for other users’ jobs.

5.5 SLURM environment variables

When your job script runs, SLURM sets several variables inside it that describe the job — its ID, the node(s) it landed on, and the resources you requested. These are handy for logging, naming output files, or writing scripts that adapt to whatever resources were allocated. An example:

#!/bin/bash
#SBATCH --job-name=env_demo
#SBATCH --time=00:05:00

echo "Job ID: $SLURM_JOB_ID"
echo "Job name: $SLURM_JOB_NAME"
echo "Running on node: $SLURMD_NODENAME"
echo "CPUs allocated: $SLURM_CPUS_ON_NODE"

Variable	What it holds
`SLURM_JOB_ID`	The numeric ID of the running job
`SLURM_JOB_NAME`	The job’s name (set with `--job-name`, or the script filename)
`SLURMD_NODENAME`	The hostname of the node the job is running on
`SLURM_CPUS_ON_NODE`	Number of CPUs allocated on the current node
`SLURM_MEM_PER_NODE`	Requested memory per node, in MB
`SLURM_SUBMIT_DIR`	The directory you ran `sbatch` from

Try it

Add the echo lines above to a job script, submit it with sbatch, and check the output file to see the values SLURM filled in.

5.6 Requesting GPU nodes

GPU nodes are in separte partitions than CPU nodes. GPU nodes are used for highly parallelizable tasks, like machine learning applications. Add the following SBATCH directives to your job script to request GPU nodes:

#SBATCH --partition=gpu_partners
#SBATCH --qos=short_gpu
#SBATCH --gres=gpu:a10:1

Unlike CPU nodes, you have to request specific types of GPUs with the --gres directive. In the above example we are requesting 1 a10 node. See which other GPUs you can request on hazel.

6 Command Reference

Command	What it does	Example
`sbatch`	Submit a batch job script	`sbatch hello_slurm.sh`
`sinfo`	Show available partitions and node states	`sinfo`
`squeue`	Show the job queue	`squeue -u $USER`
`sacct`	Show all jobs, including completed	`sacct -u $USER`
`scancel`	Cancel a job	`scancel 1234567`
`srun`	Run interactively on a compute node	`srun --pty bash`
`seff`	Show efficiency report for a finished job	`seff 1234567`