Day 2. From Pipeline to Production

Author
Affiliation

Mery Touceda-Suárez

Bioinformatics Research Center

Published

October 5, 2026

1. Setting up

2. Making your nextflow workflow configurable

Yesterday you built a working 3-step pipeline — but every path is hardcoded: data/sample_*_*.fastq is baked into the script, and if a labmate wanted to run this on their data, they’d need to open the file and edit it by hand. That’s fine for a workshop. It’s not fine for a pipeline you intend to reuse, share, or hand off. This block is about closing that gap: turning “a script that works for me” into “a pipeline other people can run.”

1. params: naming your pipeline’s dials

Nextflow lets you declare parameters — named variables the workflow reads from, instead of literal values baked into the code. Any variable prefixed with params. is a pipeline parameter:

At the top of your final script from yesterday (fastqc_example_3.nf) you can add the following:

params.input      = '/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_*_*.fastq.gz'
params.outdir     = 'results'
Note

Notice the default value itself also changes here, from yesterday’s local toy files to the shared training dataset on the cluster (/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course) — this is the real dataset you’ll run the rest of today’s pipeline against, now that it’s becoming something you’d actually reuse rather than a one-off script.

Inside the workflow block, you swap the hardcoded string for the parameter (see scripts/fastc_trimmomatic_params.nf) .

#!/usr/bin/env nextflow

params.input      = '/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_*_*.fastq.gz'
params.outdir     = 'results'

process fastqc_raw {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'params.outdir/qc_raw', mode: 'copy'

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

process fastqc_trimmed {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'params.outdir/qc_trimmed', mode: 'copy'

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

process trim {

    module 'trimmomatic/0.40'
    stageInMode 'copy'
    publishDir 'params.outdir/trimmed', mode: 'copy'

    input:
    path reads

    output:
    path "trimmed_${reads}"

    script:
    """
    trimmomatic SE -threads 4 ${reads} trimmed_${reads} \\
        SLIDINGWINDOW:4:20 MINLEN:36
    """
}

workflow {

    main:
    reads_ch = Channel.fromPath(params.input)

    fastqc_raw(reads_ch)
    trim(reads_ch)
    fastqc_trimmed(trim.out)
}

If you try running fastqc_trimmomatic_params.nf you will observe that the output directory is not called results/, but rather it is called params.outdir/. But we created the parameter and we gave it the value results, so why is that? > Here publishDir 'params.outdir/qc_raw' created a folder with that literal name. In Groovy (the language underlying Nextflow’s grammar), single-quoted strings are never interpolated. > The fix is a double-quoted string: publishDir "${params.outdir}/qc_raw".

Nothing about the pipeline’s logic changed — you’re just naming the thing that used to be hardcoded, so it can be set from outside the script.

Note

Defaults still matter. Declaring params.input = ‘…’ gives it a default value. If nobody overrides it, the pipeline still runs — this is what lets you keep a working example script instead of one that errors out until every flag is supplied.

2. nextflow.config: defaults live outside the script

Right now those params.* lines live at the top of your .nf file. Nextflow also recognizes a special file, nextflow.config, sitting next to your script, and automatically loads it every time you run nextflow run. Anything declared there is available as params.* without you writing a single line in the workflow script itself:

See scripts/fastqc_trimmomatic_params/main.nf and scripts/fastqc_trimmomatic_params/nextflow.config.

// nextflow.config
params {
    input  = '/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_*_*.fastq.gz'
    outdir = 'results'
}

This matters for a simple reason: your pipeline logic (the .nf file) and your pipeline’s settings (the .config file) are now separate. You can hand someone the .nf script and let them bring their own nextflow.config — or none at all, and get your defaults.

Important

In fact, the consensus is that pipelines should be built so that any user only has to modify the settings in the nextflow.config or config files, and no modifications of the processes or workflows are necessary.

Tip

This is also where later today’s container and SLURM settings will live — nextflow.config is going to grow into the single place that controls how and where a pipeline runs, while the .nf file stays focused on what it does.

3.getManifest(): naming every output once

Right now, every publishDir you write picks its own path — results/fastqc, results/trimmed, whatever seemed reasonable at the time. That’s fine for one pipeline run by one person. It breaks down the moment you have several pipelines, run by several people, whose outputs need to sit next to each other in a shared location — say, on /rs1, where everyone’s assemblies, QC reports, and annotations are meant to accumulate into one browsable, queryable collection instead of a pile of one-off folder names.

The fix is to stop deciding paths process-by-process and declare them all once, in a single Groovy map:

def getManifest() {
    [
        trimmed_r1  : "fastq/trimmed_r1.fq.gz",
        trimmed_r2  : "fastq/trimmed_r2.fq.gz",
        contigs     : "assembly/contigs.fa",
        qc_html     : "log/fastp_log.html",
        qc_json     : "log/fastp_log.json",
    ]
}

You can see more examples of getManifest() in the manifests/ folder under day2_material

Every process’s publishDir then references a key from this map instead of a hand-typed string. Nothing here is special Nextflow syntax — getManifest() is a plain Groovy function you write, returning a plain map. The value is entirely in the convention: one place decides where every output goes, for every pipeline that adopts it.

Important

This is not Nextflow’s built-in manifest {} block in nextflow.config (that one holds pipeline metadata — name, author, version — read by nextflow pull/nextflow run <repo>). This is a project convention that happens to reuse the word “manifest.” Different mechanism, different purpose — don’t confuse the two if you go looking at Nextflow’s docs.

Why bother? Two payoffs, both downstream of the same fact — the path for any given output is now known in advance, not something you have to run the pipeline and go look for:

  • A shared data lake. If every pipeline that touches /rs1 writes contigs.fa to the same relative spot, outputs from different projects, different people, and different runs land in a predictable structure — the collection becomes something you can query across, instead of something you have to relearn per project.
  • Machine-readable by default. A script — or an AI agent — that needs to find “the assembly output” doesn’t have to parse your Groovy code to figure out where it went. It reads the manifest. This is also the foundation for emitting a manifest.json alongside your results at the end of a run: a small file that says exactly what was produced and where, so any downstream tool can consume your pipeline’s outputs without knowing anything about how the pipeline itself works.
Tip

We won’t wire getManifest() into publishDir directives in this workshop — that’s a natural next step once you’re writing pipelines of your own, past the point where 5–6 processes fit comfortably in your head. The pattern is worth knowing now so you recognize it when your outputs start needing to live somewhere other than your own laptop.

4. Overriding from the command line

Any params.x — whether it’s default-set in the script or in nextflow.config — can be overridden at runtime with a double-dash flag of the same name:

nextflow run pipeline.nf --input '/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_2_*.fastq.gz' --outdir results_sample2
Warning

One dash vs. two. This trips almost everyone up at least once: your parameters use two dashes (–input), because they’re user-defined. Nextflow’s own built-in options use a single dash (-resume, -profile, -work-dir). If a run mysteriously ignores a flag you passed, check dash count first.

5. -resume: don’t redo finished work

Remember from yesterday: every nextflow run creates a fresh, hash-named subdirectory in work/ for each task, and old ones are never overwritten. That record of “what already ran, with what inputs” is exactly what makes the next feature possible.

Add -resume to any run:

nextflow run pipeline.nf -resume

Nextflow checks each task against its cached record. If a task’s inputs and command are byte-for-byte identical to a previous successful run, it skips re-executing it and reuses the cached output straight from work/. Only tasks that are new, changed, or previously failed actually run.

Example of command output:

executor >  local (2)
[a1/2b3c4d] fastqc_raw (1)   | 5 of 5, cached: 5 ✔
[e5/6f7a8b] trim (1)         | 5 of 5, cached: 1 ✔

That cached: 3 is the payoff — three tasks didn’t run at all, they were served from work/ because nothing about them changed.

Why this is a big deal in practice: imagine a 6-step pipeline where step 5 crashes because of a typo in a parameter. Without -resume, fixing the typo means re-running steps 1–4 from scratch — potentially hours of compute you don’t need to redo. With -resume, you fix step 5 and rerun; Nextflow recognizes steps 1–4 are untouched and skips straight to the failure point.

Note

What actually triggers a re-run. Caching is based on a hash of the task’s inputs and its exact command (including interpolated variables). Change the input file, change a parameter that feeds into the script block, or edit the process itself, and that task’s hash changes — Nextflow correctly reruns it instead of quietly serving stale output.

6. Profiles: one pipeline, swappable environments

So far, everything you’ve configured is a value (a path, a filename). Profiles let you configure behavior — bundling up a set of config settings under a name, so you can flip between entire configurations with one flag.

// nextflow.config
profiles {
      hazel {
           process.executor = 'slurm'
           process.cpus     = 2
       }
       test {
           // placeholder
       }
}

Select one at runtime with -profile:

nextflow run pipeline.nf -profile hazel

7. Exercise: parameterize the pipeline

Using your Day 1 3-step pipeline (fastqc_raw → trim → fastqc_trimmed) as a starting point:

  1. Replace the hardcoded Channel.fromPath('data/sample_*_*.fastq') with Channel.fromPath(params.input), and point its default at the shared training dataset (/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_*_*.fastq.gz) instead of yesterday’s local toy files.
  2. Add params.input and params.outdir to a nextflow.config file.
  3. Update each process’s publishDir to build off params.outdir instead of a hardcoded folder name (e.g. publishDir "${params.outdir}/trimmed", mode: 'copy').
  4. Run the pipeline once normally, then run it again with -resume and confirm you see cached: in the output.
  5. Add a second, empty profile (test {} alongside hazel {}) to nextflow.config, and run once with -profile test just to confirm the flag is accepted — nothing will look different yet, and that’s the point.

Take a few minutes before checking the solution.

// nextflow.config
params {
    input  = '/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_*_*.fastq.gz'
    outdir = 'results'
}

profiles {
      hazel {
           process.executor = 'slurm'
           process.cpus     = 2
       }
       test {
           // placeholder
       }
}
// main.nf
#!/usr/bin/env nextflow
process fastqc_raw {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir "${params.outdir}/qc_raw", mode: 'copy' //

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t ${task.cpus} ${reads}
    """
}

process fastqc_trimmed {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir "${params.outdir}/qc_raw", mode: 'copy'

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t ${task.cpus} ${reads}
    """
}

process trim {

    module 'trimmomatic/0.40'
    stageInMode 'copy'
    publishDir "${params.outdir}/qc_raw", mode: 'copy'

    input:
    path reads

    output:
    path "trimmed_${reads}"

    script:
    """
    trimmomatic SE -threads ${task.cpus} ${reads} trimmed_${reads} \\
        SLIDINGWINDOW:4:20 MINLEN:36
    """
}

workflow {
    main:
    reads_ch = Channel.fromPath(params.input)

    fastqc_raw(reads_ch)
    trim(reads_ch)
    fastqc_trimmed(trim.out)
}

(each process’s publishDir becomes, e.g., publishDir "${params.outdir}/qc_raw", mode: 'copy')

Run:

nextflow run main.nf
nextflow run main.nf -resume
nextflow run main.nf -profile test

The pipeline’s logic hasn’t changed at all — same three processes, same channel wiring. What changed is that paths, output locations, and (soon) execution environment are now things you set from outside the script, not things you edit inside it. That’s the difference between a script you wrote for yourself and a pipeline someone else can pick up and run.

ImportantNote on ${task.cpus}

We’ve gone a bit further in our parameterization: the scripts now use ${task.cpus} instead of a hardcoded thread count. task.cpus is the value of the cpus directive for that task, so it comes from your config, not from the script.

With -profile hazel, the single line process.cpus = 2 does two jobs at once. It asks SLURM for 2 cores, and it passes -t 2 to FastQC and -threads 2 to Trimmomatic. What you request and what the tool uses can’t drift apart.

Without a profile that sets cpus (a plain nextflow run main.nf, or -profile test), task.cpus falls back to 1, and the same script runs single-threaded. The code is the same and only the resources change.

To give one process more cores, change the config, not the script:

process {
    withName: 'trim' { cpus = 4 }
}

withName settings take priority over the generic process.cpus. resourceLimits caps whatever you request, so asking for more than 24 CPUs gets trimmed to 24 instead of leaving a job that never schedules.

3. Hello Modules: Organizing for Reuse

Right now, your entire pipeline — every process block and the workflow block — lives in one .nf file. That’s been fine for three processes. It stops being fine once a pipeline grows to ten, twenty, or fifty steps: the file becomes hard to navigate, hard to review in a pull request, and hard for a collaborator to find “the one process that does trimming” without scrolling past everything else.

Modules are Nextflow’s answer: each process (or a logically related group of processes) lives in its own .nf file, and your main workflow script includes them instead of defining them inline.

1. Splitting a process into its own file

Take your trim process out of the main script entirely and put it in its own

// modules/trim.nf

process trim {

    module 'trimmomatic/0.40'
    stageInMode 'copy'
    publishDir "${params.outdir}/trimmed", mode: 'copy'

    input:
    path reads

    output:
    path "trimmed_${reads}"

    script:
    """
    trimmomatic SE -threads ${task.cpus} ${reads} trimmed_${reads} \\
        SLIDINGWINDOW:4:20 MINLEN:36
    """
}

Nothing about the process itself changed — same directives, same input/output, same script. It just moved.

Back in your main script, pull it in with an include statement: file:

// pipeline.nf
include { trim } from './modules/trim.nf'

workflow {
    main:
    reads_ch = Channel.fromPath(params.input)

    fastqc_raw(reads_ch)
    trim(reads_ch)
    fastqc_trimmed(trim.out)
}

trim is now callable in the workflow exactly as before — Nextflow resolves it from modules/trim.nf at runtime. The main script’s job shrinks down to: import the pieces, wire them together.

Note

The path in include is relative to the importing script, not your current working directory. ./modules/trim.nf means “a modules/ folder next to this .nf file” — this trips people up if they run nextflow run from a different directory than expected.

2. A directory layout for a modular pipeline

Once you do this for every process, a typical layout looks like:

my_pipeline/
├── main.nf          ← just the workflow block + includes
├── nextflow.config
└── modules/
    ├── fastqc.nf
    ├── trim.nf
    └── assembly.nf          ← added this afternoon

pipeline.nf becomes short and readable almost as documentation: it tells you what steps exist and in what order, without burying that logic under every process’s implementation detail.

Tip

This is also what makes a process reusable across pipelines. A modules/fastqc.nf file doesn’t care which pipeline includes it — you can drop the same module into a completely different project and include it there too, instead of copy-pasting the process definition by hand. This is exactly how nf-core (which you’ll hear about at the end of today) organizes hundreds of shared, community-maintained modules.

3. Reusing one module under two names: include ... as

Recall yesterday’s stretch task: to get fastqc_raw and fastqc_trimmed as two separately-named steps with two different publishDirs, you duplicated the entire process body into two near-identical process definitions — flagged at the time as “not ideal, but a reasonable trade-off for a 1.5-day workshop.”

Modules give you a cleaner way to do this, using include ... as to import the same process definition under two different local names:

include { fastqc as fastqc_raw     } from './modules/fastqc.nf'
include { fastqc as fastqc_trimmed } from './modules/fastqc.nf'

Both fastqc_raw and fastqc_trimmed now refer to the exact same process — one module file, no duplicated code — but each call is tracked and logged separately in the run output, since each include ... as creates its own named entry point into that process.

Warning

This solves code duplication, not configuration duplication — both aliases still share the same hardcoded publishDir inside modules/fastqc.nf, so their outputs would collide unless you generalize the publishDir (e.g. driving the subfolder name from an input value). We won’t do that generalization together, but you can try the Stretch task at the end of this section if you want to work on the solution. Here we are just flagging it so the trade-off is explicit rather than something you discover by accident.

Why this matters as a pipeline grows

Three processes in one file is a style choice. Fifteen processes in one file is a maintenance problem — nobody wants to Ctrl-F through five hundred lines to find one process, and code review on a single massive file is painful compared to reviewing a focused, one-process diff. Modules are how real, production Nextflow pipelines (including everything under nf-core) stay navigable as they grow. Additionally, separating your processes into modules allows you to reuse them in different workflows or pipelines without having to replicate them constantly.

4. utils/ and bin/: helpers that aren’t processes

Everything in modules/ so far has been a process — a unit of work with its own input:, output:, and script: block. Not everything you want to reuse fits that shape. Two common cases, two different homes:

utils/ — Groovy helper functions. Plain logic, included the same way as a process:

// utils/archive.nf
def untarCmd(String archivePath) {
    "tar -xzf ${archivePath}"
}
// main.nf
include { untarCmd } from './utils/archive.nf'

process extract {
    input:
    path archive

    script:
    "${untarCmd(archive.name)}"
}

bin/ — standalone scripts, any language. If a pipeline author has written a Python script — say, retrieve_fasta.py, for pulling a reference sequence from a database — it doesn’t go in utils/. It goes in a bin/ directory at the pipeline root. Nextflow automatically adds bin/ to every process’s PATH, so the script is callable by name, with no include and no full path:

my_pipeline/
├── main.nf
├── nextflow.config
├── modules/
├── utils/
└── bin/
    └── retrieve_fasta.py
process fetch_reference {
    output:
    path "reference.fa"

    script:
    """
    retrieve_fasta.py --accession GCF_000001405.40 -o reference.fa
    """
}

(The script needs its executable bit set — chmod +x bin/retrieve_fasta.py — and a shebang line, same as running it standalone.)

Tip

Same scoping rule as modules/: helpers used by one pipeline stay nested inside it (utils/, bin/); helpers useful across several pipelines move up into a shared location each pipeline’s include or PATH can reach.

TipA note on pipeline layout

As your pipeline grows past one .nf file, where things live starts to matter. A common convention:

my_pipeline/
├── main.nf
├── nextflow.config
├── modules/
│ └── fastqc.nf
└── utils/
└── helpers.nf

Everything belonging to this pipeline — its entry point, its config, and any process or helper only this pipeline uses — lives under one named directory. That’s the unit you’d hand to a collaborator, or point nextflow run at.

If a module or helper function is useful across several pipelines — say, a fastqc.nf module every pipeline in your group reuses — it moves up a level, out of any one pipeline’s directory, into a shared location that each pipeline’s include statement points to:

// inside my_pipeline/main.nf
include { fastqc } from '../shared_modules/fastqc.nf'

The rule of thumb: scope decides location. Pipeline-specific stays nested inside the pipeline; anything meant to be reused moves up and out. This is the same instinct behind the manifest convention from earlier — predictable structure is what makes a pipeline (and its outputs) something other people, or other pipelines, can actually build on.

5. Exercise: refactor into modules

Using your parameterized 3-step pipeline from this morning:

  • Create a modules/ directory next to your pipeline script.
  • Move fastqc_raw, trim, and fastqc_trimmed each into their own file inside modules/ (fastqc.nf, trim.nf — your choice whether fastqc_raw/fastqc_trimmed share one file or get two).
  • Add the corresponding include { ... } from './modules/...' statements at the top of your main pipeline script.
  • Confirm your main script’s workflow block is now the only thing left in that file besides the include lines.
  • Run the pipeline again — output should be identical to before refactoring.
// main.nf
include { fastqc as fastqc_raw     } from './modules/fastqc.nf'
include { fastqc as fastqc_trimmed } from './modules/fastqc.nf'
include { trim                     } from './modules/trim.nf'

workflow {
    main:
    reads_ch = Channel.fromPath(params.input)

    fastqc_raw(reads_ch)
    trim(reads_ch)
    fastqc_trimmed(trim.out)
}

(modules/fastqc.nf contains a single process fastqc { ... } definition; modules/trim.nf is unchanged from section 1 above, just moved)

Stretch task: Parameterize publishDiracross aliased calls

If you tried the include {...} as approach in section 3, you may have noticed a wrinkle: fastqc_raw and fastqc_trimmed both point to the same modules/fastqc.nf, so they share the same hardcoded publishDir — you lose the clean qc_raw/ vs qc_trimmed/ separation from Day 1’s stretch task.

Try resolving this by passing the output subfolder in as a process input, rather than hardcoding it inside the module:

// modules/fastqc.nf
process fastqc {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir { "${params.outdir}/${qc_stage}" }, mode: 'copy'

    input:
    val qc_stage
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t ${task.cpus} ${reads}
    """
}
// pipeline.nf
include { fastqc as fastqc_raw     } from './modules/fastqc.nf'
include { fastqc as fastqc_trimmed } from './modules/fastqc.nf'
include { trim                     } from './modules/trim.nf'

workflow {
    main:
    reads_ch = Channel.fromPath(params.input)

    fastqc_raw(Channel.value('qc_raw'), reads_ch)
    trim(reads_ch)
    fastqc_trimmed(Channel.value('qc_trimmed'), trim.out)
}

Notice that qc_stage is declared with the val qualifier (val qc_stage) rather than path — this tells Nextflow it’s a plain value (a string), not a file to be staged into the task’s working directory. Since fastqc now expects two inputs positionally (label, then reads), each call supplies both: Channel.value('qc_raw') wraps a single string so it lines up as a channel item, just like reads_ch does for the file input.

ImportantpublishDir has to be a closure here

publishDir "${params.outdir}/${qc_stage}" looks like the same double-quoted interpolation from earlier (§2.1.1) — but this time it fails at runtime with No such variable: qc_stage. Directives are evaluated once, when Nextflow parses the process definition, before any input value exists. A plain interpolated string can only see things that already exist at that point (like params.outdir), not a per-task input like qc_stage.

The fix is to wrap the value in a closure — publishDir { "${params.outdir}/${qc_stage}" } — so Nextflow defers evaluating it until the task actually runs, once qc_stage has a real value. Any directive that depends on an input needs this same closure treatment, not just publishDir.

Note. This is genuinely one module now doing the job that took two separate process definitions on Day 1 — no duplicated process body, and each call still gets its own labeled output folder. The trade-off: the process signature is slightly less obvious at a glance (two positional inputs instead of one), which is exactly why this was left as a stretch task rather than the default path through this section.

4. Hello Containers: Portability

Every process you’ve written so far has assumed the tool it needs (fastqc, trimmomatic, …) is already installed and on the PATH (in our case because we were using HPC modules). That works fine as long as everyone runs this on our cluster with our exact module set — but the moment someone runs it on a different HPC, their own laptop, or even just a different module version six months from now, that assumption is gone. In that case, the pipeline either fails outright or worse, runs quietly with the wrong tool version and gives them different results than yours. Containers fix this by packaging the tool and everything it depends on into one portable unit, so “works on our cluster” becomes “works anywhere.”

4.1 Why containers

Two tools rarely agree on which version of a third tool they need. salmon 1.4 and salmon 1.10 can give you meaningfully different quantifications — and “works on my machine” isn’t a reproducible method. This is dependency hell: manageable for one tool on one machine, unmanageable for a pipeline with a dozen tools running on a cluster, a laptop, and a collaborator’s machine six months from now.

A container packages a tool and its exact runtime environment — OS libraries, dependencies, version — into a single portable image. Point a process at a container instead of a module, and “works on my machine” becomes “works on any machine with a container runtime.” That’s the whole pitch: reproducibility across machines, across time, and across collaborators who don’t share your module system.

Note

This is also why publishDir’s brittleness (Day 1, §2.1.1) matters more once you’re using containers — the container guarantees the tool is reproducible, but you still own getting the outputs somewhere sane. Keep that in mind for the stretch task later today.

4.2 Docker vs. Singularity/Apptainer

Docker is the tool most people mean when they say “containers” — and it’s what you’ll see in almost every published pipeline’s documentation. But Docker requires a root-level daemon running on the host, which HPC administrators will not grant you on a shared, multi-tenant cluster. That’s not a policy quirk — it’s a real privilege-escalation risk when hundreds of users share a machine.

Apptainer (the actively maintained fork of Singularity — you’ll see both names, they’re the same lineage) was built specifically to solve this: it runs containers as your own user, no daemon, no elevated privileges. It’s also built to read Docker images directly, so you don’t lose access to the huge existing library of Docker-based bioinformatics containers (e.g. everything on BioContainers) — Apptainer just converts them on the fly.

This is why HPC environments standardize on Apptainer/Singularity and not Docker: same portability guarantee, none of the daemon/root problem.

4.2.1 Running a container by hand

Before Nextflow handles this for you, it’s worth seeing what actually happens under the hood. Apptainer runs a container with exec:

module load apptainer
apptainer exec /usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_fastqc:0.12.1--hdfd78af_0.sif fastqc --help

You should see FastQC’s help output print out, run from inside the container, without FastQC being installed anywhere on the host. We keep a library of pre-pulled .sif images on /usr/local/usrapps/brc/brc_modules/images/ for exactly this reason — you don’t need to pull from Docker Hub or Quay live on the cluster.

This is the same idea as module load fastqc — get a tool onto your PATH — but instead of relying on our cluster’s module system, you’re pointing directly at a self-contained image that would run identically on any machine with Apptainer or Docker installed.

The catch: doing this for every process, by hand, for every sample, doesn’t scale — that’s exactly the kind of bookkeeping Nextflow exists to take off your hands. Which brings us back to profiles.

4.3 Turning container use on: it’s just another profile

You already met profiles in §2 as a way to swap settings without touching your pipeline code — a test profile with small inputs, a hazel profile with executor settings. Container engines slot into the exact same mechanism. docker and singularity (or apptainer) are just profile names that flip a container engine on:

// nextflow.config
apptainer {
    enabled    = true
    autoMounts = true
}

autoMounts handles the common case — your work directory and inputs get bound into the container automatically. Run with -profile apptainer (or roll this into whichever profile you’re already using for the cluster) and every process that declares a container directive will run inside it.

Adding a container to a process looks like this:

process fastqc {
    container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_fastqc:0.12.1--hdfd78af_0.sif'

    input:
    path reads

    output:
    path "*_fastqc.html"

    script:
    """
    fastqc -t ${task.cpus} ${reads}
    """
}

Notice this is the exact image you just ran by hand in §4.2.1 — Nextflow isn’t doing anything magical here, it’s running the same apptainer exec call for you, once per item in the channel, instead of you typing it out per sample.

4.4 [Our HPC] Reference the container path directly, not the module

ImportantThis is specific to NCSU cluster — read this before the exercise

Some of what you’d expect to be a normal HPC “module” here is actually a wrapped Singularity/Apptainer image — module load <tool> on those doesn’t give you a native binary, it gives you a shim around a container. When Nextflow also tries to manage containers for that same process, the two layers of staging conflict, and jobs can fail or silently stage the wrong files.

The fix: skip the module entirely and point the container directive at the image path directly.

process fastqc {
    container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_fastqc:0.12.1--hdfd78af_0.sif'

    input:
    tuple val(sample_id), path(reads)

    output:
    tuple val(sample_id), path("*_fastqc.{html,zip}")

    script:
    """
    fastqc -t ${task.cpus} ${reads}
    """
}

Requirements for this to work:

  1. Enable Apptainer in nextflow.config (§4.3, above) — apptainer.enabled = true.
  2. The apptainer module still needs to be loaded on the host before you launch Nextflow — Nextflow calls out to the apptainer binary to run the image, so the binary itself has to be on PATH even though the process isn’t using module load anymore.
   module load apptainer
   nextflow run main.nf -profile apptainer
  1. Bind paths that aren’t auto-mounted. With autoMounts = true, Nextflow binds the directories of each task’s declared inputs, and Apptainer binds your home directory and the current directory by default. That’s not everything. If a tool reaches /rs1 (our shared storage) through a path Nextflow doesn’t know about, like a database path passed as a string or a symlink pointing into /rs1, the container can’t see it. If a file “isn’t found” inside the container even though it exists on disk, that’s almost always a missing bind, not a missing file.

Because /rs1 is a Hazel detail, not a FastQC detail, the bind goes in the profile, not the process:

profiles {
    apptainer {
        enabled    = true
        autoMounts = true
        runOptions = '--bind /rs1:/rs1'
    }
}

The process itself only names its image:

process fastqc {
    container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_fastqc:0.12.1--hdfd78af_0.sif'
    ...
}

The quote style decides who expands ${...}, and when.

Single quotes are literal text. Groovy doesn’t interpret anything inside them, so '--bind ${TMPDIR:-/tmp}' is kept exactly as typed. Nextflow copies that text into the job script (.command.run). When the job starts on the compute node, bash expands ${TMPDIR:-/tmp}: “use $TMPDIR, or /tmp if it’s unset.”

Double quotes are Groovy interpolation. In "...", Groovy treats every ${...} as Groovy code and fills it in when the config is read. That happens once, on the node where you ran nextflow run, before any job exists. "--bind ${TMPDIR:-/tmp}" breaks in two ways:

  1. TMPDIR:-/tmp is bash syntax, not Groovy, so the config fails to parse.
  2. Even the Groovy version, "--bind ${System.getenv('TMPDIR')}", binds the wrong directory. It captures the launch node’s $TMPDIR, but SLURM gives each job its own $TMPDIR on the compute node.

Compare cacheDir, a few lines up, which uses double quotes on purpose. Nextflow uses that directory itself on the launch node, so reading the environment once at launch is exactly right.

  1. Symlinked inputs can fail silently inside the container. (We did this on Day 1 and it works on our modules too). Nextflow normally stages inputs into work/ as symlinks pointing back to the original files — fine for a native process, but a container can’t follow a symlink that points outside its mounted paths, even with the right binds in place. The fix is stageInMode, which forces Nextflow to copy the file into work/ instead of symlinking it:
   process fastqc {
       container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_fastqc:0.12.1--hdfd78af_0.sif'
       stageInMode 'copy'                   // this line
       ...
   }

Costs a little extra disk and I/O per task, but it’s the reliable fix when a process reports a missing input file that you can plainly see sitting in the directory it’s staged from.

Quick diagnostic: container job failing on a file that clearly exists? - No such file or directory for something outside your working directory → missing --bind (point 3). - Input file “not found” even though it’s sitting right there in the process’s own input directory → symlink staging issue, add stageInMode 'copy' (point 4).

5. Running on SLURM

Everything so far has run one task at a time on whatever node you happened to be on. That’s fine for FastQC on one sample — not fine for a more demanding task, such as an assembly step across 40 samples. This section connects your pipeline to the actual scheduler the cluster runs on.

5.1 The executor directive

Nextflow doesn’t submit jobs to SLURM by default — it runs tasks locally, one process execution at a time, same as everything you’ve done today. The executor setting tells Nextflow to hand each task off to SLURM instead:

process.executor = 'slurm'

Once this is set, every task Nextflow runs — every sample through every process — becomes its own sbatch submission behind the scenes. You don’t write sbatch scripts by hand for each task; Nextflow generates and submits them for you, one per process execution, using the resource directives you give it.

5.2 Writing a SLURM profile

You already used profiles {} in §4.3 to turn a container engine on without touching pipeline code. Same mechanism, different job: a SLURM profile bundles executor + resource defaults so you can switch between “run locally” and “run on the cluster” with one flag.

// nextflow.config
profiles {
    hazel {
        process.executor = 'slurm'
        process.cpus      = 2
        process.memory    = 4.GB
        process.time      = 1.h
    }
}

You could have created two profiles, one with the resource information and one with the apptainer information and run nextflow with nextflow run main.nf -profile hazel,apptainer... to combine both — profiles stack, so your container settings from §4 and your scheduler settings here apply together without redefining anything.

5.3 Submitting: interactive session vs. sbatch-wrapped

There are two ways to launch nextflow run itself, and they’re not the same thing as the SLURM jobs Nextflow submits for your tasks (§5.1) — this trips people up, so it’s worth being explicit:

  • From an interactive session (the same srun session from Day 1, §2 — same flags, no need to redo that here): you type nextflow run main.nf -profile hazel directly. Nextflow’s own head process runs live in your terminal, submitting each task as a separate SLURM job as the pipeline progresses. Good for development and debugging — you see output immediately, -resume is easy to reach for.
  • Wrapped in an sbatch script: you submit the head process itself as a SLURM job, and it runs unattended:
  #!/bin/bash
  #SBATCH --job-name=nf-head
  #SBATCH --time=04:00:00
  #SBATCH --cpus-per-task=1
  #SBATCH --mem=2G

  module load apptainer nextflow
  nextflow run main.nf -profile hazel -resume

Good for anything long enough to outlast your terminal connection — the head process survives even if you log off, and each task it launches is still its own independent SLURM job.

Important

The head process itself only needs to stay alive and track state — it’s not doing the heavy computational work. Don’t request assembly-sized resources (§5.5) for the wrapper job; a nf-head job requesting 8 CPUs and 32 GB is wasting an allocation while just watching other jobs run.

5.5 Exercise: give assembly more resources than FastQC

FastQC is light. Assembly is not. Requesting the same CPUs/memory for both wastes allocation on the light step and starves the heavy one. withName (or withLabel, if you’ve tagged processes with labels) lets you set resources per process instead of one blanket default.

Your 1st task: add a withName block for your assembly process that requests more CPUs and memory than FastQC’s defaults, then re-run with -profile hazel and confirm in the HAZEL job log (sacct -j <jobid>) that the assembly task actually requested the resources you gave it.

// nextflow.config
process {
    withName: 'fastqc' {
        cpus   = 1
        memory = 2.GB
    }
    withName: 'assembly' {
        cpus   = 8
        memory = 32.GB
        time   = 2.h
    }
}
NoteDefaults vs. overrides: how process.cpus and withName interact

The process.cpus = 2 / process.memory = 4.GB lines you wrote in §5.2 aren’t a separate setting from the withName blocks — they’re the default every process gets unless something more specific overrides it. Nextflow always prefers the most specific match: a withName block for a given process wins over the profile-wide default.

So in the config above: fastqc explicitly gets cpus = 1, memory = 2.GB from its own withName block, assembly explicitly gets cpus = 8, memory = 32.GB, and any other process in your pipeline without a withName entry falls back to the process.cpus = 2, process.memory = 4.GB default. Nothing is added together or averaged — the more specific rule simply replaces the general one for that process.

Your 2nd task: inside the hazel profile you wrote in §5.2, add a withName block for your assembly process that requests more CPUs and memory than FastQC’s defaults, then re-run with -profile hazel and confirm in the SLURM job log (sacct -j <jobid>) that the assembly task actually requested the resources you gave it.

// nextflow.config
profiles {
    hazel {
        process.executor = 'slurm'
        process.queue     = 'normal'
        process.cpus      = 2
        process.memory    = 4.GB
        process.time      = 1.h

        process {
            withName: 'fastqc' {
                cpus   = 1
                memory = 2.GB
            }
            withName: 'assembly' {
                cpus   = 8
                memory = 32.GB
                time   = 2.h
            }
        }
    }
}
NoteWhy withName lives inside the hazel profile, not outside it

You could technically put a withName block at the top level of nextflow.config, outside profiles {} entirely — but then it applies to every run, regardless of -profile. That means a local, non-SLURM run would still try to hand assembly 8 CPUs and 32 GB, even though nothing about queue or executor would apply outside the profile. You’d end up with SLURM-shaped resource requests running on your laptop or login node.

Nesting withName inside profiles { hazel { ... } } scopes it correctly: these per-process resource rules only take effect when you actually run with -profile hazel. Anything SLURM-specific — resource sizing included — belongs inside the profile that turns SLURM on.

Putting it together: one profile for Hazel

You now have two profiles that only make sense together on Hazel: apptainer (from §4) tells Nextflow how to run each tool, and hazel (from this section) tells it where to run each task. On Hazel you always want both, so combine them into one profile:

profiles {

    local {
        process.executor = 'local'
    }

    hazel {
        // --- where tasks run (§5) ---
        process.executor = 'slurm'
        // keep the rest of your §5 SLURM settings here

        // --- how tools run (§4) ---
        apptainer.enabled    = true
        apptainer.autoMounts = true
        apptainer.runOptions = '--bind /rs1:/rs1'
    }
}

Delete the standalone apptainer profile. From here on, every Hazel run is:

module load apptainer
nextflow run main.nf -profile hazel

The rule of thumb: one profile per place you run, not one profile per feature. Your command stays short, and you can’t forget half the setup.

Nextflow also lets you combine profiles at runtime with a comma:

nextflow run main.nf -profile slurm,apptainer

Profiles are applied left to right. If two profiles set the same option, the one listed later wins. Stacking is useful when features vary independently, for example the same cluster with or without containers. For this workshop, one hazel profile is simpler.

6. Capstone: Assemble Your Reads

This is where everything today comes together. You’re extending the pipeline you already have — fastqc → trim → fastqc(trimmed) — with a 4th step: de novo assembly with MEGAHIT. When you’re done, you’ll have a real, if small, pipeline: fastqc → trim → fastqc → assemble, containerized, running on SLURM.

6.1 Guided steps

  1. Write the megahit process, taking trim.out as input:
   process megahit {
       container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_megahit:1.2.9--h8b12597_0.sif
       publishDir "${params.outdir}/assembly", mode: 'copy'

       input:
       path reads

       output:
       path "megahit_out/final.contigs.fa"

       script:
       """
       megahit -r ${reads} -o megahit_out
       """
   }
  1. Add it to your workflow block, same chaining pattern from Day 1 §1 (.out from one process feeds the next):
    // pipeline.nf
    include { trim } from './modules/trim.nf'
    include { fastqc as fastqc_raw     } from './modules/fastqc.nf'
    include { fastqc as fastqc_trimmed } from './modules/fastqc.nf'
    include { megahit } from './modules/megahit.nf'

   workflow {
       main:
       reads_ch = Channel.fromPath('/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course/sample_*_*.fastq.gz')

       fastqc(reads_ch)
       trim(reads_ch)
       fastqc(trim.out)
       megahit(trim.out)
   }
  1. Confirm the container path is correct for your account (§4.2.1–4.4) — this is the most common place to get stuck.
  2. Run on the hazel profile:
   nextflow run main.nf -profile hazel
  1. Give assembly its own resource block (you should already have done this in 5.5) in the hazel profile (§5.5) — assembly is memory-hungry, more so than fastqc or trim:
   withName: 'assembly' {
       cpus   = 8
       memory = 32.GB
       time   = 2.h
   }
Tip

Work at your own pace from here. If you get a container or staging error, check §4.4’s diagnostic table before asking — most issues at this stage are a missing bind path or a missing stageInMode 'copy', not a new problem.

6.2 Stretch task: aggregate QC with MultiQC

If you finish early: add a process, multiqc, that gathers every FastQC and Trimmomatic report into one summary — the same .collect() pattern from Day 1 §3.3 (wait for every upstream item, then emit one combined item).

process multiqc {
    container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_multiqc:1.21--pyhdfd78af_0.sif'
    publishDir "${params.outdir}/multiqc", mode: 'copy'

    input:
    path reports

    output:
    path "multiqc_report.html"

    script:
    """
    multiqc .
    """
}

Your task: figure out how to combine the outputs of both fastqc calls and trim into a single channel multiqc can consume — you’ll need .collect() and possibly .mix() (Day 1, §2’s branching-vs-chaining callout mentions .mix() in passing; this is where you actually need it).

6.3 Another (ADVANCED) stretch task: align reads with BWA to host genome and extract unaligned reads with SAMtools

This task introduces a few things we haven’t used yet today: a process with more than one input (reads and a reference genome), a channel for a reference file rather than sample data, and a cluster-specific constraint around downloading things ahead of time since compute nodes have no internet access. Each is explained inline as it comes up.

A common real-world step before assembly: align your trimmed reads against a host or contaminant genome (e.g. corn, human) you want to remove, then keep only the reads that didn’t align — those are the ones you actually assemble. This is a two-tool step (BWA, then SAMtools), and it changes what feeds assembly — instead of trim.out going straight in, it’ll go through this filtering step first.

ImportantWhen compute nodes have no internet access (NCSU HPC)

Every reference we’ve used so far has already been on disk at a known shared path (/gpfs_backup/bioinfo_data/training_data/nextflow_crash_course) — but a host genome isn’t something we’ve provided, and compute nodes on our cluster can’t reach the internet to download one mid-job. If a task tries to fetch a reference from NCBI/Ensembl/etc. while running on a compute node, it will just hang or fail with a connection error that has nothing to do with your Nextflow code.

Anything your pipeline needs from outside the cluster — reference genomes, databases, container images not already in our shared library — has to be downloaded ahead of time, from the login node (which does have internet access), and referenced by its path on disk. This is why our FastQC/Trimmomatic/MEGAHIT containers all point at pre-pulled .sif files in /usr/local/usrapps/brc/brc_modules/images/ rather than pulling docker://... live (§4.2.1) — same reasoning applies to reference genomes.

For this task, download your host genome from the login node first:

# from the login node, NOT a compute node
wget -O host_genome.fa.gz https://example.com/path/to/host_genome.fa.gz
gunzip host_genome.fa.gz

Then reference host_genome.fa by its path in the config below — don’t try to download it inside a process.

A process with two inputs.

Every process you’ve written today has taken a single input — one file, one channel. align_host needs two: the reads and the reference genome to align against. Nextflow processes can take multiple inputs; just list them as separate lines in the input: block, and they get filled positionally by whatever you pass in when you call the process:

process align_host {
    container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_bwa:0.7.17--hed695b0_7.sif'
    stageInMode 'copy'

    input:
    path reads
    path reference

    output:
    path "aligned.sam"

    script:
    """
    bwa index ${reference}
    bwa mem ${reference} ${reads} > aligned.sam
    """
}

Then, extract only the reads that didn’t align. samtools view -f 4 filters for unmapped reads (the 4 flag in a SAM/BAM file marks “read did not align”), and samtools fastq converts that back into FASTQ format that assembly can consume:

process extract_unaligned {
    container '/usr/local/usrapps/brc/brc_modules/images/quay.io_biocontainers_samtools:1.19--h50ea8bc_0.sif'
    stageInMode 'copy'
    publishDir "${params.outdir}/host_removed", mode: 'copy'

    input:
    path sam

    output:
    path "unaligned.fastq"

    script:
    """
    samtools view -f 4 -b ${sam} | samtools fastq - > unaligned.fastq
    """
}

A channel for the reference genome.

align_host needs a second channel to supply reference — same Channel.fromPath factory you already know from Day 1 §2.1, just pointed at one file instead of a glob matching many:

host_reference_ch = Channel.fromPath('/path/to/host_genome.fa')

Since this channel only ever emits one item (the reference genome doesn’t change per-sample the way reads do), it gets reused across every task align_host runs — Nextflow pairs it with each incoming read file automatically.

Your task:

  1. Download a host genome from the login node (see the callout above) and build the host_reference_ch channel pointing at it.
  2. Add both processes to your pipeline, chained together — align_host’s output feeds extract_unaligned.
  3. Rewire assembly to take extract_unaligned.out instead of trim.out directly:
   trim(reads_ch)
   align_host(trim.out, host_reference_ch)
   extract_unaligned(align_host.out)
   assembly(extract_unaligned.out)
Warning

-f 4 is a SAM flag filter, not something Nextflow-specific — worth a quick samtools flags or a peek at the SAM spec if flag values are new to you. Flag 4 means “segment unmapped”; combining flags (e.g. filtering out both unmapped reads and their mapped mates) uses different flag arithmetic, out of scope for today but worth knowing exists.

Note: Think about if BWA + SAMtools should be a subworkflow since you will probably use them together a lot