Day 1. Foundations: Thinking in Channels and Processes

Inspired in Hello Nextflow by Seqera

Author

Affiliation

María (Mery) Touceda-Suárez

Published

August 28, 2026

In this first half-day, we’ll go from “what is a Nextflow process” to building a real, if small, 3-step linear pipeline — entirely by hand, before introducing any of the conveniences (config files, command-line parameters, resume/caching) that make Nextflow pleasant to use day-to-day. Those come tomorrow; today is about the core mental model everything else sits on top of.

By the end of this session, you will have:

run your first Nextflow process (a single fastqc call on one file)
learned how channels move data through a pipeline, and why a channel with more than one item gives you parallelism for free — no loops, no job arrays
learned how to keep paired-end files grouped together with Channel.fromFilePairs, and how to inspect and reshape channel contents with .view() and .map()
chained two processes together (Trimmomatic → FASTQC) so the output of one becomes the input of the next, and extended that into a 3-step pipeline (raw QC → trim → post-trim QC)

A note on pace

Today is deliberately hands-on and incremental — each new script is a small, working modification of the last one, not a leap to a “real” production pipeline. If a concept doesn’t fully click the first time you see it (this is very normal for channels), it will come back around at least once more before the day is over.

Before we get to any of that, though, we need to get you a working environment: cloning the workshop repository, starting an interactive session on the HPC, and taking a first look at the toy dataset we’ll be using throughout the day.

1. Setting up

If you haven’t done this during the Day 0 prep, you will need to clone the repository with the workshop materials (data and code) and to load certain modules. To do this, once you have logged in to your HPC you can run the setup script we have created.

Not working on NCSU HPC:

This script was created for the NCSU HPC environment, some things might need to be modified if you are working from a different system. Make sure you load the necessary modules (apptainer, nextflow) and that you clone the repository with the workshop’s materials:

git clone https://github.com/hurwitzlab/nextflow_crash_course.git "path/to/nextflow_crash_course"

NCSU HPC file system

If you are working in the NCSU HPC (Hazel), we recommend that you work in your scratch space (/share/$GROUP/$USER). Files in scratch will be automatically deleted after 30 days. If you would like to save any files for later use, you can move those to your Research Storage space (rs1/researchers/initial/PIname).

2. Starting an interactive session

We can’t do any computation in the login node of an HPC system, so we need to start an interactive session first. The exact command depends on your HPC system and the scheduler it uses (e.g., SLURM, PBS, LSF). For example, on a SLURM-based system, you might run:

srun --pty --time=4:00:00 --cpus-per-task=4 --mem=4G bash

This might take a second, especially if we all do it at the same time, but in a minute or two you should see your command line prompt change from something like:

(base) [mtouced@login02 ~]$

To something like:

[mtouced@c207n01 ~]$

This shows us that we have “moved” from the login02, a login node; to the compute node c207n01. Which means we are now interactively running a job on that compute node and we have free range to run more computational exhaustive processes.

3. Examining our data

Finally, let’s take a look at the data we will be working with in this workshop. If you do:

ls /path/to/cloned/repo/day1_material/data

You should see the following sequence data files:

sample_001_R1.fastq
sample_001_R2.fastq
sample_002_R1.fastq
sample_002_R2.fastq
sample_003_R1.fastq
sample_003_R2.fastq

We can move to the day 1 repository since we will be working there:

cd /path/to/cloned/repo/day1_material

Hello FASTQC: Your First Process

1. Warmup: run FASTQC on one sample directly

# if you are using the NCSU HPC and you haven't done this already, run: 
# this allows you to use the software provided by the BRC as modules
module use /usr/local/usrapps/brc/brc_modules/modules/

# load fastqc
module load fastqc/0.12.1 

# test it
fastqc -h

# run it on one data sample
fastqc data/sample_001_R1.fastq

Examine the results…

Let’s now run it with nextflow.

2. Examine the nextflow script and run it

You are provided a fully functional, minimalist workflow script named fastqc_example.nf that does the same thing as before (run FASTQC on one sample) but with Nextflow.

2.1. Examine the code

Open fastqc_example.nf in your editor of choice.

nano scripts/fastqc_example.nf

Full code file — fastqc_example.nf:

#!/usr/bin/env nextflow

/*
 * Run FASTQC
 */
process fastqc {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'fastqc', mode: 'copy'

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

workflow {

    main:
    // channel: grab one fastq file
    reads_ch = Channel.fromPath('data/sample_001_1.fastq')
    

    // run FASTQC
    fastqc(reads_ch)
}

A simple Nextflow workflow script typically includes one or more process definitions and the workflow itself, plus a few optional blocks we’ll introduce later.

Each process describes what operation(s) the corresponding pipeline step should accomplish, while the workflow describes the dataflow logic that connects the various steps.

2.1.1. The `process` definition

process fastqc {
    
    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'fastqc', mode: 'copy'

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

Here we have a process called fastqc that runs FastQC on a single input file and produces its standard HTML report and zip archive of results.

This process definition contains:

a module directive: tells Nextflow to run module load fastqc/0.12.1 before executing the script, so the fastqc command is available on the compute node without you having to load it yourself
a stageInMode directive: forces Nextflow to copy the input file into the work directory instead of symlinking it — necessary here because fastqc/0.12.1 runs inside a Singularity container, which can’t follow symlinks pointing outside its mounted paths.
a publishDir directive: tells Nextflow to copy this process’s declared outputs into the fastqc/ folder in your working directory, so you don’t have to go digging through work/ to find your results
an input block: uses the path qualifier, telling Nextflow that reads refers to a file that needs to be staged into the task’s working directory before the script runs
an output block: uses the path qualifier, telling Nextflow that reads refers to a file that needs to be staged into the task’s working directory before the script runs.
the script to execute

Important

The output definition does not determine what output will be created — it declares the expected output so Nextflow can look for it once execution completes. This verifies the task executed successfully and allows passing output to downstream processes. Output that doesn’t match what’s declared in the output block is not passed downstream.

Notice that the script block refers to ${reads} rather than a hardcoded filename. This variable is populated by whatever file Nextflow passes in through the input block, so the same process definition works regardless of which file it’s given.

The output pattern, "*_fastqc.{html,zip}", is a glob rather than an exact filename. This is necessary because FastQC generates its own output filenames based on the input file’s name — we don’t control them directly, so we tell Nextflow to look for anything matching that pattern once the task completes.

Warning

This process assumes exactly one file will be provided per task and does no validation of what that file actually is (a real fastq file? something else entirely?). Nextflow will attempt to run FastQC on whatever it’s handed.

Warning

Here module works because NCSU HPC (Hazel) has fastqc/0.12.1 available as an environment module — the exact name/version has to match what’s on your system (module avail fastqc to check). We’ll see an alternative to this in Day 2, using containers instead of HPC modules, which is more portable across systems.

Note

A note on publishDir: this is the traditional way to publish outputs in Nextflow, and you’ll see it everywhere in existing pipelines (including most nf-core modules) — but it’s technically the older pattern. A newer, recommended syntax (output {} blocks) exists, but since publishDir is still extremely common in the wild, we’re using it throughout this workshop.

2.2. Run the workflow

2.2.1. Launch the workflow and monitor execution

# just in case you haven't done the set up day load nextflow:
module load nextflow/26.04.3 

# run the nextflow pipeline
nextflow run scripts/fastqc_example_1.nf

You should get something like the following:

Command output:

Nextflow 26.04.6 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 26.04.3

Launching `scripts/fastqc_example.nf` [determined_wescoff] revision: bef29c91e6

executor >  local (1)
[1e/a42796] process > fastqc (1) [  0%] 0 of 1 ✘
ERROR ~ Error executing process > 'fastqc (1)'

Caused by:
  Process `fastqc (1)` terminated with an error exit status (127)


Command executed:

  fastqc -t 4 sample001_1.fastq

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 2: fastqc: command not found

Work dir:
  /gpfs_common/share03/ivirus/mtouced/nextflow_workshop/day1_material/work/1e/a42796584247e8b5e17c6c0d485967

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

Reading the output:

[determined_wescoff] — the run name (needed for -resume)
revision: bef29c91e6 — a hash of the script version run
executor > local (1) — where tasks run
[1e/a42796] — the work directory hash for this task (work/1e/a42796.../)
[ 0%] 0 of 1 ✘ — progress; ✔ = success, ✘ = failure

When a process fails, Nextflow adds an error block:

Caused by — one-line summary (here, exit status 127)
Command executed — the exact command that ran, variables already substituted
Command exit status — the raw exit code:

Code Meaning

127 Command not found (missing module/dependency)

137 Killed — often OOM

139 Segmentation fault
Command error — stderr from the failed command; usually the most useful line for debugging
Work dir — contains .command.sh (script run), .command.run (launch wrapper), .command.out/.command.err (full logs)

Code	Meaning
127	Command not found (missing module/dependency)
137	Killed — often OOM
139	Segmentation fault

Quick debugging steps: read Command error → check the exit code → cd into Work dir and inspect .command.sh → rerun with bash .command.run if still unclear.

As you can see, we got an error! This is because the fastqc process definition in fastqc_example_1.nf is missing the line:

module 'fastqc/0.12.1'

Which tells nextflow how to access/find the software for running FastQC.

Try running this instead:

nextflow run scripts/fastqc_example_2.nf

Command output:

N E X T F L O W ~ version 26.04.4

Launching fastqc_example.nf [goofy_torvalds] revision: c33d41f479

executor > local (1)
[65/7be2fa] fastqc | 1 of 1 ✔

If your console output looks something like that, congratulations — you just ran your first Nextflow workflow!

The most important line:

[65/7be2fa] fastqc | 1 of 1 ✔

This tells us the fastqc process was successfully executed once (1 of 1 ✔), and points to where the output was written.

2.2.2. Find the output and logs in the `work` directory

The first time Nextflow runs in a given directory, it creates a work/ directory where it writes all files (and symlinks) generated during execution. Within work/, Nextflow creates a nested, hash-named subdirectory per process call, where it stages inputs (via symlinks by default), writes helper files, and writes logs and outputs.

The truncated hash shown in square brackets (e.g. [65/7be2fa]) maps to the full directory path, e.g.: work/65/7be2fa7be2fad5e71e5f49998f795677fd68

tree -a work

Directory contents:

work
└── ff
    └── bb3b82315d28a45a7dff3bd8d7264d
        ├── .command.begin
        ├── .command.err
        ├── .command.log
        ├── .command.out
        ├── .command.run
        ├── .command.sh
        ├── .exitcode
        ├── sample_001_R1.fastq
        ├── sample_001_R1_fastqc.html
        └── sample_001_R1_fastqc.zip

Note

Don’t see the same thing? Exact subdirectory names will differ on your system. Log files are hidden in the terminal — use tree -a work or ls -a to see them.

Open sample_001_R1_fastqc.html and you should see the familiar FastQC report.

HEADS UP! That may seem like a lot of scaffolding for the same report you could get by running FastQC directly — its value becomes clear once we start chaining multiple steps together and running things across many files at once, which is exactly what the rest of today builds toward.

Of the other files in that directory, .command.sh is the one you’ll reach for most — it shows the actual command Nextflow executed, without any of the bookkeeping:

#!/bin/bash -ue
fastqc -t 4 sample_001_R1.fastq

This matches what we ran manually in the warmup. Here it’s simple because the filename was still hardcoded, but soon you’ll see commands built from variable interpolation, and .command.sh becomes very useful for troubleshooting exactly what Nextflow ran under the hood.

Because we also set publishDir 'fastqc', mode: 'copy' in the process, a copy of the report files is placed in a fastqc/ folder in your working directory — so you don’t need to go digging through work/ to find your results day to day.

Optional activity

Re-run the same command a few times and peek inside work/ again. Notice a new hash-named subdirectory appears each time, rather than the old one being overwritten. Nextflow always keeps a full record of every run. We’ll come back to why that matters — and how to make Nextflow skip work it’s already done — on Day 2.

2.3. Exercise: modify the process

Try the following on your own, using the fastqc_example.nf script as a starting point:

Change the process so it runs FASTQC on a different file in data/ instead of sample_001_R1.fastq.

Run your modified script with:

nextflow run path/to/fastqc_solution.nf

Take a few minutes to try this yourself before checking the solution below.

Solution

#!/usr/bin/env nextflow

/*
 * Run FASTQC
 */
process fastqc {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'fastqc', mode: 'copy'

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

workflow {

    main:
    // channel: grab a different fastq file
    reads_ch = Channel.fromPath('data/sample_002_R1.fastq')

    // run FASTQC
    fastqc(reads_ch)
}

Note that the process itself didn’t need to change at all — only the channel’s source line changed. This is a preview of a bigger idea: processes describe what to do with data, channels control which data flows in. You’ll see this separation again and again in Nextflow, and tomorrow you’ll learn how to make that channel source configurable from the command line itself (no more editing the script by hand).

3. Scaling up: from one sample to many

Notice we only processed one file. What if we wanted FastQC parallely on all the reads in data/? In bash, you’d write a loop. In Nextflow, you change one line:

reads_ch = Channel.fromPath('data/sample_*_*.fastq')

That’s it. Nextflow automatically fans this out into one parallel task per file — no loop, no manual job array, no extra code in the process itself.

Hello Channels: How Data Flows

1. Channels as conveyor belts

So far, we’ve only fed our fastqc process a single hardcoded file. But real pipelines need to move many files — possibly many different kinds of files — between processes, often many at once and in parallel.

This is exactly what channels are for. A channel is both the data itself (files, values, or tuples) and the flow logic — because in Nextflow, the shape of the data (how many items, how they’re grouped) is what determines the flow: how many tasks run, and which pieces of data move together.

Another way to think of channels is as a conveyor belts: items (files, values, or combinations of both) are placed onto it, and it carries them — one at a time — into whatever process is connected downstream. A process doesn’t “loop over” a channel the way you might loop over files in bash; instead, Nextflow automatically starts a separate task for each item that arrives, and can run many of these tasks concurrently.

This is why, back in Section 3, changing a single line —

from

reads_ch = Channel.fromPath('data/sample_001_R1.fastq')

reads_ch = Channel.fromPath('data/sample_*_*.fastq')

— was enough to go from “process one file” to “process every file in parallel.” You didn’t add a loop. You changed what gets placed on the belt.

Tip

If you’re coming from bash, the instinct is to reach for a for loop whenever you want to do something to multiple files. But a bash loop doesn’t give you parallelism on its own — each iteration runs one after another, waiting for the previous one to finish. To actually parallelize a bash loop on an HPC, you’d typically need to build a job array or background/fan out each iteration yourself.

In Nextflow, you instead reach for a channel with more than one item in it — the parallelism comes for free, without any extra scheduling code on your part.

If you would like how to create job arrays for your project you should attend one of our HPC workshops

2. Channel Factories:

A channel factory is what creates a channel in the first place — it’s how data gets onto the conveyor belt to begin with. In other words, channel factories take something outside Nextflow (a glob pattern matching files on disk) and turn it into a channel of items. More on channel factories here.

2.1. `Channel.fromPath`: one file (or many) per item

You’ve already seen the simplest channel factory, Channel.fromPath, which emits one item per file matching a path or glob pattern:

// a single file
reads_ch = Channel.fromPath('data/sample_001_R1.fastq')

// every file matching the pattern — one channel item per file
reads_ch = Channel.fromPath('data/sample_*_*.fastq')

Each item that comes out of this channel is a file path, and each one will trigger a separate call to whatever process it’s connected to.

But what is a “glob pattern”?

In Nextflow, a glob pattern is a string containing wildcard characters used to match and select files or directories based on specific name patterns rather than writing out explicit file name

2.2 `Channel.fromFilePairs`: keeping paired-end reads together

FastQ data is often paired-end: each sample has two files (e.g. sample_001_R1.fastq and sample_001_R2.fastq, representing forward and reverse reads). These two files need to travel through the pipeline together — you never want to accidentally process sample001_R1.fastq against sample002_R2.fastq.

Channel.fromPath won’t help here, since it treats every file as its own independent item. Instead, Nextflow provides Channel.fromFilePairs, which groups matching files by a shared sample name and emits them as a single item: a tuple of (sample_id, [file1, file2]).

read_pairs_ch = Channel.fromFilePairs('data/sample_*_R{1,2}.fastq')

The {1,2} glob pattern tells Nextflow which part of the filename distinguishes the pair members; everything else in the filename is treated as the shared sample identifier.

Note

Try running just this line with .view() (introduced next) — the output makes the tuple structure very concrete, and is worth doing live before diving into the exercise.

3. Channel Operators

A channel operator transforms or inspects a channel that already exists — it doesn’t create data from scratch, it takes items already on the belt and does something to them as they pass by. In other words, operators take an existing channel and perform some action on it. There are many operators in Nextflow, grouped in categories based on their function:

Filtering Operators: Reduce or select elements based on conditions. Examples include filter, first, last, and unique.
Transforming Operators: Modify the data inside a channel. Common examples are map, flatmap, flatten, and collect.
Combining Operators: Join or merge multiple channels together. Examples include join, combine, merge, and groupTuple.
Splitting Operators: Break complex data structures or files into smaller chunks, such as splitText or splitCsv.
Maths Operators: Perform basic numerical calculations on channel values like count, min, max, and sum.

More on channel operators here. We are going to look at 3 of the most used ones: .view(), .map(), and .collect().

3.1. Peeking inside a channel: `.view()`

Channels are a bit of a black box until you look inside them. The .view() operator lets you inspect items flowing through a channel without changing them — extremely useful for debugging and for building an intuition for what shape your data is in.

// channel_view.nf
Channel
    .fromFilePairs('data/sample_*_R{1,2}.fastq')
    .view()

Running this (with no process attached at all) will print each tuple to the console, e.g.:

[sample1, [data/sample_001_R1.fastq, data/sample_001_R2.fastq]]
[sample2, [data/sample_002_R1.fastq, data/sample_002_R2.fastq]]

Tip

Get in the habit of adding .view() while building a pipeline, then removing it once you’re confident the channel looks the way you expect.

3.2. Reshaping items with `.map()`

Sometimes the shape a channel factory gives you isn’t quite the shape a process needs. The .map() operator lets you transform each item flowing through a channel — much like map in Python or R’s purrr::map — by applying a function to every item.

For example, say you only want the sample ID from each pair, discarding the file paths:

// channel_map.nf
Channel
    .fromFilePairs('data/sample_*_R{1,2}.fastq')
    .map { sample_id, files -> sample_id }
    .view()

sample1
sample2

.map() is one of the most commonly used channel operators — you’ll reach for it constantly to reshape data between one process’s output and the next process’s expected input.

3.3. Gathering items with `.collect()`

Every operator so far still emits one item at a time. But sometimes you want the opposite: to wait until every item has arrived, then gather them all into a single list — for example, once every sample has been FastQC’d individually, you might want a single downstream step (like MultiQC) that summarizes all reports together.

// channel_collect.nf
Channel
    .fromPath('data/sample_*_R1.fastq')
    .collect()
    .view()

[data/sample_001_1.fastq, data/sample_002_1.fastq, data/sample_003_1.fastq]

Notice the difference: instead of three separate items flowing through (one per file), .collect() produces a single item — a list containing everything that arrived. This is exactly the operator you reach for when a process needs to see all upstream outputs at once rather than being launched once per file.

Warning

.collect() has to wait for every upstream item to arrive before it can emit anything. In a pipeline with many samples, this means the process downstream of a .collect() won’t start until the slowest upstream task finishes — a useful thing to keep in mind when thinking about where parallelism starts and stops in your pipeline.

4. Exercise: what happens when the channel shape changes?

You’ve been running fastqc on single files with Channel.fromPath. Now swap in the paired-end channel from the previous section — same process, no changes — and run it:

reads_ch = Channel.fromFilePairs('data/sample_*_R{1,2}.fastq')
fastqc(reads_ch)

What do you see?

What I see

It crashes! With something like:

ERROR ~ Error executing process > 'fastqc (1)'

Caused by:
  Not a valid path value: 'sample_003'



Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Before checking the solution, look at the error and try to answer: what type of value is the process actually receiving, and what does the path reads input declaration assume it’s receiving instead?

Tip

You can cd into the failed task’s work directory and check .command.sh — does it look like it ran on a sample ID, or on files?

Another way to trouble shoot this is adding the .view() operator to your channel like we learned above and seeing what it the shape of your output. Compare that with the shape of the input that the fastqc input is expecting.

Solution

fromFilePairs emits [sample_id, [file1, file2]] — a tuple, not a bare path. The process only declares one input (path reads), so Nextflow has no slot for the sample ID and tries to treat it as a path, hence:

    Not a valid path value: 'sample_003'

The fix is to make the input declaration match the channel’s actual shape:

// fastqc_solution.nf
    input:
    tuple val(sample_id), path(reads)

    output:
    tuple val(sample_id), path("*_fastqc.{html,zip}")

Now we should get an output of 3 tasks instead of 6:

executor >  local (3)
[57/da6072] process > fastqc (1) [100%] 3 of 3 ✔

Same fan-out rule, different grouping

Running fastqc_example_3.nf on 6 individual files with Channel.fromPath produced 6 tasks — one per file:

executor >  local (6)
[e5/f4b3d3] process > fastqc (6) [100%] 6 of 6 ✔

Now run the paired-end solution above, on the same 6 underlying files, but grouped with Channel.fromFilePairs:

executor >  local (3)
[57/da6072] process > fastqc (1) [100%] 3 of 3 ✔

Same files, same “one task per channel item” rule from §1.1 — but only 3 tasks instead of 6. fromFilePairs doesn’t emit one item per file; it emits one item per sample, with both reads already bundled together. Fewer emissions means fewer parallel tasks. What changed isn’t the fan-out mechanism — it’s what counts as one item.

Hello Workflow: Chaining Processes

1. Passing outputs from one process to the next

Every process call in Nextflow produces an output channel, accessible via .out. That output channel can be handed straight to another process as its input — this is the entire mechanism behind building a multi-step pipeline.

firstProcess(input_ch)
secondProcess(firstProcess.out)

No file names change hands manually, no intermediate variables need to track “where did that file end up” — Nextflow wires the two processes together using the channel itself.

Important

This is the core idea of the rest of the workshop: a pipeline is just a sequence of process calls, where later calls consume the .out of earlier ones. Everything else (parallelism, staging, resuming) is built on top of this one mechanism.

Branching vs. chaining

It’s easy to conflate two different things that both involve “using a channel in more than one place”:

Branching: the same channel is passed to two different processes (e.g. both fastqc(reads_ch) and trim(reads_ch)). Neither process depends on the other’s output — they just both read from the same source. Nextflow handles this automatically; you don’t need to do anything special to “duplicate” a channel.
Chaining: one process’s .out becomes another process’s input (e.g. fastqc(trim.out)). Here, the second process genuinely cannot start until the first one finishes for that item.

Keep an eye out for which one you’re doing below — the live-code section starts with branching, and the exercise is where we turn it into real chaining.

2. Branching out: raw QC + trimming

We’ll build on fastqc_example_3.nf. Alongside the existing fastqc process, add a new one for trimming with Trimmomatic:

// fastqc_trimmomatic.nf
process trim {

    module 'trimmomatic/0.40'
    stageInMode 'copy'
    publishDir 'trimmed', mode: 'copy'

    input:
    path reads

    output:
    path "trimmed_${reads}"

    script:
    """
    trimmomatic SE -threads 4 ${reads} trimmed_${reads} \\
        SLIDINGWINDOW:4:20 MINLEN:36
    """
}

A few things to notice, since this looks almost identical to fastqc:

Unlike FastQC, we choose the output filename ourselves (trimmed_${reads}), so the output: block here is an exact name rather than a glob. Nextflow will fail the task if that exact file isn’t produced — a stricter, more informative check than a glob when you’re the one naming the file.
${reads} is reused inside the output filename itself. Since reads is whatever file Nextflow staged in for this task, trimmed_${reads} will always correspond to that same task’s input, even when many tasks are running in parallel.

Now wire both processes to the same input channel in the workflow block:

// fastqc_trimmomatic.nf
workflow {

    main:
    reads_ch = Channel.fromPath('data/sample_*_*.fastq')

    fastqc(reads_ch)
    trim(reads_ch)
}

Run it:

nextflow run scripts/fastqc_example_3.nf

Command output:

executor >  local (12)
[b4/24bb55] process > fastqc (5) [100%] 6 of 6 ✔
[64/53358c] process > trim (6)   [100%] 6 of 6 ✔

You should now have two output folders, fastqc/ and trimmed/, each populated independently. Take a look inside trimmed/ — the trimmed .fastq files are there, prefixed as expected.

Tip

Try commenting out the fastqc(reads_ch) line and re-running. trim still runs fine on its own — confirming that these two processes really are independent branches right now, not a chain.

3. Exercise: add a third step — FASTQC on trimmed reads

Now let’s make this a genuine 3-step linear pipeline by adding a check on the trimmed reads: does trimming actually improve quality?

Try the obvious thing first:

Call fastqc a second time in the workflow block, passing it trim.out instead of reads_ch.
Run the pipeline.

// workflow_solution_1.nf
workflow {

    main:
    reads_ch = Channel.fromPath('data/sample_*_*.fastq')

    fastqc(reads_ch)
    trim(reads_ch)
    fastqc(trim.out)
}

You’ll get this:

[-        ] process > fastqc -
[-        ] process > trim   -
Process 'fastqc' has been already used -- If you need to reuse the same component, include it with a different name or include it in a different workflow context

 -- Check script 'scripts/workflow_solution.nf' at line: 47 or see '.nextflow.log' file for more details

Why this happens

In DSL2, a process can only be invoked once per workflow. It doesn’t matter that you’re feeding it a different channel — Nextflow tracks calls by process name, and fastqc was already called on line one. This is different from the channel-shape problems you hit in §2; this is a rule about how many times a given process name can appear in a workflow {} block, full stop.

The real fix: give the second call its own process name.

#!/usr/bin/env nextflow

// workflow_solution_2.nf

process fastqc_raw {

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'results/qc_raw', mode: 'copy'       //this line

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

process fastqc_trimmed {                            //this new process

    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'results/qc_trimmed', mode: 'copy'   //this line

    input:
    path reads

    output:
    path "*_fastqc.{html,zip}"

    script:
    """
    fastqc -t 4 ${reads}
    """
}

process trim {

    module 'trimmomatic/0.40'
    stageInMode 'copy'
    publishDir 'results/trimmed', mode: 'copy'       //this line

    input:
    path reads

    output:
    path "trimmed_${reads}"

    script:
    """
    trimmomatic SE -threads 4 ${reads} trimmed_${reads} \\
        SLIDINGWINDOW:4:20 MINLEN:36
    """
}

workflow {

    main:
    reads_ch = Channel.fromPath('data/sample_*_*.fastq')

    fastqc_raw(reads_ch)
    trim(reads_ch)
    fastqc_trimmed(trim.out)
}

This duplicates the process body, which isn’t ideal — but it uses nothing beyond what’s already in your toolkit, and for a 1.5-day workshop that’s a completely reasonable trade-off.

Your output should look like something like this:

executor >  local (18)
[8c/a16e15] process > fastqc_raw (6)     [100%] 6 of 6 ✔
[83/34b1f9] process > trim (5)           [100%] 6 of 6 ✔
[65/e5cc65] process > fastqc_trimmed (6) [100%] 6 of 6 ✔

And your results/ directory will have this structure:

results/
├── qc_raw/
├── trimmed/
└── qc_trimmed/

Takeaway

You’ve now built a real linear pipeline: raw QC → trim → post-trim QC, all from three process calls and one reused process definition. The mental model to leave with: channels decide the shape of the pipeline; processes stay generic. Tomorrow we’ll build on this by parameterizing pipelines from the command line and introducing resume/caching so re-runs don’t redo finished work.

Stretch task: paired-end reads (try this at home if we’re short on time)

Everything today used Channel.fromPath — one file per emission. Real paired-end data usually needs Channel.fromFilePairs instead, which you already built back in §2. This task asks you to rebuild today’s pipeline around that.

Tier 1 — reuse what you already fixed. Swap reads_ch to Channel.fromFilePairs(...), and update fastqc_raw/fastqc_trimmed to take the tuple input, the same fix from §2’s crash exercise:

input:
tuple val(sample_id), path(reads)

output:
tuple val(sample_id), path("*_fastqc.{html,zip}")

Confirm you get 3 tasks per FastQC step instead of 6 — same as the callout from §2, just inside a full pipeline this time.

executor >  local (9)
[de/ddd301] process > fastqc_raw (1)     [100%] 3 of 3 ✔
[fb/a9ba71] process > trim (1)           [100%] 3 of 3 ✔
[fb/3722cb] process > fastqc_trimmed (2) [100%] 3 of 3 ✔

Tier 2 — go all the way: paired-end trimming. trim is harder than it looks with paired data. Trimmomatic’s PE mode takes two input files and produces four outputs — forward paired, forward unpaired, reverse paired, reverse unpaired — not the single in/out you used with SE. You’ll need to:

Look up Trimmomatic’s PE command syntax (not covered in these materials — this is the “go read the docs” part of a stretch task)
Decide what your process’s output: block needs to declare to capture all four files
Decide which of the four outputs actually feeds fastqc_trimmed afterward (hint: it’s the two paired outputs, not the unpaired ones)

This one’s open-ended on purpose — there’s more than one reasonable way to structure the outputs. If you get stuck on the tuple/channel side rather than the Trimmomatic side, the fix is the same pattern as Tier 1.

Solution

#!/usr/bin/env nextflow

// workflow_stretch.nf

process fastqc_raw {
    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'results/qc_raw', mode: 'copy'

    input:
    tuple val(sample_id), path(reads)

    output:
    tuple val(sample_id), path("*_fastqc.{html,zip}")

    script:
    """
    fastqc -t 4 ${reads}
    """
}

process trim {
    module 'trimmomatic/0.40'
    stageInMode 'copy'
    publishDir 'results/trimmed', mode: 'copy'

    input:
    tuple val(sample_id), path(reads)

    output:
    tuple val(sample_id),
          path("${sample_id}_R1_paired.fastq"),
          path("${sample_id}_R2_paired.fastq")

    script:
    """
    trimmomatic PE -threads 4 \\
        ${reads[0]} ${reads[1]} \\
        ${sample_id}_R1_paired.fastq ${sample_id}_R1_unpaired.fastq \\
        ${sample_id}_R2_paired.fastq ${sample_id}_R2_unpaired.fastq \\
        SLIDINGWINDOW:4:20 MINLEN:36
    """
}

process fastqc_trimmed {
    module 'fastqc/0.12.1'
    stageInMode 'copy'
    publishDir 'results/qc_trimmed', mode: 'copy'

    input:
    tuple val(sample_id), path(r1_paired), path(r2_paired)

    output:
    tuple val(sample_id), path("*_fastqc.{html,zip}")

    script:
    """
    fastqc -t 4 ${r1_paired} ${r2_paired}
    """
}

workflow {

    main:
    read_pairs_ch = Channel.fromFilePairs('data/sample_*_R{1,2}.fastq')

    fastqc_raw(read_pairs_ch)
    trim(read_pairs_ch)
    fastqc_trimmed(trim.out)
}

1. Setting up

2. Starting an interactive session

3. Examining our data

Hello FASTQC: Your First Process

1. Warmup: run FASTQC on one sample directly

2. Examine the nextflow script and run it

2.1. Examine the code

2.1.1. The process definition

2.2. Run the workflow

2.2.1. Launch the workflow and monitor execution

2.2.2. Find the output and logs in the work directory

2.3. Exercise: modify the process

3. Scaling up: from one sample to many

Hello Channels: How Data Flows

1. Channels as conveyor belts

2. Channel Factories:

2.1. Channel.fromPath: one file (or many) per item

2.2 Channel.fromFilePairs: keeping paired-end reads together

3. Channel Operators

3.1. Peeking inside a channel: .view()

3.2. Reshaping items with .map()

3.3. Gathering items with .collect()

4. Exercise: what happens when the channel shape changes?

Hello Workflow: Chaining Processes

1. Passing outputs from one process to the next

2. Branching out: raw QC + trimming

3. Exercise: add a third step — FASTQC on trimmed reads

Takeaway

Stretch task: paired-end reads (try this at home if we’re short on time)

2.1.1. The `process` definition

2.2.2. Find the output and logs in the `work` directory

2.1. `Channel.fromPath`: one file (or many) per item

2.2 `Channel.fromFilePairs`: keeping paired-end reads together

3.1. Peeking inside a channel: `.view()`

3.2. Reshaping items with `.map()`

3.3. Gathering items with `.collect()`