Cloud Companion: Choosing Workflow Tools And Running Nextflow On AWS

Takeaway: Nextflow is not the only workflow manager, but it is one of the strongest choices when you need one pipeline to run on a laptop, HPC, and cloud with containers, caching, and many samples.

Week 5 introduced Nextflow from the user side. Week 6 walked through a small custom ONT pipeline. This guide zooms out: when should a project use Nextflow, when should it consider alternatives, and how can AWS runs stay cost-aware from the first test?

Open the AWS and workflow-manager resource pack on GitHub.

The Problem Workflow Tools Solve

A bioinformatics workflow manager is useful when your project stops being one command and becomes a dependency graph:

flowchart LR
  fastq["FASTQ files"] --> qc["QC"]
  fastq --> align["Alignment or quantification"]
  align --> counts["Counts or alignments"]
  qc --> multiqc["Summary report"]
  counts --> stats["Statistical analysis"]
  stats --> report["Figures and interpretation"]

Without a workflow manager, people often lose track of:

A good workflow tool makes the dataflow explicit. It does not make the biology correct by itself.

The Short Comparison

A practical selection rule:

Use Bash to learn commands.
Use Snakemake if your lab thinks in Python and file rules.
Use WDL when the ecosystem already expects WDL.
Use CWL when standards-based portability is the priority.
Use Nextflow when portability, containers, cloud, and nf-core matter together.

Choose By Team, Not Fashion

The best workflow language is the one your collaborators can test and maintain after the original author leaves.

Why Nextflow Often Wins For Omics

Nextflow is especially strong for omics because it separates three things that beginners often mix together:

That separation is the main advantage. You can write pipeline logic once, then change where it runs by changing a profile.

plugins {
  id 'nf-amazon@3.10.0'
}

profiles {
  local {
    process.executor = 'local'
  }

  awsbatch {
    process.executor = 'awsbatch'
    process.queue = 'nextflow-demo-queue'
    workDir = 's3://your-bucket/nextflow-work/demo'
    aws.region = 'us-west-2'
    aws.batch.jobRole = 'YOUR_JOB_ROLE_ARN'
    aws.batch.executionRole = 'YOUR_EXECUTION_ROLE_ARN'
  }
}

Nextflow also caches completed tasks. If a run fails after alignment, -resume can reuse completed work instead of starting everything again. That is a scientific and financial feature.

How Nextflow Runs On AWS

On AWS, Nextflow usually runs like this:

flowchart TB
  user["Your laptop or small coordinator VM"] --> nf["Nextflow command"]
  nf --> s3["S3 work directory"]
  nf --> batch["AWS Batch job queue"]
  batch --> ec2["EC2 or Fargate workers"]
  ec2 --> tasks["Containerized pipeline tasks"]
  tasks --> s3
  s3 --> results["Final results in S3"]

Plain English:

The key point: AWS Batch does not make your pipeline free. It helps run tasks at scale. You still pay for the AWS resources used underneath, although AWS Batch itself has no extra job-scheduling charge.

Cost And Time Best Practices

Use these habits before running real data:

For real projects, treat cost like a QC metric. A pipeline that gives the right answer once but cannot be rerun affordably is not production-ready.

A Credit-Bounded AWS Learning Path

As of October 2026, AWS describes its Free Tier for new customers as credit-based, with up to USD 200 in credits over 6 months depending on plan and activities. AWS service access differs by account plan, and cloud offers change. Check the current AWS Free Tier page before assuming a tutorial run will cost nothing.

Cost note:

This is a tiny AWS Batch learning run designed to stay within new-account credits.
It is not a guarantee of zero cost for every account.
Set a budget alert first.
Delete resources when finished.

Best AWS service for real Nextflow execution: AWS Batch. It is the native AWS executor documented by Nextflow and is designed for containerized batch jobs.

Tutorial: Local First, AWS Second

Start with a tiny local command:

# Create the cloud demo folder.
mkdir -p nextflow-aws-demo
cd nextflow-aws-demo

# Create the smallest possible Nextflow script.
cat > main.nf <<'EOF'
process SAY_HELLO {
  output:
  path "hello.txt"

  script:
  """
  echo "hello from Nextflow" > hello.txt
  """
}

workflow {
  SAY_HELLO()
}
EOF

# Run locally first.
nextflow run main.nf

Expected output:

executor >  local
[.../...] process > SAY_HELLO [100%] 1 of 1

Check the result:

cat work/*/*/hello.txt

Expected:

hello from Nextflow

Now add an AWS profile:

profiles {
  awsbatch {
    process.executor = 'awsbatch'
    process.queue = 'YOUR_AWS_BATCH_QUEUE'
    workDir = 's3://YOUR_BUCKET/nextflow-work/hello'
    aws.region = 'us-west-2'
    process.container = 'YOUR_ECR_REPOSITORY_URI:2026.10'
  }
}

Save that as nextflow.config. Then run only after AWS Batch, S3, IAM, and budget alerts are configured:

nextflow run main.nf -profile awsbatch

If you are learning, stop here. This tiny run proves the coordinator can submit to AWS Batch, the worker can run a container, and S3 can store work files. Do not test cloud infrastructure for the first time with a full RNA-seq or long-read dataset.

Create A Bounded Learning Environment

The resource pack includes aws-batch-learning-stack.yml. It creates an encrypted S3 work bucket, a four-vCPU Fargate Spot environment, a two-vCPU On-Demand fallback, a Batch queue, and narrow task roles. You supply an existing VPC, outbound-capable subnets, and a security group.

Before starting, confirm the coordinator computer has AWS CLI v2, Docker with Buildx, and Nextflow:

aws --version
docker buildx version
nextflow -version
aws configure

aws configure stores a local profile; it does not put credentials in this repository. Prefer temporary credentials or an approved identity flow for organizational accounts.

# Validate the template locally before creating resources.
aws cloudformation validate-template \
  --template-body file://content/resources/nextflow-aws-companion/aws-batch-learning-stack.yml

# Deploy after replacing the three networking values.
aws cloudformation deploy \
  --stack-name biofield-nextflow-learning \
  --template-file content/resources/nextflow-aws-companion/aws-batch-learning-stack.yml \
  --parameter-overrides \
    VpcId=vpc-REPLACE \
    SubnetIds='subnet-REPLACE' \
    SecurityGroupIds='sg-REPLACE' \
  --capabilities CAPABILITY_IAM

# Retrieve the queue and bucket values for nextflow.config.
aws cloudformation describe-stacks \
  --stack-name biofield-nextflow-learning \
  --query 'Stacks[0].Outputs' \
  --output table

Expected: the stack reaches CREATE_COMPLETE and prints BucketName, EcrRepositoryUri, JobQueueName, JobRoleArn, and ExecutionRoleArn. Nextflow’s coordinator still needs permission to submit and inspect Batch jobs; do not put permanent AWS keys in the pipeline repository.

Build the supplied task image and push it to the new ECR repository:

# Read the repository URI from the stack output.
REPOSITORY_URI=$(aws cloudformation describe-stacks \
  --stack-name biofield-nextflow-learning \
  --query "Stacks[0].Outputs[?OutputKey=='EcrRepositoryUri'].OutputValue" \
  --output text)

# Authenticate Docker to this account's ECR registry.
aws ecr get-login-password --region us-west-2 \
  | docker login --username AWS --password-stdin "${REPOSITORY_URI%%/*}"

# Build for the x86_64 Fargate environment and push the immutable tutorial tag.
docker buildx build \
  --platform linux/amd64 \
  --file content/resources/nextflow-aws-companion/Dockerfile.aws-batch-demo \
  --tag "${REPOSITORY_URI}:2026.10" \
  --push \
  content/resources/nextflow-aws-companion

Why not use a plain Bash image? With an S3 work directory, Nextflow’s AWS Batch tasks use the AWS CLI to stage task files unless a different staging design such as Wave and Fusion is configured. The supplied image therefore contains Bash, ps, and a pinned AWS CLI v2.

Put the five stack outputs into nextflow-aws-demo.config, then run the tiny workflow. Its 2 GB memory request is valid for a one-vCPU Fargate task.

After the test, preserve any needed outputs and remove the compute stack:

aws cloudformation delete-stack --stack-name biofield-nextflow-learning
aws cloudformation wait stack-delete-complete --stack-name biofield-nextflow-learning

The work bucket is intentionally retained to prevent accidental data loss. Inspect it before emptying or deleting it.

When AWS Batch Misbehaves

The downloadable failure map contains the same checks for use during an incident.

AWS Checklist Before Running Real Data

When Not To Use AWS

Use local or HPC instead when:

• the dataset fits comfortably on your laptop
• you are still debugging commands
• protected data cannot leave an approved environment
• your institution already has a well-supported cluster
• the cloud setup would take longer than the analysis

Cloud is a scaling tool, not a badge of seriousness.

What To Save From Every Workflow Run

For reproducibility, archive:

That habit carries into larger data platforms, where run records, storage layout, and metadata become part of the scientific audit trail.

Next In The Series

Return to Week 7 for pipeline testing with nf-test, GitHub Actions, and Jenkins. Continue with Week 8 for big bioinformatics data architecture on AWS: S3 storage zones, Glue catalogs, Athena queries, Lambda event handlers, Step Functions orchestration, AWS Batch, and Spot instances.

Credits and References

• Nextflow documentation: https://docs.seqera.io/nextflow/
• Nextflow executors documentation: https://www.nextflow.io/docs/latest/executor.html
• Nextflow AWS documentation: https://www.nextflow.io/docs/latest/aws.html
• nf-core documentation: https://nf-co.re/
• Snakemake documentation: https://snakemake.readthedocs.io/
• Common Workflow Language: https://www.commonwl.org/
• WDL specification: https://github.com/openwdl/wdl
• Cromwell documentation: https://cromwell.readthedocs.io/
• AWS Free Tier: https://aws.amazon.com/free/
• AWS Batch pricing: https://aws.amazon.com/batch/pricing/