nf-core/configs: charite
Scientific Computing Charité HPC Cluster Profile | Developed by Wassim Salam (@wassimsalam01) | Maintained by Scientific Computing, GB IT, Charité (@ScientificComputingCharite)
nf-core/configs: Charité Configuration
This page documents the charite Nextflow profile for running Nextflow and nf-core pipelines on the Charité HPC. See the full configuration at conf/charite.config. For general HPC information, see Charité Science IT.
Official Contact: Scientific Computing, GB IT, Charité - sc-hpc-helpdesk@charite.de
Table of Contents
- Contributors and Acknowledgments
- 1. Quick Overview and Usage Examples
- 2. Requesting Charité HPC Access
- 3. Environment Setup
- 4. Running Nextflow Workflows on the Charité HPC
- 5. Job Assignment Logic and GPU Acceleration
- 6. More Practical Examples
- 7. Known Limitations and Issues
Contributors and Acknowledgments
- Developer: Wassim Salam - BIH/MDC Genomics Platform Data Management Team
- Maintainers: Magnus Hagdorn & Andreas Reppas - Scientific Computing HPC Team
@wassimsalam01 thanks the following individuals for their support and guidance in developing the profile:
- Scientific Computing HPC Team (@ScientificComputingCharite)
- BIH/MDC Genomics Platform Data Management Team (@BIH-MDC-Genomics-Platform)
- Marten Jäger (@martenj)
- Cornelius Fischer (@cofischer)
- Namuun Battur (@NmnBttr)
Citation
Please reference the HPC @Charité in any research report, journal, or publication that requires citation of authors’ work. Recognition of the Charité compute resources you used to perform research is important for acquiring funding for the next generation of hardware, support services, and our research and development activities in HPC, data storage and other related infrastructure.
Our suggested acknowledgement is as follows:
The authors acknowledge the Scientific Computing of the IT Division at the Charité - Universitätsmedizin Berlin for providing computational resources that have contributed to the research results reported in this paper. URL: https://www.charite.de/en/research/research_support_services/research_infrastructure/science_it/#c30646061
1. Quick Overview and Usage Examples
- Scheduler: SLURM
- Container Engine: Apptainer
- Default QOS:
normal - Default Queue:
compute - Resource Limits of Default Queue:
- Maximum CPUs: 192 cores
- Maximum memory: 1TB
- Maximum walltime: 2 days
- Additional Queues:
dragen_cc05,dragen_cc15,dragen_gp,genomics,highmem - GPU Queues:
gpu,pgpu
See Job Assignment Logic and GPU Acceleration for more info.
- Usage Examples:
# nf-core Pipeline
nextflow run nf-core/<pipeline-name> -profile charite ...
# Test nf-core Demo Pipeline (small-scale workflow)
# Specifying '-profile test' uses dummy data provided by nf-core for that particular pipeline
nextflow run nf-core/demo -profile test,charite --outdir results/demo-test
For the sake of simplicity, this documentation will assume usage of nf-core pipelines. To use this config with a custom Nextflow workflow, see the following links for more info:
- Using nf-core/configs outside nf-core
- Create a Pipeline with nf-core CLI (nf-core CLI already available within Shared Nextflow Conda Environment)
2. Requesting Charité HPC Access
The Charité internal user documentation/wiki requires a Charité GitLab account. To request access:
- Email sc-hpc-helpdesk@charite.de to request access to the Charité HPC. This initiates the onboarding process.
- If you do not have access to git.bihealth.org, contact health-data@charite.de to request GitLab access for the Charité HPC User Documentation and Onboarding Guide.
3. Environment Setup
[!NOTE] In order for the config to work as intended, some environment variables must first be exported before running any workflow. This section provides a detailed overview of those variables.
A complete practical example is later provided in the next section, which integrates the exporting of the variables explained together with a run of a nf-core pipeline.
Click here to skip ahead to ‘Running Nextflow Workflows on the Charité HPC’
3.1. Global Utility Variables
The Charité HPC profile uses project-specific scratch storage for Nextflow and Apptainer cache and temporary files whenever available. Storage paths are determined automatically based on the PROJ_ABBR environment variable:
- If
PROJ_ABBRis non-empty:PROJ_SCRATCHDIR_CHARITE_USERis set to/sc-scratch/sc-scratch-${PROJ_ABBR}. - If
PROJ_ABBRis empty:PROJ_SCRATCHDIR_CHARITE_USERfalls back to${HOME}, which has a 100 GB storage quota.
The profile requires PROJ_SCRATCHDIR_CHARITE_USER to be defined and will fail if it is missing.
Using project scratch storage is strongly recommended, particularly for large pipelines, as Apptainer caches and Nextflow working directories can quickly exhaust the home-directory quota.
For more information on managing project and scratch storage, refer to the Charité HPC documentation:
- Existing project members: How to access project and scratch directories
- Users without an assigned project: How to apply for project and scratch directories
- Project administrators: How to manage project and scratch directories
# Project abbreviation (leave empty if you don't have one)
export PROJ_ABBR=""
if [[ -n "${PROJ_ABBR}" ]]; then
export PROJ_SCRATCHDIR_CHARITE="/sc-scratch/sc-scratch-${PROJ_ABBR}"
# Validate project scratch directory
if [[ ! -d "${PROJ_SCRATCHDIR_CHARITE}" || ! -x "${PROJ_SCRATCHDIR_CHARITE}" ]]; then
echo "[ERROR] Project scratch directory does not exist or is inaccessible: ${PROJ_SCRATCHDIR_CHARITE}"
echo "[ERROR] Please verify your PROJ_ABBR ('${PROJ_ABBR}') and project access."
exit 1
fi
export PROJ_SCRATCHDIR_CHARITE_USER="${PROJ_SCRATCHDIR_CHARITE}/${USER}"
else
export PROJ_SCRATCHDIR_CHARITE_USER="${HOME}"
fi
# Fail-safe: ensure the required environment variable is not empty
if [[ -z "${PROJ_SCRATCHDIR_CHARITE_USER:-}" ]]; then
echo "[ERROR] PROJ_SCRATCHDIR_CHARITE_USER is not set or is empty."
exit 1
fi
3.2. Apptainer-specific Variables
| Variable | Purpose | Value |
|---|---|---|
APPTAINER_CACHEDIR |
Cache used internally by Apptainer when downloading and building images | ${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer |
APPTAINER_TMPDIR |
Temporary files generated during image creation | ${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer/tmp |
NXF_APPTAINER_CACHEDIR (Nextflow equivalent apptainer.cacheDir) |
Location of built Apptainer images | ${PROJ_SCRATCHDIR_CHARITE_USER}/apptainer-images |
Apptainer is readily available on Charité HPC compute nodes and does not require loading a module.
The configuration distinguishes between Apptainer’s internal cache and temporary files and Nextflow’s cached container images. These are stored separately to allow users to customize image storage without affecting Apptainer’s internal operations.
Apptainer Cache and Temporary Files
The following environment variables define where Apptainer stores its internal cache and temporary files:
# Apptainer cache and temporary files (user-specific)
export APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer"
export APPTAINER_TMPDIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer/tmp"
mkdir -p "${APPTAINER_TMPDIR}"
Apptainer Image Cache
The location of built Apptainer container images is managed through through NXF_APPTAINER_CACHEDIR and apptainer.cacheDir.
apptainer {
// ...
cacheDir = System.getenv("NXF_APPTAINER_CACHEDIR") ?: "${params.proj_scratchdir_charite_user}/apptainer-images"
// ...
}
By default, container images are stored in the following user-specific directory: ${PROJ_SCRATCHDIR_CHARITE_USER}/apptainer-images
The resulting default structure looks like this:
/sc-scratch/sc-scratch-${PROJ_ABBR}/
├── user1/
│ ├── .apptainer/
│ │ ├── cache/
│ │ └── tmp/
│ └── apptainer-images/
└── user2/
│ ├── .apptainer/
│ │ ├── cache/
│ │ └── tmp/
│ └── apptainer-images/
...
Users can optionally override this location like so:
# Adjust path as needed, this is just an example
export NXF_APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE}/apptainer-images"
For projects with multiple collaborators, container images can be stored in a shared project directory rather than duplicated in each user’s scratch directory. This allows collaborators to reuse already downloaded images, reducing redundant downloads and storage consumption.
/sc-scratch/sc-scratch-${PROJ_ABBR}/
├── apptainer-images/ # Shared container images (optional)
├── user1/
│ └── .apptainer/
│ ├── cache/
│ └── tmp/
└── user2/
│ └── .apptainer/
│ ├── cache/
│ └── tmp/
...
Note: Shared image directories require appropriate read/write permissions. Collaborators should coordinate access and avoid modifying or deleting images that others may be using. Please discuss this with your team.
The following diagram illustrates how Nextflow determines where to store container images:
---
config:
flowchart:
wrappingWidth: 1000
---
flowchart TD
A(["Apptainer Image Directory"])
B["export NXF_APPTAINER_CACHEDIR='...' set by user?"]
C["User-specific default directory"]
D["Custom directory"]
E["${PROJ_SCRATCHDIR_CHARITE_USER}/apptainer-images"]
F["${NXF_APPTAINER_CACHEDIR}"]
A --> B
B -->|"No"| C
B -->|"Yes"| D
C --> E
D --> F
classDef start fill:#dbeafe,stroke:#2563eb,color:#1e40af
classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
classDef path fill:#dcfce7,stroke:#16a34a,color:#166534
classDef category fill:#f1f5f9,stroke:#64748b,color:#334155
class A start
class B decision
class C,D category
class E,F path
3.3. Nextflow-specific Variables
The following environment variables configure Nextflow’s home, cache, and temporary directories. By default, these are stored under ${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow, keeping Nextflow-generated files outside the limited home directory whenever project scratch space is available.
| Variable | Purpose | Value |
|---|---|---|
NXF_HOME |
Nextflow home directory (configuration, plugins, and other internal files) | ${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow |
NXF_CACHE_DIR |
Nextflow’s execution cache, used for task metadata and workflow resumption (-resume) |
${PROJ_SCRATCHDIR_CHARITE_USER} |
NXF_TEMP |
Temporary directory used by Nextflow | ${PROJ_SCRATCHDIR_CHARITE_USER}/tmp |
NXF_OPTS |
JVM memory settings for the Nextflow launcher | -Xms1g -Xmx4g |
NXF_WORK |
Nextflow working directory for task execution, managed automatically by charite.config |
Configured via workDir in charite.config |
# Nextflow home, cache, and temporary directories
export NXF_HOME="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow"
export NXF_CACHE_DIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow"
export NXF_TEMP="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow/tmp"
mkdir -p "${NXF_TEMP}"
# Limit memory allocation for the Nextflow JVM
export NXF_OPTS="-Xms1g -Xmx4g"
The JVM memory settings follow nf-core recommendations to prevent Nextflow from consuming excessive memory.
With an assigned project and the default NXF_APPTAINER_CACHEDIR, each user’s scratch directory contains their Nextflow and Apptainer files:
/sc-scratch/sc-scratch-${PROJ_ABBR}/
├── user1/
│ ├── .apptainer/
│ │ ├── cache/
│ │ └── tmp/
│ ├── apptainer-images/
│ ├── .nextflow/
│ │ ├── cache/
│ │ └── tmp/
│ └── nxf-work/
├── user2/
│ ├── .apptainer/
│ │ ├── cache/
│ │ └── tmp/
│ ├── apptainer-images/
│ ├── .nextflow/
│ │ ├── cache/
│ │ └── tmp/
│ └── nxf-work/
...
For additional information, see the Nextflow environment variable reference.
3.4. Nextflow Strict Syntax
[!IMPORTANT] Nextflow v26.04 and later uses the v2 syntax parser by default as part of the transition toward strict syntax.
Some nf-core pipelines may not yet be fully compatible with the v2 parser. If a pipeline fails due to syntax incompatibilities, you can temporarily switch to the legacy parser:
export NXF_SYNTAX_PARSER=v1This workaround should only be used when necessary and removed once the affected pipeline supports the v2 parser.
For more information, see the Nextflow Strict Syntax documentation.
3.5. Shared Nextflow Conda Environment
The Conda environment nf-core-2026-1 is provided by the SC HPC Team as part of the Shared Conda Environments catalogue, which you can readily use. It contains both the nf-core CLI (v4.2.0) as well as Nextflow (v26.10.4).
conda activate nf-core-2026-1
4. Running Nextflow Workflows on the Charité HPC
[!CAUTION] Do not attempt to start resource-heavy jobs on frontend nodes. The memory on the frontends is restricted to 1 GB per user. Your session will get killed (see Frontend Memory Restrictions). It is therefore recommended to either work in an interactive session or submit your jobs via Batch scripts.
Option 1: Interactive Session
Interactive Session via srun
First, ssh into a frontend node via your terminal as you normally would (see Access), then:
# Interactive sessions launched with srun are capped at 8 hours runtime
srun --job-name=run-nf-core --partition=compute --ntasks=1 --cpus-per-task=4 --mem=16G --time=08:00:00 --pty bash
Interactive Session via Open OnDemand Portal
It is also possible to jump right into an interactive session using the Open OnDemand Portal (Open OnDemand Wiki). You must be connected the Charité network to use this, either via Ethernet cable or OpenVPN Connect.
Interactive Session Checklist
[!NOTE]
Activate Nextflow with
conda activate nf-core-2026-1>>> Click here to reveal the hidden code snippet and copy it for your interactive session! <<<
# Global Utility Variables ## Project abbreviation (leave empty if you don't have one) export PROJ_ABBR="" if [[ -n "${PROJ_ABBR}" ]]; then export PROJ_SCRATCHDIR_CHARITE="/sc-scratch/sc-scratch-${PROJ_ABBR}" # Validate project scratch directory if [[ ! -d "${PROJ_SCRATCHDIR_CHARITE}" || ! -x "${PROJ_SCRATCHDIR_CHARITE}" ]]; then echo "[ERROR] Project scratch directory does not exist or is inaccessible: ${PROJ_SCRATCHDIR_CHARITE}" echo "[ERROR] Please verify your PROJ_ABBR ('${PROJ_ABBR}') and project access." exit 1 fi export PROJ_SCRATCHDIR_CHARITE_USER="${PROJ_SCRATCHDIR_CHARITE}/${USER}" else export PROJ_SCRATCHDIR_CHARITE_USER="${HOME}" fi # Fail-safe: ensure the required environment variable is not empty if [[ -z "${PROJ_SCRATCHDIR_CHARITE_USER:-}" ]]; then echo "[ERROR] PROJ_SCRATCHDIR_CHARITE_USER is not set or is empty." exit 1 fi # Apptainer-specific Variables # Apptainer cache and temporary files (user-specific) export APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer" export APPTAINER_TMPDIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer/tmp" mkdir -p "${APPTAINER_TMPDIR}" ## Optional: uncomment the line below if you wish to override the default profile path where Apptainer images will be stored ## Example path provided assigns a shared location for all collaborators of a project. Change as needed ## export NXF_APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE}/apptainer-images" # Nextflow-specific Variables ## Nextflow home, cache, and temporary directories export NXF_HOME="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow" export NXF_CACHE_DIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow" export NXF_TEMP="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow/tmp" mkdir -p "${NXF_TEMP}" ## Limit memory allocation for the Nextflow JVM export NXF_OPTS="-Xms1g -Xmx4g" ## Optional: Temporary workaround for Nextflow strict syntax ## Uncomment the line below, if the pipeline you are using is not compatible with NXF_SYNTAX_PARSER=v2 (default in Nextflow >=26.04) to force the legacy v1 parser ## export NXF_SYNTAX_PARSER=v1 # Define your workspace directory (i.e. where output will be saved) // Adjust as needed MY_NXF_WORKSPACE="${PROJ_SCRATCHDIR_CHARITE_USER}/nxf-workspace" mkdir -p "${MY_NXF_WORKSPACE}"
Option 1: Save to
${HOME}/setup-nxf.sh. With this method, you must then runsource ${HOME}/setup-nxf.shevery time you start a new interactive session.Option 2: Add to your
${HOME}/.bashrc. With this method, you must then runsource ${HOME}/.bashrconly once, or simply start a new interactive session; the variables are then already exported into your environment (PS: be mindful of what you add into your${HOME}/.bashrcfile, as it may unexpectedly alter the expected behavior of other existing applications).Update the value of
PROJ_ABBRif you have project scratch available to youOptional: Update the path to your workspace via
MY_NXF_WORKSPACE- Recommended is to keep it set to${PROJ_SCRATCHDIR_CHARITE_USER}/nxf-workspaceOptional: Uncomment
export NXF_APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE}/apptainer-images"if you wish to override the default path of the images cacheOptional: Uncomment
export NXF_SYNTAX_PARSER=v1if the pipeline you are using is not compatible with the v2 strict parser
# Navigate to the workspace
cd "${MY_NXF_WORKSPACE}"
# Launch the pipeline
nextflow run nf-core/demo \
-profile test,charite \
--outdir results/demo-test
# -r <version> \ # Optional: pin a specific revision of a pipeline
# -resume \ # Optional: resume an interrupted workflow
# -c custom.config \ # Optional: add an extra config file which contains additional user-defined parameters
Option 2: Batch Script
First, ssh into a frontend node via your terminal as you normally would (see Access).
If setting up for the first time, create a file with the name run-nf-core-demo-test.sh in your home directory, which will be submitted via sbatch run-nf-core-demo-test.sh.
Batch Script Checklist
[!NOTE]
>>> Click here to reveal the hidden code snippet and copy it for your Batch script! <<<
#!/usr/bin/env bash #SBATCH --job-name=run-nf-core-demo-test #SBATCH --partition=compute #SBATCH --qos=normal #SBATCH --ntasks=1 #SBATCH --cpus-per-task=2 #SBATCH --mem=8G #SBATCH --time=00:30:00 #SBATCH --error=slurm/error/%x.%j.err #SBATCH --output=slurm/output/%x.%j.out # Initialize Conda eval "$(conda shell.bash hook)" # Activate the environment conda activate nf-core-2026-1 # Global Utility Variables ## Project abbreviation (leave empty if you don't have one) export PROJ_ABBR="" if [[ -n "${PROJ_ABBR}" ]]; then export PROJ_SCRATCHDIR_CHARITE="/sc-scratch/sc-scratch-${PROJ_ABBR}" # Validate project scratch directory if [[ ! -d "${PROJ_SCRATCHDIR_CHARITE}" || ! -x "${PROJ_SCRATCHDIR_CHARITE}" ]]; then echo "[ERROR] Project scratch directory does not exist or is inaccessible: ${PROJ_SCRATCHDIR_CHARITE}" echo "[ERROR] Please verify your PROJ_ABBR ('${PROJ_ABBR}') and project access." exit 1 fi export PROJ_SCRATCHDIR_CHARITE_USER="${PROJ_SCRATCHDIR_CHARITE}/${USER}" else export PROJ_SCRATCHDIR_CHARITE_USER="${HOME}" fi # Fail-safe: ensure the required environment variable is not empty if [[ -z "${PROJ_SCRATCHDIR_CHARITE_USER:-}" ]]; then echo "[ERROR] PROJ_SCRATCHDIR_CHARITE_USER is not set or is empty." exit 1 fi # Apptainer-specific Variables # Apptainer cache and temporary files (user-specific) export APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer" export APPTAINER_TMPDIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.apptainer/tmp" mkdir -p "${APPTAINER_TMPDIR}" ## Optional: uncomment the line below if you wish to override the default profile path where Apptainer images will be stored ## Example path provided assigns a shared location for all collaborators of a project. Change as needed ## export NXF_APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE}/apptainer-images" # Nextflow-specific Variables ## Nextflow home, cache, and temporary directories export NXF_HOME="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow" export NXF_CACHE_DIR="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow" export NXF_TEMP="${PROJ_SCRATCHDIR_CHARITE_USER}/.nextflow/tmp" mkdir -p "${NXF_TEMP}" ## Limit memory allocation for the Nextflow JVM export NXF_OPTS="-Xms1g -Xmx4g" ## Optional: Temporary workaround for Nextflow strict syntax ## Uncomment the line below, if the pipeline you are using is not compatible with NXF_SYNTAX_PARSER=v2 (default in Nextflow >=26.04) to force the legacy v1 parser ## export NXF_SYNTAX_PARSER=v1 # Define your workspace directory (i.e. where output will be saved) // Adjust as needed # For example, if project scratch is available to you MY_NXF_WORKSPACE="${PROJ_SCRATCHDIR_CHARITE_USER}/nxf-workspace" mkdir -p "${MY_NXF_WORKSPACE}" # Navigate to the workspace cd "${MY_NXF_WORKSPACE}" # Do the work nextflow run nf-core/demo \ -profile test,charite \ --outdir results/demo-test # -r <version> \ # Optional: pin a specific revision of a pipeline # -resume \ # Optional: resume an interrupted workflow # -c custom.config \ # Optional: add an extra config file which contains additional user-defined parametersUpdate the value of
PROJ_ABBRif you have project scratch available to youUpdate
#SBATCH --timeto expected pipeline runtime (e.g. for a big dataset, consider setting to12:00:00or more)Optional: Add
#SBATCH --mail-user=<your-email>and#SBATCH --mail-type=ALLto get status updatesOptional: Update the path to your workspace via
MY_NXF_WORKSPACE- Recommended is to keep it set to${PROJ_SCRATCHDIR_CHARITE_USER}/nxf-workspaceOptional: Uncomment
export NXF_APPTAINER_CACHEDIR="${PROJ_SCRATCHDIR_CHARITE}/apptainer-images"if you wish to override the default path of the images cacheOptional: Uncomment
export NXF_SYNTAX_PARSER=v1if the pipeline you are using is not compatible with the v2 strict parser
If project scratch is available, future pipeline launches should be launched directly from your newly created MY_NXF_WORKSPACE folder (set here to ${PROJ_SCRATCHDIR_CHARITE_USER}/nxf-workspace), so that SLURM logs get saved there instead of in your home directory.
5. Job Assignment Logic and GPU Acceleration
| Queue | compute |
dragen_cc05 |
dragen_cc15 |
dragen_gp |
genomics |
highmem |
gpu |
pgpu |
|---|---|---|---|---|---|---|---|---|
| Max CPUs | 192 | 64 | 64 | 48 | 192 | 192 | 64 | 128 |
| Max Memory | 1 TB | 500 GB | 500 GB | 240 GB | 1 TB | 4 TB | 500 GB | 2 TB |
| Max Time | 2 days | 7 days | 7 days | 7 days | 7 days | 2 days | 2 days | 2 days |
[!NOTE] Max CPUs and Memory refer to the highest possible ceiling from all nodes within a partition (see Resource Limits), and not that this is all of the CPUs and Memory available for that partition. For more in-depth information about the number and types of nodes available on the Charité HPC, see the following internal Wiki links:
---
config:
flowchart:
wrappingWidth: 1000
---
flowchart TB
A(["Task Queue Routing"])
B["Task Label 'process_gpu'?"]
A --> B
subgraph ROUTING[" "]
direction LR
subgraph CPU["CPU routing"]
direction TB
F["Profile?"]
M["Memory Requested > 1.TB?\n (i.e. task.memory)"]
G["compute"]
H["dragen_cc05"]
I["dragen_cc15"]
J["dragen_gp"]
K["genomics"]
L["highmem"]
P["compute"]
F -->|"-profile charite"| G
F -->|"-profile charite,dragen_cc05"| H
F -->|"-profile charite,dragen_cc15"| I
F -->|"-profile charite,dragen_gp"| J
F -->|"-profile charite,genomics"| K
F -->|"-profile charite,highmem"| M
M -->|"No"|P
M -->|"Yes"|L
end
subgraph GPU["GPU routing"]
direction TB
C["GPUs Requested?\n (i.e. task.accelerator)"]
D["gpu"]
E["pgpu"]
C -->|"1"| D
C -->|"2 or more"| E
end
end
B -->|"No"| F
B -->|"Yes — any profile"| C
D --> N(["SLURM"])
E --> N
G --> N
H --> N
I --> N
J --> N
K --> N
L --> N
P --> N
classDef start fill:#dbeafe,stroke:#2563eb,color:#1e40af
classDef gpu fill:#dcfce7,stroke:#16a34a,color:#166534
classDef cpu fill:#dcfce7,stroke:#16a34a,color:#166534
classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f
classDef qos fill:#dbeafe,stroke:#2563eb,color:#1e40af
class A,N start
class D,E gpu
class G,H,I,J,K,L,P cpu
class B,C,F,M decision
style ROUTING fill:none,stroke:none
style GPU fill:none,stroke:none
style CPU fill:none,stroke:none
Pictured above is the logic that the config follows to determine to which partition SLURM should submit a job to. Regardless of the profile(s) chosen, jobs that can benefit from GPU acceleration will be handled accordingly and submitted either to gpu or pgpu, depending on how many GPUs are requested, thanks to the withLabel:process_gpu specification in conf/charite.config.
[!IMPORTANT]
“Order matters. Profiles load in sequence. Later profiles overwrite earlier ones.” (see Configuration Options for more info)
For example
nextflow run nf-core/rnaseq -profile charite,test ...is not the same as
nextflow run nf-core/rnaseq -profile test,charite ...The last example shown is how you should correctly test pipelines with the
testdataset provided by nf-core.
6. More Practical Examples
Each of the nf-core pipelines has different input requirements, so you should thoroughly read its corresponding documentation for a full overview on how to customize your run. Below are a few more examples on how to run some on the cluster:
- Run nf-core/rnaseq with test profile
nextflow run nf-core/rnaseq \
-profile test,charite \
--outdir results/rnaseq-test \
- Run nf-core/rnaseq with test profile and Parabricks GPU acceleration for STAR (see Jobs Pending in
gpuPartition for more info oncustom.config)
nextflow run nf-core/rnaseq \
-profile test,charite \
--outdir results/rnaseq-parabricks-star-test \
--use_parabricks_star \
-c custom.config
- Run nf-core/methylseq with test profile and Bismark aligner (default)
nextflow run nf-core/methylseq \
-profile test,charite \
--outdir results/methylseq-bismark-test
- Run nf-core/methylseq with test profile and BWA-Meth aligner (GPU) (see Jobs Pending in
gpuPartition for more info oncustom.config)
nextflow run nf-core/methylseq \
-profile test,gpu,charite \
--aligner bwameth \
--outdir results/methylseq-bwameth-test \
-c custom.config
7. Known Limitations and Issues
Jobs Pending in gpu Partition
Jobs requesting exactly 1 GPU may remain pending for submission to the gpu queue for a long time. The SC HPC Team is aware of this issue and is working on a solution. One temporary workaround is to force submission to the pgpu queue by requesting at least 2 GPUs.
process {
withLabel:process_gpu {
accelerator = 2
}
}
Save the snippet above to custom.config, and then run your pipeline like so:
nextflow run nf-core/<pipeline-name> -profile charite --outdir results/<pipeline-name> -c custom.config ...
See Custom Configuration Files to learn more.
Config file
// ---------------------------------------------// Nextflow nf-core config file for Charité HPC// ---------------------------------------------//// - Note on Usage// -------------------------------// Before using this config, please read the documentation (docs/charite.md) fully in order to setup your environment correctly// If the environment variable PROJ_SCRATCHDIR_CHARITE_USER is not defined, your pipeline launch with this config will fail// See docs/charite.md > "3. Environment Setup" to learn more// -------------------------------//// - Note on Nextflow Strict Syntax// -------------------------------// The migration of Nextflow to strict syntax has been taken into account during the creation of this config// It is therefore compatible with NXF_SYNTAX_PARSER=v2 (which is the default parser in Nextflow >=26.04)// -------------------------------
// Profile metadataparams { config_profile_contact = 'Scientific Computing, GB IT, Charité (@ScientificComputingCharite) | sc-hpc-helpdesk@charite.de' config_profile_description = 'Scientific Computing Charité HPC Cluster Profile | Developed by Wassim Salam (@wassimsalam01) | Maintained by Scientific Computing, GB IT, Charité (@ScientificComputingCharite)' config_profile_name = "charite" config_profile_url = "https://www.charite.de/en/research/research_support_services/research_infrastructure/science_it"
proj_scratchdir_charite_user = System.getenv("PROJ_SCRATCHDIR_CHARITE_USER")}
// Instruct Nextflow to ignore custom parametersvalidation { ignoreParams = ["proj_scratchdir_charite_user"]}
// Process defaults (see https://docs.seqera.io/nextflow/reference/process/directives for more info)process { clusterOptions = "--qos=normal" executor = "slurm" maxRetries = 3 queue = "compute" resourceLimits = [ cpus: 192, memory: 1.TB, time: 2.d ] scratch = true stageInMode = "symlink" stageOutMode = "rsync" withLabel:process_gpu { clusterOptions = { "--gres=gpu:${task.accelerator.request} --qos=normal" } containerOptions = "--nv" queue = { task.accelerator.request > 1 ? "pgpu" : "gpu" } resourceLimits = { task.accelerator.request > 1 ? [ cpus: 128, memory: 2.TB, time: 2.d ] : [ cpus: 64, memory: 500.GB, time: 2.d ] } }}
// Slurm defaults (see https://docs.seqera.io/nextflow/reference/config/executor for more info)executor { pollInterval = 10.s queueSize = 200 queueStatInterval = 2.m submitRateLimit = '2/1sec'}
// Environment variables (see https://docs.seqera.io/nextflow/reference/config/env for more info)env { APPTAINER_CACHEDIR = "${params.proj_scratchdir_charite_user}/.apptainer" APPTAINER_TMPDIR = "${params.proj_scratchdir_charite_user}/.apptainer/tmp"}
// Container engine defaults (see https://docs.seqera.io/nextflow/reference/config/apptainer for more info)// Specify where Apptainer will save built images.// 1) If "NXF_APPTAINER_CACHEDIR" is defined by the user as an environment variable, it will be used as the image directory.// 2) If not defined, the image directory used will default to "${params.proj_scratchdir_charite_user}/apptainer-images";// this is determined by the definition of the "PROJ_SCRATCHDIR_CHARITE_USER" variable.// See docs/charite.md for more info on how to set this variable.apptainer { autoMounts = true cacheDir = System.getenv("NXF_APPTAINER_CACHEDIR") ?: "${params.proj_scratchdir_charite_user}/apptainer-images" enabled = true pullTimeout = 2.h}
// Specify Nextflow's work directoryworkDir = "${params.proj_scratchdir_charite_user}/nxf-work"
// ----------------------// Pipeline run analytics// ----------------------
// Enable "dag" feature and allow overwriting of existing dag files (see https://docs.seqera.io/nextflow/reference/config/dag for more info)dag { enabled = true overwrite = true}
// Enable "report" feature and allow overwriting of existing report files (see https://docs.seqera.io/nextflow/reference/config/report for more info)report { enabled = true overwrite = true}
// Enable "trace" feature and allow overwriting of existing trace files (see https://docs.seqera.io/nextflow/reference/config/trace for more info)trace { enabled = true overwrite = true}
// Enable "timeline" feature and allow overwriting of existing timeline files (see https://docs.seqera.io/nextflow/reference/config/timeline for more info)timeline { enabled = true overwrite = true}// ----------------------
// Additional profiles// Some of the partitions below are not accessible to all user groups// If you are having trouble running jobs on a partition you should have access to, get in touch with sc-hpc-helpdesk@charite.deprofiles { dragen_cc05 { process { queue = "dragen_cc05" resourceLimits = [ cpus: 64, memory: 500.GB, time: 7.d ] } } dragen_cc15 { process { queue = "dragen_cc15" resourceLimits = [ cpus: 64, memory: 500.GB, time: 7.d ] } } dragen_gp { process { queue = "dragen_gp" resourceLimits = [ cpus: 48, memory: 240.GB, time: 7.d ] } } genomics { process { clusterOptions = "--exclude=s-sc-genomics[01]" // Reserved for demultiplexing queue = "genomics" resourceLimits = [ cpus: 192, memory: 1.TB, time: 7.d ] } } highmem { process { queue = { task.memory > 1.TB ? "highmem" : "compute" } resourceLimits = { task.memory > 1.TB ? [ cpus: 192, memory: 4.TB, time: 2.d ] : [ cpus: 192, memory: 1.TB, time: 2.d ] } } } // Disable automatic cleanup debug { cleanup = false }}
// Cleanup after successful runcleanup = true