EBI Codon Cluster Configuration
All nf-core pipelines have been successfully configured for use on the codon cluster at the European Bioinformatics Institute.
To use, run the pipeline with -profile ebi_codon. This will download and launch the ebi_codon.config which has been pre-configured with a setup suitable for the codon cluster.
You should not run Nextflow on the login nodes. You should submit a batch job that executes Nextflow.
Loading the required modules
Before running the pipeline you will need to load Nextflow and Singularity using the environment module system on the codon cluster. You can do this by issuing the commands below:
## Load Nextflow and Singularity environment modules
module purge
module load nextflow/22.10.1
module load singularityce/3.10.3
You may want to add those module load commands to your shell configuration file if you use them often.
Installing mamba
Run the following:
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
bash Mambaforge-$(uname)-$(uname -m).sh
Follow the instructions here for more details.
Setting up a suitable path for the Nextflow software cache
It is recommended to install conda environments and singularity containers in your /hps/software directory. To achieve this, add to your ~/.nextflow/config file the following lines:
singularity.cacheDir = "/hps/software/users/<group>/<user_id>/nextflow_software_cache/singularity"
conda.cacheDir = "/hps/software/users/<group>/<user_id>/nextflow_software_cache/conda"
Config file
/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Nextflow config file for EMBL-EBI Codon Cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Author: Saul Pierotti
Mail: saul@ebi.ac.uk
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*/
params {
config_profile_contact = "Saul Pierotti (@saulpierotti)"
config_profile_description = "The European Bioinformatics Institute HPC cluster (codon) profile"
config_profile_url = "https://www.ebi.ac.uk/"
}
singularity {
enabled = true
// the default is 20 minutes and fails with large images
pullTimeout = "3 hours"
}
process {
queue = {
if (task.time > 7.day && task.memory > 300.GB) {
"long_bigmem"
}
else if (task.memory > 300.GB) {
"bigmem"
}
else if (task.time > 7.day) {
"long"
}
else if (task.time > 1.day) {
"research"
}
else {
"short"
}
}
// this is to avoid errors for missing files due to shared filesystem latency
maxRetries = 3
errorStrategy = { task.exitStatus == 0 ? "retry" : "terminate" }
cache = "lenient"
afterScript = "sleep 60"
}
executor {
name = "lsf"
queueSize = 2000
submitRateLimit = "10/1sec"
exitReadTimeout = "30 min"
jobName = {
task.name
.replace("[", "(")
.replace("]", ")")
.replace(" ", "_")
}
}
homepage
contact
