Setting up a pipeline sync retrospectively

Use this guide to set up a correct TEMPLATE branch for pipelines that were not created with nf-core pipelines create. If you created your pipeline with nf-core pipelines create, your TEMPLATE branch is already configured correctly.

Alternatively, consider restarting your pipeline project by running nf-core pipelines create and copying your modifications into the newly created pipeline.

Create the TEMPLATE branch

1. Clone your pipeline into a new directory:

   mkdir <temp-directory>
   cd <temp-directory>
   git clone https://github.com/nf-core/<pipeline-name>.git
   cd <pipeline-name>

2. Create the new TEMPLATE branch and delete all files to create a completely empty branch:

   git checkout --orphan TEMPLATE && git rm -rf '*'

3. Verify your branch is completely empty:

   git status

   You should see:

   On branch TEMPLATE
   
   No commits yet
   
   nothing to commit (create/copy files and use "git add" to track)

4. Regenerate your pipeline from scratch using the most recent template:

   nf-core pipelines create --no-git

   If your pipeline already has versioned releases (for example, you are not currently on 1.0dev), specify the version number:

   nf-core pipelines create --no-git --version 1.3dev

   Note

   The version you choose should match the branch you intend to merge with. If you already have a release, use the version number specified in your dev branch.

5. Follow the prompts to enter the pipeline name, description, and authors. Use the exact text from your existing nextflow.config file (manifest.name etc.).

6. Move the newly created template files into your root git directory:

   mv nf-core-<pipeline-name>/* .
   mv nf-core-<pipeline-name>/.[!.]* .
   rmdir nf-core-<pipeline-name>

7. Verify the newly created files are in the correct place:

   git status

   You should see output similar to:

   On branch TEMPLATE
   
   No commits yet
   
   Untracked files:
     (use "git add <file>..." to include in what will be committed)
   
     .editorconfig
     .gitattributes
     .github/
     .gitignore
     .markdownlint.yml
     CHANGELOG.md
     CITATIONS.md
     CODE_OF_CONDUCT.md
     LICENSE
     README.md
     assets/
     bin/
     conf/
     docs/
     lib/
     main.nf
     modules.json
     modules/
     nextflow.config
     nextflow_schema.json
     subworkflows/
     workflows/
   
   nothing added to commit but untracked files present (use "git add" to track)

8. Commit the template files:

   git add .
   git commit -m "Initial template commit"

9. Push the TEMPLATE branch to the upstream nf-core repository:

   git push --set-upstream origin TEMPLATE

Merge TEMPLATE into main branches

Merge the TEMPLATE branch into your main pipeline branches. This requires manually resolving all merge conflicts.

1. Check out your development branch and create a new branch for the merge:

   git checkout dev
   git checkout -b template_merge

2. Merge the TEMPLATE branch:

   git merge TEMPLATE --allow-unrelated-histories

   Note

   If the merge command shows entire files as new, try adding the -Xignore-space-at-eol flag.

3. Resolve merge conflicts. You may see many conflicts:

   Auto-merging nextflow.config
   CONFLICT (add/add): Merge conflict in nextflow.config
   Auto-merging main.nf
   CONFLICT (add/add): Merge conflict in main.nf
   Auto-merging environment.yml
   CONFLICT (add/add): Merge conflict in environment.yml
   Auto-merging docs/usage.md
   CONFLICT (add/add): Merge conflict in docs/usage.md

   Go through each file to resolve the merge conflicts. Use a visual merge tool to avoid mistakes when handling many merge markers.

4. Commit and push the resolved changes:

   git commit -m "Merged vanilla TEMPLATE branch into main pipeline"
   git push origin template_merge

5. Create a pull request from your template_merge branch to the main nf-core repository.

6. Once merged into the dev branch, future automatic template syncs will work correctly. When nf-core releases new versions of nf-core/tools, pull requests will automatically be created to merge updates into your pipeline.