Introduction to Nextflow for Data Intensive Pipelines
10:00am - 11:00am
This is a hybrid event – online and in-person at the Pawsey Supercomputing Research Centre
- Does your research require the setup of complex workflows to analyse ever growing amounts of data?
- Do you find it time consuming to write and maintain your computational pipelines, and to deploy them on different infrastructures?
- Has prototyping, testing and deploying your containerised workflow led to code repetition – making your workflow readable only to you?
Finally, what if you could use a framework to create quick prototypes by simply swapping configurations – enabling you to run the same workflow locally using Docker or on HPC using Slurm and Singularity. Would you be interested in learning more?
Nextflow is such a framework. Popular within the bioinformatics community, Nextflow can be used in any data intensive research domain, including planetary sciences, space sciences, radio astronomy, oceanography, geological sciences, and more. In this session, we’ll present a use case in planetary science.
The purpose of this event is to provide mostly, but not exclusively, HPC users with an overview on how to develop new and redevelop existing workflows using not only Nextflow but also industry accepted best practices, including code modularity and reuse, rapid prototyping and deployment, and platform independence. We will use a real use case of development and redevelopment of an HPC-focused workflow. Practical issues that commonly arise in such a project will be discussed.
Join us to learn more:
- Dr. Marco De La Pierre will present an overview and the fundamental concepts of the Nextflow framework.
- Mr. Kosta Servis will present his experience with Nextflow to refactor an existing bash/Docker/Singularity workflow.
- We’ll include a short presentation of the Crater Detection Algorithm workflow structure, the main goals and project history from black box magic to bash scripts to Nextflow.
- There will be a simple walkthrough using a basic example.
- We’ll close with your questions and an opportunity for you to share your experiences.
NOTE: This event is planned as a hybrid event. Choose your attendance preference during registration:
- In-person: For individuals located in Perth, you are invited to join us onsite at the Pawsey Supercomputing Research Centre. The Centre has attendance limits; please only register for in-person if you plan to attend.
- Online: For individuals who will not join in-person at Pawsey, we will livestream the event. A Zoom invite will be sent closer to the event date.