Growing Pawsey expertise on container orchestration solutions

Opportunities to work with Kubernetes container orchestration, building on Pawsey’s existing container technology for its HPC systems, are some of the features offered by the new refreshed Nimbus cloud, available at Pawsey from April 1st.  The new high-throughput computing infrastructure represents a significant improvement from the original Nimbus Cloud, in CPU, memory and storage, providing enhanced computational flexibility, accessibility and speed.

To extend the current services to simpler and more contemporary frameworks and workflow managers, the Centre partnered with HPCNow! to upskill the operations and end-user support teams in the Kubernetes open-source container- orchestration system.

Kubernetes is a widely-used general-purpose container- orchestration solution for automating application deployment, scaling, and management. Moreover, it has evolved to result in a powerful solution for HPC and Artificial Intelligence (AI) workflows, suitable to complement and interact with the current supercomputing and HPC capabilities of Pawsey.

A 10-day Kubernetes training course was held at the Centre. The program was split into two groups of Pawsey attendees to ensure continuity of our service delivery. HPCNow! covered topics ranging from a generic Kubernetes vision through to HPC and AI-oriented batch job scheduling. By the end of the training course, Pawsey staff achieved the primary of productively interacting with a production-ready solution. Additionally, attendees were able to test a variety of on-premises solutions and to run prototype Nextflow and Kubeflow workflows on top of them.

“This training has been highly relevant for my role at Pawsey. I have learnt the key aspects of the Kubernetes world, including its jargon, components, and procedures. I now feel confident I can cooperate with our Operations team in the deployment of Kubernetes solutions. Most crucially, I am ready to assist researchers in porting their workflows onto Kubernetes clusters”, says Marco De La Pierre, Supercomputing Applications Specialist at Pawsey.

Knowledge and expertise in containerisation via Kubernetes are expected to play an important role within the bioinformatics community. Containerisation of software and research workflows have been identified as some of the techniques that can enable bioinformaticians, among other communities, to readily leverage HPC and the power provided through its effective use.

This training is one among several other development opportunities available to Pawsey staff during the $70 million Capital Refresh project. Pawsey continues to upskill its staff in alignment with the new technologies available and the latest trends on the HPC landscape. Previous offerings have focused on topics such as containers, workflow managers and GPU programming.