Pawsey Capital Refresh Project
In 2018 the Australian Government awarded $70 million to upgrade Pawsey’s supercomputing infrastructure, on top of the $80 million granted in 2009 to establish a petascale supercomputing facility.
The Pawsey upgrade, as a major part of the national HPC infrastructure, is ensuring Australia continues to enable computationally-intense research.
Pawsey capital refresh is a complex upgrade will be a staged process. Some ancillary systems, including storage and network infrastructure, have been procured prior to the main system.
The delivery of the main supercomputing is expected to occur in two-phases.
The first phase will provide researchers with a system that is at least equivalent in capacity to what they are currently using, with the latest generation of processors and increased memory per node.
During this phase, researchers with an active allocation on Magnus will transition to the new system.
Phase one is due to be commissioned in mid-2021.
The second phase is expected to be in production by mid-2022. It will provide an exponential expansion in capacity and state-of-the-art technology.
You can see all the parts that make the Capital Refresh Project on the infographics displayed at the bottom of this page or they can be download here.
Monthly update
March 2020
The Pawsey Supercomputer System (PSS) Tender closed on 25 February and evaluation of the tenders has commenced.
The MWA Compute Cluster procurement was awarded to HPE during February. The new 78-node cluster will provide a dedicated system for astronomers to process in excess of 30 PB of MWA telescope data using Pawsey infrastructure. The new cluster will provide users with enhanced GPU capabilities to power AI, computational work, machine learning workflows and data analytics.
The RFQ for the procurement of the ASKAP ingest nodes was released 7 February 2020 and closed 18 March.
The RFI for long-term storage was released 7 February 2020 and closed 1 April.
The upgrade to the Nimbus high-throughput computing (HTC) infrastructure was completed in February. Early adopters will be trialing the upgrade during March and the migration of all users will commence in April 2020. Users can expect the upgraded Cloud Service to provide improved computational flexibility, accessibility and speed. The upgrade will allow researchers to process and analyse even larger amounts of data through additional object storage and the Kubernetes container orchestrator, building on Pawsey’s existing container technology for its HPC systems.
In parallel with the above activities the Pawsey team has been focusing on Change Management planning with the following plans on target to be finalised in March 2020:
- User Experience Change Management Plan (UXCMP)
- Training Plan
- User Workflow Migration Plan
- Communications Plan (updated)
Detailed planning has been carried out for the migration of users to the upgraded Nimbus High Throughput Computing (HTC). As mentioned above, early adopters will be accessing Nimbus HTC in March and access for all users (current and new) will be available in April. Training updates and the support documentation updates are being finalized at present. A new application portal has been developed to request access. The new platform is more friendly and will now capture Field of Research (FoR) codes.
The migration is (relatively) straight forward as existing instances can be migrated directly by each research group. Documentation will be released to guide the research groups through this process. If users need help to move or rebuild their instances on the new infrastructure and move any data across from their old instances, researchers will be able to apply for expert support through one of three sprints designed to help with migration. These sprints will run for 2 weeks each during April and May 2020.
Planning is also in the early stages for the migration to the MWA Compute Cluster which is expected to commence towards the end of Q2 2020. Pawsey staff are currently reviewing existing documentation to identify updates required and are looking to establish a working group with MWA to plan the details of the migration.
Finally, Pawsey is drafting and reviewing a COVID-19 response and management plan to deal with the issue as it develops. Measures are being put in place now to support business and project continuity.
News Stories
To read the stories published related to the project milestones referred to the information below:
- New Pawsey Nimbus Cloud infrastructure available for Australian researchers 10/03/2020
- HPE to deliver a dedicated system for astronomy needs 28/02/2020
- Pawsey Capital Refresh Boosts Cloud Infrastructure 21/11/2019
- Tender released for Australia’s new research supercomputer 14/11/2019
- Three times more storage and performance for SKA pathfinders 11/11/2019
- Pawsey Capital Refresh – Reference Groups Established 5/04/2019
- New funding to accelerate science and innovation 28/04/2018
Pawsey is committed to engage with its diverse stakeholders and keep it update regarding the procurement. Some of the channels the Centre has established to achieve this are the Pawsey user forums, Capital Refresh Update for potential vendors, Pawsey newsletters and more recently our podcasts.
You can access the outcomes from the Pawsey user forums via this link, read the Centre’s newsletters on our website.
Podcasts
You can listen to the Capital Refresh Podcast from the list below
- Episode 4: The year that was, the year ahead
- Episode 3: HTC Cloud Procurement
- Episode 2: Capital Refresh Status Update
- Episode 1: What is the Pawsey Capital Refresh?
Infographic
Find below an infographic regarding the current status of the Project (last updated on 28/02/2020). They can also be download here
Pawsey Capital Refresh Status
Tape library expansion Additional tape storage has been procured to expand the existing tape libraries from 50 to 63 Petabytes in each library. Long term storage Server and hard disk storage and networking will be refreshed, along with the tape library’s total maximum storage capacity. Both ingest and egress data transfer services will be upgraded, improving upload and download times. |
Pawsey is partnering with Dell EMC to expand its current cloud system with 5x more memory and 25x more storage to form a new cutting-edge flexible compute system. This expansion aims to better service emerging research areas and communities who benefit more from a high throughput compute. |
|
Astronomy high-speed storage: 3x more storage and performance. The existing Astro filesystem was expanded to service the MWA community. Powered by HPE, it has been upgraded to 2.7 PB of usable space and capable of reading/writing at 30 GB/s. The New buffer filesystem, a dedicated resource for ASKAP researchers, provides 3.7 PB of usable space and is capable of reading/writing at 40 GB/s. It is manufactured by Dell. High-speed storage filesystems Designed to deal with thousands of users accessing them at the same time, high-speed filesystems will be procured to increase speed and storage capability to general-purpose science. |
The 546 TeraFlops MWA cluster will be a resource better tuned to MWA’s needs, powered by HPE. Procured ahead of the Main Supercomputer, this cluster will allow ASKAP to use the full CPU partition of Galaxy. |
|
The high-speed interconnect ties all the pieces of the Pawsey ‘puzzle’ together. When procured, all parts will sit on the same fabric as first-class citizens, allowing Pawsey researchers to run their workflows quicker. |
The remote visualisation capability will be procured as part of the main supercomputer. When the new capabilities become available, researchers will be able to visualise their science in real-time, while being processed. This new capability will allow researchers to steer their visualisation while the data is processing and fine-tuned to the desired outcome. |
|
Delivered in two phases, the Pawsey Supercomputer phase 1 will be delivered by mid-2021, it will provide researchers with a system that is at least equivalent in capacity to what they are currently using. Phase 2 is expected to be in production by mid-2022; it will provide an exponential expansion in capacity and the latest state-of-the-art technology. |
Pawsey Data Workflow