P’Con – Exascale Data Management with ADIOS
9:30am - 11:30am
Speaker: Scott Klasky, Norbert Podhorszki (Oak Ridge National Laboratories)
ADIOS is a high performance publish/subscribe I/O framework which has been designed and developed for the exascale computing era. In a snapshot, ADIOS:
- Is integrated into most of the popular analysis and visualization packages.
- Has strict continuous integration practices, providing stable, portable and efficient I/O services.
- Has a programming interface designed for easy switching from files to streams (on-HPC machines) to streams over the Wide Area Network.
In this presentation, the speaker will introduce the ADIOS I/O framework and describe how it is being used, including:
- In data intensive supercomputing simulations, using it for data storage and retrieval moving petabytes of data in a single job,
- In coupled simulations using it for data movement between the simulation codes and to in situ analysis and in situ visualization services,
- In AI applications, for collecting training data from ensembles of computations on the fly, and
- In collaborations between experimental facilities and HPC centers to stream experimental data for near-real-time decision making.
We will present how ADIOS was used in the 2020 Gordon Bell finalist paper from Pawsey and ORNL about the workflow for simulating and processing the full-scale low-frequency telescope data of SKA Phase 1 on the Summit supercomputer at ORNL.
ADIOS is also a research framework for new I/O technologies, pushing the boundaries beyond current use cases. Our research focuses on data reduction with trusted lossy compression techniques, data refactoring for easier on-demand retrieval, hierarchical storage to speed up access to the most important data, asynchronous I/O techniques and others.
The techniques in ADIOS have been developed for over 15 years through collaborations with science applications, and we are always seeking new collaborations and new challenges that will define the I/O landscape of the future.
We hope this presentation will allow the audience to learn about how others are using extreme I/O in their fields and also stimulate the audience to bring new challenges to us.
Exascale Data Management with ADIOS session, is part of the first PaCER Conference – P’con: A week where Pawsey continues setting the pace for exascale.
Registration is needed