A key objective of the Visualization and Data Analysis project is to develop new visualization libraries and systems to meet capability requirements for ASC simulations. This work is required to address ASC workloads: massive data sizes, complex results, and the use of unique supercomputing architectures. ASC simulations are currently producing massive amounts of data that threaten to outstrip the ability to visualize and analyze it. Therefore, it is important to understand how to triage data within the simulation as it is generated using techniques such as in-situ analysis for data reduction and visualization kernels that run on the supercomputing platform, including data analysis, visualization, and rendering methods.
The ASC Production project supports a wide range of production activities for the LANL ASC program. It helps users understand their data with visualization and data analysis on exceptionally large data. It provides software on supercomputers, support and training for that software, and can produce analyses for users as well. It also identifies needed visualization and data analysis research and development tasks in order to facilitate needs of end users. ASC users can get routed to us through the normal LANL supercomputing support channels, or they can call us directly.
ASC Burst Buffer
The ASC Burst Buffer Next Generation Computing Technology project is aimed at high-risk, high-reward investigations that can enable ASC codes on new system architectures. The CSSE Burst Buffer investigation will look into alternative hierarchical storage technologies for support of in situ analysis, out-of-core processing, and data management. With the progressive march towards ever larger and faster HPC platforms, the discrepancy between the bandwidth available to the collective set of compute nodes and the bandwidth available on traditional parallel file systems utilizing hard disks has become mismatched. Rather than purchasing additional disks to increase bandwidth (beyond what is required for capacity requirements), the concept of a burst buffer has been proposed to impedance match the compute nodes to the storage system. A burst buffer is an allocation of fast, likely solid state, storage that is capable of absorbing a burst of I/O activity, which can then be slowly drained to a parallel file system while computation resumes within the running application. The original design intent of such a system was to handle the I/O workload commonly seen in the checkpoint-restart process which many current HPC applications use to handle faults. In ASC CSSE R&D, the CCS-7 data science at scale team has been exploring alternative uses for burst buffers to support Trinity, the first supercomputer to utilize burst buffer systems. We seek to use Trinity burst-buffers to improve scale-out performance of visualization and analysis tasks, which are typically I/O bound. This scale-out performance requires innovative design changes in the supercomputer architecture, such as burst buffers, and changes in the typical HPC analysis workflow, from traditional post-processing, to new in situ and in transit analytics. For example, the development of an ‘in-transit’ solution for data analytics, that utilize burst buffer technology, requires an understanding of the burst buffer’s capabilities. In support of this, a prototype burst buffer system has been constructed using estimates of expected technology and deployment strategy from the LANL Trinity project. Initial testing was completed on this prototype burst buffer in order to determine bandwidth capabilities (for both file read and write) under various I/O conditions, to determine how quickly data could be ingested and read back into memory for use, along with testing under analysis workload conditions. Future work will continue development on prototype Trinity hardware to develop new workflows to support exascale supercomputing.
PINION is a portable, data-parallel software framework for physics simulations. PINION data structures allow scientists to program in a way that maps easily to the problem domain, while PINION operators provide data-parallel implementations of analysis and computational functions often used in physics simulations. Backend implementations of data parallel primitives are being optimized for emerging supercomputer hardware architectures such as Intel Xeon Phi (MIC).
The fundamental exascale and ‘big data’ analysis challenge is that there is simply too much data and too little cognitive (human capacity for understanding) and computational bandwidth. In situ and online techniques that process and reduce data at the source are a promising approach forward. We propose to explore and evaluate two quantitative, sampling-based data-reduction approaches. The first is to intelligently down-scale the ‘raw’ data, producing reduced data representations for later visualization and analysis. The second is to intelligently sample the set of possible visualizations and analyses that can be generated, producing a database of selected data products for exploration. These intelligent data reduction approaches will significantly reduce the need for data movement, thus reducing associated supercomputer power costs, and increase scientific productivity by directing eyeballs to selected data.
ASCR Scalable Data Management, Analysis, and Visualization (SDAV) VTK-m
The goal of the SDAV VTK-m project within the Scalable Data Management, Analysis, and Visualization (SDAV) SciDAC Institute is to deliver multi-/many-core enabled visualization and analysis algorithms to scientific codes. VTK-m leverages the strengths of the Dax project at Sandia National Laboratory, the EAVL project at Oak Ridge National Laboratory, and the PISTON project at Los Alamos National Laboratory. The PISTON component of VTK-m focuses on developing data-parallel algorithms that are portable across multi-core and many-core architectures for use by LCF codes of interest, such as the HACC cosmology code. PISTON consists of a library of visualization and analysis algorithms implemented using NVIDIA’s Thrust library, as well as our set of extensions to Thrust. PISTON algorithms are integrated into LCF codes in-situ either directly or though ParaView Catalyst.
ASCR Co-Design Center for Exascale Simulation of Advanced Reactors (CESAR)
The Center for Exascale Simulation of Advanced Reactors (CESAR) is one of three Department of Energy-funded Co-Design Centers. The goal of CESAR is twofold: to both drive architectural decisions and adapt algorithms to the next generation HPC computer architectures on the path to exascale systems. CESAR’s particular focus is on the algorithms that underlie the high-fidelity analysis of nuclear reactors: namely, neutron transport (Boltzmann and Monte Carlo) and conjugate heat transfer (Navier Stokes). Thus, the CESAR co-design process involves continually evaluating complex architectural and algorithmic tradeoffs aimed ultimately at the design of both exascale computers and algorithms that can efficiently leverage them. LANL’s primary involvement is through the co-design of visualization and analysis codes at scale. When coupling two different mesh-based codes, for example with in situ analytics, the typical strategy is to explicitly copy data (deep copy) from one implementation to another, doing translation in the process. This is necessary because codes usually do not share data model interfaces or implementations. The drawback is that data duplication results in an increased memory footprint for the coupled code. An alternative strategy, which we study in this project, is to share mesh data through on-demand, fine-grained, run-time data model translation. This saves memory, which is an increasingly scarce resource at exascale, for the increased use of in situ analysis and decreasing memory per core. We study the performance of our method compared against a deep copy with in situ analysis at scale.
ASCR SciDac Plasma Surface Interactions (PSI)
Gaining physics understanding and predictive capabilities to describe the evolution of plasma facing components (PFC) requires simultaneously addressing complex and diverse physics occurring over a wide range of length and time scales, as well as integrating extensive physical processes across the plasma – surface – bulk materials boundaries. This requires development not only of detailed physics models and computational strategies at each of these scales, but computer science algorithms and methods to strongly couple them in a way that can be robustly validated through comparisons against available data and new experiments. Therefore, the objective of this project is to develop robust, high-fidelity simulation tools capable of predicting the PFC operating lifetime and the PFC impact on plasma contamination, recycling of hydrogenic species, and tritium retention in future magnetic fusion devices, with a focus on tungsten based material systems. Deploying these tools requires the development of a leadershipscale computational code, as well as a host of simulations that span the multiple scales needed to address complex physical and computational issues at the plasma – surface interface and the transition below the surface where neutron damage processes in the bulk material dominate behavior in multiple-component materials systems. Successful development will enable improved prediction of PFC performance needed to ensure magnetic fusion energy development beyond ITER.
ASCR SciDAC Cosmology
Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model — dark matter and dark energy — remain mysterious. Next-generation sky surveys will map billions of galaxies to explore the physics of the ‘Dark Universe’. Science requirements for these surveys demand simulations at extreme scales; these will be delivered by the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework. HACC’s novel algorithmic structure allows tuning across diverse architectures, including accelerated and multi-core systems, such as LANL’s Roadrunner in the past, and Argonne’s BG/Q in the present. HACC simulations at these scales will for the first time enable tracking individual galaxies over the entire volume of a cosmological survey. Analysing the results from future cosmological surveys, which promise measurements at the percent level accuracy, will be a daunting task. Theoretical predictions have to be at least as accurate, preferably even more accurate than the measurements themselves. For large scale structure probes, which explore the nonlinear regime of structure formation, this level of accuracy can only be achieved with costly high-performance simulations. Unlike in CMB analysis where one can generate a large number (tens to hundred thousand) of power spectra with fast codes like CAMB relatively easily this is impossible for large scale structure simulations. It is therefore important to develop fast prediction tools — emulators — based on a relatively small number of high-precision simulations which replace the simulator in the analysis work. To understand the emulator outputs is a daunting task itself, as they are multi-dimensional inputs which result in multi-dimensional outputs. The CCS-7 Data at Science Scale team is working on developing high-dimensional visualization tools to help with the exploration task of understanding the emulation process and science data. Images shown here are examples of a prototypical visualization system that connects directly to the emulator. Here, scientists are able to interactively use a wide variety of visualization techniques to understand the cosmological data, parameter study, and sensitivities of different cosmological parameters.
BER Accelerated Climate Modeling for Energy (ACME)
The Accelerated Climate Modeling for Energy (ACME) project is a newly launched project sponsored by the Earth System Modeling (ESM) program within U.S. Department of Energy’s (DOE’s) Office of Biological and Environmental Research. ACME is an unprecedented collaboration among eight national laboratories and six partner institutions to develop and apply the most complete, leading-edge climate and Earth system models to challenging and demanding climate-change research imperatives. It is the only major national modeling project designed to address DOE mission needs to efficiently utilize DOE leadership computing resources now and in the future.
LDRD Mesoscale Materials Science of Ductile Damage in Four Dimensions
The recent development of novel synchrotron x-ray diffraction techniques is now enabling in-situ 3-D characterization of polycrystalline materials, providing tomography, crystal orientation fields and local stress mapping. Here we report our initial efforts to draw upon these techniques, integrating and combining them with appropriate modeling and data analysis formulations, to discover relationships between microstructure and ductile damage in polycrystalline aggregates. Our ultimate goal is to develop a predictive tool for materials discovery that can master defect meso-structure and its evolution to control structural properties.
Build and Execution Environment
The goal of Build and Execution Environment (BEE) is to create a unified software stack to containerize HPC applications. BEE utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers define the runtime that isolates all software dependency from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. Containers are placed in open-source virtual machine (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space.
The ALPINE proposal has four major development areas:
- Deliver exascale visualization and analysis algorithms that will be critical for ECP Applications as the dominant analysis paradigm shifts from post hoc (post-processing) to in situ (processing data in a code as it is generated).
- Deliver an exascale-capable infrastructure for the development of in situ algorithms and deployment into existing applications, libraries, and tools.
- Engage with ECP Applications to integrate our algorithms and infrastructure into their software.
- Engage with ECP Software Technologies to integrate their exascale software into our infrastructure.
We believe these focus areas are critically important to the success of ECP data analysis and visualization effort and therefore proposal team members are assigned to two of these four areas, one from a capability building area (algorithms or infrastructure) and one from an integration area (application or software technology). Many high performance simulation codes are using post hoc processing, meaning they write data to disk and then visualize and analyze it afterwards. Given exascale I/O constraints, in situ processing will be necessary. In situ data analysis and visualization selects, analyzes, reduces, and generates extracts from scientific simulation results during the simulation runs to overcome bandwidth and storage bottlenecks associated with writing out full simulation results to disk. Since in situ processing is in “early days” in terms of production usage within the DOE, we need to add new types of in situ algorithms. For example, we will develop algorithms that can automate which data analysis and visualization routines are applied and how they are applied to focus on the most important aspects of the simulation. We will also develop algorithms that can transform simulation data in a way that massively reduces it and yet allows for the integrity of the underlying information to be preserved. Our team will deploy these algorithms to ECP application scientists via our infrastructure.