Conference Paper

24/7 Characterization of petascale I/O workloads

Math. & Comput. Sci. Div., Argonne Nat. Lab., Argonne, IL, USA
DOI: 10.1109/CLUSTR.2009.5289150 Conference: Cluster Computing and Workshops, 2009. CLUSTER '09. IEEE International Conference on
Source: IEEE Xplore


Developing and tuning computational science applications to run on extreme scale systems are increasingly complicated processes. Challenges such as managing memory access and tuning message-passing behavior are made easier by tools designed specifically to aid in these processes. Tools that can help users better understand the behavior of their application with respect to I/O have not yet reached the level of utility necessary to play a central role in application development and tuning. This deficiency in the tool set means that we have a poor understanding of how specific applications interact with storage. Worse, the community has little knowledge of what sorts of access patterns are common in today's applications, leading to confusion in the storage research community as to the pressing needs of the computational science community. This paper describes the Darshan I/O characterization tool. Darshan is designed to capture an accurate picture of application I/O behavior, including properties such as patterns of access within files, with the minimum possible overhead. This characterization can shed important light on the I/O behavior of applications at extreme scale. Darshan also can enable researchers to gain greater insight into the overall patterns of access exhibited by such applications, helping the storage community to understand how to best serve current computational science applications and better predict the needs of future applications. In this work we demonstrate Darshan's ability to characterize the I/O behavior of four scientific applications and show that it induces negligible overhead for I/O intensive jobs with as many as 65,536 processes.

Download full-text


Available from: Katherine Riley, Aug 04, 2015
  • Source
    • "That said, there are library calls available that can query various aspects of memory usage, such as the size of the text, data, and BSS segments, together with the maximum amount of memory consumed on the heap. We also use Darshan [3], a resource monitoring tool with two major design points. First, it was explicitly aimed at parallel I/O, since there are no well accepted tools for doing so. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Robust high throughput computing requires effective monitoring and enforcement of a variety of resources including CPU cores, memory, disk, and network traffic. Without effective monitoring and enforcement, it is easy to overload machines, causing failures and slowdowns, or underload machines, which results in wasted opportunities. This paper explores how to describe, measure, and enforce resources used by computational tasks. We focus on tasks running in distributed execution systems, in which a task requests the resources it needs, and the execution system ensures the availability of such resources. This presents two non-trivial problems: how to measure the resources consumed by a task, and how to monitor and report resource exhaustion in a robust and timely manner. For both of these tasks, operating systems have a variety of mechanisms with different degrees of availability, accuracy, overhead, and intrusiveness. We develop a model to describe various forms of monitoring and map the available mechanisms in contemporary operating systems to that model. Based on this analysis, we present two specific monitoring tools that choose different tradeoffs in overhead and accuracy, and evaluate them on a selection of benchmarks. We conclude by describing our experience in collecting large quantities of monitoring data for complex workflows.
    2nd Workshop on Monitoring and Analysis for High Performance Computing Systems Plus Applications, IEEE Cluster 2015, Chicago, IL; 01/2015
  • Source
    • "Darshan's minimal collection of data (1-2% overhead, depending on the app [10][23]) allows it to be enabled for all jobs by default. This allows us to observe a platform at workload scale and to identify its jobs and apps that can most benefit from follow-up analyses with I/O tracing and other performance analysis tools. "
    [Show abstract] [Hide abstract]
    ABSTRACT: We examine the I/O behavior of thousands of supercomputing applications " in the wild, " by analyzing the Darshan logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. We mined these logs to analyze the I/O behavior of applications across all their runs on a platform; the evolution of an application's I/O behavior across time, and across platforms; and the I/O behavior of a platform's entire workload. Our analysis techniques can help developers and platform owners improve I/O performance and I/O system utilization, by quickly identifying underperforming applications and offering early intervention to save system resources. We summarize our observations regarding how jobs perform I/O and the throughput they attain in practice.
    HPDC; 01/2015
  • Source
    • "These bursts result from defensive I/O strategies (e.g., checkpoints that can be used to restart calculations following a system fault) or storage of simulation output for subsequent analysis (e.g., recording time series data for use in visualization). To quantify this behavior on Intrepid, we analyzed one month of production I/O activity from December 2011 using the Darshan lightweight I/O characterization tool [9]. Darshan captures application-level access pattern information with per process and per file granularity. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The largest-scale high-performance (HPC) systems are stretching parallel file systems to their limits in terms of aggregate bandwidth and numbers of clients. To further sustain the scalability of these file systems, researchers and HPC storage architects are exploring various storage system designs. One proposed storage system design integrates a tier of solid-state burst buffers into the storage system to absorb application I/O requests. In this paper, we simulate and explore this storage system design for use by large-scale HPC systems. First, we examine application I/O patterns on an existing large-scale HPC system to identify common burst patterns. Next, we describe enhancements to the CODES storage system simulator to enable our burst buffer simulations. These enhancements include the integration of a burst buffer model into the I/O forwarding layer of the simulator, the development of an I/O kernel description language and interpreter, the development of a suite of I/O kernels that are derived from observed I/O patterns, and fidelity improvements to the CODES models. We evaluate the I/O performance for a set of multiapplication I/O workloads and burst buffer configurations. We show that burst buffers can accelerate the application perceived throughput to the external storage system and can reduce the amount of external storage bandwidth required to meet a desired application perceived throughput goal.
    Proceedings of the 2012 IEEE Conference on Massive Data Storage; 04/2012
Show more