Adaptive data placement for staging-based coupled scientific workflows

Qian Sun, Tong Jin, Melissa Romanus, Hoang Bui, Fan Zhang, Hongfeng Yu, Hemanth Kolla, Scott Klasky, Jacqueline Chen, Manish Parashar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Scopus citations

Abstract

Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches improve end-to-end performance by enabling runtime data sharing between coupled simulations and data analytics components of the workflow. However, the complex and dynamic data exchange patterns exhibited by the workflows coupled with the varied data access behaviors make efficient data placement within the staging area challenging. In this paper, we present an adaptive data placement approach to address these challenges. Our approach adapts data placement based on application-specific dynamic data access patterns, and applies access pattern-driven and location-aware mechanisms to reduce data access costs and to support efficient data sharing between the multiple workflow components. We experimentally demonstrate the effectiveness of our approach on Titan Cray XK7 using a real combustion-analyses workflow. The evaluation results demonstrate that our approach can effectively improve data access performance and overall efficiency of coupled scientific workflows.

Original languageEnglish (US)
Title of host publicationProceedings of SC 2015
Subtitle of host publicationThe International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherIEEE Computer Society
ISBN (Electronic)9781450337236
DOIs
StatePublished - Nov 15 2015
EventInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015 - Austin, United States
Duration: Nov 15 2015Nov 20 2015

Publication series

NameInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
Volume15-20-November-2015
ISSN (Print)2167-4329
ISSN (Electronic)2167-4337

Other

OtherInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015
CountryUnited States
CityAustin
Period11/15/1511/20/15

    Fingerprint

Keywords

  • adaptive data placement
  • coupled scientific workflows
  • data access pattern
  • data staging
  • in-situ/in-transit

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Cite this

Sun, Q., Jin, T., Romanus, M., Bui, H., Zhang, F., Yu, H., Kolla, H., Klasky, S., Chen, J., & Parashar, M. (2015). Adaptive data placement for staging-based coupled scientific workflows. In Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis [a65] (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 15-20-November-2015). IEEE Computer Society. https://doi.org/10.1145/2807591.2807669