Adaptive data placement for staging-based coupled scientific workflows

Qian Sun, Tong Jin, Melissa Romanus, Hoang Bui, Fan Zhang, Hongfeng Yu, Hemanth Kolla, Scott Klasky, Jacqueline Chen, Manish Parashar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Citations (Scopus)

Abstract

Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches improve end-to-end performance by enabling runtime data sharing between coupled simulations and data analytics components of the workflow. However, the complex and dynamic data exchange patterns exhibited by the workflows coupled with the varied data access behaviors make efficient data placement within the staging area challenging. In this paper, we present an adaptive data placement approach to address these challenges. Our approach adapts data placement based on application-specific dynamic data access patterns, and applies access pattern-driven and location-aware mechanisms to reduce data access costs and to support efficient data sharing between the multiple workflow components. We experimentally demonstrate the effectiveness of our approach on Titan Cray XK7 using a real combustion-analyses workflow. The evaluation results demonstrate that our approach can effectively improve data access performance and overall efficiency of coupled scientific workflows.

Original languageEnglish (US)
Title of host publicationProceedings of SC 2015
Subtitle of host publicationThe International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherIEEE Computer Society
ISBN (Electronic)9781450337236
DOIs
StatePublished - Nov 15 2015
EventInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015 - Austin, United States
Duration: Nov 15 2015Nov 20 2015

Publication series

NameInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
Volume15-20-November-2015
ISSN (Print)2167-4329
ISSN (Electronic)2167-4337

Other

OtherInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015
CountryUnited States
CityAustin
Period11/15/1511/20/15

Fingerprint

Electronic data interchange
Costs

Keywords

  • adaptive data placement
  • coupled scientific workflows
  • data access pattern
  • data staging
  • in-situ/in-transit

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Cite this

Sun, Q., Jin, T., Romanus, M., Bui, H., Zhang, F., Yu, H., ... Parashar, M. (2015). Adaptive data placement for staging-based coupled scientific workflows. In Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis [a65] (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 15-20-November-2015). IEEE Computer Society. https://doi.org/10.1145/2807591.2807669

Adaptive data placement for staging-based coupled scientific workflows. / Sun, Qian; Jin, Tong; Romanus, Melissa; Bui, Hoang; Zhang, Fan; Yu, Hongfeng; Kolla, Hemanth; Klasky, Scott; Chen, Jacqueline; Parashar, Manish.

Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2015. a65 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 15-20-November-2015).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sun, Q, Jin, T, Romanus, M, Bui, H, Zhang, F, Yu, H, Kolla, H, Klasky, S, Chen, J & Parashar, M 2015, Adaptive data placement for staging-based coupled scientific workflows. in Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis., a65, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, vol. 15-20-November-2015, IEEE Computer Society, International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, Austin, United States, 11/15/15. https://doi.org/10.1145/2807591.2807669
Sun Q, Jin T, Romanus M, Bui H, Zhang F, Yu H et al. Adaptive data placement for staging-based coupled scientific workflows. In Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2015. a65. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). https://doi.org/10.1145/2807591.2807669
Sun, Qian ; Jin, Tong ; Romanus, Melissa ; Bui, Hoang ; Zhang, Fan ; Yu, Hongfeng ; Kolla, Hemanth ; Klasky, Scott ; Chen, Jacqueline ; Parashar, Manish. / Adaptive data placement for staging-based coupled scientific workflows. Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2015. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).
@inproceedings{45279ae8e220465989e010cf9e7e552d,
title = "Adaptive data placement for staging-based coupled scientific workflows",
abstract = "Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches improve end-to-end performance by enabling runtime data sharing between coupled simulations and data analytics components of the workflow. However, the complex and dynamic data exchange patterns exhibited by the workflows coupled with the varied data access behaviors make efficient data placement within the staging area challenging. In this paper, we present an adaptive data placement approach to address these challenges. Our approach adapts data placement based on application-specific dynamic data access patterns, and applies access pattern-driven and location-aware mechanisms to reduce data access costs and to support efficient data sharing between the multiple workflow components. We experimentally demonstrate the effectiveness of our approach on Titan Cray XK7 using a real combustion-analyses workflow. The evaluation results demonstrate that our approach can effectively improve data access performance and overall efficiency of coupled scientific workflows.",
keywords = "adaptive data placement, coupled scientific workflows, data access pattern, data staging, in-situ/in-transit",
author = "Qian Sun and Tong Jin and Melissa Romanus and Hoang Bui and Fan Zhang and Hongfeng Yu and Hemanth Kolla and Scott Klasky and Jacqueline Chen and Manish Parashar",
year = "2015",
month = "11",
day = "15",
doi = "10.1145/2807591.2807669",
language = "English (US)",
series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",
publisher = "IEEE Computer Society",
booktitle = "Proceedings of SC 2015",

}

TY - GEN

T1 - Adaptive data placement for staging-based coupled scientific workflows

AU - Sun, Qian

AU - Jin, Tong

AU - Romanus, Melissa

AU - Bui, Hoang

AU - Zhang, Fan

AU - Yu, Hongfeng

AU - Kolla, Hemanth

AU - Klasky, Scott

AU - Chen, Jacqueline

AU - Parashar, Manish

PY - 2015/11/15

Y1 - 2015/11/15

N2 - Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches improve end-to-end performance by enabling runtime data sharing between coupled simulations and data analytics components of the workflow. However, the complex and dynamic data exchange patterns exhibited by the workflows coupled with the varied data access behaviors make efficient data placement within the staging area challenging. In this paper, we present an adaptive data placement approach to address these challenges. Our approach adapts data placement based on application-specific dynamic data access patterns, and applies access pattern-driven and location-aware mechanisms to reduce data access costs and to support efficient data sharing between the multiple workflow components. We experimentally demonstrate the effectiveness of our approach on Titan Cray XK7 using a real combustion-analyses workflow. The evaluation results demonstrate that our approach can effectively improve data access performance and overall efficiency of coupled scientific workflows.

AB - Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches improve end-to-end performance by enabling runtime data sharing between coupled simulations and data analytics components of the workflow. However, the complex and dynamic data exchange patterns exhibited by the workflows coupled with the varied data access behaviors make efficient data placement within the staging area challenging. In this paper, we present an adaptive data placement approach to address these challenges. Our approach adapts data placement based on application-specific dynamic data access patterns, and applies access pattern-driven and location-aware mechanisms to reduce data access costs and to support efficient data sharing between the multiple workflow components. We experimentally demonstrate the effectiveness of our approach on Titan Cray XK7 using a real combustion-analyses workflow. The evaluation results demonstrate that our approach can effectively improve data access performance and overall efficiency of coupled scientific workflows.

KW - adaptive data placement

KW - coupled scientific workflows

KW - data access pattern

KW - data staging

KW - in-situ/in-transit

UR - http://www.scopus.com/inward/record.url?scp=84966495088&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84966495088&partnerID=8YFLogxK

U2 - 10.1145/2807591.2807669

DO - 10.1145/2807591.2807669

M3 - Conference contribution

AN - SCOPUS:84966495088

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2015

PB - IEEE Computer Society

ER -