Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows

Tong Jin, Fan Zhang, Qian Sun, Hoang Bui, Manish Parashar, Hongfeng Yu, Scott Klasky, Norbert Podhorszki, Hasan Abbasi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Citations (Scopus)

Abstract

As system scales and application complexity grow, managing and processing simulation data has become a signif-icant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management require-ments. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic place-ment and scheduling of data processing kernels, and (3) dy-namic allocation of in-transit resources. We also exploit co-ordinated approaches that dynamically combine these adap-tations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the In-trepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and in-creasing resource efficiency.

Original languageEnglish (US)
Title of host publicationProceedings of SC 2013
Subtitle of host publicationThe International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherIEEE Computer Society
ISBN (Print)9781450323789
DOIs
StatePublished - Jan 1 2013
Event2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013 - Denver, CO, United States
Duration: Nov 17 2013Nov 22 2013

Publication series

NameInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
ISSN (Print)2167-4329
ISSN (Electronic)2167-4337

Other

Other2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013
CountryUnited States
CityDenver, CO
Period11/17/1311/22/13

Fingerprint

Information management
Scheduling
Processing

Keywords

  • Coupled simulation workows
  • Cross-layer adaptation
  • Data management
  • In-situ/in-transit
  • Staging

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Cite this

Jin, T., Zhang, F., Sun, Q., Bui, H., Parashar, M., Yu, H., ... Abbasi, H. (2013). Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows. In Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis [74] (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). IEEE Computer Society. https://doi.org/10.1145/2503210.2503301

Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows. / Jin, Tong; Zhang, Fan; Sun, Qian; Bui, Hoang; Parashar, Manish; Yu, Hongfeng; Klasky, Scott; Podhorszki, Norbert; Abbasi, Hasan.

Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2013. 74 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jin, T, Zhang, F, Sun, Q, Bui, H, Parashar, M, Yu, H, Klasky, S, Podhorszki, N & Abbasi, H 2013, Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows. in Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis., 74, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, IEEE Computer Society, 2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, Denver, CO, United States, 11/17/13. https://doi.org/10.1145/2503210.2503301
Jin T, Zhang F, Sun Q, Bui H, Parashar M, Yu H et al. Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows. In Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2013. 74. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). https://doi.org/10.1145/2503210.2503301
Jin, Tong ; Zhang, Fan ; Sun, Qian ; Bui, Hoang ; Parashar, Manish ; Yu, Hongfeng ; Klasky, Scott ; Podhorszki, Norbert ; Abbasi, Hasan. / Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows. Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2013. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).
@inproceedings{fbe5b8341f1b4290ae1685e7a5f8e198,
title = "Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows",
abstract = "As system scales and application complexity grow, managing and processing simulation data has become a signif-icant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management require-ments. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic place-ment and scheduling of data processing kernels, and (3) dy-namic allocation of in-transit resources. We also exploit co-ordinated approaches that dynamically combine these adap-tations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the In-trepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and in-creasing resource efficiency.",
keywords = "Coupled simulation workows, Cross-layer adaptation, Data management, In-situ/in-transit, Staging",
author = "Tong Jin and Fan Zhang and Qian Sun and Hoang Bui and Manish Parashar and Hongfeng Yu and Scott Klasky and Norbert Podhorszki and Hasan Abbasi",
year = "2013",
month = "1",
day = "1",
doi = "10.1145/2503210.2503301",
language = "English (US)",
isbn = "9781450323789",
series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",
publisher = "IEEE Computer Society",
booktitle = "Proceedings of SC 2013",

}

TY - GEN

T1 - Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows

AU - Jin, Tong

AU - Zhang, Fan

AU - Sun, Qian

AU - Bui, Hoang

AU - Parashar, Manish

AU - Yu, Hongfeng

AU - Klasky, Scott

AU - Podhorszki, Norbert

AU - Abbasi, Hasan

PY - 2013/1/1

Y1 - 2013/1/1

N2 - As system scales and application complexity grow, managing and processing simulation data has become a signif-icant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management require-ments. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic place-ment and scheduling of data processing kernels, and (3) dy-namic allocation of in-transit resources. We also exploit co-ordinated approaches that dynamically combine these adap-tations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the In-trepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and in-creasing resource efficiency.

AB - As system scales and application complexity grow, managing and processing simulation data has become a signif-icant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management require-ments. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic place-ment and scheduling of data processing kernels, and (3) dy-namic allocation of in-transit resources. We also exploit co-ordinated approaches that dynamically combine these adap-tations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the In-trepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and in-creasing resource efficiency.

KW - Coupled simulation workows

KW - Cross-layer adaptation

KW - Data management

KW - In-situ/in-transit

KW - Staging

UR - http://www.scopus.com/inward/record.url?scp=84899670896&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899670896&partnerID=8YFLogxK

U2 - 10.1145/2503210.2503301

DO - 10.1145/2503210.2503301

M3 - Conference contribution

SN - 9781450323789

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2013

PB - IEEE Computer Society

ER -