GoldRush: Resource efficient in situ scientific data analytics using fine-grained interference aware execution

Fang Zheng, Hongfeng Yu, Can Hantas, Matthew Wolf, Greg Eisenhauer, Karsten Schwan, Hasan Abbasi, Scott Klasky

Research output: Chapter in Book/Report/Conference proceedingConference contribution

37 Citations (Scopus)

Abstract

Severe I/O bottlenecks on High End Computing platforms call for running data analytics in situ. Demonstrating that there exist considerable resources in compute nodes un-used by typical high end scientific simulations, we leverage this fact by creating an agile runtime, termed GoldRush, that can harvest those otherwise wasted, idle resources to efficiently run in situ data analytics. GoldRush uses fine-grained scheduling to "steal" idle resources, in ways that minimize interference between the simulation and in situ analytics. This involves recognizing the potential causes of on-node resource contention and then using scheduling methods that prevent them. Experiments with representative science applications at large scales show that resources harvested on compute nodes can be leveraged to perform useful analytics, significantly improving resource efficiency, reducing data movement costs incurred by alternate solutions, and posing negligible impact on scientific simulations.

Original languageEnglish (US)
Title of host publicationProceedings of SC 2013
Subtitle of host publicationThe International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherIEEE Computer Society
ISBN (Print)9781450323789
DOIs
StatePublished - Jan 1 2013
Event2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013 - Denver, CO, United States
Duration: Nov 17 2013Nov 22 2013

Publication series

NameInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
ISSN (Print)2167-4329
ISSN (Electronic)2167-4337

Other

Other2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013
CountryUnited States
CityDenver, CO
Period11/17/1311/22/13

Fingerprint

Scheduling
Costs
Experiments

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Cite this

Zheng, F., Yu, H., Hantas, C., Wolf, M., Eisenhauer, G., Schwan, K., ... Klasky, S. (2013). GoldRush: Resource efficient in situ scientific data analytics using fine-grained interference aware execution. In Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis [78] (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). IEEE Computer Society. https://doi.org/10.1145/2503210.2503279

GoldRush : Resource efficient in situ scientific data analytics using fine-grained interference aware execution. / Zheng, Fang; Yu, Hongfeng; Hantas, Can; Wolf, Matthew; Eisenhauer, Greg; Schwan, Karsten; Abbasi, Hasan; Klasky, Scott.

Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2013. 78 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zheng, F, Yu, H, Hantas, C, Wolf, M, Eisenhauer, G, Schwan, K, Abbasi, H & Klasky, S 2013, GoldRush: Resource efficient in situ scientific data analytics using fine-grained interference aware execution. in Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis., 78, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, IEEE Computer Society, 2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, Denver, CO, United States, 11/17/13. https://doi.org/10.1145/2503210.2503279
Zheng F, Yu H, Hantas C, Wolf M, Eisenhauer G, Schwan K et al. GoldRush: Resource efficient in situ scientific data analytics using fine-grained interference aware execution. In Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2013. 78. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). https://doi.org/10.1145/2503210.2503279
Zheng, Fang ; Yu, Hongfeng ; Hantas, Can ; Wolf, Matthew ; Eisenhauer, Greg ; Schwan, Karsten ; Abbasi, Hasan ; Klasky, Scott. / GoldRush : Resource efficient in situ scientific data analytics using fine-grained interference aware execution. Proceedings of SC 2013: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2013. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).
@inproceedings{88acb273f54246b2b19d832f90328ece,
title = "GoldRush: Resource efficient in situ scientific data analytics using fine-grained interference aware execution",
abstract = "Severe I/O bottlenecks on High End Computing platforms call for running data analytics in situ. Demonstrating that there exist considerable resources in compute nodes un-used by typical high end scientific simulations, we leverage this fact by creating an agile runtime, termed GoldRush, that can harvest those otherwise wasted, idle resources to efficiently run in situ data analytics. GoldRush uses fine-grained scheduling to {"}steal{"} idle resources, in ways that minimize interference between the simulation and in situ analytics. This involves recognizing the potential causes of on-node resource contention and then using scheduling methods that prevent them. Experiments with representative science applications at large scales show that resources harvested on compute nodes can be leveraged to perform useful analytics, significantly improving resource efficiency, reducing data movement costs incurred by alternate solutions, and posing negligible impact on scientific simulations.",
author = "Fang Zheng and Hongfeng Yu and Can Hantas and Matthew Wolf and Greg Eisenhauer and Karsten Schwan and Hasan Abbasi and Scott Klasky",
year = "2013",
month = "1",
day = "1",
doi = "10.1145/2503210.2503279",
language = "English (US)",
isbn = "9781450323789",
series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",
publisher = "IEEE Computer Society",
booktitle = "Proceedings of SC 2013",

}

TY - GEN

T1 - GoldRush

T2 - Resource efficient in situ scientific data analytics using fine-grained interference aware execution

AU - Zheng, Fang

AU - Yu, Hongfeng

AU - Hantas, Can

AU - Wolf, Matthew

AU - Eisenhauer, Greg

AU - Schwan, Karsten

AU - Abbasi, Hasan

AU - Klasky, Scott

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Severe I/O bottlenecks on High End Computing platforms call for running data analytics in situ. Demonstrating that there exist considerable resources in compute nodes un-used by typical high end scientific simulations, we leverage this fact by creating an agile runtime, termed GoldRush, that can harvest those otherwise wasted, idle resources to efficiently run in situ data analytics. GoldRush uses fine-grained scheduling to "steal" idle resources, in ways that minimize interference between the simulation and in situ analytics. This involves recognizing the potential causes of on-node resource contention and then using scheduling methods that prevent them. Experiments with representative science applications at large scales show that resources harvested on compute nodes can be leveraged to perform useful analytics, significantly improving resource efficiency, reducing data movement costs incurred by alternate solutions, and posing negligible impact on scientific simulations.

AB - Severe I/O bottlenecks on High End Computing platforms call for running data analytics in situ. Demonstrating that there exist considerable resources in compute nodes un-used by typical high end scientific simulations, we leverage this fact by creating an agile runtime, termed GoldRush, that can harvest those otherwise wasted, idle resources to efficiently run in situ data analytics. GoldRush uses fine-grained scheduling to "steal" idle resources, in ways that minimize interference between the simulation and in situ analytics. This involves recognizing the potential causes of on-node resource contention and then using scheduling methods that prevent them. Experiments with representative science applications at large scales show that resources harvested on compute nodes can be leveraged to perform useful analytics, significantly improving resource efficiency, reducing data movement costs incurred by alternate solutions, and posing negligible impact on scientific simulations.

UR - http://www.scopus.com/inward/record.url?scp=84899670941&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899670941&partnerID=8YFLogxK

U2 - 10.1145/2503210.2503279

DO - 10.1145/2503210.2503279

M3 - Conference contribution

AN - SCOPUS:84899670941

SN - 9781450323789

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2013

PB - IEEE Computer Society

ER -