Eunomia

A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters

Wei Zhou, K. Preston White, Hongfeng Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Due to hardware upgrades and server consolidation, it is not uncommon to witness a few generations of servers deployed in the same datacenters. As a result, variants of fair job schedulers are proposed to enforce fairness for constrained jobs that have hardware or software constraints on task placement. However, the other important characteristics resulted from server heterogeneity, performance variation, is unfortunately overlooked by state-of-art fair job schedulers with placement constraints. In this paper, we propose Eunomia, a performance-variation-aware fair job scheduler, to address the unfairness issue due to performance variation in heterogenous clusters. Eunomia introduces a key metric, called progress share, which is defined as the ratio between the accumulated task progress given the current allocation and the accumulated task progress if the user can monopolize the cluster. Eunomia aims to equalize progress share of jobs as much as possible, so as to achieve the same slowdown of jobs from different users due to resource sharing and placement constraints, regardless of performance variation. Evaluation results show that Eunomia is able to deliver better share fairness compared with state-of-art schedulers without performance loss.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018
PublisherIEEE Computer Society
Pages1034-1039
Number of pages6
ISBN (Electronic)9781538673089
DOIs
StatePublished - Feb 19 2019
Event24th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2018 - Singapore, Singapore
Duration: Dec 11 2018Dec 13 2018

Publication series

NameProceedings of the International Conference on Parallel and Distributed Systems - ICPADS
Volume2018-December
ISSN (Print)1521-9097

Conference

Conference24th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2018
CountrySingapore
CitySingapore
Period12/11/1812/13/18

Fingerprint

Servers
Consolidation
Computer hardware
Hardware

Keywords

  • Big Data
  • Fair Scheduler
  • Placement Constraint

ASJC Scopus subject areas

  • Hardware and Architecture

Cite this

Zhou, W., White, K. P., & Yu, H. (2019). Eunomia: A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters. In Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018 (pp. 1034-1039). [8645046] (Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS; Vol. 2018-December). IEEE Computer Society. https://doi.org/10.1109/PADSW.2018.8645046

Eunomia : A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters. / Zhou, Wei; White, K. Preston; Yu, Hongfeng.

Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018. IEEE Computer Society, 2019. p. 1034-1039 8645046 (Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS; Vol. 2018-December).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhou, W, White, KP & Yu, H 2019, Eunomia: A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters. in Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018., 8645046, Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS, vol. 2018-December, IEEE Computer Society, pp. 1034-1039, 24th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2018, Singapore, Singapore, 12/11/18. https://doi.org/10.1109/PADSW.2018.8645046
Zhou W, White KP, Yu H. Eunomia: A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters. In Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018. IEEE Computer Society. 2019. p. 1034-1039. 8645046. (Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS). https://doi.org/10.1109/PADSW.2018.8645046
Zhou, Wei ; White, K. Preston ; Yu, Hongfeng. / Eunomia : A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters. Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018. IEEE Computer Society, 2019. pp. 1034-1039 (Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS).
@inproceedings{12774be0948f423e98a0535e8bbd55d4,
title = "Eunomia: A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters",
abstract = "Due to hardware upgrades and server consolidation, it is not uncommon to witness a few generations of servers deployed in the same datacenters. As a result, variants of fair job schedulers are proposed to enforce fairness for constrained jobs that have hardware or software constraints on task placement. However, the other important characteristics resulted from server heterogeneity, performance variation, is unfortunately overlooked by state-of-art fair job schedulers with placement constraints. In this paper, we propose Eunomia, a performance-variation-aware fair job scheduler, to address the unfairness issue due to performance variation in heterogenous clusters. Eunomia introduces a key metric, called progress share, which is defined as the ratio between the accumulated task progress given the current allocation and the accumulated task progress if the user can monopolize the cluster. Eunomia aims to equalize progress share of jobs as much as possible, so as to achieve the same slowdown of jobs from different users due to resource sharing and placement constraints, regardless of performance variation. Evaluation results show that Eunomia is able to deliver better share fairness compared with state-of-art schedulers without performance loss.",
keywords = "Big Data, Fair Scheduler, Placement Constraint",
author = "Wei Zhou and White, {K. Preston} and Hongfeng Yu",
year = "2019",
month = "2",
day = "19",
doi = "10.1109/PADSW.2018.8645046",
language = "English (US)",
series = "Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS",
publisher = "IEEE Computer Society",
pages = "1034--1039",
booktitle = "Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018",

}

TY - GEN

T1 - Eunomia

T2 - A Performance-Variation-Aware Fair Job Scheduler with Placement Constraints for Heterogeneous Datacenters

AU - Zhou, Wei

AU - White, K. Preston

AU - Yu, Hongfeng

PY - 2019/2/19

Y1 - 2019/2/19

N2 - Due to hardware upgrades and server consolidation, it is not uncommon to witness a few generations of servers deployed in the same datacenters. As a result, variants of fair job schedulers are proposed to enforce fairness for constrained jobs that have hardware or software constraints on task placement. However, the other important characteristics resulted from server heterogeneity, performance variation, is unfortunately overlooked by state-of-art fair job schedulers with placement constraints. In this paper, we propose Eunomia, a performance-variation-aware fair job scheduler, to address the unfairness issue due to performance variation in heterogenous clusters. Eunomia introduces a key metric, called progress share, which is defined as the ratio between the accumulated task progress given the current allocation and the accumulated task progress if the user can monopolize the cluster. Eunomia aims to equalize progress share of jobs as much as possible, so as to achieve the same slowdown of jobs from different users due to resource sharing and placement constraints, regardless of performance variation. Evaluation results show that Eunomia is able to deliver better share fairness compared with state-of-art schedulers without performance loss.

AB - Due to hardware upgrades and server consolidation, it is not uncommon to witness a few generations of servers deployed in the same datacenters. As a result, variants of fair job schedulers are proposed to enforce fairness for constrained jobs that have hardware or software constraints on task placement. However, the other important characteristics resulted from server heterogeneity, performance variation, is unfortunately overlooked by state-of-art fair job schedulers with placement constraints. In this paper, we propose Eunomia, a performance-variation-aware fair job scheduler, to address the unfairness issue due to performance variation in heterogenous clusters. Eunomia introduces a key metric, called progress share, which is defined as the ratio between the accumulated task progress given the current allocation and the accumulated task progress if the user can monopolize the cluster. Eunomia aims to equalize progress share of jobs as much as possible, so as to achieve the same slowdown of jobs from different users due to resource sharing and placement constraints, regardless of performance variation. Evaluation results show that Eunomia is able to deliver better share fairness compared with state-of-art schedulers without performance loss.

KW - Big Data

KW - Fair Scheduler

KW - Placement Constraint

UR - http://www.scopus.com/inward/record.url?scp=85063344081&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063344081&partnerID=8YFLogxK

U2 - 10.1109/PADSW.2018.8645046

DO - 10.1109/PADSW.2018.8645046

M3 - Conference contribution

T3 - Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS

SP - 1034

EP - 1039

BT - Proceedings - 2018 IEEE 24th International Conference on Parallel and Distributed Systems, ICPADS 2018

PB - IEEE Computer Society

ER -