Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations

Xiao Qin, Hong Jiang, Yifeng Zhu, David R. Swanson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

While previous CPU- or memory-centric load balancing schemes are capable of achieving the effective usage of global CPU and memory resources in a cluster system, the cluster exhibits significant performance drop under I/O-intensive workload conditions due to the imbalance of I/O load. To tackle this problem, we have developed two simple yet effective I/O-aware load-balancing schemes, which make it possible to balance I/O load by assigning I/O intensive sequential and parallel jobs to nodes with light I/O loads. Moreover, the proposed schemes judiciously take into account both CPU and memory load sharing in the cluster, thereby maintaining a high performance for a wide spectrum of workload. Using a set of real I/O-intensive parallel applications in addition to synthetic parallel jobs, we show that the proposed schemes consistently outperform the existing non-I/O-aware load-balancing schemes for a diverse set of workload conditions. Importantly, the performance improvement becomes much more pronounced when the applications are I/O-intensive.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages100-107
Number of pages8
ISBN (Electronic)0769520669
DOIs
StatePublished - Jan 1 2003
EventIEEE International Conference on Cluster Computing, CLUSTER 2003 - Hong Kong, China
Duration: Dec 1 2003Dec 4 2003

Publication series

NameProceedings - IEEE International Conference on Cluster Computing, ICCC
Volume2003-January
ISSN (Print)1552-5244

Other

OtherIEEE International Conference on Cluster Computing, CLUSTER 2003
CountryChina
CityHong Kong
Period12/1/0312/4/03

Fingerprint

Computer workstations
Resource allocation
Program processors
Data storage equipment
Computer systems

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Signal Processing

Cite this

Qin, X., Jiang, H., Zhu, Y., & Swanson, D. R. (2003). Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations. In Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003 (pp. 100-107). [1253305] (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2003-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTR.2003.1253305

Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations. / Qin, Xiao; Jiang, Hong; Zhu, Yifeng; Swanson, David R.

Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003. Institute of Electrical and Electronics Engineers Inc., 2003. p. 100-107 1253305 (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2003-January).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Qin, X, Jiang, H, Zhu, Y & Swanson, DR 2003, Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations. in Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003., 1253305, Proceedings - IEEE International Conference on Cluster Computing, ICCC, vol. 2003-January, Institute of Electrical and Electronics Engineers Inc., pp. 100-107, IEEE International Conference on Cluster Computing, CLUSTER 2003, Hong Kong, China, 12/1/03. https://doi.org/10.1109/CLUSTR.2003.1253305
Qin X, Jiang H, Zhu Y, Swanson DR. Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations. In Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003. Institute of Electrical and Electronics Engineers Inc. 2003. p. 100-107. 1253305. (Proceedings - IEEE International Conference on Cluster Computing, ICCC). https://doi.org/10.1109/CLUSTR.2003.1253305
Qin, Xiao ; Jiang, Hong ; Zhu, Yifeng ; Swanson, David R. / Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations. Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003. Institute of Electrical and Electronics Engineers Inc., 2003. pp. 100-107 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).
@inproceedings{f3ede819f0084f47880183a56ec9a2b0,
title = "Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations",
abstract = "While previous CPU- or memory-centric load balancing schemes are capable of achieving the effective usage of global CPU and memory resources in a cluster system, the cluster exhibits significant performance drop under I/O-intensive workload conditions due to the imbalance of I/O load. To tackle this problem, we have developed two simple yet effective I/O-aware load-balancing schemes, which make it possible to balance I/O load by assigning I/O intensive sequential and parallel jobs to nodes with light I/O loads. Moreover, the proposed schemes judiciously take into account both CPU and memory load sharing in the cluster, thereby maintaining a high performance for a wide spectrum of workload. Using a set of real I/O-intensive parallel applications in addition to synthetic parallel jobs, we show that the proposed schemes consistently outperform the existing non-I/O-aware load-balancing schemes for a diverse set of workload conditions. Importantly, the performance improvement becomes much more pronounced when the applications are I/O-intensive.",
author = "Xiao Qin and Hong Jiang and Yifeng Zhu and Swanson, {David R.}",
year = "2003",
month = "1",
day = "1",
doi = "10.1109/CLUSTR.2003.1253305",
language = "English (US)",
series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "100--107",
booktitle = "Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003",

}

TY - GEN

T1 - Towards load balancing support for I/O-intensive parallel jobs in a cluster of workstations

AU - Qin, Xiao

AU - Jiang, Hong

AU - Zhu, Yifeng

AU - Swanson, David R.

PY - 2003/1/1

Y1 - 2003/1/1

N2 - While previous CPU- or memory-centric load balancing schemes are capable of achieving the effective usage of global CPU and memory resources in a cluster system, the cluster exhibits significant performance drop under I/O-intensive workload conditions due to the imbalance of I/O load. To tackle this problem, we have developed two simple yet effective I/O-aware load-balancing schemes, which make it possible to balance I/O load by assigning I/O intensive sequential and parallel jobs to nodes with light I/O loads. Moreover, the proposed schemes judiciously take into account both CPU and memory load sharing in the cluster, thereby maintaining a high performance for a wide spectrum of workload. Using a set of real I/O-intensive parallel applications in addition to synthetic parallel jobs, we show that the proposed schemes consistently outperform the existing non-I/O-aware load-balancing schemes for a diverse set of workload conditions. Importantly, the performance improvement becomes much more pronounced when the applications are I/O-intensive.

AB - While previous CPU- or memory-centric load balancing schemes are capable of achieving the effective usage of global CPU and memory resources in a cluster system, the cluster exhibits significant performance drop under I/O-intensive workload conditions due to the imbalance of I/O load. To tackle this problem, we have developed two simple yet effective I/O-aware load-balancing schemes, which make it possible to balance I/O load by assigning I/O intensive sequential and parallel jobs to nodes with light I/O loads. Moreover, the proposed schemes judiciously take into account both CPU and memory load sharing in the cluster, thereby maintaining a high performance for a wide spectrum of workload. Using a set of real I/O-intensive parallel applications in addition to synthetic parallel jobs, we show that the proposed schemes consistently outperform the existing non-I/O-aware load-balancing schemes for a diverse set of workload conditions. Importantly, the performance improvement becomes much more pronounced when the applications are I/O-intensive.

UR - http://www.scopus.com/inward/record.url?scp=84944893513&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84944893513&partnerID=8YFLogxK

U2 - 10.1109/CLUSTR.2003.1253305

DO - 10.1109/CLUSTR.2003.1253305

M3 - Conference contribution

AN - SCOPUS:84944893513

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

SP - 100

EP - 107

BT - Proceedings - IEEE International Conference on Cluster Computing, CLUSTER 2003

PB - Institute of Electrical and Electronics Engineers Inc.

ER -