Workload characteristic oriented scheduler for MapReduce

Peng Lu; Young Choon Lee; Chen Wang; Bing Bing Zhou; Junliang Chen; Albert Y. Zomaya

doi:10.1109/ICPADS.2012.31

Workload characteristic oriented scheduler for MapReduce

Peng Lu^*, Young Choon Lee, Chen Wang, Bing Bing Zhou, Junliang Chen, Albert Y. Zomaya

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

28 Citations (Scopus)

Abstract

Applications in many areas are increasingly developed and ported using the MapReduce framework (more specifically, Hadoop) to exploit (data) parallelism. The application scope of MapReduce has been extended beyond the original design goal which was large-scale data processing. This extension inherently makes a need for scheduler to explicitly take into account characteristics of job for two main goals of efficient resource use and performance improvement. In this paper, we study MapReduce scheduling strategies to effectively deal with different workload characteristics - CPU intensive and I/O intensive. We present the Workload Characteristic Oriented Scheduler (WCO), which strives for co-locating tasks of possibly different MapReduce jobs with complementing resource usage characteristics. WCO is characterized by its essentially dynamic and adaptive scheduling decisions using information obtained from its characteristic estimator. Workload characteristics of tasks are primarily estimated by sampling with the help of some static task selection strategies, e.g., Java bytecode analysis. Results obtained from extensive experiments using 11 benchmarks in a 4-node local cluster and a 51-node Amazon EC2 cluster show 17% performance improvement on average in terms of throughput in the situation of co-existing diverse workloads.

Original language	English
Title of host publication	Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, ICPADS 2012
Place of Publication	Piscataway, NJ
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	156-163
Number of pages	8
ISBN (Print)	9780769549033
DOIs	https://doi.org/10.1109/ICPADS.2012.31
Publication status	Published - 2012
Externally published	Yes
Event	18th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2012 - Singapore, Singapore Duration: 17 Dec 2012 → 19 Dec 2012

Other

Other	18th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2012
Country/Territory	Singapore
City	Singapore
Period	17/12/12 → 19/12/12

Access to Document

10.1109/ICPADS.2012.31

Cite this

Lu, P., Lee, Y. C., Wang, C., Zhou, B. B., Chen, J., & Zomaya, A. Y. (2012). Workload characteristic oriented scheduler for MapReduce. In Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, ICPADS 2012 (pp. 156-163). Article 6413701 Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ICPADS.2012.31

@inproceedings{94d00431b902427a9e13b21cdad52bd1,

title = "Workload characteristic oriented scheduler for MapReduce",

abstract = "Applications in many areas are increasingly developed and ported using the MapReduce framework (more specifically, Hadoop) to exploit (data) parallelism. The application scope of MapReduce has been extended beyond the original design goal which was large-scale data processing. This extension inherently makes a need for scheduler to explicitly take into account characteristics of job for two main goals of efficient resource use and performance improvement. In this paper, we study MapReduce scheduling strategies to effectively deal with different workload characteristics - CPU intensive and I/O intensive. We present the Workload Characteristic Oriented Scheduler (WCO), which strives for co-locating tasks of possibly different MapReduce jobs with complementing resource usage characteristics. WCO is characterized by its essentially dynamic and adaptive scheduling decisions using information obtained from its characteristic estimator. Workload characteristics of tasks are primarily estimated by sampling with the help of some static task selection strategies, e.g., Java bytecode analysis. Results obtained from extensive experiments using 11 benchmarks in a 4-node local cluster and a 51-node Amazon EC2 cluster show 17% performance improvement on average in terms of throughput in the situation of co-existing diverse workloads.",

author = "Peng Lu and Lee, {Young Choon} and Chen Wang and Zhou, {Bing Bing} and Junliang Chen and Zomaya, {Albert Y.}",

year = "2012",

doi = "10.1109/ICPADS.2012.31",

language = "English",

isbn = "9780769549033",

pages = "156--163",

booktitle = "Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, ICPADS 2012",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "18th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2012 ; Conference date: 17-12-2012 Through 19-12-2012",

}

Lu, P, Lee, YC, Wang, C, Zhou, BB, Chen, J & Zomaya, AY 2012, Workload characteristic oriented scheduler for MapReduce. in Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, ICPADS 2012., 6413701, Institute of Electrical and Electronics Engineers (IEEE), Piscataway, NJ, pp. 156-163, 18th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2012, Singapore, Singapore, 17/12/12. https://doi.org/10.1109/ICPADS.2012.31

Workload characteristic oriented scheduler for MapReduce. / Lu, Peng; Lee, Young Choon; Wang, Chen et al.
Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, ICPADS 2012. Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE), 2012. p. 156-163 6413701.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Workload characteristic oriented scheduler for MapReduce

AU - Lu, Peng

AU - Lee, Young Choon

AU - Wang, Chen

AU - Zhou, Bing Bing

AU - Chen, Junliang

AU - Zomaya, Albert Y.

PY - 2012

Y1 - 2012

N2 - Applications in many areas are increasingly developed and ported using the MapReduce framework (more specifically, Hadoop) to exploit (data) parallelism. The application scope of MapReduce has been extended beyond the original design goal which was large-scale data processing. This extension inherently makes a need for scheduler to explicitly take into account characteristics of job for two main goals of efficient resource use and performance improvement. In this paper, we study MapReduce scheduling strategies to effectively deal with different workload characteristics - CPU intensive and I/O intensive. We present the Workload Characteristic Oriented Scheduler (WCO), which strives for co-locating tasks of possibly different MapReduce jobs with complementing resource usage characteristics. WCO is characterized by its essentially dynamic and adaptive scheduling decisions using information obtained from its characteristic estimator. Workload characteristics of tasks are primarily estimated by sampling with the help of some static task selection strategies, e.g., Java bytecode analysis. Results obtained from extensive experiments using 11 benchmarks in a 4-node local cluster and a 51-node Amazon EC2 cluster show 17% performance improvement on average in terms of throughput in the situation of co-existing diverse workloads.

AB - Applications in many areas are increasingly developed and ported using the MapReduce framework (more specifically, Hadoop) to exploit (data) parallelism. The application scope of MapReduce has been extended beyond the original design goal which was large-scale data processing. This extension inherently makes a need for scheduler to explicitly take into account characteristics of job for two main goals of efficient resource use and performance improvement. In this paper, we study MapReduce scheduling strategies to effectively deal with different workload characteristics - CPU intensive and I/O intensive. We present the Workload Characteristic Oriented Scheduler (WCO), which strives for co-locating tasks of possibly different MapReduce jobs with complementing resource usage characteristics. WCO is characterized by its essentially dynamic and adaptive scheduling decisions using information obtained from its characteristic estimator. Workload characteristics of tasks are primarily estimated by sampling with the help of some static task selection strategies, e.g., Java bytecode analysis. Results obtained from extensive experiments using 11 benchmarks in a 4-node local cluster and a 51-node Amazon EC2 cluster show 17% performance improvement on average in terms of throughput in the situation of co-existing diverse workloads.

UR - http://www.scopus.com/inward/record.url?scp=84874104491&partnerID=8YFLogxK

U2 - 10.1109/ICPADS.2012.31

DO - 10.1109/ICPADS.2012.31

M3 - Conference proceeding contribution

AN - SCOPUS:84874104491

SN - 9780769549033

SP - 156

EP - 163

BT - Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems, ICPADS 2012

PB - Institute of Electrical and Electronics Engineers (IEEE)

CY - Piscataway, NJ

T2 - 18th IEEE International Conference on Parallel and Distributed Systems, ICPADS 2012

Y2 - 17 December 2012 through 19 December 2012

ER -

Workload characteristic oriented scheduler for MapReduce

Abstract

Other

Access to Document

Other files and links

Fingerprint

Cite this