Dynamic workload balancing for hadoop MapReduce

Xiaofei Hou; Ashwin T K Kumar; Johnson P. Thomas; Vijay Varadharajan

doi:10.1109/BDCloud.2014.103

Dynamic workload balancing for hadoop MapReduce

Xiaofei Hou, Ashwin T K Kumar, Johnson P. Thomas, Vijay Varadharajan

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

17 Citations (Scopus)

Abstract

Hadoop has two components which are HDFS and MapReduce. HDFS is a distributed file system for storing data for users of Hadoop and MapReduce is the framework that executes jobs from users. Hadoop stores user data based on space utilization of data nodes on the cluster rather than the processing capability of the data nodes. Furthermore Hadoop runs in a heterogeneous environment as all data nodes may not be homogeneous. For these reasons, workload imbalances will occur when Hadoop runs resulting in poor performance. In this paper, we propose a dynamic algorithm to balance the workload between different racks on a Hadoop cluster based on information obtained from analyzing the log files of Hadoop. Moving tasks from the busiest rack to another rack improves the performance of Hadoop MapReduce by reducing the running time of jobs. Our simulations indicate that using our algorithm, we can decrease by more than 50% the remaining time of the tasks belonged to a job running on the busiest rack.

Original language	English
Title of host publication	Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014
Editors	Jinjun Chen, Laurence T. Yang
Place of Publication	Picataway, NJ
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	56-62
Number of pages	7
ISBN (Electronic)	9781479967193
ISBN (Print)	9781479967209
DOIs	https://doi.org/10.1109/BDCloud.2014.103
Publication status	Published - 2014
Event	4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 - Sydney, Australia Duration: 3 Dec 2014 → 5 Dec 2014

Other

Other	4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014
Country/Territory	Australia
City	Sydney
Period	3/12/14 → 5/12/14

Keywords

Hadoop
MapReduce
Dynamic Workload balancing
OpenFlow

Access to Document

10.1109/BDCloud.2014.103

Cite this

Hou, X., Kumar, A. T. K., Thomas, J. P., & Varadharajan, V. (2014). Dynamic workload balancing for hadoop MapReduce. In J. Chen, & L. T. Yang (Eds.), Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014 (pp. 56-62). Article 7034766 Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/BDCloud.2014.103

Hou, Xiaofei ; Kumar, Ashwin T K ; Thomas, Johnson P. et al. / Dynamic workload balancing for hadoop MapReduce. Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014. editor / Jinjun Chen ; Laurence T. Yang. Picataway, NJ : Institute of Electrical and Electronics Engineers (IEEE), 2014. pp. 56-62

@inproceedings{cb7273d459c7412dae0f2f331311a036,

title = "Dynamic workload balancing for hadoop MapReduce",

abstract = "Hadoop has two components which are HDFS and MapReduce. HDFS is a distributed file system for storing data for users of Hadoop and MapReduce is the framework that executes jobs from users. Hadoop stores user data based on space utilization of data nodes on the cluster rather than the processing capability of the data nodes. Furthermore Hadoop runs in a heterogeneous environment as all data nodes may not be homogeneous. For these reasons, workload imbalances will occur when Hadoop runs resulting in poor performance. In this paper, we propose a dynamic algorithm to balance the workload between different racks on a Hadoop cluster based on information obtained from analyzing the log files of Hadoop. Moving tasks from the busiest rack to another rack improves the performance of Hadoop MapReduce by reducing the running time of jobs. Our simulations indicate that using our algorithm, we can decrease by more than 50% the remaining time of the tasks belonged to a job running on the busiest rack.",

keywords = "Hadoop, MapReduce, Dynamic Workload balancing, OpenFlow",

author = "Xiaofei Hou and Kumar, {Ashwin T K} and Thomas, {Johnson P.} and Vijay Varadharajan",

year = "2014",

doi = "10.1109/BDCloud.2014.103",

language = "English",

isbn = "9781479967209",

pages = "56--62",

editor = "Jinjun Chen and Yang, {Laurence T.}",

booktitle = "Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 ; Conference date: 03-12-2014 Through 05-12-2014",

}

Hou, X, Kumar, ATK, Thomas, JP & Varadharajan, V 2014, Dynamic workload balancing for hadoop MapReduce. in J Chen & LT Yang (eds), Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014., 7034766, Institute of Electrical and Electronics Engineers (IEEE), Picataway, NJ, pp. 56-62, 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014, Sydney, Australia, 3/12/14. https://doi.org/10.1109/BDCloud.2014.103

Dynamic workload balancing for hadoop MapReduce. / Hou, Xiaofei; Kumar, Ashwin T K; Thomas, Johnson P. et al.
Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014. ed. / Jinjun Chen; Laurence T. Yang. Picataway, NJ: Institute of Electrical and Electronics Engineers (IEEE), 2014. p. 56-62 7034766.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Dynamic workload balancing for hadoop MapReduce

AU - Hou, Xiaofei

AU - Kumar, Ashwin T K

AU - Thomas, Johnson P.

AU - Varadharajan, Vijay

PY - 2014

Y1 - 2014

N2 - Hadoop has two components which are HDFS and MapReduce. HDFS is a distributed file system for storing data for users of Hadoop and MapReduce is the framework that executes jobs from users. Hadoop stores user data based on space utilization of data nodes on the cluster rather than the processing capability of the data nodes. Furthermore Hadoop runs in a heterogeneous environment as all data nodes may not be homogeneous. For these reasons, workload imbalances will occur when Hadoop runs resulting in poor performance. In this paper, we propose a dynamic algorithm to balance the workload between different racks on a Hadoop cluster based on information obtained from analyzing the log files of Hadoop. Moving tasks from the busiest rack to another rack improves the performance of Hadoop MapReduce by reducing the running time of jobs. Our simulations indicate that using our algorithm, we can decrease by more than 50% the remaining time of the tasks belonged to a job running on the busiest rack.

AB - Hadoop has two components which are HDFS and MapReduce. HDFS is a distributed file system for storing data for users of Hadoop and MapReduce is the framework that executes jobs from users. Hadoop stores user data based on space utilization of data nodes on the cluster rather than the processing capability of the data nodes. Furthermore Hadoop runs in a heterogeneous environment as all data nodes may not be homogeneous. For these reasons, workload imbalances will occur when Hadoop runs resulting in poor performance. In this paper, we propose a dynamic algorithm to balance the workload between different racks on a Hadoop cluster based on information obtained from analyzing the log files of Hadoop. Moving tasks from the busiest rack to another rack improves the performance of Hadoop MapReduce by reducing the running time of jobs. Our simulations indicate that using our algorithm, we can decrease by more than 50% the remaining time of the tasks belonged to a job running on the busiest rack.

KW - Hadoop

KW - MapReduce

KW - Dynamic Workload balancing

KW - OpenFlow

UR - http://www.scopus.com/inward/record.url?scp=84924404074&partnerID=8YFLogxK

U2 - 10.1109/BDCloud.2014.103

DO - 10.1109/BDCloud.2014.103

M3 - Conference proceeding contribution

SN - 9781479967209

SP - 56

EP - 62

BT - Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014

A2 - Chen, Jinjun

A2 - Yang, Laurence T.

PB - Institute of Electrical and Electronics Engineers (IEEE)

CY - Picataway, NJ

T2 - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014

Y2 - 3 December 2014 through 5 December 2014

ER -

Hou X, Kumar ATK, Thomas JP, Varadharajan V. Dynamic workload balancing for hadoop MapReduce. In Chen J, Yang LT, editors, Proceedings - 4th IEEE International Conference on Big Data and Cloud Computing, BDCloud 2014 with the 7th IEEE International Conference on Social Computing and Networking, SocialCom 2014 and the 4th International Conference on Sustainable Computing and Communications, SustainCom 2014. Picataway, NJ: Institute of Electrical and Electronics Engineers (IEEE). 2014. p. 56-62. 7034766 doi: 10.1109/BDCloud.2014.103

Dynamic workload balancing for hadoop MapReduce

Abstract

Other

Keywords

Access to Document

Other files and links

Cite this