Local resource shaper for MapReduce

Peng Lu, Young Choon Lee, Vincent Gramoli, Luke M. Leslie, Albert Y. Zomaya

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

4 Citations (Scopus)

Abstract

Resource capacity is often over provisioned to primarily deal with short periods of peak load. Shaping these peaks by shifting them to low utilization periods (valleys) is referred to as 'resource consumption shaping'. While originally aimed at the data center level, the resource consumption shaping we consider focuses on local resources, like CPU or I/O as we have identified that individual jobs also incur load peaks and valleys on these resources. In this paper, we present Local Resource Shaper (LRS), which limits fairness in resource sharing between co-located MapReduce tasks. LRS enables Hadoop to maximize resource utilization and minimize resource contention independently of job type. Co-located MapReduce tasks are often prone to resource contention (i.e., Load peak) due to similar resource usage patterns particularly with traditional fair resource sharing. In essence, LRS differentiates co-located tasks through active and passive slots that serve as containers for interchangeable map or reduce tasks. LRS lets an active slot consume as much resources as possible, and a passive slot make use of any unused resources. LRS leverages such slot differentiation with its new scheduler, Interleave. Our results show that LRS always outperforms the best static slot configuration with three Hadoop schedulers in terms of both resource utilization and performance.

Original languageEnglish
Title of host publicationProceedings of the International Conference on Cloud Computing Technology and Science
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages483-490
Number of pages8
DOIs
Publication statusPublished - 2014
Externally publishedYes
EventIEEE 6th International Conference on Cloud Computing Technology and Science
- Singapore, Singapore, Singapore
Duration: 15 Dec 201418 Dec 2014

Conference

ConferenceIEEE 6th International Conference on Cloud Computing Technology and Science
CitySingapore
Period15/12/1418/12/14

Fingerprint Dive into the research topics of 'Local resource shaper for MapReduce'. Together they form a unique fingerprint.

  • Projects

  • Cite this

    Lu, P., Lee, Y. C., Gramoli, V., Leslie, L. M., & Zomaya, A. Y. (2014). Local resource shaper for MapReduce. In Proceedings of the International Conference on Cloud Computing Technology and Science (pp. 483-490). Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE). Proceedings of the International Conference on Cloud Computing Technology and Science, CloudCom https://doi.org/10.1109/CloudCom.2014.55