Local resource shaper for MapReduce

Peng Lu, Young Choon Lee, Vincent Gramoli, Luke M. Leslie, Albert Y. Zomaya

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

4 Citations (Scopus)


Resource capacity is often over provisioned to primarily deal with short periods of peak load. Shaping these peaks by shifting them to low utilization periods (valleys) is referred to as 'resource consumption shaping'. While originally aimed at the data center level, the resource consumption shaping we consider focuses on local resources, like CPU or I/O as we have identified that individual jobs also incur load peaks and valleys on these resources. In this paper, we present Local Resource Shaper (LRS), which limits fairness in resource sharing between co-located MapReduce tasks. LRS enables Hadoop to maximize resource utilization and minimize resource contention independently of job type. Co-located MapReduce tasks are often prone to resource contention (i.e., Load peak) due to similar resource usage patterns particularly with traditional fair resource sharing. In essence, LRS differentiates co-located tasks through active and passive slots that serve as containers for interchangeable map or reduce tasks. LRS lets an active slot consume as much resources as possible, and a passive slot make use of any unused resources. LRS leverages such slot differentiation with its new scheduler, Interleave. Our results show that LRS always outperforms the best static slot configuration with three Hadoop schedulers in terms of both resource utilization and performance.

Original languageEnglish
Title of host publicationProceedings - 2014 IEEE 6th International Conference on Cloud Computing Technology and Science, CloudCom 2014
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages8
Publication statusPublished - 2014
Externally publishedYes
EventIEEE 6th International Conference on Cloud Computing Technology and Science
- Singapore, Singapore, Singapore
Duration: 15 Dec 201418 Dec 2014

Publication series

NameInternational Conference on Cloud Computing Technology and Science
ISSN (Print)2330-2194


ConferenceIEEE 6th International Conference on Cloud Computing Technology and Science


  • Hadoop
  • MapReduce
  • Resource management
  • Scheduling


Dive into the research topics of 'Local resource shaper for MapReduce'. Together they form a unique fingerprint.

Cite this