Projects per year
Resource capacity is often over provisioned to primarily deal with short periods of peak load. Shaping these peaks by shifting them to low utilization periods (valleys) is referred to as 'resource consumption shaping'. While originally aimed at the data center level, the resource consumption shaping we consider focuses on local resources, like CPU or I/O as we have identified that individual jobs also incur load peaks and valleys on these resources. In this paper, we present Local Resource Shaper (LRS), which limits fairness in resource sharing between co-located MapReduce tasks. LRS enables Hadoop to maximize resource utilization and minimize resource contention independently of job type. Co-located MapReduce tasks are often prone to resource contention (i.e., Load peak) due to similar resource usage patterns particularly with traditional fair resource sharing. In essence, LRS differentiates co-located tasks through active and passive slots that serve as containers for interchangeable map or reduce tasks. LRS lets an active slot consume as much resources as possible, and a passive slot make use of any unused resources. LRS leverages such slot differentiation with its new scheduler, Interleave. Our results show that LRS always outperforms the best static slot configuration with three Hadoop schedulers in terms of both resource utilization and performance.
|Title of host publication||Proceedings - 2014 IEEE 6th International Conference on Cloud Computing Technology and Science, CloudCom 2014|
|Place of Publication||Piscataway, NJ|
|Publisher||Institute of Electrical and Electronics Engineers (IEEE)|
|Number of pages||8|
|Publication status||Published - 2014|
|Event||IEEE 6th International Conference on Cloud Computing Technology and Science|
- Singapore, Singapore, Singapore
Duration: 15 Dec 2014 → 18 Dec 2014
|Name||International Conference on Cloud Computing Technology and Science|
|Conference||IEEE 6th International Conference on Cloud Computing Technology and Science|
|Period||15/12/14 → 18/12/14|
- Resource management
FingerprintDive into the research topics of 'Local resource shaper for MapReduce'. Together they form a unique fingerprint.
- 1 Active