Local Resource Consumption Shaping: A Case for MapReduce

Today, resource capacity is no longer an issue for running large-scale distributed systems, such as MapReduce. As a result, it is often the case that resources are provisioned, for such systems, at the level of peak loads. This overprovisioning has become a serious efficiency issue in cloud data centers with poor resource utilization. The improvement of resource utilization can be achieved by concurrently running tasks sharing a common set of resources. However, many distributed systems spawn a large number of tasks that exhibit similar resource consumption patterns causing much performance interference/degradation that is primarily due to fair resource sharing. In this study, we consider the problem of "local resource consumption shaping" - an alternative to fair resource sharing - at the local node/core level.