Multi-tier resource allocation for data-intensive computing

Thomas Ryan, Young Choon Lee*

*Corresponding author for this work

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

As distributed computing systems are used more widely, driven by trends such as 'big data' and cloud computing, they are being used for an increasingly wide range of applications. With this massive increase in application heterogeneity, the ability to have a general purpose resource management technique that performs well in heterogeneous environments is becoming increasingly important.In this paper, we present Multi-Tier Resource Allocation (MTRA) as a novel fine-grained resource management technique for distributed systems. The core idea is based on allocating resources to individual tasks in a tiered or layered approach. To account for heterogeneity, we propose a dynamic resource allocation method that adjusts resource allocations to individual tasks on a cluster node based on resource utilisation levels. We demonstrate the efficacy of this technique in a data-intensive computing environment, MapReduce data processing framework in Hadoop YARN. Our results demonstrate that MTRA is an effective general purpose resource management technique particularly for data-intensive computing environments. On a range of MapReduce benchmarks in a Hadoop YARN environment, our MTRA technique improves performance by up to 18%. In a Facebook workload model it improves job execution times by 10% on average, and up to 56% for individual jobs.

Original languageEnglish
Pages (from-to)110-116
Number of pages7
JournalBig Data Research
Volume2
Issue number3
DOIs
Publication statusPublished - Sep 2015

Keywords

  • Application heterogeneity
  • Big data
  • Cloud computing
  • Data-intensive computing
  • MapReduce
  • Resource allocation

Fingerprint Dive into the research topics of 'Multi-tier resource allocation for data-intensive computing'. Together they form a unique fingerprint.

  • Cite this