Resource-efficient workflow scheduling in clouds

Young Choon Lee*, Hyuck Han, Albert Y. Zomaya, Mazin Yousif

*Corresponding author for this work

Research output: Contribution to journalArticle

61 Citations (Scopus)

Abstract

Workflow applications in science and engineering have steadily increased in variety and scale. Coinciding with this increase has been the relentless effort to improve the performance of these applications through exploiting the abundance of resources in hyper-scale clouds and with little attention to resources efficiency. The inefficient use of resources when executing scientific workflows results from both the excessive amount of resources provisioned and the wastage from unused resources among task runs. In this paper, we address the problem of resource-efficient workflow scheduling. To this end, we present the Maximum Effective Reduction (MER) algorithm, a resource efficiency solution that optimizes the resource usage of a workflow schedule generated by any particular scheduling algorithm. MER trades the minimal makespan increase for the maximal resource usage reduction by consolidating tasks with the exploitation of resource inefficiency in the original workflow schedule. The main novelty of MER lies in its identification of "near-optimal" trade-off point between makespan increase and resource usage reduction. Finding such a point is of great practical importance and can lead to: (1) improvements in resource utilization, (2) reductions in resource provisioning, and (3) savings in energy consumption. Another significant contribution of this work is MER's broad applicability. In essence, MER can be applied to any environments that deal with the execution of (scientific) workflows of many precedence-constrained tasks although MER best suits for the IaaS cloud model. Based on results obtained from our extensive simulations using scientific workflow traces, we demonstrate MER is capable of reducing the amount of actual resources used by 54% with an average makespan increase of less than 10%. The efficacy of MER is further verified by results (from a comprehensive set of experiments with varying makespan delay limits) that show the resource usage reduction, makespan increase and the trade-off between them for various workflow applications.

Original languageEnglish
Pages (from-to)153-162
Number of pages10
JournalKnowledge-Based Systems
Volume80
DOIs
Publication statusPublished - 1 May 2015

Keywords

  • Cloud computing
  • Resource efficiency
  • Resource management
  • Scientific workflows
  • Workflow scheduling

Fingerprint Dive into the research topics of 'Resource-efficient workflow scheduling in clouds'. Together they form a unique fingerprint.

Cite this