Hybrid scheduling for scientific workflows on hybrid clouds

Amirmohammad Pasdar, Young Choon Lee, Khaled Almi'ani*

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Scientific workflows consist of many interdependent tasks dictated by their data dependencies. As these workflows are becoming resource-intensive in both data and computing, private clouds struggle to cope with their resource requirements. Private cloud's limitations are claimed to be addressed by public clouds, however, the complete offloading of workflow execution to public clouds causes excessive data transfer. In this paper, we address the problem of scheduling scientific workflows on hybrid clouds aiming at cost reduction while improving execution time, with the consideration of public cloud costs, by minimizing data movements between private cloud and public cloud. To this end, we develop Hybrid Scheduling for Hybrid Clouds (HSHC) as a novel data-locality aware scheduling algorithm. HSHC adopts a hybrid approach consisting of static phase and dynamic phase. The former solves the problem of workflow scheduling, using an extended genetic algorithm, with static information of workflows and resources. The latter dynamically adapts the static schedule in response to changing execution conditions, such as locality of intermediate output data and performance degradation of tasks and resources. We evaluate HSHC with both random workflows and real-world scientific applications in execution time and cost. Experimental results compared with seven state-of-the-art algorithms demonstrate HSHC significantly reduces the cost by up to 40% and improves the execution time by up to 25%.

Original languageEnglish
Article number107438
Number of pages16
JournalComputer Networks
Volume181
DOIs
Publication statusPublished - 9 Nov 2020

Keywords

  • Data placement
  • Scientific workflows
  • Task scheduling
  • Hybrid cloud

Fingerprint Dive into the research topics of 'Hybrid scheduling for scientific workflows on hybrid clouds'. Together they form a unique fingerprint.

Cite this