Serverless execution of scientific workflows

Qingye Jiang*, Young Choon Lee, Albert Y. Zomaya

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

11 Citations (Scopus)

Abstract

In this paper, we present a serverless workflow execution system (DEWE v31) with Function-as-a-Service (FaaS aka serverless computing) as the target execution environment. DEWE v3 is designed to address problems of (1) execution of large-scale scientific workflows and (2) resource underutilization. At its core is our novel hybrid (FaaS and dedicated/local clusters) job dispatching approach taking into account resource consumption patterns of different phases of workflow execution. In particular, the hybrid approach deals with the maximum execution duration limit, memory limit, and storage space limit. DEWE v3 significantly reduces the efforts needed to execute large-scale scientific workflow applications on public clouds. We have evaluated DEWE v3 on both AWS Lambda and Google Cloud Functions and demonstrate that FaaS offers an ideal solution for scientific workflows with complex precedence constraints. In our large-scale evaluations, the hybrid execution model surpasses the performance of the traditional cluster execution model with significantly less execution cost.

Original languageEnglish
Title of host publicationService-Oriented Computing
Subtitle of host publication15th International Conference, ICSOC 2017, Proceedings
EditorsMichael Maximilien, Antonio Vallecillo, Jianmin Wang, Marc Oriol
Place of PublicationCham
PublisherSpringer, Springer Nature
Pages706-721
Number of pages16
ISBN (Electronic)9783319690353
ISBN (Print)9783319690346
DOIs
Publication statusPublished - 1 Jan 2017
Event15th International Conference on Service-Oriented Computing, ICSOC 2017 - Malaga, Spain
Duration: 13 Nov 201716 Nov 2017

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10601 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference15th International Conference on Service-Oriented Computing, ICSOC 2017
CountrySpain
CityMalaga
Period13/11/1716/11/17

Keywords

  • Function-as-a-service
  • Scientific workflow
  • Serverless computing

Fingerprint

Dive into the research topics of 'Serverless execution of scientific workflows'. Together they form a unique fingerprint.

Cite this