Effective resource multiplexing for scientific workflows

Thomas Ryan, Young Choon Lee

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

Abstract

Scientific workflows feature complex precedence constraints that are mostly dictated by data dependencies between tasks. The inter-task communication (data staging) in these complex workflow applications incurs significant overheads resulting in a major hindering factor of high performance and effective resource utilization. As the scale of these applications becomes increasingly large due primarily to the recent explosive growth of data, addressing this hindrance is of great practical importance. In this paper, we present a resource multiplexing (RM) technique, which leverages data staging aiming to minimize idle times between execution of tasks due to inter-task communication overheads. In particular, we incorporate RM into our DEWE framework 1 with making a set of extensions to the framework. The rationale behind RM is each slot or core pairs up the actual workflow task and the RM-enabled file loading DEWE extension (File Client) in the way that their resource usage is complementary. We demonstrate the efficacy of our multiplexing technique in a data-intensive computing environment using an astronomy application. Our results from experiments conducted in Amazon EC2 demonstrate that our multiplexing technique is effective with the reduction in resource idle time between jobs by 57% on average and up to 91%.

Original languageEnglish
Title of host publication17th Asia-Pacific Network Operations and Management Symposium: Managing a Very Connected World, APNOMS 2015
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages232-237
Number of pages6
ISBN (Electronic)9784885522963
DOIs
Publication statusPublished - 24 Sep 2015
Event17th Asia-Pacific Network Operations and Management Symposium, APNOMS 2015 - Busan, Korea, Republic of
Duration: 19 Aug 201521 Aug 2015

Other

Other17th Asia-Pacific Network Operations and Management Symposium, APNOMS 2015
CountryKorea, Republic of
CityBusan
Period19/08/1521/08/15

Fingerprint Dive into the research topics of 'Effective resource multiplexing for scientific workflows'. Together they form a unique fingerprint.

Cite this