Extensible pipeline development powered by PyCPL and PyEsorex

Brent Miszalski*, Amir E. Bazkiaei, Jesse van de Sande, Simon J. O'Toole, Anthony Horton, James Tocknell

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

Abstract

Despite Python being the preferred programming language of choice for most astronomers, building or extending data reduction pipelines in the language can be problematic. A common approach is to write Python functions or classes as wrappers, calling individual pipeline recipes underneath, but this does not scale well with increasing pipeline complexity. Data management is also fraught since housekeeping code must be written to carefully handle input and output products between recipes. We have addressed these issues by creating an extensible pipeline development framework that leverages the Python bindings for the ESO Common Pipeline Library (PyCPL) toolkit. Pipeline recipes can be defined in a regulated manner using existing ESO pipeline recipes or new Python recipes compliant with ESO standards. Users can easily build their own pipeline workflows for execution by the PyCPL companion package PyEsorex. The ability to define Python recipes offers a powerful means to extend existing ESO pipelines or develop entirely new pipelines. An overview of the framework is presented along with an illustrative MUSE pipeline workflow.

Original languageEnglish
Title of host publicationSoftware and Cyberinfrastructure for Astronomy VIII
EditorsJorge Ibsen, Gianluca Chiozzi
Place of PublicationWashington
PublisherSPIE
Pages1-6
Number of pages6
ISBN (Electronic)9781510675261
ISBN (Print)9781510675254
DOIs
Publication statusPublished - 2024
EventSoftware and Cyberinfrastructure for Astronomy VIII 2024 - Yokohama, Japan
Duration: 16 Jun 202421 Jun 2024

Publication series

NameProceedings of SPIE
PublisherSPIE
Volume13101
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X

Conference

ConferenceSoftware and Cyberinfrastructure for Astronomy VIII 2024
Country/TerritoryJapan
CityYokohama
Period16/06/2421/06/24

Keywords

  • Manuscript format
  • template
  • SPIE Proceedings
  • LaTeX

Cite this