Testing learning-enabled cyber-physical systems with large-language models: a formal approach

Xi Zheng; Aloysius K. Mok; Ruzica Piskac; Yong Jae Lee; Bhaskar Krishnamachari; Dakai Zhu; Oleg Sokolsky; Insup Lee

doi:10.1145/3663529.3663779

Testing learning-enabled cyber-physical systems with large-language models: a formal approach

Xi Zheng, Aloysius K. Mok, Ruzica Piskac, Yong Jae Lee, Bhaskar Krishnamachari, Dakai Zhu, Oleg Sokolsky, Insup Lee

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

1 Citation (Scopus)

27 Downloads (Pure)

Abstract

The integration of machine learning into cyber-physical systems (CPS) promises enhanced efficiency and autonomous capabilities, revolutionizing fields like autonomous vehicles and telemedicine. This evolution necessitates a shift in the software development life cycle, where data and learning are pivotal. Traditional verification and validation methods are inadequate for these AI-driven systems. This study focuses on the challenges in ensuring safety in learning-enabled CPS. It emphasizes the role of testing as a primary method for verification and validation, critiques current methodologies, and advocates for a more rigorous approach to assure formal safety.

Original language	English
Title of host publication	FSE Companion '24
Subtitle of host publication	companion proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering
Editors	Marcelo D'Amorim
Place of Publication	New York, NY
Publisher	Association for Computing Machinery (ACM)
Pages	467-471
Number of pages	5
ISBN (Electronic)	9798400706585
DOIs	https://doi.org/10.1145/3663529.3663779
Publication status	Published - 2024
Event	ACM International Conference on the Foundations of Software Engineering (32nd : 2024) - Porto de Galinhas, Brazil Duration: 15 Jul 2024 → 19 Jul 2024

Conference

Conference	ACM International Conference on the Foundations of Software Engineering (32nd : 2024)
Country/Territory	Brazil
City	Porto de Galinhas
Period	15/07/24 → 19/07/24

Bibliographical note

Copyright the Author(s) 2024. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

AI-based Systems
LLM-based Testing
automata-learning
model-based testing

Access to Document

10.1145/3663529.3663779

Publisher version (open access)Final published version, 587 KBLicence: CC BY

Cite this

Zheng, X., Mok, A. K., Piskac, R., Lee, Y. J., Krishnamachari, B., Zhu, D., Sokolsky, O., & Lee, I. (2024). Testing learning-enabled cyber-physical systems with large-language models: a formal approach. In M. D'Amorim (Ed.), FSE Companion '24: companion proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (pp. 467-471). Association for Computing Machinery (ACM). https://doi.org/10.1145/3663529.3663779

Zheng, Xi ; Mok, Aloysius K. ; Piskac, Ruzica et al. / Testing learning-enabled cyber-physical systems with large-language models : a formal approach. FSE Companion '24: companion proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. editor / Marcelo D'Amorim. New York, NY : Association for Computing Machinery (ACM), 2024. pp. 467-471

@inproceedings{b5e180e511a245d3bbedb21a4a77738c,

title = "Testing learning-enabled cyber-physical systems with large-language models: a formal approach",

abstract = "The integration of machine learning into cyber-physical systems (CPS) promises enhanced efficiency and autonomous capabilities, revolutionizing fields like autonomous vehicles and telemedicine. This evolution necessitates a shift in the software development life cycle, where data and learning are pivotal. Traditional verification and validation methods are inadequate for these AI-driven systems. This study focuses on the challenges in ensuring safety in learning-enabled CPS. It emphasizes the role of testing as a primary method for verification and validation, critiques current methodologies, and advocates for a more rigorous approach to assure formal safety.",

keywords = "AI-based Systems, LLM-based Testing, automata-learning, model-based testing",

author = "Xi Zheng and Mok, {Aloysius K.} and Ruzica Piskac and Lee, {Yong Jae} and Bhaskar Krishnamachari and Dakai Zhu and Oleg Sokolsky and Insup Lee",

note = "Copyright the Author(s) 2024. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.; ACM International Conference on the Foundations of Software Engineering (32nd : 2024) ; Conference date: 15-07-2024 Through 19-07-2024",

year = "2024",

doi = "10.1145/3663529.3663779",

language = "English",

pages = "467--471",

editor = "Marcelo D'Amorim",

booktitle = "FSE Companion '24",

publisher = "Association for Computing Machinery (ACM)",

address = "United States",

}

Zheng, X, Mok, AK, Piskac, R, Lee, YJ, Krishnamachari, B, Zhu, D, Sokolsky, O & Lee, I 2024, Testing learning-enabled cyber-physical systems with large-language models: a formal approach. in M D'Amorim (ed.), FSE Companion '24: companion proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. Association for Computing Machinery (ACM), New York, NY, pp. 467-471, ACM International Conference on the Foundations of Software Engineering (32nd : 2024), Porto de Galinhas, Brazil, 15/07/24. https://doi.org/10.1145/3663529.3663779

Testing learning-enabled cyber-physical systems with large-language models: a formal approach. / Zheng, Xi; Mok, Aloysius K.; Piskac, Ruzica et al.
FSE Companion '24: companion proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. ed. / Marcelo D'Amorim. New York, NY: Association for Computing Machinery (ACM), 2024. p. 467-471.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Testing learning-enabled cyber-physical systems with large-language models

T2 - ACM International Conference on the Foundations of Software Engineering (32nd : 2024)

AU - Zheng, Xi

AU - Mok, Aloysius K.

AU - Piskac, Ruzica

AU - Lee, Yong Jae

AU - Krishnamachari, Bhaskar

AU - Zhu, Dakai

AU - Sokolsky, Oleg

AU - Lee, Insup

N1 - Copyright the Author(s) 2024. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2024

Y1 - 2024

N2 - The integration of machine learning into cyber-physical systems (CPS) promises enhanced efficiency and autonomous capabilities, revolutionizing fields like autonomous vehicles and telemedicine. This evolution necessitates a shift in the software development life cycle, where data and learning are pivotal. Traditional verification and validation methods are inadequate for these AI-driven systems. This study focuses on the challenges in ensuring safety in learning-enabled CPS. It emphasizes the role of testing as a primary method for verification and validation, critiques current methodologies, and advocates for a more rigorous approach to assure formal safety.

AB - The integration of machine learning into cyber-physical systems (CPS) promises enhanced efficiency and autonomous capabilities, revolutionizing fields like autonomous vehicles and telemedicine. This evolution necessitates a shift in the software development life cycle, where data and learning are pivotal. Traditional verification and validation methods are inadequate for these AI-driven systems. This study focuses on the challenges in ensuring safety in learning-enabled CPS. It emphasizes the role of testing as a primary method for verification and validation, critiques current methodologies, and advocates for a more rigorous approach to assure formal safety.

KW - AI-based Systems

KW - LLM-based Testing

KW - automata-learning

KW - model-based testing

UR - https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=mq-pure-production&SrcAuth=WosAPI&KeyUT=WOS:001273423600048&DestLinkType=FullRecord&DestApp=WOS_CPL

UR - http://www.scopus.com/inward/record.url?scp=85199085406&partnerID=8YFLogxK

U2 - 10.1145/3663529.3663779

DO - 10.1145/3663529.3663779

M3 - Conference proceeding contribution

SP - 467

EP - 471

BT - FSE Companion '24

A2 - D'Amorim, Marcelo

PB - Association for Computing Machinery (ACM)

CY - New York, NY

Y2 - 15 July 2024 through 19 July 2024

ER -

Zheng X, Mok AK, Piskac R, Lee YJ, Krishnamachari B, Zhu D et al. Testing learning-enabled cyber-physical systems with large-language models: a formal approach. In D'Amorim M, editor, FSE Companion '24: companion proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. New York, NY: Association for Computing Machinery (ACM). 2024. p. 467-471 doi: 10.1145/3663529.3663779

Testing learning-enabled cyber-physical systems with large-language models: a formal approach

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Cite this