DeepCT: tomographic combinatorial testing for deep learning systems

Lei Ma*, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

29 Citations (Scopus)

Abstract

Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

Original languageEnglish
Title of host publicationSANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering
EditorsXinyu Wang, David Lo, Emad Shihab
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages614-618
Number of pages5
ISBN (Electronic)9781728105918
DOIs
Publication statusPublished - 2019
Event26th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2019 - Hangzhou, China
Duration: 24 Feb 201927 Feb 2019

Publication series

NameSANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering

Conference

Conference26th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2019
CountryChina
CityHangzhou
Period24/02/1927/02/19

Keywords

  • combinatorial testing
  • Deep learning
  • robustness

Fingerprint Dive into the research topics of '<i>DeepCT</i>: tomographic combinatorial testing for deep learning systems'. Together they form a unique fingerprint.

Cite this