Optimizing cancer classification in high-dimensional data by integrating generative AI with feature selection

High-dimensional datasets, such as gene expression data, provide a variety of advantages in the healthcare sector, including enhanced patient survival rates and early cancer diagnosis. Yet, they also pose significant challenges to prediction, particularly due to insufficient training data and high computational complexity. While generative AI techniques, such as Conditional Tabular Generative Adversarial Networks (CTGAN), can help address this issue by generating synthetic data, they encounter challenges when working with high-dimensional datasets. To address these complexities and overcome the challenge of insufficient training data, we combine generative AI with feature selection, focusing on augmenting the reduced feature dataset with synthetic data to enhance the predictive accuracy of cancer classification models. We commence by employing feature selection techniques to reduce the dimensionality of gene expression data, concentrating on the most important features for cancer classification. After augmenting the reduced feature dataset with synthetic data, the approach can improve classification accuracy in some cases and generally reduce variability across repeated runs compared with using either the reduced or original datasets alone in several cases. In particular, we illustrate the efficacy of this methodology by employing publicly available gene expression data to classify breast and brain cancer. The classification process is conducted using a Feedforward Neural Network, with the Random Forest Classifier and Distance Correlation Sure Independence Screening as the feature selection methods that are employed. Our findings indicate that integrating feature selection with generative AI and augmenting reduced datasets with synthetic data can help address challenges associated with high-dimensional data and can lead to improved predictive performance.