From chest X-rays to radiology reports

a multimodal machine learning approach

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

Abstract

Interpreting medical images and summarising them in the form of radiology reports is a challenging, tedious, and complex task. A radiologist provides a complete description of a medical image in the form of radiology report by describing normal or abnormal findings and providing a summary for decision making. Research shows that the radiology practice is error-prone due to the limited number of experts, increasing patient volumes, and the subjective nature of human perception. To reduce the number of diagnostic errors and to alleviate the task of radiologists, there is a need for a computer-aided report generation system that can automatically generate a radiology report for a given medical image. We propose an encoder-decoder based framework that can automatically generate radiology reports from medical images. Specifically, we use a Convolutional Neural Network as an encoder coupled with a multi-stage Stacked Long Short-Term Memory as a decoder to generate reports. We perform experiments on the Indiana University Chest X-ray collection, a publicly available dataset, to measure the effectiveness of our model. Experimental results show the effectiveness of our model in automatically generating radiology reports from medical images.

Original languageEnglish
Title of host publicationProceedings - 2019 Digital Image Computing: Techniques and Applications, DICTA 2019
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages8
ISBN (Electronic)9781728138572
DOIs
Publication statusPublished - 2019
Event2019 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2019 - Perth, Australia
Duration: 2 Dec 20194 Dec 2019

Conference

Conference2019 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2019
CountryAustralia
CityPerth
Period2/12/194/12/19

    Fingerprint

Keywords

  • Artificial Intelligence
  • Computer Vision
  • Computer-aided Report Generation
  • Convolutional Neural Network
  • Medical Imaging
  • Natural Language Processing
  • Radiology
  • Recurrent Neural Network

Cite this

Singh, S., Karimi, S., Ho-Shon, K., & Hamey, L. (2019). From chest X-rays to radiology reports: a multimodal machine learning approach. In Proceedings - 2019 Digital Image Computing: Techniques and Applications, DICTA 2019 Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/DICTA47822.2019.8945819