MS-GAN: GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging

Chaoyi Zhang, Yang Song, Sidong Liu, Scott Lill, Chenyu Wang, Zihao Tang, Yuyi You, Yang Gao, Alexander Klistorner, Michael Barnett, Weidong Cai

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionResearchpeer-review

Abstract

Automated segmentation of multiple sclerosis (MS) lesions in brain imaging is challenging due to the high variability in lesion characteristics. Based on the generative adversarial network (GAN), we propose a semantic segmentation framework MS-GAN to localize MS lesions in multimodal brain magnetic resonance imaging (MRI), which consists of one multimodal encoder-decoder generator G and multiple discriminators D corresponding to the multiple input modalities. For the design of the generator, we adopt an encoder-decoder deep learning architecture with bypass of spatial information from encoder to the corresponding decoder, which helps to reduce the network parameters while improving the localization performance. Our generator is also designed to integrate multimodal imaging data in end-to-end learning with multi-path encoding and cross-modality fusion. An additional classification-related constraint is proposed for the adversarial training process of the GAN model, with the aim of alleviating the hard-to-converge issue in classification-based image-to-image translation problems. For evaluation, we collected a database of 126 cases from patients with relapsing MS. We also experimented with other semantic segmentation models as well as patch-based deep learning methods for performance comparison. The results show that our method provides more accurate segmentation than the state-of-the-art techniques.

LanguageEnglish
Title of host publication2018 International Conference on Digital Image Computing
Subtitle of host publicationTechniques and Applications, DICTA 2018
EditorsManzur Murshed, Manoranjan Paul, Md Asikuzzaman, Mark Pickering, Ambarish Natu, Antonio Robles-Kelly, Shaodi You, Lihong Zheng, Ashfaqur Rahman
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages84-91
Number of pages8
ISBN (Electronic)9781538666029
DOIs
Publication statusPublished - 2018
Event2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018 - Canberra, Australia
Duration: 10 Dec 201813 Dec 2018

Conference

Conference2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018
CountryAustralia
CityCanberra
Period10/12/1813/12/18

Fingerprint

Magnetic resonance
Semantics
Multiple Sclerosis
Brain
Magnetic Resonance Imaging
Imaging techniques
Learning
Discriminators
Multimodal Imaging
Fusion reactions
Neuroimaging
Databases
Deep learning

Keywords

  • GAN
  • Image-to-image Translation
  • Multiple Sclerosis
  • Semantic Segmentation

Cite this

Zhang, C., Song, Y., Liu, S., Lill, S., Wang, C., Tang, Z., ... Cai, W. (2018). MS-GAN: GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging. In M. Murshed, M. Paul, M. Asikuzzaman, M. Pickering, A. Natu, A. Robles-Kelly, S. You, L. Zheng, ... A. Rahman (Eds.), 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018 (pp. 84-91). [8615771] Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/DICTA.2018.8615771
Zhang, Chaoyi ; Song, Yang ; Liu, Sidong ; Lill, Scott ; Wang, Chenyu ; Tang, Zihao ; You, Yuyi ; Gao, Yang ; Klistorner, Alexander ; Barnett, Michael ; Cai, Weidong. / MS-GAN : GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging. 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018. editor / Manzur Murshed ; Manoranjan Paul ; Md Asikuzzaman ; Mark Pickering ; Ambarish Natu ; Antonio Robles-Kelly ; Shaodi You ; Lihong Zheng ; Ashfaqur Rahman. Piscataway, NJ : Institute of Electrical and Electronics Engineers (IEEE), 2018. pp. 84-91
@inproceedings{0bc9c4cf173e4d2eaa9cdc76a34c5bba,
title = "MS-GAN: GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging",
abstract = "Automated segmentation of multiple sclerosis (MS) lesions in brain imaging is challenging due to the high variability in lesion characteristics. Based on the generative adversarial network (GAN), we propose a semantic segmentation framework MS-GAN to localize MS lesions in multimodal brain magnetic resonance imaging (MRI), which consists of one multimodal encoder-decoder generator G and multiple discriminators D corresponding to the multiple input modalities. For the design of the generator, we adopt an encoder-decoder deep learning architecture with bypass of spatial information from encoder to the corresponding decoder, which helps to reduce the network parameters while improving the localization performance. Our generator is also designed to integrate multimodal imaging data in end-to-end learning with multi-path encoding and cross-modality fusion. An additional classification-related constraint is proposed for the adversarial training process of the GAN model, with the aim of alleviating the hard-to-converge issue in classification-based image-to-image translation problems. For evaluation, we collected a database of 126 cases from patients with relapsing MS. We also experimented with other semantic segmentation models as well as patch-based deep learning methods for performance comparison. The results show that our method provides more accurate segmentation than the state-of-the-art techniques.",
keywords = "GAN, Image-to-image Translation, Multiple Sclerosis, Semantic Segmentation",
author = "Chaoyi Zhang and Yang Song and Sidong Liu and Scott Lill and Chenyu Wang and Zihao Tang and Yuyi You and Yang Gao and Alexander Klistorner and Michael Barnett and Weidong Cai",
year = "2018",
doi = "10.1109/DICTA.2018.8615771",
language = "English",
pages = "84--91",
editor = "Manzur Murshed and Manoranjan Paul and Md Asikuzzaman and Mark Pickering and Ambarish Natu and Antonio Robles-Kelly and Shaodi You and Lihong Zheng and Ashfaqur Rahman",
booktitle = "2018 International Conference on Digital Image Computing",
publisher = "Institute of Electrical and Electronics Engineers (IEEE)",
address = "United States",

}

Zhang, C, Song, Y, Liu, S, Lill, S, Wang, C, Tang, Z, You, Y, Gao, Y, Klistorner, A, Barnett, M & Cai, W 2018, MS-GAN: GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging. in M Murshed, M Paul, M Asikuzzaman, M Pickering, A Natu, A Robles-Kelly, S You, L Zheng & A Rahman (eds), 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018., 8615771, Institute of Electrical and Electronics Engineers (IEEE), Piscataway, NJ, pp. 84-91, 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018, Canberra, Australia, 10/12/18. https://doi.org/10.1109/DICTA.2018.8615771

MS-GAN : GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging. / Zhang, Chaoyi; Song, Yang; Liu, Sidong; Lill, Scott; Wang, Chenyu; Tang, Zihao; You, Yuyi; Gao, Yang; Klistorner, Alexander; Barnett, Michael; Cai, Weidong.

2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018. ed. / Manzur Murshed; Manoranjan Paul; Md Asikuzzaman; Mark Pickering; Ambarish Natu; Antonio Robles-Kelly; Shaodi You; Lihong Zheng; Ashfaqur Rahman. Piscataway, NJ : Institute of Electrical and Electronics Engineers (IEEE), 2018. p. 84-91 8615771.

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionResearchpeer-review

TY - GEN

T1 - MS-GAN

T2 - GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging

AU - Zhang, Chaoyi

AU - Song, Yang

AU - Liu, Sidong

AU - Lill, Scott

AU - Wang, Chenyu

AU - Tang, Zihao

AU - You, Yuyi

AU - Gao, Yang

AU - Klistorner, Alexander

AU - Barnett, Michael

AU - Cai, Weidong

PY - 2018

Y1 - 2018

N2 - Automated segmentation of multiple sclerosis (MS) lesions in brain imaging is challenging due to the high variability in lesion characteristics. Based on the generative adversarial network (GAN), we propose a semantic segmentation framework MS-GAN to localize MS lesions in multimodal brain magnetic resonance imaging (MRI), which consists of one multimodal encoder-decoder generator G and multiple discriminators D corresponding to the multiple input modalities. For the design of the generator, we adopt an encoder-decoder deep learning architecture with bypass of spatial information from encoder to the corresponding decoder, which helps to reduce the network parameters while improving the localization performance. Our generator is also designed to integrate multimodal imaging data in end-to-end learning with multi-path encoding and cross-modality fusion. An additional classification-related constraint is proposed for the adversarial training process of the GAN model, with the aim of alleviating the hard-to-converge issue in classification-based image-to-image translation problems. For evaluation, we collected a database of 126 cases from patients with relapsing MS. We also experimented with other semantic segmentation models as well as patch-based deep learning methods for performance comparison. The results show that our method provides more accurate segmentation than the state-of-the-art techniques.

AB - Automated segmentation of multiple sclerosis (MS) lesions in brain imaging is challenging due to the high variability in lesion characteristics. Based on the generative adversarial network (GAN), we propose a semantic segmentation framework MS-GAN to localize MS lesions in multimodal brain magnetic resonance imaging (MRI), which consists of one multimodal encoder-decoder generator G and multiple discriminators D corresponding to the multiple input modalities. For the design of the generator, we adopt an encoder-decoder deep learning architecture with bypass of spatial information from encoder to the corresponding decoder, which helps to reduce the network parameters while improving the localization performance. Our generator is also designed to integrate multimodal imaging data in end-to-end learning with multi-path encoding and cross-modality fusion. An additional classification-related constraint is proposed for the adversarial training process of the GAN model, with the aim of alleviating the hard-to-converge issue in classification-based image-to-image translation problems. For evaluation, we collected a database of 126 cases from patients with relapsing MS. We also experimented with other semantic segmentation models as well as patch-based deep learning methods for performance comparison. The results show that our method provides more accurate segmentation than the state-of-the-art techniques.

KW - GAN

KW - Image-to-image Translation

KW - Multiple Sclerosis

KW - Semantic Segmentation

UR - http://www.scopus.com/inward/record.url?scp=85062235873&partnerID=8YFLogxK

U2 - 10.1109/DICTA.2018.8615771

DO - 10.1109/DICTA.2018.8615771

M3 - Conference proceeding contribution

SP - 84

EP - 91

BT - 2018 International Conference on Digital Image Computing

A2 - Murshed, Manzur

A2 - Paul, Manoranjan

A2 - Asikuzzaman, Md

A2 - Pickering, Mark

A2 - Natu, Ambarish

A2 - Robles-Kelly, Antonio

A2 - You, Shaodi

A2 - Zheng, Lihong

A2 - Rahman, Ashfaqur

PB - Institute of Electrical and Electronics Engineers (IEEE)

CY - Piscataway, NJ

ER -

Zhang C, Song Y, Liu S, Lill S, Wang C, Tang Z et al. MS-GAN: GAN-based semantic segmentation of multiple sclerosis lesions in brain magnetic resonance imaging. In Murshed M, Paul M, Asikuzzaman M, Pickering M, Natu A, Robles-Kelly A, You S, Zheng L, Rahman A, editors, 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018. Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE). 2018. p. 84-91. 8615771 https://doi.org/10.1109/DICTA.2018.8615771