Speaker-independent visual speech recognition with the inception v3 model

Timothy Santos; Andrew Abel; Nick Wilson; Yan Xu

Speaker-independent visual speech recognition with the inception v3 model

Timothy Santos, Andrew Abel, Nick Wilson, Yan Xu

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution

Abstract

The natural process of understanding speech involves combining auditory and visual cues. CNN based lip reading systems have become very popular in recent years. However, many of these systems consider lipreading to be a black box problem, with limited detailed performance analysis. In this paper, we performed transfer learning by training the Inception v3 CNN model, which has pre-trained weights produced from IMAGENET, with the GRID corpus, delivering good speech recognition results, with 0.61 precision, 0.53 recall, and 0.51F1-score. The lip reading model was able to automatically learn pertinent features, demonstrated using visualisation, and achieve good speaker-independent results. We also identify limitations that match those of humans, therefore limiting potential deep learning performance in real world situations.

Original language	English
Title of host publication	Proceedings of SLT 2021
Subtitle of host publication	IEEE Spoken Language Technology Workshop
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Publication status	Accepted/In press - 22 Jan 2021
Event	IEEE Workshop on Spoken Language Technology - Shenzhen, China Duration: 19 Jan 2021 → 22 Jan 2021 http://2021.ieeeslt.org/

Conference

Conference	IEEE Workshop on Spoken Language Technology
Abbreviated title	SLT2021
Country	China
City	Shenzhen
Period	19/01/21 → 22/01/21
Internet address	http://2021.ieeeslt.org/

Keywords

deep learning
lip-reading
visual speech recognition

Fingerprint Dive into the research topics of 'Speaker-independent visual speech recognition with the inception v3 model'. Together they form a unique fingerprint.

Cite this

@inproceedings{0dda3d4bfe6d432dbc684842e61840de,

title = "Speaker-independent visual speech recognition with the inception v3 model",

abstract = "The natural process of understanding speech involves combining auditory and visual cues. CNN based lip reading systems have become very popular in recent years. However, many of these systems consider lipreading to be a black box problem, with limited detailed performance analysis. In this paper, we performed transfer learning by training the Inception v3 CNN model, which has pre-trained weights produced from IMAGENET, with the GRID corpus, delivering good speech recognition results, with 0.61 precision, 0.53 recall, and 0.51F1-score. The lip reading model was able to automatically learn pertinent features, demonstrated using visualisation, and achieve good speaker-independent results. We also identify limitations that match those of humans, therefore limiting potential deep learning performance in real world situations.",

keywords = "deep learning, lip-reading, visual speech recognition",

author = "Timothy Santos and Andrew Abel and Nick Wilson and Yan Xu",

year = "2021",

month = "1",

day = "22",

language = "English",

booktitle = "Proceedings of SLT 2021",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

}

TY - GEN

T1 - Speaker-independent visual speech recognition with the inception v3 model

AU - Santos, Timothy

AU - Abel, Andrew

AU - Wilson, Nick

AU - Xu, Yan

PY - 2021/1/22

Y1 - 2021/1/22

N2 - The natural process of understanding speech involves combining auditory and visual cues. CNN based lip reading systems have become very popular in recent years. However, many of these systems consider lipreading to be a black box problem, with limited detailed performance analysis. In this paper, we performed transfer learning by training the Inception v3 CNN model, which has pre-trained weights produced from IMAGENET, with the GRID corpus, delivering good speech recognition results, with 0.61 precision, 0.53 recall, and 0.51F1-score. The lip reading model was able to automatically learn pertinent features, demonstrated using visualisation, and achieve good speaker-independent results. We also identify limitations that match those of humans, therefore limiting potential deep learning performance in real world situations.

AB - The natural process of understanding speech involves combining auditory and visual cues. CNN based lip reading systems have become very popular in recent years. However, many of these systems consider lipreading to be a black box problem, with limited detailed performance analysis. In this paper, we performed transfer learning by training the Inception v3 CNN model, which has pre-trained weights produced from IMAGENET, with the GRID corpus, delivering good speech recognition results, with 0.61 precision, 0.53 recall, and 0.51F1-score. The lip reading model was able to automatically learn pertinent features, demonstrated using visualisation, and achieve good speaker-independent results. We also identify limitations that match those of humans, therefore limiting potential deep learning performance in real world situations.

KW - deep learning

KW - lip-reading

KW - visual speech recognition

M3 - Conference proceeding contribution

BT - Proceedings of SLT 2021

PB - Institute of Electrical and Electronics Engineers (IEEE)

ER -