Abstract
The natural process of understanding speech involves combining auditory and visual cues. CNN based lip reading systems have become very popular in recent years. However, many of these systems consider lipreading to be a black box problem, with limited detailed performance analysis. In this paper, we performed transfer learning by training the Inception v3 CNN model, which has pre-trained weights produced from IMAGENET, with the GRID corpus, delivering good speech recognition results, with 0.61 precision, 0.53 recall, and 0.51 F1-score. The lip reading model was able to automatically learn pertinent features, demonstrated using visualisation, and achieve speaker-independent results comparable to human lip readers on the GRID corpus. We also identify limitations that match those of humans, therefore limiting potential deep learning performance in real world situations.
Original language | English |
---|---|
Title of host publication | Proceedings of SLT 2021 |
Subtitle of host publication | IEEE Spoken Language Technology Workshop |
Place of Publication | Piscataway, NJ |
Publisher | Institute of Electrical and Electronics Engineers (IEEE) |
Pages | 613-620 |
Number of pages | 8 |
ISBN (Electronic) | 9781728170664 |
DOIs | |
Publication status | Published - 2021 |
Event | IEEE Workshop on Spoken Language Technology - Shenzhen, China Duration: 19 Jan 2021 → 22 Jan 2021 http://2021.ieeeslt.org/ |
Publication series
Name | IEEE Workshop on Spoken Language Technology |
---|---|
Publisher | IEEE |
ISSN (Print) | 2639-5479 |
Conference
Conference | IEEE Workshop on Spoken Language Technology |
---|---|
Abbreviated title | SLT2021 |
Country/Territory | China |
City | Shenzhen |
Period | 19/01/21 → 22/01/21 |
Internet address |
Keywords
- deep learning
- lip-reading
- visual speech recognition