Fellowship: Robust Self Supervised Audio Visual Speech Recognition

#artificialintelligence #arxiv #datascience #encoding #machinelearning #deeplearning #speechrecognition
Link to paper: paperswithcode.com/paper/robu...
Paper by: Bowen Shi, Wei-Ning Hsu, Abdelrahman Mohamed
Presentation by Fellowship.ai team: www.fellowship.ai/
Fellowship.ai is brought to you by Launchpad.ai: www.launchpad.ai/
Launchpad brings cutting-edge technologies and AI applications to organizations, to learn more about our products and services check: www.launchpad.ai/ai-developme...
_______________________________________________________________
Abstract: Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe. Audio-visual speech recognition (AVSR) systems improve robustness by complementing the audio stream with the visual information that is invariant to noise and helps the model focus on the desired speaker. However, previous AVSR work focused solely on the supervised learning setup; hence the progress was hindered by the amount of labeled data available. In this work, we present a self-supervised AVSR framework built upon Audio-Visual HuBERT (AV-HuBERT), a state-of-the-art audio-visual speech representation learning model. On the largest available AVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by ~50% (28.0% vs. 14.1%) using less than 10% of labeled data (433hr vs. 30hr) in the presence of babble noise, while reducing the WER of an audio-based model by over 75% (25.8% vs. 5.8%) on average. Code and demo are available at github.com/facebookresearch/a...