TY - JOUR
T1 - Making sense of Doppler effect for multi-modal hand motion detection
AU - Ruan, Wenjie
AU - Sheng, Quan Z.
AU - Xu, Peipei
AU - Yang, Lei
AU - Gu, Tao
AU - Shangguan, Longfei
PY - 2018/9/1
Y1 - 2018/9/1
N2 - Hand gesture is becoming an increasingly popular means of interacting with consumer electronic devices, such as mobile phones, tablets and laptops. In this paper, we present AudioGest, a device-free gesture recognition system that can accurately sense the hand in-air movement around user's devices. Compared to the state-of-the-art techniques, AudioGest is superior in using only one pair of built-in speaker and microphone, without any extra hardware or infrastructure support and with no training, to achieve multi-modal hand detection. Specifically, our system is not only able to accurately recognize various hand gestures, but also reliably estimate the hand in-air duration, average moving speed and waving range. We achieve this by transforming the device into an active sonar system that transmits inaudible audio signal and decodes the echoes of hand's movement at its microphone. We address various challenges including cleaning the noisy reflected sound signal, interpreting the echo spectrogram into hand gestures, decoding the Doppler frequency shifts into the hand waving speed and range, as well as being robust to the environmental motion and signal drifting. We extensively evaluate our system on three electronic devices under four real-world scenarios using overall 3,900 hand gestures collected by five users for more than two weeks. Our results show that AudioGest detects six hand gestures with an accuracy up to 96 percent. By distinguishing the gesture attributions, it can provide more fine-grained control commands for various applications.
AB - Hand gesture is becoming an increasingly popular means of interacting with consumer electronic devices, such as mobile phones, tablets and laptops. In this paper, we present AudioGest, a device-free gesture recognition system that can accurately sense the hand in-air movement around user's devices. Compared to the state-of-the-art techniques, AudioGest is superior in using only one pair of built-in speaker and microphone, without any extra hardware or infrastructure support and with no training, to achieve multi-modal hand detection. Specifically, our system is not only able to accurately recognize various hand gestures, but also reliably estimate the hand in-air duration, average moving speed and waving range. We achieve this by transforming the device into an active sonar system that transmits inaudible audio signal and decodes the echoes of hand's movement at its microphone. We address various challenges including cleaning the noisy reflected sound signal, interpreting the echo spectrogram into hand gestures, decoding the Doppler frequency shifts into the hand waving speed and range, as well as being robust to the environmental motion and signal drifting. We extensively evaluate our system on three electronic devices under four real-world scenarios using overall 3,900 hand gestures collected by five users for more than two weeks. Our results show that AudioGest detects six hand gestures with an accuracy up to 96 percent. By distinguishing the gesture attributions, it can provide more fine-grained control commands for various applications.
KW - audio signal
KW - device-free
KW - FFT normalization
KW - Hand gesture recognition
KW - segmentation
KW - sonar
UR - http://www.scopus.com/inward/record.url?scp=85051180439&partnerID=8YFLogxK
U2 - 10.1109/TMC.2017.2762677
DO - 10.1109/TMC.2017.2762677
M3 - Article
AN - SCOPUS:85051180439
SN - 1536-1233
VL - 17
SP - 2087
EP - 2100
JO - IEEE Transactions on Mobile Computing
JF - IEEE Transactions on Mobile Computing
IS - 9
M1 - 8067452
ER -