DynaKey: dynamic keystroke tracking using a head-mounted camera device

Hao Zhang, Yafeng Yin*, Lei Xie, Tao Gu, Minghui You, Sanglu Lu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

Mobile and wearable devices have become more and more popular. However, the tiny touch screen leads to inefficient interaction with these devices, especially for text input. In this article, we propose DynaKey, which allows people to type on a virtual keyboard printed on a piece of article or drawn on a desk, for inputting text into a head-mounted camera device (e.g., smart glasses). By using the built-in camera and gyroscope, we capture image frames during typing and detect possible head movements, then track keys, detect fingertips, and locate keystrokes. To track the changes of keys' coordinates in images caused by natural head (i.e., camera) movements, we introduce perspective transformation to transform keys' coordinates among different frames. To detect and locate keystrokes, we utilize the variation of fingertip's coordinates across multiple frames to detect possible keystrokes for localization. To reduce the time cost, we combine gyroscope and camera to adaptively track the keys, and introduce a series of optimizations, such as keypoint detection, frame skipping, multithread processing, etc. Finally, we implement DynaKey on Android-powered devices. The extensive experimental results show that our system can efficiently track and locate the keystrokes in real time. Specifically, the average tracking deviation of the keyboard layout is less than 3 pixels and the Intersection over Union (IoU) of a key in two consecutive images is above 93%. The average keystroke localization accuracy reaches 95.5%.

Original languageEnglish
Pages (from-to)6563-6577
Number of pages15
JournalIEEE Internet of Things Journal
Volume9
Issue number9
DOIs
Publication statusPublished - 1 May 2022

Fingerprint

Dive into the research topics of 'DynaKey: dynamic keystroke tracking using a head-mounted camera device'. Together they form a unique fingerprint.

Cite this