Temporal relation between speech and co-verbal iconic gestures in multimodal interface design

An important issue, in order to build multimodal interfaces that enable the user to communicate with computer systems in a more natural manner, especially using speech and gestures, is how to relate speech with co-verbal gestures. This paper investigates how speech and iconic gestures are related with each other at lexical level. At lexical level we study the timing relationships of iconic hand gestures and the related words in speech. We also try to find common hand gestures used in object descriptions. Our experimental results show that the overwhelming majority of hand gestures in object descriptions precede the related words in users’ speech within 2 seconds at lexical level. The results also indicate that people use some similar gestures when they describe the same parts of the objects.