Multimodal analysis of interaction

Human communication is multimodal. When people gather together, they talk, position their bodies, shape their hands, contort their faces, direct their gaze, inter alia, in the course of communicative acts. Few would question that these behaviors are important for communication, but recognizing and embracing multimodality as a defining property of human communication has far‐reaching consequences for its study. This chapter will introduce theoretical and methodological resources for exploring multimodality in real‐time, interactive human communication. Although there is a growing body of research exploring multimodal interaction instrumentally and experimentally (see, e.g., Holler, 2022), the concepts and analytic units that form the basis of these studies are commonly derived from observational studies of interaction, and particularly those employing conversation analysis (see Chapter 6). As such, we will principally draw on multimodal conversation analysis (and related work) in this chapter, and set out methodological strategies suited to observational research. Importantly, too, we will argue that observational studies employing systematic, principled approaches to multimodality and interaction are essential for deepening our understanding of communication disability. The unique constellations of semiotic resources that people with communication disabilities use in their interactions require careful consideration with reference to the foundational meaning‐making processes that are intrinsic to, and enabling of, all forms of human communication.