Watching viewers watch subtitled videos. Audiovisual and linguistic factors in subtitle processing

How is reading subtitles different from reading a book? Unlike when reading a book, when watching subtitled videos, people are busy processing the sequences of images, sounds and text, at the same time trying to understand the action and construe the narrative. Although film watching may seem like a passive activity, it is in fact extremely complex and requires the use of multiple cognitive resources. Typically, viewers move their eyes to different points on screen about two to five times per second, which in total amounts to over 20 thousand eye movements in an average 90-minute film (Smith, 2013). But what are the factors that determine where viewers look and for how long? How is viewers’ gaze related to the audiovisual nature of the film and to the linguistic and technical parameters of subtitles? What is the role of viewers’ individual characteristics in this highly complex cognitive process? Our project addresses these questions by undertaking a series of eye tracking experiments that help us watch viewers watch subtitled videos in order to investigate key factors involved in subtitle processing.
The main goal of this project is to understand the impact of audiovisual and linguistic factors in the processing of subtitled videos. Our international team of researchers will conduct four eye-tracking experiments with viewers in three different countries (Poland, the UK and Australia) on two continents. The experiments will be carried out in three incremental stages, building up on each other. By conducting this project, we will gain novel insights into the role of four key factors in subtitle processing: concurrent video, concurrent sound, subtitle speed and the language of the subtitles.
The project answers the following research questions:
1. What is the impact of video characteristics, such as the presence of concurrent video and concurrent sound, on subtitle processing by viewers?
2. What is the role of subtitle parameters, such as subtitle speed and the language of subtitles, in the processing of subtitled videos?
3. Can individual viewers’ characteristics in any way predict subtitle processing and viewing patterns?
The outcome variables include indicators of online processing measured with eye tracking as well as comprehension, cognitive load and immersion. We will also explore the impact of viewers’ individual characteristics, such as language proficiency, working memory capacity and immersive tendency, with a view to discovering how factors related to the nature of audiovisual stimuli are modulated by individual viewers’ characteristics. Applying a mixed-methods approach, we will also conduct semi-structured interviews with participants to gain insights into their perception and opinion on different aspects of subtitle reception that are not discernible from quantitative data.