Building a multi-sensor lab for interactive communication research: challenges, workarounds, and lessons learned

Designing multi-sensor experiments to study interactive communication is costly, complex, and often poorly documented, particularly in terms of implementation challenges and the rationale behind decision-making. Here, we aim to demystify the process by detailing the development of a custom multi-sensor lab designed to monitor comprehensive behavioural and physiological responses during interactive communication. To illustrate the system architecture, we use an implemented paradigm as a worked example: two adults engaged in natural conversation while listening to realistic background noise scenes delivered via open Sennheiser HD-800 headphones. Multi-sensor data were collected using DPA headset microphones, Tobii Pro Glasses 3, a Vicon motion capture system (with Tobii integration), BIOPAC amplifiers (PPG, EDA, ECG, respiration, temperature), a Fitbit Sense 2 smartwatch, and six Logitech BRIO 4K ultra HD video cameras recording via Open Broadcaster Software (OBS). Due to high bandwidth demands and varying sampling rates (1 Hz to 48 kHz) among the different devices, all systems recorded independently which posed a significant synchronisation challenge. A central RME soundcard delivered trigger pulses to the Tobii, BIOPAC, OBS, and Vicon systems to mark stimulus onset/offset. Redundant synchronisation mechanisms were implemented to mitigate the risk of trigger failure and bespoke processing pipelines were developed to synchronise data streams. We share key challenges and creative solutions encountered in building such a lab, to help researchers understand and anticipate common pain points and make more informed decisions when implementing their own multi-sensor setups.