Self-supervised human-object interaction of complex scenes with context-aware mixing: towards in-store consumer behavior analysis

Recognizing human-object interactions (HOIs) in physical retail stores, such as picking up a product, can provide valuable information about non-purchasers, and is an important aspect of understanding customer behaviors. However, there are often complex scenes in physical retail stores with numerous similar objects in the shelf, making the task of recognizing the interacting object challenging. To address the drawback of complex background scenes, we propose a method using image mixing and self-supervised techniques to train the model to differentiate objects that interact with background objects. The proposed method generates images without the object's influence based on the input image using Context-aware image mixing. Then, we introduce a self-supervised method using the generated images to learn the difference between the actual and the background objects. We evaluated the network's performance using public and private retail dataset. We confirmed that when applied to physical retail scenes, the performance overcame the recent HOI detection methods including the recent state-of-the-art method. To the best of our knowledge, this is the first study to apply a self-supervised technique to control the target of interaction for the HOI detection model, demonstrating promising potential for use in in-store consumer behavior analysis.