A contextualized transformer-based method for cyberbullying detection

Automatic detection of Cyberbullying is a challenging task due to the availability of limited trained data, which is usually noisy and inherently multimodal. Transfer learning over pre-trained BERT-based language models has succeeded in various complex use cases like sequence-to-sequence translation and text classification. These methods mainly utilize transformer models to learn the word and sentence-level relationships. While they have demonstrated promising results, they only focus on textual features without taking contextual and structural information into account. Moreover, due to the data-heavy nature of BERT-based models, they may fail to model all the desired relationships if not adequate training data is provided to them during the fine-tuning process. In this paper, we propose a novel Session-level Contextualized Transformer-based architecture for Cyberbullying Detection (SECTR-CD), which can leverage transfer learning for modeling word-level attention while also being able to model sentence-level relationships in large bodies of text. The model is also capable of utilizing other contextual features from various modalities like images and social information. Our experimental results indicate remarkable improvement in the Cyberbullying detection task even in the presence of limited training samples.