Face-Cap: image captioning using facial expression analysis

Omid Mohamad Nezami*, Mark Dras, Peter Anderson, Leonard Hamey

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

8 Citations (Scopus)


Image captioning is the process of generating a natural language description of an image. Most current image captioning models, however, do not take into account the emotional aspect of an image, which is very relevant to activities and interpersonal relationships represented therein. Towards developing a model that can produce human-like captions incorporating these, we use facial expression features extracted from images including human faces, with the aim of improving the descriptive ability of the model. In this work, we present two variants of our Face-Cap model, which embed facial expression features in different ways, to generate image captions. Using all standard evaluation metrics, our Face-Cap models outperform a state-of-the-art baseline model for generating image captions when applied to an image caption dataset extracted from the standard Flickr 30 K dataset, consisting of around 11 K images containing faces. An analysis of the captions finds that, perhaps surprisingly, the improvement in caption quality appears to come not from the addition of adjectives linked to emotional aspects of the images, but from more variety in the actions described in the captions. Code related to this paper is available at: https://github.com/omidmn/Face-Cap.

Original languageEnglish
Title of host publicationMachine Learning and Principles and Practice of Knowledge Discovery in Databases
Subtitle of host publicationEuropean Conference, ECML-PKDD 2018. Proceedings, Part I
EditorsMichele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, Georgiana Ifrim
Place of PublicationCham, Switzerland
PublisherSpringer, Springer Nature
Number of pages15
ISBN (Electronic)9783030109257
ISBN (Print)9783030109240
Publication statusPublished - 2019
EventEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2018 - Dublin, Ireland
Duration: 10 Sep 201814 Sep 2018

Publication series

NameLecture Notes in Artificial Intelligence
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2018


  • Image captioning
  • Facial expression recognition
  • Sentiment analysis
  • Deep learning


Dive into the research topics of 'Face-Cap: image captioning using facial expression analysis'. Together they form a unique fingerprint.

Cite this