Partially-supervised image captioning

Peter Anderson*, Stephen Gould, Mark Johnson

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

4 Citations (Scopus)


Image captioning models are becoming increasingly successful at describing the content of images in restricted domains. However, if these models are to function in the wild - for example, as assistants for people with impaired vision - a much larger number and variety of visual concepts must be understood. To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets. Since image labels and object classes can be interpreted as partial captions, we formulate this problem as learning from partially specified sequence data. We then propose a novel algorithm for training sequence models, such as recurrent neural networks, on partially-specified sequences which we represent using finite state automata. In the context of image captioning, our method lifts the restriction that previously required image captioning models to be trained on paired image-sentence corpora only, or otherwise required specialized model architectures to take advantage of alternative data modalities. Applying our approach to an existing neural captioning model, we achieve state of the art results on the novel object captioning task using the COCO dataset. We further show that we can train a captioning model to describe new visual concepts from the Open Images dataset while maintaining competitive COCO evaluation scores.

Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 31 (NIPS 2018)
EditorsS. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett
Place of PublicationSan Diego
PublisherNeural Information Processing Systems (NIPS) Foundation
Number of pages12
Publication statusPublished - 2018
Event32nd Conference on Neural Information Processing Systems (NIPS) - Montreal, Canada
Duration: 2 Dec 20188 Dec 2018

Publication series

NameAdvances in Neural Information Processing Systems
ISSN (Print)1049-5258


Conference32nd Conference on Neural Information Processing Systems (NIPS)

Cite this

Anderson, P., Gould, S., & Johnson, M. (2018). Partially-supervised image captioning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 31 (NIPS 2018) (pp. 1-12). (Advances in Neural Information Processing Systems; Vol. 31). San Diego: Neural Information Processing Systems (NIPS) Foundation.