Knowledge graph construction in hyperbolic space for automatic image annotation

Fariba Lotfi*, Mansour Jamzad, Hamid Beigy, Helia Farhood, Quan Z. Sheng, Amin Beheshti

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Downloads (Pure)

Abstract

Automatic image annotation (AIA) is a fundamental and challenging task in computer vision. Considering the correlations between tags can lead to more accurate image understanding, benefiting various applications, including image retrieval and visual search. While many attempts have been made to incorporate tag correlations in annotation models, the method of constructing a knowledge graph based on external knowledge sources and hyperbolic space has not been explored. In this paper, we create an attributed knowledge graph based on vocabulary, integrate external knowledge sources such as WordNet, and utilize hyperbolic word embeddings for the tag representations. These embeddings provide a sophisticated tag representation that captures hierarchical and complex correlations more effectively, enhancing the image annotation results. In addition, leveraging external knowledge sources enhances contextuality and significantly enriches existing AIA datasets. We exploit two deep learning-based models, the Relational Graph Convolutional Network (R-GCN) and the Vision Transformer (ViT), to extract the input features. We apply two R-GCN operations to obtain word descriptors and fuse them with the extracted visual features. We evaluate the proposed approach using three public benchmark datasets. Our experimental results demonstrate that the proposed architecture achieves state-of-the-art performance across most metrics on Corel5k, ESP Game, and IAPRTC-12.

Original languageEnglish
Article number105293
Pages (from-to)1-12
Number of pages12
JournalImage and Vision Computing
Volume151
DOIs
Publication statusPublished - Nov 2024

Bibliographical note

Copyright the Author(s) 2024. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

  • Automatic image annotation
  • Attributed knowledge graph
  • External knowledge sources
  • Hyperbolic space
  • Relational graph convolutional network
  • Vision transformer

Cite this