Explicit knowledge integration for knowledge-aware visual question answering about named entities - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Explicit knowledge integration for knowledge-aware visual question answering about named entities

Résumé

Recent years have shown an unprecedented growth of interest in Vision-Language related tasks, with the need to address the inherent challenges of integrating linguistic and visual information to solve real-world applications. Such a typical task is Visual Question Answering (VQA), which aims at answering questions about visual content. The limitations of the VQA task in terms of question redundancy and poor linguistic variability encouraged researchers to propose Knowledge-aware Visual Question Answering tasks as a natural extension of VQA. In this paper, we tackle the KVQAE (Knowledge-based Visual Question Answering about named Entities) task, which proposes to answer questions about named entities defined in a knowledge base and grounded in a visual content. In particular, beside the textual and visual information, we propose to leverage the structural information extracted from syntactic dependency trees and external knowledge graphs to help answer questions about a large spectrum of entities of various types. Thus, by combining contextual and graph-based representations using Graph Convolutional Networks (GCNs), we are able to learn meaningful embeddings for information retrieval tasks. Experiments on the KVQAE public dataset show how our approach improves the state-of-the art baselines while demonstrating the interest of injecting external knowledge to enhance multimodal information retrieval.
Fichier principal
Vignette du fichier
ICMR2023_NoteACM.pdf (3.49 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-04172061 , version 1 (27-07-2023)

Licence

Paternité

Identifiants

Citer

Omar Adjali, Paul Grimal, Olivier Ferret, Sahar Ghannay, Hervé Le Borgne. Explicit knowledge integration for knowledge-aware visual question answering about named entities. ICRM'23 - ACM International Conference on Multimedia Retrieval, ACM, Jun 2023, Thessalonique, Greece. pp.29-38, ⟨10.1145/3591106.3592227⟩. ⟨cea-04172061⟩
36 Consultations
91 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More