What does KnowBert-UMLS forget? - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

What does KnowBert-UMLS forget?

Résumé

Integrating a source of structured prior knowledge, such as a knowledge graph, into transformer-based language models is an increasingly popular method for increasing data efficiency and adapting them to a target domain. However, most methods for integrating structured knowledge into language models require additional training in order to adapt the model to the non-textual modality. This process typically leads to some amount of catastrophic forgetting on the general domain. KnowBert is one such knowledge integration method which can incorporate information from a variety of knowledge graphs to enhance the capabilities of transformer-based language models such as BERT. We conduct a qualitative analysis of the results of KnowBert-UMLS, a biomedically specialized KnowBert model, on a variety of linguistic tasks. Our results reveal that its increased understanding of biomedical concepts comes at the cost, specifically, of general common-sense knowledge and understanding of casual speech.
Fichier principal
Vignette du fichier
AICCSA_2023_Paper_IEEE_Guilhem_Piat_NoteIEEE.pdf (223.35 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-04559677 , version 1 (25-04-2024)

Identifiants

Citer

Guilhem Piat, Nasredine Semmar, Julien Tourille, Alexandre Allauzen, Hassane Essafi. What does KnowBert-UMLS forget?. AICCSA 2023 - 20th ACS/IEEE International Conference on Computer Systems and Applications, Dec 2023, Gizeh, Egypt. pp.1-8, ⟨10.1109/AICCSA59173.2023.10479333⟩. ⟨cea-04559677⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More