Transformer-based language models for factoid question answering at BioASQ9b

Urvashi Khanna; Diego Mollá

Transformer-based language models for factoid question answering at BioASQ9b

Urvashi Khanna^*, Diego Mollá

^*Corresponding author for this work

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

2 Citations (Scopus)

435 Downloads (Pure)

Abstract

In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-tuning and gradual unfreezing techniques on transformer-based language models. For factoid questions, our ALBERT-based systems ranked first in test batch 1 and fourth in test batch 2. Our DistilBERT systems outperformed the ALBERT variants in test batches 4 and 5 despite having 81% fewer parameters than ALBERT. However, we observed that gradual unfreezing had no significant impact on the model's accuracy compared to standard fine-tuning.

Original language	English
Title of host publication	CLEF 2021 Working Notes
Subtitle of host publication	Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum
Editors	Guglielmo Faggioli, Nicola Ferro, Alexis Joly, Maria Maistro, Florina Piroi
Place of Publication	Aachen, Germany
Publisher	CEUR
Pages	247-257
Number of pages	11
Publication status	Published - 2021
Event	2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 - Virtual, Bucharest, Romania Duration: 21 Sept 2021 → 24 Sept 2021

Publication series

Name	CEUR Workshop Proceedings
Publisher	CEUR
Volume	2936
ISSN (Electronic)	1613-0073

Conference

Conference	2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021
Country/Territory	Romania
City	Virtual, Bucharest
Period	21/09/21 → 24/09/21

Bibliographical note

Publisher Copyright:
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

Keywords

ALBERT
BioASQ9b
DistilBERT
Question answering
Transfer learning

Access to Document

Publisher version (open access)Final published version, 563 KBLicence: CC BY

Cite this

Khanna, Urvashi ; Mollá, Diego. / Transformer-based language models for factoid question answering at BioASQ9b. CLEF 2021 Working Notes: Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum. editor / Guglielmo Faggioli ; Nicola Ferro ; Alexis Joly ; Maria Maistro ; Florina Piroi. Aachen, Germany : CEUR, 2021. pp. 247-257 (CEUR Workshop Proceedings).

@inproceedings{a23f5ef4c3ae4df5a7601b0cb2c2b9b7,

title = "Transformer-based language models for factoid question answering at BioASQ9b",

abstract = "In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-tuning and gradual unfreezing techniques on transformer-based language models. For factoid questions, our ALBERT-based systems ranked first in test batch 1 and fourth in test batch 2. Our DistilBERT systems outperformed the ALBERT variants in test batches 4 and 5 despite having 81\% fewer parameters than ALBERT. However, we observed that gradual unfreezing had no significant impact on the model's accuracy compared to standard fine-tuning.",

keywords = "ALBERT, BioASQ9b, DistilBERT, Question answering, Transfer learning",

author = "Urvashi Khanna and Diego Moll{\'a}",

note = "Publisher Copyright: {\textcopyright} 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).; 2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 ; Conference date: 21-09-2021 Through 24-09-2021",

year = "2021",

language = "English",

series = "CEUR Workshop Proceedings",

publisher = "CEUR",

pages = "247--257",

editor = "Guglielmo Faggioli and Nicola Ferro and Alexis Joly and Maria Maistro and Florina Piroi",

booktitle = "CLEF 2021 Working Notes",

}

Khanna, U & Mollá, D 2021, Transformer-based language models for factoid question answering at BioASQ9b. in G Faggioli, N Ferro, A Joly, M Maistro & F Piroi (eds), CLEF 2021 Working Notes: Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, vol. 2936, CEUR, Aachen, Germany, pp. 247-257, 2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021, Virtual, Bucharest, Romania, 21/09/21.

Transformer-based language models for factoid question answering at BioASQ9b. / Khanna, Urvashi; Mollá, Diego.
CLEF 2021 Working Notes: Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum. ed. / Guglielmo Faggioli; Nicola Ferro; Alexis Joly; Maria Maistro; Florina Piroi. Aachen, Germany: CEUR, 2021. p. 247-257 (CEUR Workshop Proceedings; Vol. 2936).

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Transformer-based language models for factoid question answering at BioASQ9b

AU - Khanna, Urvashi

AU - Mollá, Diego

PY - 2021

Y1 - 2021

N2 - In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-tuning and gradual unfreezing techniques on transformer-based language models. For factoid questions, our ALBERT-based systems ranked first in test batch 1 and fourth in test batch 2. Our DistilBERT systems outperformed the ALBERT variants in test batches 4 and 5 despite having 81% fewer parameters than ALBERT. However, we observed that gradual unfreezing had no significant impact on the model's accuracy compared to standard fine-tuning.

AB - In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-tuning and gradual unfreezing techniques on transformer-based language models. For factoid questions, our ALBERT-based systems ranked first in test batch 1 and fourth in test batch 2. Our DistilBERT systems outperformed the ALBERT variants in test batches 4 and 5 despite having 81% fewer parameters than ALBERT. However, we observed that gradual unfreezing had no significant impact on the model's accuracy compared to standard fine-tuning.

KW - ALBERT

KW - BioASQ9b

KW - DistilBERT

KW - Question answering

KW - Transfer learning

UR - https://www.scopus.com/pages/publications/85113447473

UR - http://ceur-ws.org/Vol-2936/

M3 - Conference proceeding contribution

AN - SCOPUS:85113447473

T3 - CEUR Workshop Proceedings

SP - 247

EP - 257

BT - CLEF 2021 Working Notes

A2 - Faggioli, Guglielmo

A2 - Ferro, Nicola

A2 - Joly, Alexis

A2 - Maistro, Maria

A2 - Piroi, Florina

PB - CEUR

CY - Aachen, Germany

T2 - 2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021

Y2 - 21 September 2021 through 24 September 2021

ER -

Transformer-based language models for factoid question answering at BioASQ9b

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this