Boot and switch: alternating distillation for zero-shot dense retrieval

Fan Jiang, Qiongkai Xu, Tom Drummond, Trevor Cohn

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

Abstract

Neural 'dense' retrieval models are state of the art for many datasets, however these models often exhibit limited domain transfer ability. Existing approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models. We present ABEL, a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings. Our technique follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever. By iterating this loop, the two components mutually enhance one another's performance. Experimental results demonstrate that our unsupervised ABEL model outperforms both leading supervised and unsupervised retrievers on the BEIR benchmark. Meanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training. By either fine-tuning ABEL on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-art results.

Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics
Subtitle of host publicationEMNLP 2023
Place of PublicationStroudsburg, PA
PublisherAssociation for Computational Linguistics (ACL)
Pages912-931
Number of pages20
ISBN (Electronic)9798891760615
DOIs
Publication statusPublished - 2023
Externally publishedYes
Event2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Singapore, Singapore
Duration: 6 Dec 202310 Dec 2023

Conference

Conference2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/TerritorySingapore
CitySingapore
Period6/12/2310/12/23

Fingerprint

Dive into the research topics of 'Boot and switch: alternating distillation for zero-shot dense retrieval'. Together they form a unique fingerprint.

Cite this