TY - GEN
T1 - Investigating the learning behaviour of in-context learning
T2 - 26th European Conference on Artificial Intelligence, ECAI 2023
AU - Wang, Xindi
AU - Wang, Yufei
AU - Xu, Can
AU - Geng, Xiubo
AU - Zhang, Bowen
AU - Tao, Chongyang
AU - Rudzicz, Frank
AU - Mercer, Robert E.
AU - Jiang, Daxin
N1 - Copyright the Author(s) 2023. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.
PY - 2023
Y1 - 2023
N2 - Large language models (LLMs) have shown remarkable capacity for in-context learning (ICL), where learning a new task from just a few training examples is done without being explicitly pre-trained. However, despite the success of LLMs, there has been little understanding of how ICL learns the knowledge from the given prompts. In this paper, to make progress toward understanding the learning behaviour of ICL, we train the same LLMs with the same demonstration examples via ICL and supervised learning (SL), respectively, and investigate their performance under label perturbations (i.e., noisy labels and label imbalance) on a range of classification tasks. First, via extensive experiments, we find that gold labels have significant impacts on the downstream in-context performance, especially for large language models; however, imbalanced labels matter little to ICL across all model sizes. Second, when comparing with SL, we show empirically that ICL is less sensitive to label perturbations than SL, and ICL gradually attains comparable performance to SL as the model size increases.
AB - Large language models (LLMs) have shown remarkable capacity for in-context learning (ICL), where learning a new task from just a few training examples is done without being explicitly pre-trained. However, despite the success of LLMs, there has been little understanding of how ICL learns the knowledge from the given prompts. In this paper, to make progress toward understanding the learning behaviour of ICL, we train the same LLMs with the same demonstration examples via ICL and supervised learning (SL), respectively, and investigate their performance under label perturbations (i.e., noisy labels and label imbalance) on a range of classification tasks. First, via extensive experiments, we find that gold labels have significant impacts on the downstream in-context performance, especially for large language models; however, imbalanced labels matter little to ICL across all model sizes. Second, when comparing with SL, we show empirically that ICL is less sensitive to label perturbations than SL, and ICL gradually attains comparable performance to SL as the model size increases.
UR - http://www.scopus.com/inward/record.url?scp=85174481476&partnerID=8YFLogxK
U2 - 10.3233/FAIA230559
DO - 10.3233/FAIA230559
M3 - Conference proceeding contribution
AN - SCOPUS:85174481476
SN - 9781643684369
T3 - Frontiers in Artificial Intelligence and Applications
SP - 2543
EP - 2551
BT - ECAI 2023
A2 - Gal, Kobi
A2 - Nowé, Ann
A2 - Nalepa, Grzegorz J.
A2 - Fairstein, Roy
A2 - Rădulescu, Roxana
PB - IOS Press
CY - Netherlands
Y2 - 30 September 2023 through 4 October 2023
ER -