Credit assignment in movement-dependent reinforcement learning

Samuel D. McDougle*, Matthew J. Boggess, Matthew J. Crossley, Darius Parvin, Richard B. Ivry, Jordan A. Taylor

*Corresponding author for this work

Research output: Contribution to journalArticle

18 Citations (Scopus)

Abstract

When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model whereinmovement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tractable solution to the credit assignment problem.

Original languageEnglish
Pages (from-to)6797-6802
Number of pages6
JournalProceedings of the National Academy of Sciences of the United States of America
Volume113
Issue number24
DOIs
Publication statusPublished - 14 Jun 2016
Externally publishedYes

Keywords

  • decision-making
  • reinforcement learning
  • sensory prediction error
  • reward prediction error
  • cerebellum

Fingerprint Dive into the research topics of 'Credit assignment in movement-dependent reinforcement learning'. Together they form a unique fingerprint.

  • Cite this