| The overall objective of this project is to develop a robotic system to automatically train primates (either humans or monkeys) with a neurologic injury to perform a desired reaching movement for rehabilitation purposes, without explicit instruction about the target movement. We developed a system (the RL trainer) that uses a Reinforcement Learning approach to deliver a reward (food or video game points) at a speed proportional to the similarity of the user's movement to the target movement. Monkeys quickly learned the target movement with the RL trainer, but if they were too impaired, reward delivery was slow, which led to frustration. We therefore developed an assistive algorithm, the "Reward Equalizer Algorithm" (RE), to reward users with different levels of impairment at comparable rates. To pilot-test the RE, we measured 21 unimpaired human subjects as they tried to learn two target movements with the basic RL trainer, RL+RE, or RL+RE plus an anti-slacking watchdog (ASW). RE allowed normal learning of the first target movement and it increased reward when we imposed a virtual impairment on the subjects, as desired. However, when we asked subjects to learn a second target movement, RE slowed learning, and ASW further slowed learning. This was because both RE and ASW decreased exploration, quantified as movement variability. These results suggest that machine algorithms that assist trainees in achieving rewards and/or prevent slacking can discourage the exploration-based strategies needed to learn new movement. This concept is related to the exploration versus exploitation tradeoff in Reinforcement Learning theory, and has implications for design of machine-assisted motor learning and rehabilitation. |
