Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors

Inferring the intentions behind behavior

SWIRL (Switching Inverse Reinforcement Learning) infers changing goals from behavioral trajectories. It models long sequences as transitions between hidden decision-making modes, each with its own reward function and policy.

Method

• Goal switching: transitions between hidden modes depend on the previous mode and the animal’s current state.
• Behavioral memory: rewards and policies can depend on recent states, capturing how past experience shapes actions.
• Joint inference: expectation–maximization alternates between inferring behavioral modes and learning their transitions, rewards, and policies.

Experimental findings

• Gridworld: combining both levels of history dependence improves reward recovery and behavioral segmentation.
• Mouse navigation: recovers interpretable water-seeking, home-seeking, and exploration modes in a 127-node labyrinth.
• Spontaneous behavior: outperforms autoregressive HMM baselines in held-out likelihood. State-dependent switching helps, while adding action-level history does not improve this dataset.

Research connection

This work connects learning from demonstrations with interpretable reward inference, offering a way to study changing intentions under the complicated behavior.