Reinforcement Learning Primer: Imitation Learning and Inverse Reinforcement Learning

Teaching a robot to place a cup safely, avoid a person in a narrow aisle, or operate a complex tool is often harder than it sounds because the reward is hard to specify. Rewarding distance to a goal may encourage pushing objects aside; rewarding only final success may give exploration no useful signal. Imitation learning uses demonstrations from people, teleoperators, or an existing policy to learn actions from observations. Inverse reinforcement learning (IRL) instead tries to infer the objective—or reward—that makes demonstrated behavior sensible, then derives a policy for that objective.

Neither method becomes safe merely because it uses human data. The central questions are what a policy does in states absent from demonstrations, how labels are timed and attributed, how failures are collected, and what independently limits a physical action. This article links Behavior Cloning (BC), covariate shift, DAgger, IRL, Vision-Language-Action models, evaluation, safety, and data provenance. See Reinforcement Learning Basics, PPO and SAC, and VLA Primer for adjacent concepts.

Practical conclusion

• BC is supervised learning from demonstrated state s to expert action a. It is fast to start, but a small early error can move the policy into a state distribution it never saw.
• DAgger runs the current policy in a controlled setting, asks an expert for the action in states that policy actually visits, aggregates those labels, and retrains. It turns failure modes into data, but requires safe execution and reliable expert intervention.
• IRL estimates a reward r_\theta(s,a) from demonstrations and then optimizes a policy for it. Reward is not uniquely identified; judge it by behavior in held-out conditions and explicit safety constraints, not by a plausible-looking heatmap.
• A VLA can be a large conditional imitation policy, but language and image scale do not remove physical limits, timing, contact safety, or a safe-stop path.

Demonstrations are operational records, not just videos

A usable demonstration joins observations (images, depth, force, joint state), actions (joint command, velocity, gripper, stop), timestamps, frames, task instruction, success/failure, operator, environment conditions, and intervention events. If the logged command is what arrived at a robot after transport delay rather than what the operator intended, the delay belongs in the dataset. A 100 ms image-to-arm offset turns correct behavior into inconsistent training pairs.

Data provenance includes collector consent, permission for the recording environment, privacy, third-party works, robot safety responsibility, and the ability to trace a sample to its source. Mixing a public dataset requires checking license, commercial-use terms, redistribution, people and audio, and calibration for the intended use. “Found on the internet” is not provenance; it is an absence of traceable consent and terms.

Diagram: Duskcoil, conceptual. A rollout is real data collection; the diagram does not imply that safety monitoring, stop logic, or operator intervention may be omitted.

Behavior Cloning and covariate shift

Given expert pairs D=\{(s_i,a_i^*)\}_{i=1}^N, continuous-action BC may minimize

Discrete actions use cross-entropy. If more than one action is valid, a single squared-error prediction can average “pass left” and “pass right” into an unsafe middle action; conditional distributions, mixture models, or diffusion-style policies can represent multiple modes. Context, operator style, and task instruction matter.

BC is attractive because it can be trained offline before reward design or exploration. Its core weakness is that demonstrations come from expert state distribution d_{\pi^*}(s) while deployment produces d_{\pi_\theta}(s). Small mistakes change the next observation and can compound. In simplified analyses, one-step error \epsilon can grow to order O(T^2\epsilon) over a sequential horizon T; the exact bound depends on assumptions, but the causal point is durable: a policy creates its own future inputs.

DAgger: aggregate labels where the learner goes

Dataset Aggregation runs the learned policy under controlled conditions, requests the desired expert action a^* at state s actually visited, adds (s,a^*) to D, and retrains. Policy mixtures can be used during iterations. The method brings training distribution toward deployment distribution.

DAgger is not blanket permission to let a robot fail in public. Begin with simulation, low speed, physical guards, remote E-stop, immediate expert takeover, and an action safety filter. Labels can be actions the expert physically performed or counterfactual answers to “what should have happened here?” The latter can be valuable but adds expert workload, latency, and subjective variation. Plan collection around uncovered failure modes—not row count alone.

IRL: infer objective rather than copy action

IRL estimates reward r_\theta(s,a) or r_\theta(s,a,s') from expert policy \pi_E. The maximum-entropy intuition is that expert trajectories favor high reward without being unnecessarily deterministic among similarly good trajectories. Conceptually,

Updating \theta to increase expert trajectory likelihood yields a reward that can be paired with RL or optimal control. This may adapt better than direct BC to new initial states or constraints, because the objective can be re-optimized.

But many rewards can explain the same demonstrations; unobserved constraints and operator habits may be absorbed into a learned reward. A reward that explains training trajectories may generalize dangerously. GAIL-style methods instead match expert and learner occupancy distributions through a discriminator. Neither formulation justifies learning a prohibition—collision, pinch force, human exclusion zone—only by inference. Explicit safety rules remain explicit.

Connection to VLA

A Vision-Language-Action model conditions next action or an action chunk on images, language, and robot state. In a broad sense this is large-scale conditional imitation. Its range of objects and language does not guarantee extrapolation to new lighting, friction, camera placement, ambiguous instructions, or contact-force limits. A VLA interpreting “put the cup on the shelf” does not itself certify force, collision distance, joint range, or stopping distance; see VLA Primer.

Validate coordinate frame, units, velocity, acceleration, and freshness before an action reaches an actuator. Route it through kinematics, collision checking, and an impedance or position/velocity controller. Ambiguous language should trigger clarification, refusal, or low-speed mode; degraded perception should stop execution. Data scale does not substitute for a physical safety limit.

Evaluation, safety, and provenance

Do not approve a policy from offline action error alone. Create held-out evaluations over initial pose, objects, lighting, backgrounds, friction, latency, instruction wording, and disturbance. Report success, collision, stop, intervention count, peak force, completion time, and worst observed cases. High average success does not absorb rare human or equipment risk. Track each DAgger iteration, source demonstration, labeler, split, model, thresholds, failures, and rollback path.

References

• Pomerleau, 1989: ALVINN
• Ross et al., 2011: DAgger
• Ziebart et al., 2008: Maximum Entropy IRL
• Reinforcement Learning Basics, PPO and SAC, VLA Primer