Reinforcement learning (RL) is a way for a robot to learn which action pays off over the long run by interacting with its environment. Unlike image classification, where an input and a label arrive together, the robot observes the world, moves a motor, and receives a reward—often several seconds later. The essential loop is try, observe the result, and update the policy.
The 30-second summary
- The smallest RL transition is state (or observation) s_t, action a_t, reward r_{t+1}, and next state s_{t+1}.
- A Markov decision process (MDP) models how the current state and action produce a next state and reward. The state must summarize the history well enough for prediction.
- A policy \pi(a\mid s) chooses actions; a value function V^\pi(s) is the expected future return when that policy is followed.
- The return discounts future rewards with \gamma. A horizon that is too long can destabilize learning, while a horizon that is too short produces a short-sighted and unsafe robot.
- Exploration tries uncertain actions; exploitation chooses the action currently believed to be best. On hardware, safety constraints sit above both.
1. See the robot as an agent
Figure 1 — After an agent acts, the environment changes and returns the next observation and reward. A real robot adds communication delay, sensor noise, and actuator saturation to this loop.
For a differential-drive robot, the agent can use camera, LiDAR, and encoder data as its state and output left and right wheel speeds as actions. The environment includes vehicle dynamics, floor friction, obstacles, and battery state. Moving toward a goal can yield positive reward, while a collision or an abrupt steering change can be penalized. A single “+1 at the goal” signal is usually too sparse; distance, velocity, stopping margin, and energy must be considered together.
2. MDP: split the problem into components
An MDP is defined by a state space \mathcal{S}, action space \mathcal{A}, transition probability P(s'\mid s,a), reward function R(s,a,s'), and discount factor \gamma:
“Markov” means that, once the current state is known, the past no longer adds information needed to predict the future. A mobile robot whose state contains only position cannot distinguish a robot that is stopped from one that is sliding through the same position. Include velocity, angular rate, and sensor confidence, or use a recurrent model that retains history.
When the complete state cannot be observed, the problem is a partially observable MDP (POMDP). Almost every real robot is a POMDP because of occlusions and missing LiDAR returns. A state estimator—an EKF, factor graph, or learned model—turns observations o_t into a useful internal state. The sensor-fusion article explains this boundary, and ROS 2 Primer shows how to make it a reproducible software component.
3. Value functions and return
The discounted sum of rewards from time t is the return G_t:
The value of state s under policy \pi is
and the state-action value specifies the first action as well:
Selecting the largest Q value is a value-based design. Updating a neural policy \pi_\theta(a\mid s) directly is policy-based. Continuous steering angles and joint torques often favor policy-gradient or Actor-Critic methods, because enumerating every possible action is impossible.
4. The Bellman equation breaks a long horizon into one step
Instead of evaluating an entire future at once, split it into the immediate reward plus the value one step later. The Bellman expectation equation is
The optimal value V^*(s) obeys the Bellman optimality equation:
This is why a value target can be generated from other estimates rather than a human-provided label. The self-reference is also a source of instability. Target networks, experience replay, and reward normalization separate old estimates from the current update and reduce harmful correlations.
5. Balancing exploration and exploitation
Always selecting the action with the current highest estimate can trap the agent in a lucky local solution. Exploration tries unknown actions, but random motion on a real machine can cause a collision. Common choices are:
| Method | Intuition | Strength | Hardware concern |
|---|---|---|---|
| ε-greedy | choose randomly with probability ε | simple | abrupt changes are unsafe for continuous torque |
| Boltzmann/softmax | sample in proportion to value | favors promising options | temperature needs tuning |
| UCB | try actions with high uncertainty | explicit exploration rationale | needs uncertainty estimates |
| Noisy policy | add continuous noise to actions or weights | smoother exploration | still needs saturation and limits |
On hardware, confine exploration to a validated operating envelope. Put speed limits, joint soft limits, force/current limits, a watchdog, and an emergency stop outside the learner so that every policy output can be intercepted. Randomization in a simulator is useful; it is not permission to apply random commands to a machine.
6. Check the idea in a small grid world
A 5×5 grid makes the learning dynamics visible. Let a cell be the state, up/down/left/right be actions, the goal reward be +1, a wall be −0.1, and each step be −0.01. Initialize Q to zero and repeat the temporal-difference update:
The bracketed term is the TD error: the difference between the prediction and the one-step target. If \alpha is too large, new experiences dominate; if it is too small, the policy cannot follow a changing environment. Log success rate, average steps, collision rate, and the fraction of unvisited states—not only a single reward curve.
7. Write the reward like a specification
Reward design often matters more than an algorithmic detail. A delivery robot might use
to combine progress, collisions, input energy, and smoothness. Increasing a weight does not always improve behavior. If the collision penalty dominates, the robot may learn the safe but useless policy of never moving. Log each term separately and audit which term the policy is actually optimizing.
Reward hacking is another failure mode: a bug in the goal detector, a sensor blind spot, or a simulator-only contact rule can produce a high score without achieving the intended task. Human-readable goals, physics-based constraints, and an independent evaluation environment make these shortcuts easier to detect.
8. Where research meets a product
Value methods are data-efficient but often assume discrete states and actions. Policy gradients and Actor-Critic methods handle continuous control; SAC adds an entropy objective, while model-based RL plans with a learned or analytical dynamics model before moving the robot. Model-based methods can reduce real-world samples, but they must tolerate model error.
In production, RL is not necessarily applied to every layer from safety monitoring to motor current. A classical PID or MPC can provide the safety envelope while RL selects a grasp contact, a route preference, or a gain schedule. The VLA overview describes a similar boundary: a vision-language model can propose action chunks while a verified low-level controller limits torque and speed.
9. Before moving to hardware
- Does the state include velocity, delay, and sensor confidence, or has the Markov assumption been silently broken?
- Are reward terms logged separately, with collision rate, energy, input smoothness, and stopping distance in addition to success rate?
- Are action ranges, rate limits, watchdogs, and emergency stops independent of the learner?
- Were friction, mass, sensor delay, lighting, and packet loss randomized in simulation, and was the distribution gap measured on real logs?
- Is an unseen evaluation set kept apart from training data? Are failures included rather than filtered out?
- Does a process restart enter a safe state and avoid replaying an old command?
Summary
Reinforcement learning does not make a robot memorize a “correct motion.” It defines states, actions, transitions, and rewards as an MDP, then estimates long-term value one step at a time with Bellman equations. Exploration, reward hacking, and hardware safety must be part of the system design before a learned policy can leave simulation. The next articles in this series will compare Q-learning/DQN, policy gradients, PPO and SAC, imitation learning, and Sim-to-Real under the same framework.