Q-learning and DQN choose from a finite action set. A wheel speed, drone thrust, or manipulator torque is continuous, however. Policy-gradient methods update a policy \pi_\theta that outputs a distribution over continuous actions. Actor-Critic adds a Critic that evaluates those actions; PPO and SAC add practical stabilization used widely in robot research.
The 30-second summary
- Policy gradients increase the expected return J(\theta) directly. They handle continuous actions naturally, but single-trajectory estimates have high variance.
- Actor-Critic uses a value estimate as a baseline. Advantage measures whether an action was better than the average action in the same state.
- PPO clips the probability ratio between the old and new policies so one update cannot move too far. It is straightforward, but on-policy data is difficult to reuse.
- SAC maximizes both reward and policy entropy. It is off-policy, uses replay, and is a practical fit for continuous control.
- Neither method supplies safety limits. Keep action bounds, rate limits, low-level PID/MPC, and emergency stops outside the learner.
1. Directly optimizing a policy
Figure 1 — The Actor proposes a continuous distribution and the Critic evaluates the return. On hardware, the action goes through a verified lower-level controller rather than directly to torque.
For a stochastic policy, the policy-gradient theorem is
Actions that produced a high return become more probable. Because G_t is noisy and delayed, subtracting a state-dependent baseline reduces variance without changing the expected gradient:
2. Advantage and Actor-Critic
The Critic learns V_\phi(s) and estimates whether an action was better than the state average with A_t=Q(s_t,a_t)-V(s_t). A one-step TD approximation is
Generalized Advantage Estimation (GAE) combines several steps with \lambda. A value close to one uses a longer horizon and lower bias but higher variance. For a fast attitude loop use the task's actual delay and time scale rather than copying a benchmark setting.
The Actor and Critic losses are typically
When an image encoder is shared, one gradient changes both estimates. Separate learning rates, gradient clipping, and a fixed observation-normalization record make failures easier to diagnose.
3. PPO: do not change the policy too far at once
On-policy policy gradients collect a rollout with the current policy. Reusing it for too many updates moves the new policy away from the data distribution. PPO forms a probability ratio
and maximizes the clipped objective
Positive advantages cannot increase their probability without limit, and negative advantages cannot decrease it without limit. A small \epsilon is conservative; a large one permits bolder updates. Clipping is not a safety constraint, so joint, speed, and force limits are still separate.
An implementation collects a fixed rollout, normalizes Advantage, and trains several mini-batch epochs. Save old log probabilities, distinguish a timeout from a true terminal state, and freeze normalization statistics at evaluation. Otherwise two runs with the same weights can act differently.
4. SAC: reward plus entropy
Soft Actor-Critic (SAC) maximizes future reward and policy entropy:
The entropy term prevents equally good actions from collapsing to one too early. SAC is off-policy and uses a replay buffer, so each real-world transition can be reused. That improves sample efficiency, but stale data, reward scale, and temperature tuning matter.
For a continuous action, the Actor outputs the mean \mu_\theta(s) and standard deviation \sigma_\theta(s) of a Gaussian, then maps it into bounds with tanh or another transform. The log probability needs the Jacobian correction of that transform. Twin Q networks and the smaller Q estimate reduce overestimation. Decide whether deployment uses the deterministic mean or retains exploration noise.
5. Choosing PPO or SAC
| Aspect | PPO | SAC |
|---|---|---|
| Data | on-policy rollouts | off-policy replay |
| Exploration | policy distribution and entropy bonus | entropy is part of the objective |
| Sample efficiency | low to medium | often higher |
| Main bugs | log-probability, terminal flags, clipping | Q overestimation, reward scale, tanh correction |
| Typical fit | many parallel simulators | continuous torque and scarce hardware data |
PPO is often stable when many simulated environments can run in parallel. SAC can be attractive when every real transition is expensive. Neither method models sensor delay, motor dead zones, or actuator saturation automatically.
6. Sim-to-Real is a boundary, not a switch
The simulation-to-reality gap comes from mass and friction, backlash, exposure and sensor noise, network delay, battery sag, floor material, and lighting. Domain Randomization samples these parameters so the policy does not overfit one ideal value. The range should be measured from hardware; randomizing impossible worlds only makes training harder.
A staged transfer is safer: (1) pure simulation, (2) replay real sensor logs without motion, (3) low-power tests with the wheels lifted, (4) bounded speed and force on a clear floor, and (5) the task itself. Log the requested action separately from the action that the safety limiter actually passed. Sim-to-Real is an iterative loop of identification and evaluation, not a single deployment button.
7. Safety and evaluation
PPO clipping and SAC entropy do not prevent collisions. Independent monitors should check speed, acceleration, joint angle, contact force, hydraulic pressure, current, and temperature. On an invalid observation, NaN, expired timestamp, or communication dropout, the monitor should command a safe stop. Put it in a separate process or MCU when the risk warrants it.
Besides reward, measure success, collision rate, maximum deviation, stopping distance, energy, action smoothness, inference latency, and safety-limiter intervention rate. A high success rate with frequent intervention means the policy depends on the limiter. Repeat runs with several random seeds and report variance and worst cases, not only the mean.
Implementation checklist
- Test action units, bounds, tanh/clip ordering, and log-probability corrections.
- For PPO, save old log probabilities, Advantage, and terminal versus timeout flags.
- For SAC, monitor replay distribution, twin Q values, temperature \alpha, and reward scale.
- Pin preprocessing, normalization, seeds, weights, and environment parameters by experiment ID.
- Keep low-level PID/MPC, rate limits, watchdog, and emergency stop independent of the learner.
- Define stop conditions for passive logs, low power, and normal operation before each transition.
- Record success, collision, limiter intervention, latency, energy, and maximum deviation together.
Summary
Policy gradients optimize a continuous action distribution directly. Actor-Critic uses Advantage to reduce variance; PPO clips the update; SAC uses entropy and replay to improve exploration and data efficiency. None of them resolves hardware uncertainty or safety by itself. A verified lower-level controller, an independent monitor, staged transfer, and worst-case evaluation are part of the algorithm when a learned policy leaves simulation.