Bottom line

Model-based RL learns a world model \hat f(s,a), tests action sequences inside it, and applies only a supervised action to hardware. Model-free methods learn a policy from reward directly; MBRL can be more data-efficient and can express constraints, but model error can create simulator-only success. See RL basics, Q-learning, PPO/SAC, and MPC.

World models and planning

observationsworld modelMPC/policysafety monitor

Figure 1 — Conceptual SVG created by Duskcoil, not measured system data.

Learn \hat s_{t+1}=\hat f_\theta(s_t,a_t) and optimize a receding-horizon objective:

a^*=\arg\max\sum_{k=0}^{H-1}\gamma^k\hat r(\hat s,a)-\lambda C(\hat s,a).
Only the first action is executed, then the system is observed again. Constraints C can represent collision, joint, temperature, speed, or current limits.

Sim-to-Real, identification, and safety

One-step errors compound over long rollouts. Ensembles can estimate uncertainty; planners can avoid high-variance regions and use short horizons. Domain randomization varies mass, friction, delay, sensor noise, lighting, and camera pose. System identification measures inertia, friction, motor constants, and latency so those ranges are defensible rather than arbitrary.

Move from logging, to low-speed shadow control, to bounded supervised trials. Keep emergency stop, independent speed/force/temperature monitors, communication-loss stopping, and human separation outside the learned policy. Common failures include fixed-friction simulations causing wheel slip, lighting overfit, reward hacking that creates vibration, and unmodelled delay causing instability. Report violation count, stopping distance, uncertainty-triggered abstention, and worst-case behavior—not only mean reward.

Model-free and model-based roles

Model-free RL updates a policy or value directly from experience. It is expressive when contact dynamics are difficult to model, but every improvement may consume real trials. MBRL reuses each transition to train a predictor and evaluates many candidate sequences in that predictor. If the predictor has a systematic bias, the planner optimizes a simulator-only shortcut.

Aspect Model-free Model-based
Hardware data low to medium efficiency medium to high within the model's range
Runtime often light policy inference rollouts and optimization each cycle
Constraints usually an indirect reward/safety layer natural to put into the planning problem
Main failure poor extrapolation from sparse data long-horizon model error
Typical fit complex contact and visual policies short-horizon motion, energy, and thermal planning

Hybrid stacks are common: a learned policy proposes candidates, a learned model predicts a short horizon, and MPC enforces speed, force, and current constraints. Use the PPO/SAC article and MPC article to assign timing and safety responsibilities rather than treating algorithms as replacements.

Do not collapse uncertainty into one number

Epistemic uncertainty comes from missing training data; aleatoric uncertainty comes from irreducible sensor and environment variation. An ensemble variance is a useful signal for the first, but an ensemble can still share the same bias. When predictive variance exceeds a threshold, shorten the horizon, reduce speed, switch to a classical controller, or collect another observation before planning. This abstention policy must be specified before deployment.

For one-step residual e_t=s_{t+1}-\hat s_{t+1}, log quantiles and worst cases in addition to mean squared error. A low average error can hide a large error just before contact. Never interpret a probabilistic prediction interval as a safety certificate; keep independent collision, force, temperature, and current monitors.

Design identification experiments

System identification is an experiment design problem, not a single calibration. For a motor, measure static friction, inertia at several speeds, load-dependent torque, and communication round-trip delay separately. For a hydraulic actuator, timestamp pressure, flow, cylinder velocity, oil temperature, and valve command on one clock. Split data by day, floor, payload, lighting, and temperature—not random neighboring frames—so the final set is genuinely unseen.

A small numerical intuition

Consider a 1 kg cart whose velocity changes by 0.1u every T_s=0.1\,\mathrm{s}. A friction-free model predicts a 0.5\,\mathrm{m/s} velocity increase for u=1 over five steps. If the real cart loses 0.02\,\mathrm{m/s} per step to friction, the five-step error reaches 0.1\,\mathrm{m/s}. A shorter horizon, online identification, margin in the speed constraint, and re-planning from measurement keep this simple model useful.

Evidence to publish

Alongside equations, retain the training period, sensor rate, delay, random seed, environment parameters, solver deadline, and fallback policy. Plot prediction residuals, constraint violations, stopping distance, and uncertainty-triggered abstention by the same experiment ID as the reward. If raw logs cannot be shared, publish anonymized statistics, simulation settings, units, and a reference date so the scope of the claim remains checkable.

References

#reinforcement learning #model-based RL #Sim-to-Real #MPC