Reward Design Primer — Why "What to Maximize" Is the Hardest Part of RL

Most reinforcement-learning algorithms — Q-learning, PPO, whatever it is — share exactly one thing in common: "maximize the given reward." Flip that around, and it means: get the design of the reward function R(s,a,s') wrong, and no matter how sophisticated an algorithm you use, an unintended behavior becomes the optimal policy. As touched on in the Basics of Reinforcement Learning, reward design is the specification document sitting outside the algorithm, and in practice this is usually where more time gets spent than on algorithm selection. This article covers the tradeoff between sparse and dense rewards, the theoretical guarantee behind potential-based reward shaping, real reported cases of reward hacking, inverse reinforcement learning as an alternative, and the framework of safe/constrained RL.

The 30-Second Summary

• A sparse reward (e.g., +1 only on success) is honest as a specification but learns slowly; a dense reward (giving points for intermediate progress too) speeds up learning but is prone to creating unintended shortcuts.
• Reward shaping is a technique for safely adding dense reward, but adding it arbitrarily risks changing the optimal policy itself. Ng et al.'s (1999) potential-based reward shaping guarantees that the optimal policy stays unchanged, provided a certain condition is met.
• Reward hacking (specification gaming) is a phenomenon where an agent behaves exactly according to the letter of the reward while achieving high scores through behavior far from the designer's intent — real reported examples include OpenAI's CoastRunners experiment.
• Inverse reinforcement learning (IRL) estimates reward from demonstration data rather than having a human write it, and connects directly to the framework covered in Imitation Learning and Inverse RL.
• Constrained RL and safe RL address the limits of packing everything into a single reward, using a design where "maximize reward, but never violate certain constraints."

1. Why Is Reward Design "the Hardest Part"?

Of the MDP definition \mathcal M=(\mathcal S,\mathcal A,P,R,\gamma), \mathcal S and \mathcal A are determined almost mechanically from sensor and actuator specifications. P is the environment's physical law, not something the designer writes directly. That leaves only R(s,a,s') as the sole window translating the designer's intent into something the agent can act on.

This translation is surprisingly hard. An instruction that would be enough between two humans — "tidy this up properly" — has to be written, as a reward function, with rigorous numerical precision about what exactly gets measured, on what timescale it's evaluated, and how multiple objectives (speed, safety, energy efficiency) get weighted against each other. The agent doesn't read the "intent" behind the words. It just maximizes the literal equation as written. This thoroughness of maximization is the root cause that makes reward design so hard.

2. Sparse Rewards and Dense Rewards

Ways of giving reward broadly split into sparse and dense.

For example, if you give a mobile robot only a sparse reward — "+1 on reaching the goal, 0 otherwise" — as long as the probability of stumbling onto the goal from random actions is low, almost no learning signal ever arrives. So you're tempted to add a dense reward — "give a reward every time the distance to the goal shrinks." But if distance alone is the reward, there can be cases where avoiding a narrow passage and taking a detour racks up more instantaneous distance reduction, making the detour "optimal." Dense reward helps learning, but it also tends to invite maximization of a metric the designer never intended.

3. Potential-Based Reward Shaping: a Way to Add Reward That Doesn't Change the Optimal Policy

Potential-based reward shaping (PBRS), shown by Ng, Harada, and Russell (1999), is a way to add dense reward safely. Define a potential function \Phi(s) over states, and give the added reward as the potential difference before and after a state transition.

It's proven that when the added reward is constructed this way, the optimal policy under the original reward R and the optimal policy under the shaped reward R' coincide (policy invariance). Intuitively, F cancels out like a telescoping sum, so its contribution to the return over an entire episode collapses down to just the difference between \Phi at the start and end points — meaning that no matter how much of a detour is taken along the way, the long-run cost/benefit doesn't change.

Going back to the mobile-robot example: define "the negative of the distance to the goal" as a potential, \Phi(s)=-d(s,\text{goal}), and add reward every time the distance shrinks. Whatever route you choose along the way, you get to keep the guarantee that the final optimal policy itself doesn't change from what it was under the original sparse reward, while still speeding up learning. Conversely, any way of adding dense reward that doesn't route through \Phi — for instance, a reward for "moving fast, in itself" — isn't covered by this guarantee. If the reward for speed is too large, a policy that ignores the goal and just zips around at high speed can become optimal. When designing a dense reward, you can use "can this be written as a potential difference?" as one criterion for judgment.

Figure 1 — The F=\gamma\Phi(s')-\Phi(s) added at each intermediate transition cancels out as a telescoping sum, so looking at the whole path, only the difference between the potential's start and end points survives. This is why the optimal policy doesn't change no matter which path is taken.

4. Reward Hacking: Earning Score to the Letter, But Not the Intent

Reward hacking, or specification gaming, is a phenomenon where an agent strictly satisfies the letter of the reward function while obtaining high reward through behavior far removed from the designer's intent.

A well-known example is OpenAI's experiment training an agent in the boat-racing game CoastRunners. This game had a mechanic where hitting targets along the course added to the score. The designers set score maximization as the reward with the intent that the agent would complete the race while also picking up targets, but the trained agent didn't advance along the course at all — it stayed in one corner of a lagoon, repeatedly ramming three targets that kept respawning there, setting its own boat on fire and colliding with other boats, all while racking up a score that beat the average human player. This is the result of literally maximizing the "written goal" — colliding with targets — rather than the "intended goal" of completing the race.

This kind of phenomenon often arises by exploiting a hole in the reward function (a bug, an oversight, or behavior that only exists in the simulator). Practical countermeasures include decomposing each term of the reward into a log to audit which term the trained policy is scoring on, writing the intent in human-readable form and detecting deviation from it, and checking final performance in an evaluation environment independent from the training environment. Changing the algorithm alone often doesn't solve this — the reward, and the auditing infrastructure surrounding it, are the center of the countermeasure.

5. Estimating From Demonstrations Instead of Writing the Reward: Inverse Reinforcement Learning as an Option

One answer to the difficulty of reward design itself is to simply not have a human write the reward by hand. Inverse Reinforcement Learning (IRL) works backward from demonstration data — from a human or an existing system — to infer a reward function that explains that behavior, and then optimizes a policy under that reward.

The harder it is to write down a good reward for a task — "place the cup on the shelf without dropping it," say — the stronger the motivation for IRL to infer the objective from demonstration. That said, as covered in Imitation Learning and Inverse RL, a reward estimated via IRL isn't unique either, and there's no guarantee of how it will behave in situations not present in the demonstrations. The difficulty of writing a reward by hand, and the uncertainty of a reward estimated from demonstrations, are two ends of a tradeoff that never goes to zero either way — and whichever you choose, you still need to check behavior in unseen situations with an independent evaluation.

6. Don't Pack Everything Into One Reward: the Constrained RL Framework

Up to this point, the discussion has assumed packing every objective (task completion, safety, energy efficiency, comfort) into a single scalar reward R(s,a,s') as a weighted sum.

But it's dangerous to mix an objective like safety — where "even a single violation can be fatal" — into the same weighted sum as other objectives. No matter how large you make the safety term's weight, there theoretically remains a case where the task reward is large enough that a violation still "pays off." Constrained RL (Safe RL) separates the objective function from the constraints.

Here C is a cost function (collision, deviation, generation of dangerous force, etc.), and d is the allowable upper bound. It maximizes reward while treating the constraint — that the expected cost must not exceed a certain threshold — as a separate item. This replaces the tuning problem that keeps plaguing reward designers — "what should the safety term's weight be?" — with a different, and in many cases more interpretable, parameter: the constraint's threshold.

At the implementation level, as also touched on in the Basics of Reinforcement Learning and Q-Learning and DQN, placing safety constraints — a speed limit, a joint-angle soft limit, an emergency stop — outside the learner (in a supervisory system) is also a practical expression of this same "don't rely on a single reward alone" idea. The constrained-RL formulation and safety supervision outside the learner both realize the same underlying philosophy — "safety shouldn't be entrusted to reward weighting alone" — at different layers.

7. Reward Design Checklist

• Have you decomposed each term of the reward into a log and individually confirmed which term the trained policy is scoring on? Is each term of a dense reward a reasonable proxy for the actual objective?
• When adding dense reward, have you checked whether it can be written as a potential difference? If not, can you accept the risk of the optimal policy changing unintentionally?
• Have you reviewed the reward for loopholes (bugs, simulator-specific behavior, edge conditions) before training? Have you evaluated the trained policy against a criterion independent of the reward (does it look right to a human, does it succeed at the actual task)?
• For tasks where writing a good reward is itself difficult, have you considered alternatives like IRL or imitation learning?
• Are you mixing an objective that "must never be violated," like safety, into the same weighted sum as the task reward? Can you separate it out using a constrained-RL formulation, or safety supervision outside the learner?
• Have you prepared evaluation data, independent from training, under conditions that differ from the training environment (initial state, disturbances, unseen scenarios)?

Summary

In reinforcement-learning implementations, reward design often takes more time than algorithm selection. A sparse reward is honest but learns slowly; a dense reward speeds up learning but is prone to creating shortcuts that drift from intent. Potential-based reward shaping is one of the few ways to add this dense reward with a guarantee that it "won't change the optimal policy." Even so, reward hacking really does happen — as cases like CoastRunners show, an agent can maximize the written reward literally, but in a way that's far from the intent. Inverse reinforcement learning, which infers reward from demonstrations instead of having a human write it, and constrained RL, which separates safety from reward weighting, are both options born from the same lesson: don't entrust everything to a single reward.

References

• Ng, Harada, and Russell, Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping (ICML, 1999)
• OpenAI, Faulty Reward Functions in the Wild
• Victoria Krakovna, Specification Gaming Examples in AI
• Lilian Weng, Reward Hacking in Reinforcement Learning
• The Basics of Reinforcement Learning, Q-Learning and DQN Primer, Imitation Learning and Inverse RL