How to evaluate SLAM — ATE, RPE, runtime and failures

A visually clean SLAM map does not reveal pose error or latency. Define the reference, alignment freedoms and treatment of failures before comparing systems. The numbers here come from synthetic trajectories, not a benchmark of a particular SLAM algorithm.

What ATE and RPE measure

Let Q_i be the reference pose, P_i the estimate and S a fixed alignment. Absolute error can be written E_i=Q_i^{-1}SP_i. The root mean square of its translation norm is one common ATE measure; evo calls the broader absolute-pose metric APE.

Relative error compares motion over the interval i\rightarrow j: F_{ij}=(Q_i^{-1}Q_j)^{-1}(P_i^{-1}P_j). Results depend on whether the interval is one second, one meter or one frame. See the official evo Metrics documentation.

Check the distinction with a small calculation

Save slam_metric_lab.py in an empty directory and run python3 slam_metric_lab.py. It was checked with Python 3.12.3 and uses only the standard library. The reference is x=t meters, sampled at 101 points from 0 to 10 seconds at 0.1-second intervals, with identity rotations.

A constant translation appears in position differences but cancels from motion differences. Drift remains in both. This example applies neither alignment nor time interpolation. Download the reference and drifting estimate.

Match real data before evaluating

Diagram 1 · Scroll horizontally to read the diagram

Use meters and seconds, and do not mix sensor and body reference frames. Distinguish different recording start times from clock errors. The TUM format is timestamp x y z q_x q_y q_z q_w; verify quaternion order against the official format documentation.

SE(3) alignment corrects rotation and translation; Sim(3) also corrects scale. Scale correction can be appropriate for comparing monocular trajectory shape, but removes scale error if that is what the experiment intends to measure. Do not silently mix these evaluation settings in a ranking.

Record the evo environment

In a dedicated evo environment, save evo_ape --version and evo_rpe tum --help. These commands follow the official CLI structure; they are a procedure, not results executed on real trajectories for this article.

evo_ape tum reference.tum estimate.tum --align --save_results ape.zip
evo_rpe tum reference.tum estimate.tum --delta 1 --delta_unit s --all_pairs --save_results rpe.zip

Replace the filenames with trajectories from the same run. Fix the timestamp-association tolerance using the installed version's help and report matched sample counts. The synthetic table above is unaligned, so adding --align changes its interpretation and results.

Report more than accuracy

Per-frame processing time differs from latency including queueing. Without ground truth, do not report ATE. You can measure repeatability or consistency instead, while stating that these do not directly establish positional accuracy.

Retain difficult intervals

Removing only difficult sections leaves an accuracy score for successful tracking. Report overall success rate separately from error on evaluable intervals. First check sensor timing and frames to avoid mixing integration faults into an algorithm comparison.