Pose estimation may infer 2D joints (u,v), 3D joints, or object rotation R and translation t. Projection can be written \tilde p\sim K[R|t]P. Calibration, occlusion, left-right ambiguity, and time alignment are as important as network architecture.

Image keypoint estimationAn image is transformed to feature heatmaps and keypoint or pose coordinates.imagefeatures/heatmapskeypoints/pose

Diagram: Duskcoil, conceptual. It does not authorize identification or surveillance use.

Use heatmap or coordinate regression, PnP, and temporal filtering. Evaluate PCK, OKS, or reprojection error by occlusion, lighting, clothing, assistive devices, and demographics. For people-facing use, specify consent, purpose limitation, retention, and human review for errors.

Methods and coordinate frames

For 2D human pose, a network may select the maximum of a heatmap for each joint, regress coordinates directly, estimate one person at a time after detection (top-down), or associate all people from a full image (bottom-up). For 6D object pose, PnP estimates rotation and translation from known 3D points and image correspondences. A rotation R\in SO(3) obeys R^TR=I and \det R=1; direct Euler-angle regression has singularities and discontinuities, so normalized quaternions or continuous 6D representations are used.

Output Typical use Metric Failure
2D keypoint Pose analysis, UI PCK, OKS Occlusion, left-right swap
3D skeleton Motion analysis, robot imitation MPJPE Depth ambiguity, camera error
6D object pose Grasping, AR ADD, reprojection error Symmetry, reflection, PnP outliers

Data and failure handling

Evaluate occlusion, backlight, mirrors, multiple people, assistive devices, clothing, body shape, skin appearance, and camera height separately. A high mean OKS does not prove the worst latency and miss rate needed for fall detection or collaborative-robot avoidance. Do not repurpose keypoints for identity or emotion inference without consent. Define purpose limitation, retention, access control, deletion, and appeal procedures.

In implementation, calibrate intrinsics and extrinsics and document frame, unit, timestamp, and confidence. When confidence is low, do not treat interpolation as truth: slow the robot, stop, or request human review. Temporal smoothing reduces jitter but adds delay, which must be reflected in stopping distance.

  1. Define outputs, tolerances, and latency for the task.
  2. Split evaluation by attributes, occlusion, and lighting.
  3. Log calibration, coordinate transforms, and PnP outliers.

  4. Test stop rules for low confidence, camera dropout, and approaching people.

  5. COCO Keypoints evaluation

Calibration and time belong outside the network

Pose error is not only a network issue. A wrong focal length or principal point makes a visually correct point wrong in 3D. Keep distortion order, resize/crop, origin, and pixel-center convention identical in training and inference. Extrinsics change with camera movement and vibration, so re-check reprojection error against a reference target instead of trusting one calibration forever.

With multiple cameras or an IMU, verify timestamps before frames. One 30-fps frame of delay can move a fast hand or tool substantially. Store hardware time, receive time, inference completion time, and samples used for interpolation. A beautiful smoothed trajectory can still be too late for collision avoidance.

Pipeline choices

Pipeline Benefit Compute/data need Main failure
Monocular 2D Cheap and simple 2D labels, one camera Unknown depth, occlusion
Monocular 3D Simple installation 3D priors, large data Out-of-domain capture, scale
Multi-view triangulation Geometric 3D constraint Sync, calibration, views Wrong correspondences, out of view
RGB-D Direct depth Depth sensor, range limit Reflection, sunlight, missing depth
Keypoints + PnP Uses CAD 3D correspondences, intrinsics Symmetry, outliers, degenerate layout

Normalize 2D error by target size and state 3D units and frame. For symmetric objects, comparing to one canonical rotation can mark a physically correct pose wrong; use symmetry-aware ADD-S or reprojection and evaluate grasp clearance. Report 95th percentile, misses, high-confidence errors, and end-to-end latency in addition to mean error.

Failure case: a confident left-right mirror

Mirrors, rear views, and self-occlusion can make a left-right-swapped skeleton geometrically plausible. Smoothing may lock the error for many frames. Combine orientation, visible cues, multiple hypotheses, and a motion limit; do not rely on bone-length constraints alone. In safety applications, high confidence is not proof—use swap detection and disagreement with an independent sensor as stop conditions.

Reproducible implementation procedure

  1. Fix image, camera, world, and robot frames with a diagram and test data.
  2. Round-trip-test keypoint transforms after resize, padding, and augmentation, including a known sub-pixel error.
  3. Set PnP RANSAC threshold and minimum points; inspect inliers, reprojection error, and positive depth.
  4. Record static, walking, fast motion, occlusion, image-edge, and camera-dropout cases with latency.
  5. Test stop and reinitialization on low confidence, sudden skeleton deformation, calibration drift, and reversed time.
  6. Document purpose, access, retention, anonymization, and deletion if original frames are stored.

Pose is an observation hypothesis, not a person's identity, intention, or condition. Pass frame, time, and uncertainty downstream so that downstream systems can treat “unknown” safely.

Annotation quality and operational audits

Annotations also have uncertainty. Joint centers beneath clothing, occluded wrists, and symmetric-object orientation vary between annotators. Store visible, occluded, and outside-frame states separately; retain annotator disagreement, definition revisions, and interpolated points rather than forcing ambiguity into a hard label. Leave a real-image holdout so synthetic material and lighting cues cannot dominate.

Operations can be monitored without retaining every image forever. Aggregate confidence, missing-joint count, reprojection error, latency, camera status, and stops; retain only incident data for a short, access-controlled period. On model updates, replay identical videos and compare drop-out length, left-right swaps, and worst error in hazardous zones, then recalibrate after camera replacement.

#machine learning #pose estimation #keypoints #6D pose #computer vision