Selected work

Head-Wrist Egocentric Full-Body Pose Recovery System

An end-to-end platform for synchronized wearable capture, multi-view ground-truth reconstruction, and egocentric pose learning.

Role
Research Assistant, HKU MMLab
Date
Topics
Kalibr · RTMPose · Fisheye Camera · Multi-view Triangulation · PyTorch · SolidWorks

Overview

I built an end-to-end research platform for egocentric full-body pose recovery, spanning wearable hardware, synchronized multi-camera capture, external 3D human reconstruction, fisheye projection and optimization, and model training based on EgoRear and EgoPoseFormer.

End-to-end pipeline

  1. Design and calibrate the headband and wristband capture hardware.
  2. Record synchronized nine-camera fisheye video together with three head and wrist IMU streams.
  3. Recover high-quality external 2D and 3D human pose as ground truth from multi-view observations.
  4. Transform and project the reconstructed pose into the head-mounted fisheye camera coordinate system.
  5. Process aligned real-world sequences and train egocentric pose models in the EgoRear and EgoPoseFormer frameworks.

Mechanical design

I iterated the headband and wristband through hardware generations. The mechanical interfaces constrain each camera, fiducial tag, and motion-capture rigid body in a stable geometric chain, making the wearable setup repeatable across calibration and recording sessions.

Wearable wristband prototype with multiple fiducial markers and an inward-facing camera
Wristband prototype with camera, electronics, and calibration tags.

My contribution

  • Independently designed and built the wearable system from mechanical structure to embedded integration, combining nine fisheye cameras, three IMUs, wrist-mounted modules, AprilTag-based localization, and hardware synchronization.
  • Iterated the headband and wristband through six hardware generations, making the camera-tag-rigid-body geometry repeatable and calibratable.
  • Built the synchronized video and inertial capture stack, including external triggering, timestamp alignment, H.265 recording, and Kalibr intrinsic and extrinsic calibration with the omni fisheye model.
  • Developed the multi-view ground-truth reconstruction pipeline and the projection from the external reference frame to each head-mounted camera.
  • Collected, processed, and validated multiple real-world batches for model training and evaluation.
  • Refactored the research code into a reproducible project and prepared the public GitHub repository.

System design and synchronized capture

The acquisition system combines nine DepthAI/OAK fisheye cameras with three head and wrist IMUs. An external trigger starts all streams from a shared event, while per-frame timestamps support fine-grained temporal alignment across the nine-camera configuration. Camera intrinsics and inter-camera extrinsics are calibrated in Kalibr using an omni-directional fisheye model.

The wearable hardware was designed around a stable geometric chain between each camera, fiducial tag, and motion-capture rigid body. This makes calibration repeatable across recording sessions and allows the complete capture rig to be reconstructed in a common coordinate system.

Participant wearing the head-wrist capture system inside the external motion-capture volume
Wearable system inside the external motion-capture setup.

3D pose ground truth

External stereo and multi-view cameras run RTMPose/RTMW for 2D keypoint detection. Each keypoint is reconstructed through fisheye multi-ray triangulation, then transformed through the optical motion-capture rigid-body chain. The resulting 3D skeleton is projected into the head-mounted fisheye cameras to produce aligned 2D and 3D pose ground truth for egocentric learning.

This process connects external observations with the moving wearable coordinate frame, providing cleaner supervision than relying on a single reconstruction or projection stage.

3D pose ground truth reconstructed by triangulating detections from the synchronized external multi-camera system.

Egocentric hand pose exploration

I explored hand-pose perception from both inward-facing wrist cameras and head-mounted stereo views during object manipulation. These experiments tested whether compact egocentric cameras could recover stable 2D hand skeletons despite close-range distortion, self-occlusion, and rapid viewpoint changes.

Hand skeleton detected from an inward-facing wrist camera while manipulating a tablet Hand skeleton detected from an inward-facing wrist camera with an open palm
Hand-pose recognition from the head-mounted stereo views during close-range interaction.
Full-body pose with wrist-ground-truth-anchored stereo hand reconstruction at a yaw setting of 140 degrees.

Real-world data and model training

Across multiple capture batches, I collected approximately 10 hours of synchronized real-world motion data. I built the batch-processing workflow used to generate training-ready sequences, then re-implemented and fine-tuned EgoRear’s three-stage heatmap-based 3D pose pipeline, benchmarking against EgoPoseFormer and other egocentric methods. The reported real-world training used approximately 10 hours of data without large-scale synthetic-data pretraining.

Egocentric fisheye pose predictions compared with projected ground truth across multiple camera views
Predicted skeletons compared with projected ground truth on held-out fisheye views.
Internal test-set evaluation: model prediction on the left and the corresponding ground truth on the right.

3D Pose Estimation

The pipeline achieved 50.5 mm MPJPE on a held-out real-world test set. This is approximately 11% below the 56.94 mm EgoRear reference result listed below.

Method Dataset and camera setup MPJPE ↓
Our method Internally collected data, held-out real-world test set 50.5 mm
EgoPoseFormer SceneEgo, real-world monocular 93.0 mm
EgoPoseFormer Ego4View-RW, two front cameras 77.95 mm
EgoPoseFormer Ego4View-RW, two front and two rear cameras 63.38 mm
EgoRear Ego4View-RW, two front and two rear cameras 56.94 mm
EgoPoseFormer UnrealEgo, synthetic stereo 33.4 mm

Published results are included as literature references; datasets, camera configurations, skeleton definitions, and evaluation protocols differ. The percentage above describes the numerical difference from the reference result, rather than a controlled comparison on the same benchmark.

Simulation and supporting tools

In parallel, I developed a simulation and validation environment using Isaac Sim and BlenderProc. The tooling compares Mahony-filtered head and wrist IMU orientation against optical motion capture, provides skeleton playback for debugging, and generates randomized synthetic motion from AMASS sequences and SMPL-X bodies.

This environment supported geometry studies before hardware experiments and supplied synthetic-data generation and simulation validation tools to a parallel project team. The 50.5 mm real-world result reported above did not use large-scale synthetic-data pretraining.

Technology

Kalibr, RTMPose, fisheye cameras calibration, multi-view triangulation, PyTorch, Isaac Sim, BlenderProc, SolidWorks, ArUco/AprilTag, SMPL-X.