Selected work
Head-Wrist Egocentric Full-Body Pose Recovery System
An end-to-end platform for synchronized wearable capture, multi-view ground-truth reconstruction, and egocentric pose learning.
Overview
I built an end-to-end research platform for egocentric full-body pose recovery, spanning wearable hardware, synchronized multi-camera capture, external 3D human reconstruction, fisheye projection and optimization, and model training based on EgoRear and EgoPoseFormer.
End-to-end pipeline
- Design and calibrate the headband and wristband capture hardware.
- Record synchronized nine-camera fisheye video together with three head and wrist IMU streams.
- Recover high-quality external 2D and 3D human pose as ground truth from multi-view observations.
- Transform and project the reconstructed pose into the head-mounted fisheye camera coordinate system.
- Process aligned real-world sequences and train egocentric pose models in the EgoRear and EgoPoseFormer frameworks.
Mechanical design
I iterated the headband and wristband through hardware generations. The mechanical interfaces constrain each camera, fiducial tag, and motion-capture rigid body in a stable geometric chain, making the wearable setup repeatable across calibration and recording sessions.
My contribution
- Independently designed and built the wearable system from mechanical structure to embedded integration, combining nine fisheye cameras, three IMUs, wrist-mounted modules, AprilTag-based localization, and hardware synchronization.
- Iterated the headband and wristband through six hardware generations, making the camera-tag-rigid-body geometry repeatable and calibratable.
- Built the synchronized video and inertial capture stack, including external triggering, timestamp alignment, H.265 recording, and Kalibr intrinsic and extrinsic calibration with the omni fisheye model.
- Developed the multi-view ground-truth reconstruction pipeline and the projection from the external reference frame to each head-mounted camera.
- Collected, processed, and validated multiple real-world batches for model training and evaluation.
- Refactored the research code into a reproducible project and prepared the public GitHub repository.
System design and synchronized capture
The acquisition system combines nine DepthAI/OAK fisheye cameras with three head and wrist IMUs. An external trigger starts all streams from a shared event, while per-frame timestamps support fine-grained temporal alignment across the nine-camera configuration. Camera intrinsics and inter-camera extrinsics are calibrated in Kalibr using an omni-directional fisheye model.
The wearable hardware was designed around a stable geometric chain between each camera, fiducial tag, and motion-capture rigid body. This makes calibration repeatable across recording sessions and allows the complete capture rig to be reconstructed in a common coordinate system.
3D pose ground truth
External stereo and multi-view cameras run RTMPose/RTMW for 2D keypoint detection. Each keypoint is reconstructed through fisheye multi-ray triangulation, then transformed through the optical motion-capture rigid-body chain. The resulting 3D skeleton is projected into the head-mounted fisheye cameras to produce aligned 2D and 3D pose ground truth for egocentric learning.
This process connects external observations with the moving wearable coordinate frame, providing cleaner supervision than relying on a single reconstruction or projection stage.
Egocentric hand pose exploration
I explored hand-pose perception from both inward-facing wrist cameras and head-mounted stereo views during object manipulation. These experiments tested whether compact egocentric cameras could recover stable 2D hand skeletons despite close-range distortion, self-occlusion, and rapid viewpoint changes.
Real-world data and model training
Across multiple capture batches, I collected approximately 10 hours of synchronized real-world motion data. I built the batch-processing workflow used to generate training-ready sequences, then re-implemented and fine-tuned EgoRear’s three-stage heatmap-based 3D pose pipeline, benchmarking against EgoPoseFormer and other egocentric methods. The reported real-world training used approximately 10 hours of data without large-scale synthetic-data pretraining.
3D Pose Estimation
The pipeline achieved 50.5 mm MPJPE on a held-out real-world test set. This is approximately 11% below the 56.94 mm EgoRear reference result listed below.
| Method | Dataset and camera setup | MPJPE ↓ |
|---|---|---|
| Our method | Internally collected data, held-out real-world test set | 50.5 mm |
| EgoPoseFormer | SceneEgo, real-world monocular | 93.0 mm |
| EgoPoseFormer | Ego4View-RW, two front cameras | 77.95 mm |
| EgoPoseFormer | Ego4View-RW, two front and two rear cameras | 63.38 mm |
| EgoRear | Ego4View-RW, two front and two rear cameras | 56.94 mm |
| EgoPoseFormer | UnrealEgo, synthetic stereo | 33.4 mm |
Published results are included as literature references; datasets, camera configurations, skeleton definitions, and evaluation protocols differ. The percentage above describes the numerical difference from the reference result, rather than a controlled comparison on the same benchmark.
Simulation and supporting tools
In parallel, I developed a simulation and validation environment using Isaac Sim and BlenderProc. The tooling compares Mahony-filtered head and wrist IMU orientation against optical motion capture, provides skeleton playback for debugging, and generates randomized synthetic motion from AMASS sequences and SMPL-X bodies.
This environment supported geometry studies before hardware experiments and supplied synthetic-data generation and simulation validation tools to a parallel project team. The 50.5 mm real-world result reported above did not use large-scale synthetic-data pretraining.
Technology
Kalibr, RTMPose, fisheye cameras calibration, multi-view triangulation, PyTorch, Isaac Sim, BlenderProc, SolidWorks, ArUco/AprilTag, SMPL-X.