Show HN：基于Jax的运动学手部骨骼优化

Show HN：基于Jax的运动学手部骨骼优化
Show HN: Kinematic Hand Skeleton Optimization in Jax

原始链接: https://github.com/rerun-io/pi0-lerobot/tree/hand-kinematic-fitting

本仓库提供使用树莓派0和Lerobot训练机器人的工具和流程，重点是人体姿态动作重定向。目前支持Linux系统，并使用Pixi进行包管理。工作流程始于使用VGGT进行相机标定，以从多视角图像中获取内参、外参、深度图和点云。该流程的核心是将多相机镜头转换为轴角关节角(θ)和手指关节的度量3D位置，方便机器人重定向和精细运动技能学习。该过程涉及2D姿态估计、跟踪和三角测量，产生与机械结构无关、紧凑且梯度友好的运动数据。它使用具有关节角的运动学骨架，非常适合重定向。流程的阶段包括：2D关键点检测、3D关节三角测量/PnP、获得关节角的反向运动学(IK)、正向运动学(FK)验证以及轴角θ和度量3D关节位置的序列化。此外，还提供Jupyter Notebook教程，涵盖人体姿态/运动学、身体/手部姿态重定向以及模仿学习/遥操作。

Hacker News 新 | 旧 | 评论 | 提问 | 展示 | 招聘 | 提交登录展示 HN：JAX 中的运动学手骨架优化 (github.com/rerun-io) 19 分，作者 pablovelagomez，一天前 | 隐藏 | 旧 | 收藏 | 讨论我一直在尝试理解用于模仿学习的正向/反向运动学，因此我用 JAX 构建了一个完全可微分的运动学手骨架，并使用 reruns 的新回调系统在 Jupyter Notebook 中将其可视化。这显示了每个关节角度及其对运动学骨架的影响。加入我们，参加 6 月 16-17 日在旧金山举行的 AI 初创公司学校！指导原则 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系我们搜索：

树莓派激光雷达扫描仪 2025-04-19

Show HN：我构建了一个AI，可以将GitHub代码库转换成简单的教程 2025-04-19

Krita 人工智能广播 2023-11-21

视觉几何基础Transformer (VGGT) 2025-03-25

原文

A repo to explore training robots with Pi0 and Lerobot and human pose motion retargeting

Currently only linux is supported

Make sure you have the Pixi package manager installed

git clone https://github.com/rerun-io/pi0-lerobot.git
cd pi0-lerobot

Camera Calibration using VGGT

Get camera intrinsics/extrinsics, depth maps, and pointcloud for a set of multiview images from hocap

To run camera calibration on a sequence run

pixi run calibrate-cameras-vggt

Human Pose and Kinematics

For HOCap dataset, sample is automatically downloaded from dataset on command run. Checkout example rrd file here

For the full assembly101 dataset go to this link to get/download dataset

To run dataset visualization

pixi run visualize-hocap-dataset # recommended
pixi run visualize-assembly101-dataset

Current Pipeline

To run 2D pose estimation, tracking, and triangulation

pixi run pose-estimation-assembly101

The goal is to create a pipeline that converts time-synced multi-camera footage into axis-angle joint angles (θ) and metric 3D positions for every finger joint. This provides "motion fuel" for robotic retargeting and learning fine motor skills.

Why use a kinematic skeleton with joint angles?

Rig-agnostic Pose: Captures relative bone rotations, allowing transfer to any avatar or robot hand.
Easy Retargeting: A single Forward Kinematics (FK) pass transfers motion.
Compact Storage: Requires only ~60 floats per frame.
Built-in Constraints: Easily enforce bone lengths, joint limits, etc.
Gradient-Friendly: Differentiable FK enables optimization from 2D errors.
Sensor Fusion-Ready: Integrates well with IMUs, mocap markers, etc.

Capture-to-Angles Pipeline:

Input: Time-synced, calibrated RGB frames.
2D Keypoints: Detect pixel joint locations using a 2D keypoint detector.
3D Joints: Triangulate or use PnP with camera parameters to get metric 3D joint positions.
Inverse Kinematics (IK): Convert 3D joints into axis-angle joint parameters (θ).
Forward Kinematics (FK) Check: Rebuild 3D joints from θ for validation.
Output: Serialize axis-angle θ and metric 3D joint positions (xyz).

Jupyter Notebook Tutorials

pixi run notebook_tutorial