Tsinghua清华大学 Ph.D.博士 UCLA加州大学洛杉矶分校 Berkeley伯克利 Kandao (Start-up)看到科技 (创业) SenseTime商汤科技 IDEA (former)IDEA研究院 (前) Manifold AI流形空间 R&D Director研发总监 Senior Engineer高级工程师
Python / C++ / GoPython / C++ / Go PyTorchPyTorch 3D Vision3D视觉 Lie Algebra李代数 Bundle Adjustment光束法平差 Optimization优化理论 Vibe Coding氛围编程 Linux / GitLinux / Git Imitation Learning模仿学习 RL强化学习 World Model世界模型 Whole-Body Control全身控制 Loco-Manipulation运动操作 Tactile & Compliant Control触觉与柔顺控制 Badminton羽毛球 Dev Psychology发展心理学 VLA Robot 6D Pose 3D Detection 3DGS Object Reconstruction Hand Reconstruction Keypoint Detection Human Motion Defect Detection Person ReID NDN/Networking VR Streaming
avatar

Shock (Xiaoke) Jiang 蒋小可 (Shock Jiang)

Partner & R&D Director @ Manifold AI 流形空间(Manifold AI)技术合伙人、研发总监

Explore the world as well as our inner heart 探索世界,也探索我们的内心

A gentle black sheep. I hold warmth toward others while remaining inwardly independent. I'm drawn to clear facts, vivid details, well-reasoned arguments, measured expression paired with a kind heart. I hold no claim to absolute truth, and have little patience for context-free moralizing.

I am inclined toward order: knowledge should be hierarchical, data should be structured, and cognition is the continual process of organizing both data and knowledge. Only human beings are equal by birth; at the level of all living things, there is no place for hierarchy.
一只羊,保持对他人的尊重和善意,内心独立有态度。喜欢清晰的事实,生动的细节,有条理的逻辑,妥善的表达和善良的心灵;知自己没有掌握真理,也厌烦被脱离场景的真理训诫。

我倾向于秩序,知识要有层次,数据需要结构化,认知的过程就是不断把数据和知识进行秩序化的过程;唯有人生而平等,在万物之灵的层面没有排序。

2026-09 🔥 Hiring interns! Research topics: whole-body control, tactile sensing + VLA. Contact: shock@manifoldai.cn 🔥 招收实习生!研究方向:全身控制(whole-body control)、触觉 + VLA。联系:shock@manifoldai.cn
2026-08 Joined Manifold AI as Partner & R&D Director, working on world models and embodied manipulation. 加入流形空间(Manifold AI),任技术合伙人、研发总监,聚焦世界模型与具身操作。
2026-06 One paper accepted to ECCV 2026: GTA-VLA (Guide, Think, Act)! 一篇论文被 ECCV 2026 录用:GTA-VLA(Guide, Think, Act)!
2026-04-27 Keynote talk on SpatialPoint at the Guangdong Provincial AI Application Matchmaking Conference (coverage). 在广东省人工智能应用对接大会作 SpatialPoint 主题演讲(报道)。
2026-03 SpatialPoint released on arXiv, covered by QbitAI (量子位)! SpatialPoint 在 arXiv 发布,被量子位报道!
2025-11 DINO-XGrasp showcased at IDEA Day (demo video). DINO-XGrasp 在 IDEA Day 上亮相(演示视频)。
2025-07 One paper accepted to ICCV 2025: UniGS (UniGS: Modeling Unitary 3D Gaussians for Novel View Synthesis from Sparse-view Images)! 一篇论文被 ICCV 2025 录用:UniGS(UniGS: Modeling Unitary 3D Gaussians for Novel View Synthesis from Sparse-view Images)!
2025-06 One paper accepted to IJCAI 2025: SeqPose! 一篇论文被IJCAI 2025录用:SeqPose!
2025-02 Three papers accepted to CVPR 2025: LeanGaussian, HandOS, HumanMM! 三篇论文被CVPR 2025录用:LeanGaussian、HandOS、HumanMM!
2024-09 Geo6D accepted to IEEE Transactions on Multimedia (TMM). Geo6D 被 IEEE Transactions on Multimedia (TMM) 录用。
2023-01 Uni6Dv2 accepted to AISTATS 2023. Uni6Dv2 被 AISTATS 2023 录用。
2022-06 Shared Uni6D at SenseTime CVPR 2022 Paper Session. 在商汤 CVPR 2022 论文分享会上分享 Uni6D。
2022-03 Uni6D accepted as Oral presentation at CVPR 2022! Uni6D 被 CVPR 2022 录用为 Oral 报告!

从真实场景出发,记录 VLA 从数据、模型到现场交付的实践。 Practical notes on taking VLA from data and models to real-world operation.

VLA / 实践笔记 · 1VLA / Field Notes · 1

如何完成VLA交付:以小件拣选为例How to Deliver a VLA System: A Small-Parts Picking Case

从交付目标、SOP 与硬件选型开始,建立数据、模型、控制和真机反馈的完整闭环。全文以一个单臂从杂乱料框拣选小件、放上传送带的现场项目为例,每一步都给出实际做法和数据。Start with delivery goals, SOP design and hardware, then close the loop across data, policy, control and real-robot feedback. Every step is illustrated with a field project in which a single arm picks small parts from a cluttered bin onto a conveyor.

阅读全文 →Read the article →
VLA / 实践笔记 · 2VLA / Field Notes · 2

VLA 数据预处理:以小件拣选为例的八层流水线(L0–L7)VLA Data Preprocessing: An Eight-Layer Pipeline (L0–L7) for Small-Parts Picking

一条连续清框录像怎样变成训练样本:信号体检、切周期与关键段、按姿态判定目标数据、用 3D 轨迹对比筛选数据源、失败分类、技能标签、训练权重、停顿切除和速度归一。每一层都给出规则、阈值和在约两千万帧数据上的实测结果。How a continuous bin-clearing recording becomes training samples: signal checks, cycle and core-segment extraction, pose-based target admission, 3D trajectory comparison of data sources, failure classes, skill labels, training weights, pause removal and speed normalization. Each layer comes with its rules, thresholds and measured results on about twenty million frames.

阅读全文 →Read the article →
← 返回文章列表
VLA / 实践笔记

如何完成VLA交付:以小件拣选为例

从交付目标、SOP 与硬件选型开始,建立数据、模型、控制和真机反馈的完整闭环。全文以一个单臂从杂乱料框拣选小件、放上传送带的现场项目为例,每一步都给出实际做法和数据。

00 / 实例:单臂小件拣选

任务是让一只机械臂从杂乱的料框里逐个夹起小盒,放到传送带上。策略是 π0.5 类的视觉—语言—动作模型,先用另一站点的遥操 SOP 数据起步,再加入目标现场的数据,部署在现场的边缘 GPU 上。从第一次现场测试到客户认可的演示,用了 26 天。下面各节的“小件拣选实例”都来自这个项目。

指标结果
现场峰值节拍680 件/小时
稳定运行节拍420 件/小时
杂乱料框单次夹取成功率41% → 75–94%
服务端单次推理165 → 42 ms(机器人侧 48.5 ms)
研发自测评星1 → 4.3 星
训练 / 现场评测25 天 104 个模型;8 天 30 次现场评测

01 / 先确认交付目标

VLA(Vision-Language-Action,视觉—语言—动作)交付通常面对固定、单一的业务场景,可以利用目标物、目标位置和工位布局等先验,通过后训练与强化学习缩短迭代周期。它与追求广泛泛化的预训练、中期训练有不同的侧重点。

交付要求系统在现场持续工作,成功率、节拍、精度和异常恢复都要达到约定要求。因此,第一步是确认交付目标;第二步是设计 SOP(标准操作流程)并选择合适的硬件。数据采集应在这些决策之后启动。

  • 明确任务边界:处理什么物体、覆盖哪些位置、允许哪些环境变化,以及哪些情况需要人工协作。
  • 把成功率与节拍一起评估,提前约定测试条件和失败的判定方式。研发、采集与交付场地不一致时,要逐项对齐差异。

02 / 硬件先跑通整个工作空间

  • 臂展与夹爪:覆盖所有操作位置,检查开口宽度、闭合速度,以及异形物体能否夹稳。夹爪根部读数不一定等于指尖的实际间距。
  • 边缘与角落:验证抓取箱边、箱角物体时,夹爪和腕上相机是否与料框干涉。
  • 视觉与触觉:主相机要覆盖完整场景,末端结构不能遮挡关键信息;检查触觉信号能否区分对齐与未对齐状态。
  • 先检查机械和传感问题:夹爪松动、电机掉电、图像模糊、视频帧率不足与重复帧,都可能表现为模型效果差。

03 / SOP 是最关键、也最难的一步

先用物理结构简化任务。例如,用斜坡辅助翻面,把双手操作改成单手操作。流程越简单、状态越可区分,模型学习与后续排障就越容易。

  • 固定机器人、工作台和料框的相对布局,并使场景能够在仿真器里复现;采集时让料框在允许的小范围内变化,覆盖真实部署偏差。料框底部可增加柔性垫层。
  • 分别定义最佳流程、常规流程和失败恢复流程。用遥操作模拟典型错误,再采集顺势恢复的动作,覆盖抓空、滑落等状态。
  • 减少误触、临时换目标、犹豫停顿和不必要的姿态调整。细小动作也会进入数据,进而影响模型行为。
  • 检查动作歧义:复位与接近、抓取成功后的提升与抓空后的提升,可能在单帧中很相似。可用图像、夹爪宽度和关节角训练动作分类器,通过混淆矩阵定位难以区分的阶段。
  • 训练算法的人应亲自采集至少 10 条数据,关键相关人也建议参与,建立对流程难点和遥操作体感的共同理解。

04 / 把安全护栏放进执行链路

  • 防触底应按夹爪末端的真实几何计算,例如指尖角点或边缘,不能只看机械臂末端参考点的高度。
  • 到达深度边界或出现“持续下压指令、实际位置不动”时,停止继续下压,保留允许的开合爪和上抬动作。回退上抬距离需结合工装、物体和可用空间确定,例如在空间允许时上抬 5 cm。
  • 六维力传感器需要去重力、去加速度,并考虑负载影响。力信号只有经过校准与配套处理,才能可靠参与控制。

05 / 数据质量:正确动作与恢复动作都要有

  • 每次遥操从约定初始位置开始,剔除接手前的抖动与观察阶段。动作果断,操作对象始终处于有效视野内。
  • 标注区分 task、sub-task 与 action/skill,并为特殊动作补充环境描述。小停顿可能是关键技能边界,不能一概作为无效数据删除。
  • 清理无效静止帧和异常尖峰,按需平滑动作。模仿学习数据保留正确行为与恢复行为,但要剔除导致失败的错误动作片段。

06 / 训练和离线评测盯住关键阶段

  • 针对真机抖动带来的图像变化做数据增强,避免只适应平稳遥操画面。
  • 用反事实样本检查指令跟随;对特殊 SOP 分支补充场景与动作文本,结合动作和进度预测,检查模型是否区分不同阶段。
  • 在训练与评测中提高关键帧、关键 skill 的权重;按需加入末端执行器(EEF)相关辅助预测头与损失。整体平均误差之外,也要关注抓取、接触和恢复阶段。

07 / 从模型动作到平滑的底层控制

先单步执行,分别看目标是否抓得准、物体是否夹得稳、失败后是否能恢复。随后检查推理延迟、轨迹插值和底层控制,避免把执行链路问题全部归因于模型。

  • 以 30 Hz 的 action chunk 对接 500 Hz 底层控制为例:两者之间需要插值,五次多项式可用于构造位置、速度和加速度连续的轨迹。这些频率是实例,实际以机器人接口为准。
  • 控制器过于强调位置到达时,可能放大抖动。应结合本体能力调试 PID,并在硬件支持时评估力位混合控制。

08 / 模型与规则协作,重点是平滑接管

  • 抓取并提升到合适高度后,可由规则接管搬运、放置和返回等确定性动作,把模型能力集中在复杂交互阶段。
  • 接管点、避障必经点和放置点之间要考虑速度与加速度连续;规则与模型切换也要避免突然停顿或跳变。放置位置可在允许范围内加入变化。
  • 夹持稳定性需要单独验证。可在夹住后补充闭合动作以减少途中掉落,执行力度与次数需按物体和夹爪能力调整。

09 / 用真机反馈建立迭代闭环

  • 把空料框、没有目标、腕上相机看不到目标等分布外状态纳入测试,并明确停止、恢复或人工接管的处理方式。
  • 每次真机测试保留人工反馈,区分定位不准、夹持不稳、恢复失败、控制抖动和硬件异常,再决定改数据、模型、SOP 还是控制链路。
  • 回流真机数据,特别是抖动、模糊和失败恢复片段。真实部署图像与遥操图像存在差异,需要在下一轮采集和训练中覆盖。

10 / 交付前的核对清单

  • 目标已确认:任务范围、成功率、节拍、测试条件和人机协作边界有共同定义。
  • 流程已跑通:硬件可达、视野充分,常规 SOP 与恢复 SOP 都有对应数据和真机验证。
  • 执行可持续:安全护栏生效、规则接管平滑、空料框等异常状态有明确处理,已知硬件故障有排查方法。
  • 反馈能闭环:现场问题能归因,数据与人工反馈能进入下一轮迭代。必要时结合检测、跟踪与分割模块补足感知,并检查遮挡下不同模块输出的含义。
← 返回文章列表
← All articles
VLA / Field Notes

How to Deliver a VLA System: A Small-Parts Picking Case

Start with delivery goals, SOP design and hardware, then close the loop across data, policy, control and real-robot feedback. Every step is illustrated with a field project in which a single arm picks small parts from a cluttered bin onto a conveyor.

00 / The case: single-arm small-parts picking

A single arm picks small boxes one by one from a cluttered bin and places them on a conveyor. The policy is a π0.5-class vision-language-action model, bootstrapped from teleoperated SOP data collected at another site, then extended with data from the target site and deployed on an on-site edge GPU. It took 26 days from the first field test to a demonstration the customer accepted. Every "small-parts picking example" below comes from this project.

MetricResult
Peak throughput on site680 pieces/hour
Sustained throughput420 pieces/hour
Single-grasp success in a cluttered bin41% → 75–94%
Server-side inference per request165 → 42 ms (48.5 ms measured on the robot)
Internal star rating1 → 4.3 stars
Training / field evaluation104 models in 25 days; 30 field evaluations in 8 days

01 / Define delivery goals

VLA (Vision-Language-Action) delivery usually targets a specific operational setting. Object, destination and workstation priors can support faster iteration through post-training and reinforcement learning, with different priorities from pre-training and mid-training.

Delivery requires sustained operation on site, with agreed success rate, cycle time, accuracy and recovery behavior. Confirm these goals first, then design the standard operating procedure (SOP) and select hardware. Data collection follows those decisions.

  • Define task boundaries: objects, reachable locations, allowed environmental changes and cases requiring human assistance.
  • Evaluate success rate together with cycle time, and agree on test conditions and failure criteria. Reconcile differences across development, collection and delivery sites.

02 / Validate hardware across the workspace

  • Reach and gripper: cover all operating locations; check opening width, closing speed and grasp stability on irregular objects. A reading at the gripper base may differ from the actual fingertip gap.
  • Edges and corners: check whether the gripper or wrist camera collides with the bin while reaching boundary objects.
  • Vision and touch: the main camera must cover the scene without critical occlusion by the end effector. Check whether tactile signals distinguish aligned and misaligned contact.
  • Check mechanics and sensing early: loose grippers, motor power loss, motion blur, low camera frame rate and duplicate frames can all appear as policy failures.

03 / SOP design is the critical step

Simplify the task physically first: a ramp may assist flipping, or a two-handed operation may become a one-handed one. Simpler procedures with distinguishable states are easier to learn and debug.

  • Keep the robot, table and bin layout reproducible in simulation. Vary bin position within an allowed range during collection to cover deployment tolerances. A compliant layer at the bin bottom can reduce impact.
  • Define ideal, routine and recovery procedures separately. Reproduce typical errors through teleoperation and collect recovery actions for states such as missed grasps and slips.
  • Reduce accidental contact, target switching, hesitation and unnecessary object adjustments. Small demonstration habits can become policy behavior.
  • Check ambiguous stages: returning and approaching, or lifting after successful and failed grasps, may look alike in a single frame. An action classifier using images, gripper width and joint angles can expose confusion between stages.
  • The person training the policy should collect at least ten demonstrations; key collaborators should also participate to develop a shared understanding of the procedure.

04 / Put guardrails in the execution loop

  • Detect bottom contact using the physical fingertip geometry, such as corners or edges, rather than only the end-effector reference position.
  • At a depth limit, or when downward commands persist without corresponding motion, inhibit further descent while preserving permitted gripper and retreat actions. A 5 cm upward retreat is one example where clearance permits; actual thresholds depend on tooling, objects and clearance.
  • Six-axis force sensing requires calibration and compensation for gravity, acceleration and payload before it can reliably inform control.

05 / Collect clean demonstrations and recovery data

  • Start each demonstration from the agreed initial state. Remove takeover jitter and observation periods; execute decisively and keep relevant objects visible.
  • Distinguish task, sub-task and action/skill labels, with environmental context for special actions. Brief pauses can mark important skill boundaries and should not be removed indiscriminately.
  • Remove irrelevant stationary frames and anomalous spikes, with action smoothing where appropriate. For imitation learning, retain correct behavior and recovery while excluding the preceding erroneous actions.

06 / Focus training and offline evaluation on critical stages

  • Augment for image changes caused by robot motion so the policy does not rely solely on smooth teleoperation footage.
  • Use counterfactual samples to assess instruction following. Annotate special SOP branches with scene and action descriptions, and consider action and progress prediction to distinguish stages.
  • Weight critical frames and skills more heavily in training and evaluation, and consider end-effector auxiliary heads and losses. Inspect grasping, contact and recovery in addition to aggregate error.

07 / Turn policy actions into smooth robot control

Start with stepwise execution to assess targeting accuracy, grasp stability and recovery separately. Then inspect inference latency, trajectory interpolation and low-level control before attributing every failure to the policy.

  • Consider a 30 Hz action chunk feeding a 500 Hz controller. Interpolation bridges the rates; quintic polynomials can provide continuous position, velocity and acceleration. These rates are examples and must match the robot interface.
  • An overly stiff position controller can amplify jitter. Tune PID for the platform and evaluate hybrid force/position control when supported.

08 / Combine the policy with smooth rule-based handoffs

  • After a successful grasp and lift, rules can handle deterministic carrying, placement and return motions, leaving the policy to manage complex interactions.
  • Maintain smooth velocity and acceleration across handoff, obstacle-avoidance and placement points. Avoid abrupt stops or jumps when switching between rules and the policy. Placement can vary within the allowed region.
  • Validate grasp retention separately. Additional closing actions after grasping may reduce slips; force and repetition must suit the object and gripper.

09 / Close the loop with real-robot feedback

  • Test out-of-distribution states such as empty bins, absent targets and missing wrist-camera views, with defined stop, recovery or assistance behavior.
  • Keep human feedback for each real-robot evaluation. Distinguish targeting errors, unstable grasps, failed recovery, control jitter and hardware faults before choosing what to change.
  • Feed real-robot data back into iteration, especially jitter, blur and recovery cases. Deployment imagery differs from demonstrations and needs coverage in subsequent collection and training.

10 / Before handover

  • Goals agreed: task scope, success rate, cycle time, test conditions and human-assistance boundaries are defined.
  • Procedure validated: hardware reaches required locations, cameras provide sufficient coverage, and routine and recovery procedures have data and real-robot validation.
  • Operation supported: guardrails work, handoffs are smooth, exceptional states have defined handling and known hardware faults can be diagnosed.
  • Feedback actionable: site issues can be diagnosed, and data plus human feedback feed the next iteration. Detection, tracking and segmentation may supplement perception; check how their outputs differ under occlusion.
← All articles
← 返回文章列表
VLA / 实践笔记

VLA 数据预处理:以小件拣选为例的八层流水线(L0–L7)

一条连续清框录像怎样变成训练样本:信号体检、切周期与关键段、按姿态判定目标数据、用 3D 轨迹对比筛选数据源、失败分类、技能标签、训练权重、停顿切除和速度归一。每一层都给出规则、阈值和在约两千万帧数据上的实测结果。

01 / 为什么数据处理决定上限

模型学到的上限由数据决定。在小件拣选项目里,多次出问题,根子都在数据:头部相机实际只有 12.5 fps;一批清框数据的左右腕部相机在录制端接反;失败恢复数据用法不同,离线误差能相差 −61% 到 +30%;回位和起始接近混在一起训练,模型会把两者搞混。所以每条录像进入训练之前,都要过同一条处理流水线。

下面三条是交付篇中“数据质量”一节的原则,八层流水线把它们变成了可执行、可统计的规则:

  • 每次遥操从约定初始位置开始,剔除接手前的抖动与观察阶段。动作果断,操作对象始终处于有效视野内。
  • 标注区分 task、sub-task 与 action/skill,并为特殊动作补充环境描述。小停顿可能是关键技能边界,不能一概作为无效数据删除。
  • 清理无效静止帧和异常尖峰,按需平滑动作。模仿学习数据保留正确行为与恢复行为,但要剔除导致失败的错误动作片段。

02 / 八层总览:从一条录像到训练样本

采集方式是“连续清框”:一条录像里连续抓取多个物品,中间有成功、空抓、碰触、恢复和停顿。流水线先做 L0 信号体检,再用 L1–L7 七层把录像切开、分类、加权和对齐。左右臂各自独立处理;所有层只生成标签,不改原始数据,因此任何一层的规则都可以单独修改、重算。

层单位做什么解决什么
L0 信号体检每条录像每只手臂关节跳变、FK 一致性、夹爪读数与指令、时间戳、相机同步初筛:问题段训练权重置 0
L1.1 完整周期周期HOME → 抓取 → 抬起 → 放下 → 回 HOME非关键阶段只从目标场景取
L1.2 关键段(核心)周期物体上方 10 cm → 抓取 → 抬到物体上方 5 cm只借最难、最值得学的一段
L1.3 目标 / 辅助关键段三个时刻的关节姿态与目标场景比较多源数据的准入标准
L2 尝试抓取尝试每次下探到抓取高度算一次,无论是否闭合找出空抓、碰触等失败尝试
L3 结果尝试成功或 F1–F14 失败类型失败不学,恢复要学
L4 技能每帧三段(fall / grasp / raise)+ 12 技能阶段提示与细粒度数据分析
L5 权重每帧是否训练 0/1;接触窗权重 ×2汇总成训练权重
L6 停顿切除帧序列切掉停顿,接缝处注入过渡动作采集员的停顿习惯
L7 速度归一每段自由段速度对齐到基准采集员,接触窗不变速采集员的速度差异

03 / L0 信号体检:先把明显有毛病的去掉

参考 Qwen 机器人操作工作中的状态—动作过滤,每条录像的每只手臂先过 L0。不通过的整段或局部标记为 qa_fail,训练权重置 0。表中阈值是初值,跑完全量数据后按分布再定。

在 9,328 个文件、16,688 条单臂录像上跑完 L0:2,556 条含问题段,主要是时间戳跳变(2,412 条)和夹爪读数与指令对不上(115 条)。

检查怎么判初始阈值
关节跳变相邻帧关节差或其二阶差突增单帧 > 0.2 rad,或 > 该关节 p99.9 的 3 倍
极值 / 越限关节超出 URDF 限位或录像内 p0.1–p99.9 之外超限即标
FK 一致由记录的关节算 FK,与记录的末端位姿比较位置 > 1–2 cm 或姿态 > 5°
夹爪读数 vs 指令指令闭合 ≥ 1 s 后实测宽度应下降下降 < 6 mm 即标
状态冻结关节整段不变而指令在变> 2 s
时间戳 / 掉帧相邻时间戳差> 200 ms
相机健康与身份黑帧、冻结、重复帧、帧率;左右腕部相机是否接反同步偏移 > 2 帧

04 / L1 切周期与关键段:最难的是抓取本身

完整周期是 HOME → 抓取 → 抬起 → 放下 → 回 HOME,只保留目标场景的数据。关键段(核心)是从物体上方 10 cm 下落 → 抓取 → 抬到物体上方 5 cm(高度以闭合时刻的末端高度为物体参考),这是整个动作里最难、最值得学的部分。借用其他 SOP 或其他场景的数据时,只借这一段;从 home 出发、搬运、放下、回位这些容易学的阶段,只从目标场景取。

统一定义之前,每条流水线各用一套规则,同样叫 focus / lift 的帧在不同数据集里完全不同:一套里 focus 从抓取点上方约 1.7 cm 开始、lift 抬到约 15 cm;另一套里 focus 从约 3.6 cm 开始、lift 只抬约 4.1 cm。按新定义,在 2,582 条录像、11,329 次有效闭合上,95% 能同时找到“上方 10 cm”和“抬到 5 cm”两个锚点;剩下的取“上一次张开以来的最高点”作起点。侧向取物时,起点改为“离抓取点 10 cm”。

来源10 cm → 闭合闭合 → 抬起 5 cm关键段总长
目标现场(基准采集员)2.43 s0.53 s3.03 s
另一站点 SOP2.60 s0.60 s3.23 s
堆叠场景2.92 s1.00 s4.03 s
难例(拥挤料框)3.70 s0.67 s4.45 s
失败恢复4.33 s1.03 s5.43 s

05 / L1.3 目标还是辅助:按姿态而不是按来源

在关键段上再分一层:这次抓取和目标场景是不是“同一个动作”。判据是 HOME、抓取、抬起 5 cm 三个时刻的 7 个关节角,到最近的目标场景关键段的距离都不超过目标场景自身留一最近邻距离的 p95(HOME 0.092 rad、抓取 0.243 rad、抬起 0.227 rad)。通过的按子类训练(正常抓取、失败后的恢复抓取、现场回收的 RECAP 数据等);不通过的标为辅助,只作辅助训练。失败恢复从失败处开始,不比 HOME。

  • 目标现场自己 88% 三者都在范围内,与“半径覆盖约 80–90% 样本”的设计一致。
  • 另一站点的 SOP 名义上是同一任务,实测只有 2% 合格:HOME 的关节角整体偏 0.1–0.23 rad 且非常集中(换了起始姿态),抓取时手腕翻滚差约 0.3 rad。按关节角,它不是同一个动作。
  • 多品类小件来源和双臂数据接近 0%:它们仍然有用,但只能作辅助数据,不能当目标数据混训。
来源关键段HOME 在范围内抓取在范围内抬起在范围内标为目标
目标现场73595%95%95%646(88%)
另一站点 SOP80236%16%8%16(2%)
失败恢复132050%22%20%169(13%,不比 HOME)
难例(拥挤料框)47543%11%7%0(按来源归辅助)
堆叠场景136878%30%30%0(按来源归辅助)
双臂录像114374%1%2%0
多品类小件92430%0%0%0

06 / 3D 轨迹对比(一):抓取瞬间的姿态离目标有多远

不同来源采的数据,有的训练效果好,有的不好,判断的核心是比较 3D 轨迹。第一步看抓取瞬间:在“夹爪开始闭合”这一帧,取每次抓取的 7 个关节角、躯干坐标系下的末端位姿和躯干状态,逐个来源与目标现场的抓取集合比较。“区域内”指到最近目标抓取的距离不超过目标自身留一最近邻的 p95(关节 0.25 rad,位置 18 mm);姿态差是与位置最近的目标抓取之间的旋转角。

来源抓取数关节最近邻(中位,rad)关节区域内位置区域内位置最近邻(mm)姿态差抓取高度 z(mm)
目标现场(参照)6330.1695%95%913°−180
另一站点 SOP6530.3118%18%3720°−90
失败恢复9930.3027%64%1419°−133
旋转放置(历史项目)1920.3215%61%1422°−132
难例(拥挤料框)3270.3517%44%2022°−115
双臂录像的左臂49290.473%8%6237°−92
另一场景的右臂数据6430.760%0%21943°−192
多品类小件20770.870%0%11857°−321
新采集的小件清框7690.3233%47%1930°−178

07 / 3D 轨迹对比(二):路径形状、采集员差异与筛选

第二步看整条关键段路径:以抓取点为原点,把每次抓取从物体上方 10 cm 画到抬起 5 cm,比较俯视、侧视、正视三视图和可旋转的 3D 图。即使是同一场景,不同人采的数据差别也很大:平滑的抓取只在抓取点附近有一个极值;有人下落时来回调整,轨迹前伸、折返,空中出现多个极值。

  • 速度:以目标现场的基准采集员为准,其他采集员的自由段要提速约 1.1 到 3 倍才追平。
  • 停顿:有人在空中明显停住,有人一气呵成。
  • 极值点:目标现场平均每次 1.2 个、38% 一个都没有;另一站点 SOP 1.6 个;多品类小件 3.0 个;双臂录像 4.6 个。
  • 筛选规则:先按 L1.3 的姿态区域决定目标还是辅助;再用速度(L7)和停顿(L6)对齐;轨迹多极值、前伸折返目前只做统计,是下一步要补的处理层。
  • 去抖:遥操数据在运动时带约 10 Hz 的抖动分量。基于 DCT 的自适应平滑在 200 条 SOP 录像上去掉了 7–13 Hz 的抖动、完整保留 0–3 Hz 的动作,加加速度(jerk)降低 95%,但尚未接入训练验证。
四类来源的抓取轨迹侧视图
侧视图(向前 – 高度,单位 mm)。每个来源随机取最多 14 次抓取,从物体上方 10 cm 画到抬起 5 cm,红点为抓取点。A 目标现场基准采集员的路径最干净;C、D 的路径前伸、折返明显,空中极值多。

08 / L2–L3 抓取尝试与失败分类:F1–F14

一个周期里每次下探到抓取高度都算一次尝试,无论是否闭合。每次尝试落到“成功”或 14 类失败之一。能可靠测量的现在就处理;测不出或归不了类的标记为“已见、暂不处理”,对应段训练权重为 0。F14 是兜底类,收容高度可疑但测不出的情况。

全部数据共 38,045 次尝试:成功 24,712;抬起前重夹(F9)5,255;手指触碰(F2)3,632;下探放弃(F1)2,412;其他可疑(F14)780;空抓(F5)473;抓到窄边(F7)342;抬起时脱落(F10)322。注意:没有力信号时,F1 / F2 里可能有一部分只是重新对准、并没有碰到物体;它们之后的回升按恢复训练。

类别名称判据处理
F1下探放弃到抓取高度附近未闭合即抬起暂不处理
F2手指触碰末端在抓取高度附近局部低点、回弹 ≥ 0.5 cm、未闭合处理
F3推挤 / 碰歪未闭合时物体跟踪位移 > 0.35 个物长或 > 10°处理(有物体跟踪的数据)
F4选错目标对准的物体 ≠ 最后抓起的物体暂不处理
F5空抓闭合期间最窄宽度 ≤ 20 mm处理
F6没夹紧 / 夹偏最窄宽度 42–48 mm暂不处理 → F14
F7抓到窄边宽度 6–13 mm 且夹着移动 ≥ 20 cm处理
F8错位闭合在非物体位置闭合(需物体跟踪)暂不处理
F9抬起前重夹抬起前多于一次闭合,含夹住—拖动—重夹处理:拖动段不训,最后一次夹取起训练
F10抬起时脱落抬起中宽度 < 5 mm处理
F11搬运中掉落夹到过、放下指令前掉到 < 5 mm处理
F12空夹爪搬运从未夹到却移动 ≥ 20 cm处理
F13碰撞手臂或物体碰到料框 / 其他物体(需跟踪或力信号)暂不处理
F14其他可疑高度怀疑有问题但测不出、归不了类训练权重 0,待人工复核

09 / L4 技能标签:12 类技能与三段

一个抓放周期里有很多种动作。每一帧按动作的作用标成 12 类技能,再归并成三段(fall / grasp / raise)作为模型的阶段提示。新定义下,focus 从抓取点上方约 10 cm 开始到闭合指令为止,lift 只到抬起约 5 cm,之后到放下前算 carry。有遥操作夹爪指令的数据用规则切,没有的用腕部检测器的技能头逐帧预测再平滑,两者在共有数据上对账。

  • 停顿多少:目标现场和另一站点 SOP 有 24–25% 的帧在 wait,多品类小件只有 6%。
  • 对准花多久:focus 在目标现场占 17%,堆叠和双臂占 24–31%,后两者在物体上方调整的时间明显更长。
  • 失败和恢复在哪里:失败恢复数据有 2.7% 的帧是 touch、1.4% 是 lift_recovery、12% 是 other;目标现场几乎为 0。
  • 关键动作有多少:真正闭合抓取(close_gripper)在所有数据里只占 3% 的帧,这正是关键段要加权、不能被其他阶段淹没的原因。
三段技能含义
fallapproach从高处向物体下落接近
fallfocus在物体上方对准、微调(从约 10 cm 起)
graspclose_gripper闭合夹爪抓取
raiselift抓住后抬起(到约 5 cm)
raiselift_recovery失败后抬起、重新开始
—touch失败:手指碰到物体或料框,没抓住
—carry搬运到放置点
—open_gripper张开夹爪放下
—return回到 home(保留标签,一般不训练)
—wait停住等待
—global_observe在 home 附近观察全局
—other以上都不是,包括空抓等其他失败段

10 / L5 是否训练与权重

前面各层的结论在 L5 汇总成逐帧的训练开关和权重。权重按动作块里的每一步计算:块里落在失败帧上的步,权重为 0。公开工作里,GiB 同样把坏段的模仿损失置 0;π*0.6 的 RECAP 则不屏蔽,而是给坏动作打低优势标签——可以作为第二方案。

段落是否训练权重
成功尝试的关键段11
接触窗(接触前 45 帧到后 5 帧)12
失败后的恢复段(截断周期里也训)11
F1–F14 失败段(含暂不处理的)00
L0 问题段00
完整周期中关键段以外的部分(不含 return)目标场景 1,其他场景 01
return00(保留标签)
录像尾部未完成的周期00

11 / L6 切除停顿并注入过渡动作

在 2,876 条录像上实测,关键段内停顿长度中位 8 帧(30 Hz,p90 为 22 帧),相邻停顿间隔中位 32 帧;间隔 ≤ 10 帧的占 14%,“两三个挨得很近”的停顿确实存在。切掉停顿后,接缝处按前后平均速度注入过渡动作,用三次 Hermite 插值;注入帧只作动作目标,不作样本起点(它们没有图像)。

公开做法不同:openpi 处理 DROID 数据时不拼接、不插值,只禁止在静止段及其前约半个动作块内取样本起点,动作块仍读原始的未来。它更简单、零风险,但动作块里仍会出现停顿,适合作为 L6 的对照实验。

参数取值依据
最短切除停顿≥ 6 帧(0.2 s)更短的是手抖级别,切了反而破坏连续性
估速窗口 k5 帧与静止判据的 ±5 帧一致;间隔中位 32 帧,两侧都够用
合并规则间隔 < 10 帧的相邻切段合成一段,只注入一次否则估速窗口会跨进另一个停顿
注入速度前后平均速度,三次 Hermite 插值保证接缝处速度连续
接触窗不切,也不跨接触窗合并接触前后的细小停顿可能就是技能边界

12 / L7 速度归一到基准采集员

按关键段(10 cm → 抓取 → 5 cm)统计每位采集员的中位时长。目标现场的基准采集员最快,但这位采集员在另一站点的 SOP 里慢了 45%,所以目标速度取其在目标现场的关键段,而不是按人名一刀切。场景难度也有影响:同一位采集员在 SOP 里 3.23 s,在难例里 4.45 s。因此比例只在同场景内比,跨场景只对齐自由段。

公开工作的结论一致:DemoSpeedup 只加速自由段,均匀 2 倍反而掉成功率;ESPADA 只加速非接触段;SAIL 在实机上提速明显,但接触多的任务例外;TempoVLA 则把速度作为条件输入。

  • 分段:关键段分成自由段(10 cm → 接触窗开始、接触窗结束 → 5 cm)和接触窗。
  • 算比例:每段 r = 该段时长 / 基准采集员同段中位时长。
  • 只加速不减速:r > 1 的自由段按 r 抽帧,图像和动作同步抽,不需要合成图像;r ≤ 1 不动;接触窗保持 1 倍。
  • 顺序:先 L6 切停顿,再 L7;否则停顿会把 r 拉大。
采集员场景次数关键段总长相对基准
A(基准)目标现场7663.03 s1.00×
A另一站点 SOP1094.40 s1.45×
B另一站点 SOP7453.23 s1.07×
B堆叠场景12304.03 s1.33×
B难例5284.45 s1.47×
B失败恢复5495.43 s1.79×
C双臂录像34653.73 s1.23×
D双臂录像29114.37 s1.44×
E双臂录像2965.90 s1.95×

13 / 全流程跑在全部数据上的结果

L0–L7 在全部取放数据上跑完:9,328 条录像、16,688 只活动手臂、约 2,049 万帧,单进程约 2 分钟。只生成标签,不改训练文件。失败后的恢复段一律训练(4,410 段,之前有 46% 被失败段误置 0);L6 不切接触窗,124.8 万接触帧一帧未动。关键段帧数经 L6、L7 后共减少约 31%,目标现场只减少 10%(它本来就快)。

来源关键段帧L6 后L7 后关键段共减少
全部31363482823916217156931%
新场景小件(多来源)105474391843772640831%
目标现场68306672746118910%
另一站点 SOP85981826626917320%
难例68364610364299737%
堆叠场景18697117454112029436%
双臂录像14985681361049103914331%
失败恢复17341515891711236535%

14 / 小结与仍缺的部分

  • 先筛后用:信号、夹爪、相机的明显问题在 L0 一次查完,并作为自动检查跑在每次新数据同步上;“流是活的、但身份错了”这类问题,靠人眼发现太晚。
  • 多源数据要有准入标准:按姿态和目标场景比较(L1.3),而不是按来源整体接收或丢弃;名义上的同一任务,实测可能只有 2% 合格。
  • 只借关键段:其他来源只贡献 10 cm → 抓取 → 5 cm 的关键段,接触窗加权、不切、不加速。
  • 3D 轨迹对比是判断数据好坏的核心工具:抓取瞬间的关节与位姿说明“是不是同一个动作”,整条路径的极值和折返说明“采得干不干净”,还能追查到采集设置这样的根因。
  • 归一化是混训的前提:数值上,归一化统计量按每次的数据组合重算;时间上,L6 切停顿、L7 统一速度;空间轨迹的平滑和对齐先从关键段做起,以抓取点和上下两个高度为锚点。
  • 仍缺的部分:周期级质量分与去重;按采集员或场次划分训练集和验证集(随机按录像抽会让相似周期两边都有);按采集员或站点平衡采样;以及针对轨迹多极值、前伸折返的处理层。
← 返回文章列表
← All articles
VLA / Field Notes

VLA Data Preprocessing: An Eight-Layer Pipeline (L0–L7) for Small-Parts Picking

How a continuous bin-clearing recording becomes training samples: signal checks, cycle and core-segment extraction, pose-based target admission, 3D trajectory comparison of data sources, failure classes, skill labels, training weights, pause removal and speed normalization. Each layer comes with its rules, thresholds and measured results on about twenty million frames.

01 / Why data processing sets the ceiling

The data sets the ceiling on what a policy can learn. In the small-parts picking project, problem after problem traced back to data. The head camera actually ran at 12.5 fps; in one bin-clearing collection the left and right wrist cameras were swapped at the recorder; different uses of failure-recovery data moved offline error anywhere from −61% to +30%; and training on returning and initial approaching together confused the policy. Every recording therefore passes through the same processing pipeline before it reaches training.

The three principles below come from the data-quality section of the delivery article; the eight layers turn them into executable, measurable rules:

  • Start each demonstration from the agreed initial state. Remove takeover jitter and observation periods; execute decisively and keep relevant objects visible.
  • Distinguish task, sub-task and action/skill labels, with environmental context for special actions. Brief pauses can mark important skill boundaries and should not be removed indiscriminately.
  • Remove irrelevant stationary frames and anomalous spikes, with action smoothing where appropriate. For imitation learning, retain correct behavior and recovery while excluding the preceding erroneous actions.

02 / Overview: from one recording to training samples

Data is collected by continuous bin clearing, with many picks per recording, mixing successes, empty grasps, touches, recoveries and pauses. The pipeline first runs the L0 signal check, then seven layers, L1–L7, cut, classify, weight and align the recording. Left and right arms are processed independently. Every layer only writes labels and never edits raw data, so any layer's rules can be changed and recomputed on their own.

LayerUnitWhat it doesWhat it solves
L0 signal checkEach recording and armJoint jumps, FK consistency, gripper reading vs command, timestamps, camera syncScreening: faulty segments get zero training weight
L1.1 full cycleCycleHOME → grasp → lift → place → HOMENon-critical phases come only from the target scene
L1.2 core segmentCycle10 cm above the object → grasp → lift to 5 cm above itBorrow only the hardest, most valuable segment
L1.3 target / auxiliaryCore segmentCompare joint poses at three moments with the target sceneAdmission rule for multi-source data
L2 attemptsGrasp attemptEvery descent to grasp height counts, closed or notFind empty grasps, touches and other failed attempts
L3 outcomesAttemptSuccess or failure classes F1–F14Do not learn failures; do learn recoveries
L4 skillsFrameThree stages (fall / grasp / raise) plus 12 skillsStage prompts and fine-grained data analysis
L5 weightsFrameTrain 0/1; contact window weighted ×2Combine everything into training weights
L6 pause removalFrame sequenceCut pauses, inject bridging actions at the seamsOperators' pause habits
L7 speed normalizationSegmentAlign free-motion speed to the benchmark operator; contact window unchangedSpeed differences between operators

03 / L0 signal check: remove the obviously broken first

Following the state-action filtering in Qwen's robot-manipulation work, each arm of each recording first passes L0. Failing segments, or whole recordings, are marked qa_fail and get zero training weight. The thresholds are initial values, to be revised from the distribution after a full pass.

Across 9,328 files and 16,688 single-arm recordings, 2,556 contained faulty segments, mainly timestamp gaps (2,412) and gripper readings that did not follow commands (115).

CheckRuleInitial threshold
Joint jumpsSudden rise in frame-to-frame joint change or its second difference> 0.2 rad in one frame, or > 3× the joint's p99.9
Extremes / limit violationsJoints beyond URDF limits or outside the recording's p0.1–p99.9Flag on violation
FK consistencyFK from recorded joints vs the recorded end-effector pose> 1–2 cm or > 5°
Gripper reading vs commandMeasured width must drop after ≥ 1 s of a close commandFlag if drop < 6 mm
Frozen stateJoints constant while commands change> 2 s
Timestamps / dropped framesGap between consecutive timestamps> 200 ms
Camera health and identityBlack, frozen or duplicate frames, frame rate; swapped left/right wrist camerasSync offset > 2 frames

04 / L1 cycles and core segments: the grasp is the hard part

A full cycle runs HOME → grasp → lift → place → HOME and is kept only for the target scene. The core segment runs from 10 cm above the object down to the grasp and up to 5 cm above the object, with the end-effector height at closing as the object reference. It is the hardest and most valuable part of the motion. When borrowing data from other SOPs or scenes, borrow only this segment; the easy phases (leaving home, carrying, placing, returning) come only from the target scene.

Before the definitions were unified, each pipeline had its own rules, and frames called focus or lift meant different things in different datasets. In one, focus began about 1.7 cm above the grasp point and lift ended near 15 cm; in another, focus began near 3.6 cm and lift ended at about 4.1 cm. Under the new definition, 95% of 11,329 valid closes in 2,582 recordings have both anchors, 10 cm above and lifted 5 cm. The rest start from the highest point since the previous opening. For side picks, the start is 10 cm from the grasp point instead.

Source10 cm → closeClose → lifted 5 cmCore total
Target site (benchmark operator)2.43 s0.53 s3.03 s
SOP from another site2.60 s0.60 s3.23 s
Stacking scene2.92 s1.00 s4.03 s
Hard cases (crowded bin)3.70 s0.67 s4.45 s
Failure recovery4.33 s1.03 s5.43 s

05 / L1.3 target or auxiliary: decide by pose, not by source

One more split on the core segment: is this grasp the same move as in the target scene? The rule compares the seven joint angles at three moments, HOME, grasp and lifted 5 cm, with the nearest target-scene core. Each distance must be within the p95 of the target scene's own leave-one-out nearest-neighbour distance (HOME 0.092 rad, grasp 0.243 rad, lift 0.227 rad). Passing cores train under their subtype (normal grasp, recovery after failure, field RECAP data and so on); the rest are marked auxiliary. Recovery cores start where the failure happened, so HOME is not compared for them.

  • The target site itself has 88% within range at all three moments, consistent with radii meant to cover about 80–90% of samples.
  • The SOP from another site is nominally the same task, yet only 2% qualify. Its HOME joints are offset by 0.1–0.23 rad and tightly clustered (a different start pose), and wrist roll at the grasp differs by about 0.3 rad. By joint angles, it is not the same move.
  • Multi-item small-parts data and dual-arm data are close to 0%. They are still useful, but only as auxiliary data, not mixed in as target data.
SourceCore segmentsHOME in rangeGrasp in rangeLift in rangeLabelled target
Target site73595%95%95%646 (88%)
SOP from another site80236%16%8%16 (2%)
Failure recovery132050%22%20%169 (13%, HOME not compared)
Hard cases (crowded bin)47543%11%7%0 (auxiliary by source)
Stacking scene136878%30%30%0 (auxiliary by source)
Dual-arm recordings114374%1%2%0
Multi-item small parts92430%0%0%0

06 / 3D trajectory comparison (1): how far grasp poses are from the target

Some sources train well and some do not, and the core test is to compare 3D trajectories. Start with the moment of grasping. At the frame where the gripper starts closing, take the seven joint angles, the end-effector pose in the torso frame and the torso state, and compare each source with the target site's set of grasps. In region means the distance to the nearest target grasp is within the target's own leave-one-out p95 (0.25 rad in joints, 18 mm in position). Orientation difference is the rotation to the positionally nearest target grasp.

SourceGraspsJoint NN (median rad)Joints in regionPosition in regionPosition NN (mm)Orientation diffGrasp height z (mm)
Target site (reference)6330.1695%95%913°−180
SOP from another site6530.3118%18%3720°−90
Failure recovery9930.3027%64%1419°−133
Rotate-and-place (earlier project)1920.3215%61%1422°−132
Hard cases (crowded bin)3270.3517%44%2022°−115
Left arm of dual-arm recordings49290.473%8%6237°−92
Right-arm data from another scene6430.760%0%21943°−192
Multi-item small parts20770.870%0%11857°−321
Newly collected small-part clearing7690.3233%47%1930°−178

07 / 3D trajectory comparison (2): path shape, operator differences and selection

Next, compare whole core paths. With the grasp point as the origin, draw each grasp from 10 cm above the object to the 5 cm lift and compare top, side and front views plus a rotatable 3D view. Even within one scene, operators differ a lot. A smooth grasp has a single extremum near the grasp point; others adjust back and forth on the way down, overshooting and backtracking, with several extrema in mid-air.

  • Speed: measured against the target site's benchmark operator, other operators' free-motion segments need a 1.1× to 3× speed-up to match.
  • Pauses: some operators stop visibly in mid-air, others move in one continuous stroke.
  • Extrema: 1.2 per grasp at the target site, with none in 38% of grasps; 1.6 for the other site's SOP; 3.0 for multi-item small parts; 4.6 for dual-arm recordings.
  • Selection: L1.3 pose regions decide target versus auxiliary; speed (L7) and pauses (L6) are then aligned. Multiple extrema and overshoot are only measured so far and are the next layer to add.
  • De-jittering: teleoperation data carries a roughly 10 Hz component during motion. Adaptive DCT smoothing on 200 SOP recordings removed the 7–13 Hz jitter, kept 0–3 Hz motion intact and cut jerk by 95%; it has not yet been validated in training.
Side view of grasp trajectories from four sources
Side view (forward vs height, mm). Up to 14 random grasps per source, drawn from 10 cm above the object to the 5 cm lift; the red dot is the grasp point. The target site's benchmark operator (A) has the cleanest paths; C and D overshoot and backtrack, with many mid-air extrema.

08 / L2–L3 grasp attempts and failure classes: F1–F14

Every descent to grasp height within a cycle counts as an attempt, whether or not the gripper closes. Each attempt ends as a success or one of 14 failure classes. Classes that can be measured reliably are handled now; the rest are marked seen but unprocessed, with zero training weight. F14 is the catch-all for highly suspicious cases that cannot be measured or classified.

All data together has 38,045 attempts: 24,712 successes, 5,255 regrasps before lift (F9), 3,632 finger touches (F2), 2,412 aborted descents (F1), 780 other suspicious (F14), 473 empty closes (F5), 342 edge grasps (F7) and 322 drops during lift (F10). Without force sensing, some F1/F2 cases may just be re-alignment without contact; the rise that follows them is trained as recovery.

ClassNameRuleHandling
F1Aborted descentRises near grasp height without closingNot yet handled
F2Finger touchLocal low point near grasp height, rebound ≥ 0.5 cm, no closeHandled
F3Push / knockTracked object moves > 0.35 object lengths or > 10° before closingHandled (with object tracking)
F4Wrong targetAligned object ≠ the object finally pickedNot yet handled
F5Empty closeMinimum width ≤ 20 mm while closingHandled
F6Partial closeMinimum width 42–48 mmNot yet handled → F14
F7Edge graspWidth 6–13 mm while moving ≥ 20 cmHandled
F8Misplaced closeClosing away from any object (needs tracking)Not yet handled
F9Regrasp before liftMore than one close before lifting, including grip-drag-regripHandled: drag not trained, train from the last grasp
F10Drop during liftWidth < 5 mm while liftingHandled
F11Drop while carryingHeld once, falls below 5 mm before the release commandHandled
F12Carrying with an empty gripperNever held but moves ≥ 20 cmHandled
F13CollisionArm or object hits the bin or other objects (needs tracking or force)Not yet handled
F14Other suspiciousHighly suspicious but unmeasurable or unclassifiableZero weight, pending human review

09 / L4 skill labels: 12 skills and 3 stages

A pick-and-place cycle contains many kinds of motion. Each frame gets one of 12 skills by what the motion does, and the skills merge into three stages (fall, grasp, raise) used as stage prompts. Under the new definitions, focus runs from about 10 cm above the grasp point to the close command, lift ends at about 5 cm, and the rest until release is carry. Data with teleoperation gripper commands is segmented by rules; data without them uses a wrist detector's skill head per frame plus smoothing, and the two are reconciled where both exist.

  • How much pausing: 24–25% of frames are wait at the target site and in the other site's SOP, only 6% for multi-item small parts.
  • How long alignment takes: focus is 17% of frames at the target site versus 24–31% for stacking and dual-arm data, which spend much longer adjusting above the object.
  • Where failures and recoveries are: in failure-recovery data 2.7% of frames are touch, 1.4% lift_recovery and 12% other; at the target site these are almost zero.
  • How rare the key action is: actual closing (close_gripper) is only 3% of all frames, which is why the core segment is weighted rather than drowned out by other phases.
StageSkillMeaning
fallapproachDescend toward the object from above
fallfocusAlign and fine-tune above the object (from about 10 cm)
graspclose_gripperClose the gripper
raiseliftLift after grasping (to about 5 cm)
raiselift_recoveryLift after a failure, then start again
—touchFailure: fingers touch the object or bin without grasping
—carryCarry to the placement point
—open_gripperOpen the gripper to release
—returnReturn home (label kept, normally not trained)
—waitStop and wait
—global_observeObserve the scene near home
—otherNone of the above, including empty closes and other failures

10 / L5 what to train and how much

L5 combines the earlier layers into per-frame training switches and weights. Weights apply to each step of an action chunk; steps that land on failure frames get zero weight. Among public work, GiB likewise zeroes the imitation loss on bad segments, while π*0.6's RECAP keeps them and assigns low advantage labels instead, which is a possible second approach.

SegmentTrainWeight
Core segment of a successful attempt11
Contact window (45 frames before to 5 after contact)12
Recovery after a failure (also in truncated cycles)11
Failure segments F1–F14 (including unhandled classes)00
L0 faulty segments00
Rest of the full cycle outside the core (excluding return)1 in the target scene, 0 elsewhere1
return00 (label kept)
Unfinished cycle at the end of a recording00

11 / L6 cut pauses and inject bridging actions

Measured on 2,876 recordings, pauses inside core segments last a median 8 frames (30 Hz; p90 22 frames), and the median gap between neighbouring pauses is 32 frames, with 14% of gaps at 10 frames or less, so clusters of close pauses are real. After a pause is cut, a bridging action at the seam uses the mean velocity before and after, with cubic Hermite interpolation. Injected frames serve only as action targets, never as sample starts, because they have no images.

Public practice differs. openpi, on DROID, neither splices nor interpolates; it only forbids sample starts inside idle segments and about half a chunk before them, and chunks still read the raw future. That is simpler and risk-free, but pauses remain inside chunks, so it is a good control experiment for L6.

ParameterValueReason
Shortest pause to cut≥ 6 frames (0.2 s)Shorter pauses are hand tremor; cutting them breaks continuity
Velocity window k5 framesMatches the ±5-frame idle rule; with a 32-frame median gap, both sides fit
MergingCuts less than 10 frames apart merge, with one injectionOtherwise the velocity window reaches into another pause
Injected velocityMean of before and after, cubic Hermite interpolationKeeps velocity continuous at the seam
Contact windowNever cut, never merged acrossSmall pauses around contact can be skill boundaries

12 / L7 normalize speed to the benchmark operator

For each operator, take the median duration of core segments (10 cm → grasp → 5 cm). The target site's benchmark operator is the fastest, yet the same person is 45% slower in another site's SOP, so the target speed is the operator's target-site core segments rather than anything tied to a name. Scene difficulty matters too; one operator takes 3.23 s in the SOP but 4.45 s on hard cases. Ratios are therefore compared within a scene, and only free-motion segments are aligned across scenes.

Public work agrees. DemoSpeedup accelerates only free motion, and uniform 2× hurts success; ESPADA accelerates only casual segments; SAIL gains a lot on real robots except in contact-rich tasks; TempoVLA conditions on speed as an input.

  • Split: the core segment divides into free-motion segments (10 cm → start of contact window, end of contact window → 5 cm) and the contact window.
  • Ratio: r = segment duration / the benchmark operator's median for the same segment.
  • Speed up only: free segments with r > 1 are subsampled by r, images and actions together, so no images need to be synthesized; r ≤ 1 is left alone; the contact window stays at 1×.
  • Order: L6 pause removal before L7, otherwise pauses inflate r.
OperatorSceneGraspsCore totalRelative to benchmark
A (benchmark)Target site7663.03 s1.00×
ASOP from another site1094.40 s1.45×
BSOP from another site7453.23 s1.07×
BStacking scene12304.03 s1.33×
BHard cases5284.45 s1.47×
BFailure recovery5495.43 s1.79×
CDual-arm recordings34653.73 s1.23×
DDual-arm recordings29114.37 s1.44×
EDual-arm recordings2965.90 s1.95×

13 / Results of the full pipeline on all data

L0–L7 ran on all pick-and-place data, 9,328 recordings, 16,688 active arms and about 20.49 million frames, in roughly two minutes on a single process. It only writes labels and leaves training files untouched. Recovery after failure is always trained (4,410 segments; previously 46% had been zeroed by neighbouring failure segments). L6 never cuts the contact window, so all 1.248 million contact frames are untouched. Core frames shrink by about 31% after L6 and L7, but only 10% at the target site, which was already fast.

SourceCore framesAfter L6After L7Total core reduction
All31363482823916217156931%
Small parts in a new scene (multi-source)105474391843772640831%
Target site68306672746118910%
SOP from another site85981826626917320%
Hard cases68364610364299737%
Stacking scene18697117454112029436%
Dual-arm recordings14985681361049103914331%
Failure recovery17341515891711236535%

14 / Takeaways and what is still missing

  • Screen before use: signal, gripper and camera problems are caught once in L0 and checked automatically on every sync. Problems where the stream is alive but its identity is wrong are found too late by eye.
  • Multi-source data needs an admission rule: compare poses with the target scene (L1.3) instead of accepting or rejecting whole sources; a nominally identical task may qualify only 2% of the time.
  • Borrow only the core: other sources contribute only the 10 cm → grasp → 5 cm segment, with the contact window weighted, never cut and never sped up.
  • 3D trajectory comparison is the core tool for judging data. Joints and pose at the grasp tell whether it is the same move, extrema and backtracking along the path tell how clean the demonstration is, and the comparison can expose root causes such as collection setup.
  • Normalization comes before mixing: recompute normalization statistics for every data mix; in time, L6 removes pauses and L7 aligns speed; in space, smoothing and alignment start with the core segment, anchored at the grasp point and the two heights.
  • Still missing: cycle-level quality scores and deduplication; train/validation splits by operator or session (random splits by recording put similar cycles on both sides); balanced sampling by operator or site; and a layer that handles multiple extrema and overshoot in trajectories.
← All articles
Bold = me; * = corresponding author; † = corresponding author (others)加粗为本人;* 表示本人为通讯作者;† 表示其他通讯作者

2026

Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models
Yiran Ling, Qing Lian, Jinghang Li, Qing Jiang, Tianming Zhang, Xiaoke Jiang, Chuanxiu Liu, Jie Liu†, Lei Zhang†
European Conference on Computer Vision (ECCV) 2026CCF-A
VLARobot arXiv Project BibTeX
@article{guide,think,act2026,
  title={Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models},
  author={Yiran Ling, Qing Lian, Jinghang Li, Qing Jiang, Tianming Zhang, Xiaoke Jiang, Chuanxiu Liu, Jie Liu, Lei Zhang},
  journal={European Conference on Computer Vision (ECCV) 2026},
  year={2026}
}
SpatialPoint: Spatial-aware Point Prediction for Embodied Localization
Qiming Zhu, Zhirui Fang, Tianming Zhang, Chuanxiu Liu, Xiaoke Jiang*, Lei Zhang
arXiv 2026
VLARobot arXiv BibTeX
@article{spatialpoint2026,
  title={SpatialPoint: Spatial-aware Point Prediction for Embodied Localization},
  author={Qiming Zhu, Zhirui Fang, Tianming Zhang, Chuanxiu Liu, Xiaoke Jiang, Lei Zhang},
  journal={arXiv 2026},
  year={2026}
}

2025

3DRot: Rediscovering the Missing Primitive for RGB-Based 3D Augmentation
Shitian Yang, Deyu Li, Xiaoke Jiang*, Lei Zhang
arXiv 2025
6D Pose3D Detection arXiv BibTeX
@article{3drot2025,
  title={3DRot: Rediscovering the Missing Primitive for RGB-Based 3D Augmentation},
  author={Shitian Yang, Deyu Li, Xiaoke Jiang, Lei Zhang},
  journal={arXiv 2025},
  year={2025}
}
Coca-Splat: Collaborative Optimization for Camera Parameters and 3D Gaussians
Jiamin Wu, Hongyang Li, Xiaoke Jiang*, Yuan Yao, Lei Zhang
arXiv 2025
3DGSObject Reconstruction arXiv BibTeX
@article{coca-splat2025,
  title={Coca-Splat: Collaborative Optimization for Camera Parameters and 3D Gaussians},
  author={Jiamin Wu, Hongyang Li, Xiaoke Jiang, Yuan Yao, Lei Zhang},
  journal={arXiv 2025},
  year={2025}
}
UniGS: Modeling Unitary 3D Gaussians for Novel View Synthesis from Sparse-view Images
Jiamin Wu, Kenkun Liu, Xiaoke Jiang*, Yuan Yao, Lei Zhang
International Conference on Computer Vision (ICCV) 2025, pp. 26241-26251CCF-A
3DGSObject Reconstruction arXiv Code Project BibTeX
@article{unigs2025,
  title={UniGS: Modeling Unitary 3D Gaussians for Novel View Synthesis from Sparse-view Images},
  author={Jiamin Wu, Kenkun Liu, Xiaoke Jiang, Yuan Yao, Lei Zhang},
  journal={International Conference on Computer Vision (ICCV) 2025, pp. 26241-26251},
  year={2025}
}
LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians
Jiamin Wu, Kenkun Liu, Han Gao, Xiaoke Jiang*, Yuan Yao, Lei Zhang
Conference on Computer Vision and Pattern Recognition (CVPR) 2025CCF-A
3DGSObject Reconstruction PDF Code Project BibTeX
@article{leangaussian2025,
  title={LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians},
  author={Jiamin Wu, Kenkun Liu, Han Gao, Xiaoke Jiang, Yuan Yao, Lei Zhang},
  journal={Conference on Computer Vision and Pattern Recognition (CVPR) 2025},
  year={2025}
}
HandOS: 3D Hand Reconstruction in One Stage
Xingyu Chen, Zhuheng Song, Xiaoke Jiang, Yaoqing Hu, Junzhi Yu, Lei Zhang
Conference on Computer Vision and Pattern Recognition (CVPR) 2025CCF-A
Hand ReconstructionKeypoint Detection PDF Project BibTeX
@article{handos2025,
  title={HandOS: 3D Hand Reconstruction in One Stage},
  author={Xingyu Chen, Zhuheng Song, Xiaoke Jiang, Yaoqing Hu, Junzhi Yu, Lei Zhang},
  journal={Conference on Computer Vision and Pattern Recognition (CVPR) 2025},
  year={2025}
}
HumanMM: Global Human Motion Recovery from Multi-shot Videos
Yuhong Zhang, Guanlin Wu, Ling-Hao Chen, Zhuokai Zhao, Jing Lin, Xiaoke Jiang, Jiamin Wu, Zhuoheng Li, Hao Frank Yang, Haoqian Wang, Lei Zhang
Conference on Computer Vision and Pattern Recognition (CVPR) 2025, pp. 1973-1983CCF-A
Human MotionObject Reconstruction arXiv Code Project BibTeX
@article{humanmm2025,
  title={HumanMM: Global Human Motion Recovery from Multi-shot Videos},
  author={Yuhong Zhang, Guanlin Wu, Ling-Hao Chen, Zhuokai Zhao, Jing Lin, Xiaoke Jiang, Jiamin Wu, Zhuoheng Li, Hao Frank Yang, Haoqian Wang, Lei Zhang},
  journal={Conference on Computer Vision and Pattern Recognition (CVPR) 2025, pp. 1973-1983},
  year={2025}
}
SeqPose: An End-to-End Framework to Unify Single-frame and Video-based RGB Category-Level Pose Estimation
Yuzhu Ji, Mingshan Sun, Jianyang Shi, Xiaoke Jiang, Yiqun Zhang, Haijun Zhang
International Joint Conference on Artificial Intelligence (IJCAI) 2025CCF-A
6D Pose PDF Project BibTeX
@article{seqpose2025,
  title={SeqPose: An End-to-End Framework to Unify Single-frame and Video-based RGB Category-Level Pose Estimation},
  author={Yuzhu Ji, Mingshan Sun, Jianyang Shi, Xiaoke Jiang, Yiqun Zhang, Haijun Zhang},
  journal={International Joint Conference on Artificial Intelligence (IJCAI) 2025},
  year={2025}
}
Geo6D: Geometric-Constraints-Guided Direct Object 6D Pose Estimation Network
Jianqiu Chen, Mingshan Sun, Ye Zheng, Tianpeng Bao, Zhenyu He†, Donghai Li, Guoqiang Jin, Zhao Rui, Liwei Wu, Xiaoke Jiang
IEEE Transactions on Multimedia (TMM) 2025
6D Pose PDF Code BibTeX
@article{geo6d2025,
  title={Geo6D: Geometric-Constraints-Guided Direct Object 6D Pose Estimation Network},
  author={Jianqiu Chen, Mingshan Sun, Ye Zheng, Tianpeng Bao, Zhenyu He, Donghai Li, Guoqiang Jin, Zhao Rui, Liwei Wu, Xiaoke Jiang},
  journal={IEEE Transactions on Multimedia (TMM) 2025},
  year={2025}
}
Uni6Dv2: Noise Elimination for 6D Pose Estimation
Mingshan Sun, Ye Zheng, Tianpeng Bao, Jianqiu Chen, Guoqiang Jin, Liwei Wu, Rui Zhao, Xiaoke Jiang
International Conference on Artificial Intelligence and Statistics (AISTATS) 2025
6D PoseRobot arXiv BibTeX
@article{uni6dv22025,
  title={Uni6Dv2: Noise Elimination for 6D Pose Estimation},
  author={Mingshan Sun, Ye Zheng, Tianpeng Bao, Jianqiu Chen, Guoqiang Jin, Liwei Wu, Rui Zhao, Xiaoke Jiang},
  journal={International Conference on Artificial Intelligence and Statistics (AISTATS) 2025},
  year={2025}
}

2023

Defect classification for specular surfaces based on deflectometry and multi-modal fusion network
Jingtian Guan, Jingjing Fei, Wei Li, Xiaoke Jiang, Liwei Wu, Yakun Liu, Juntong Xi
Optics and Lasers in Engineering 163 (2023)
Defect Detection PDF Project BibTeX
@article{defectclassificationforspecularsurfacesbasedondeflectometryandmulti-modalfusionnetwork2023,
  title={Defect classification for specular surfaces based on deflectometry and multi-modal fusion network},
  author={Jingtian Guan, Jingjing Fei, Wei Li, Xiaoke Jiang, Liwei Wu, Yakun Liu, Juntong Xi},
  journal={Optics and Lasers in Engineering 163 (2023)},
  year={2023}
}

2022

Uni6D: A Unified CNN Framework without Projection Breakdown for 6D Pose Estimation
Xiaoke Jiang, Donghai Li, Hao Chen, Ye Zheng, Rui Zhao, Liwei Wu
Conference on Computer Vision and Pattern Recognition (CVPR) 2022 (Oral)ORALCCF-A
6D PoseRobot arXiv BibTeX
@article{uni6d2022,
  title={Uni6D: A Unified CNN Framework without Projection Breakdown for 6D Pose Estimation},
  author={Xiaoke Jiang, Donghai Li, Hao Chen, Ye Zheng, Rui Zhao, Liwei Wu},
  journal={Conference on Computer Vision and Pattern Recognition (CVPR) 2022 (Oral)},
  year={2022}
}

2021

SSN3D: Self-Separated Network to Align Parts for 3D Convolution in Video Person Re-Identification
Xiaoke Jiang, Yu Qiao, Junjie Yan, Qichen Li, Wanrong Zheng, Dapeng Chen
AAAI Conference on Artificial Intelligence (AAAI) 2021CCF-A
Person ReID3D Detection PDF BibTeX
@article{ssn3d2021,
  title={SSN3D: Self-Separated Network to Align Parts for 3D Convolution in Video Person Re-Identification},
  author={Xiaoke Jiang, Yu Qiao, Junjie Yan, Qichen Li, Wanrong Zheng, Dapeng Chen},
  journal={AAAI Conference on Artificial Intelligence (AAAI) 2021},
  year={2021}
}

2019

ATSRA: An Accelerated Transmission Strategy Based on Request Aggregation in NDN
Shanshan Shi, Jun Li, Haibo Wu, Yongmao Ren, Xiaoke Jiang
IEEE International Conference on Computer Communications (INFOCOM) 2019 (poster)CCF-A
NDN/Networking BibTeX
@article{atsra2019,
  title={ATSRA: An Accelerated Transmission Strategy Based on Request Aggregation in NDN},
  author={Shanshan Shi, Jun Li, Haibo Wu, Yongmao Ren, Xiaoke Jiang},
  journal={IEEE International Conference on Computer Communications (INFOCOM) 2019 (poster)},
  year={2019}
}

2018

Smart Streaming of Panoramic Video
Hongwei Ma, Xiaoke Jiang, Rui Ma, Zhiyou Ma, Yizhen Cai, Dah Ming Chiu
ACM SIGCOMM 2018 VR WorkshopCCF-A
VR Streaming BibTeX
@article{smartstreamingofpanoramicvideo2018,
  title={Smart Streaming of Panoramic Video},
  author={Hongwei Ma, Xiaoke Jiang, Rui Ma, Zhiyou Ma, Yizhen Cai, Dah Ming Chiu},
  journal={ACM SIGCOMM 2018 VR Workshop},
  year={2018}
}
Cache and delivery of VR video over named data networking
Yi Zhang, Xiaoke Jiang, Yi Wang, Kai Lei
IEEE INFOCOM Workshops 2018CCF-A
VR StreamingNDN/Networking BibTeX
@article{cacheanddeliveryofvrvideoovernameddatanetworking2018,
  title={Cache and delivery of VR video over named data networking},
  author={Yi Zhang, Xiaoke Jiang, Yi Wang, Kai Lei},
  journal={IEEE INFOCOM Workshops 2018},
  year={2018}
}

2017

NDNS: A DNS-Like Name Service for NDN
Alexander Afanasyev, Xiaoke Jiang, Yingdi Yu, Jiewen Tan, Yumin Xia, Allison Mankin, Lixia Zhang
International Conference on Computer Communications and Networks (ICCCN) 2017
NDN/Networking Code BibTeX
@article{ndns2017,
  title={NDNS: A DNS-Like Name Service for NDN},
  author={Alexander Afanasyev, Xiaoke Jiang, Yingdi Yu, Jiewen Tan, Yumin Xia, Allison Mankin, Lixia Zhang},
  journal={International Conference on Computer Communications and Networks (ICCCN) 2017},
  year={2017}
}

2016

IS: Interest Set to Enhance Flow Transmission in Named-Data Networking
Xiaoke Jiang, Jun Bi
China Communications (IEEE), Vol.13, 2016
NDN/Networking BibTeX
@article{is2016,
  title={IS: Interest Set to Enhance Flow Transmission in Named-Data Networking},
  author={Xiaoke Jiang, Jun Bi},
  journal={China Communications (IEEE), Vol.13, 2016},
  year={2016}
}

2015

A Survey on Information-Centric Networking: Rationales, Designs and Debates
Xiaoke Jiang, Jun Bi, Guoshun Nan, Zhaogeng Li
China Communications (IEEE), Vol.12, No.7, 2015
NDN/Networking BibTeX
@article{asurveyoninformation-centricnetworking2015,
  title={A Survey on Information-Centric Networking: Rationales, Designs and Debates},
  author={Xiaoke Jiang, Jun Bi, Guoshun Nan, Zhaogeng Li},
  journal={China Communications (IEEE), Vol.12, No.7, 2015},
  year={2015}
}

2014

nCDN: CDN Enhanced with NDN
Xiaoke Jiang, Jun Bi
IEEE INFOCOM 2014, NOM WorkshopCCF-A
NDN/Networking BibTeX
@article{ncdn2014,
  title={nCDN: CDN Enhanced with NDN},
  author={Xiaoke Jiang, Jun Bi},
  journal={IEEE INFOCOM 2014, NOM Workshop},
  year={2014}
}
What Benefits Does NDN Have in Supporting Mobility
Xiaoke Jiang, Jun Bi, You Wang
IEEE Symposium on Computers and Communications (ISCC) 2014
NDN/Networking BibTeX
@article{whatbenefitsdoesndnhaveinsupportingmobility2014,
  title={What Benefits Does NDN Have in Supporting Mobility},
  author={Xiaoke Jiang, Jun Bi, You Wang},
  journal={IEEE Symposium on Computers and Communications (ISCC) 2014},
  year={2014}
}
MCBS: Matrix Computation Based Simulator of NDN
Xiaoke Jiang, Jun Bi, You Wang
Journal of Computers, Vol.9, No.9, 2014
NDN/Networking BibTeX
@article{mcbs2014,
  title={MCBS: Matrix Computation Based Simulator of NDN},
  author={Xiaoke Jiang, Jun Bi, You Wang},
  journal={Journal of Computers, Vol.9, No.9, 2014},
  year={2014}
}

2013

Interest Set Mechanism to Improve the Transport of Named Data Networking
Xiaoke Jiang, Jun Bi
ACM SIGCOMM Conference on Data Communication 2013 (poster)CCF-A
NDN/Networking BibTeX
@article{interestsetmechanismtoimprovethetransportofnameddatanetworking2013,
  title={Interest Set Mechanism to Improve the Transport of Named Data Networking},
  author={Xiaoke Jiang, Jun Bi},
  journal={ACM SIGCOMM Conference on Data Communication 2013 (poster)},
  year={2013}
}
An Adaptive Probabilistic Marking Scheme for Fast and Secure Traceback
Hongcheng Tian, Jun Bi, Xiaoke Jiang
Networking Science (Springer), Vol.2, No.1-2, 2013
NDN/Networking BibTeX
@article{anadaptiveprobabilisticmarkingschemeforfastandsecuretraceback2013,
  title={An Adaptive Probabilistic Marking Scheme for Fast and Secure Traceback},
  author={Hongcheng Tian, Jun Bi, Xiaoke Jiang},
  journal={Networking Science (Springer), Vol.2, No.1-2, 2013},
  year={2013}
}
MSDN: A Mechanism for Scalable Intradomain Control Plane in SDN
MSDN: A Mechanism for Scalable Intradomain Control Plane in SDN
Pingping Lin, Jun Bi, Hongyu Hu, Xiaoke Jiang
Journal of Chinese Computer Systems, Vol.34, No.9, 2013
NDN/Networking BibTeX
@article{msdn2013,
  title={MSDN: A Mechanism for Scalable Intradomain Control Plane in SDN},
  author={Pingping Lin, Jun Bi, Hongyu Hu, Xiaoke Jiang},
  journal={Journal of Chinese Computer Systems, Vol.34, No.9, 2013},
  year={2013}
}

2012

A Content Provider Mobility Solution of Named Data Networking
Xiaoke Jiang, Jun Bi, You Wang, Pingping Lin, Zhaogeng Li
IEEE International Conference on Network Protocols (ICNP) 2012
NDN/Networking BibTeX
@article{acontentprovidermobilitysolutionofnameddatanetworking2012,
  title={A Content Provider Mobility Solution of Named Data Networking},
  author={Xiaoke Jiang, Jun Bi, You Wang, Pingping Lin, Zhaogeng Li},
  journal={IEEE International Conference on Network Protocols (ICNP) 2012},
  year={2012}
}
An Easy Matrix Computation based Simulator of NDN
Xiaoke Jiang, Jun Bi, You Wang, Pingping Lin, Zhaogeng Li
IEEE International Conference on Networking and Distributed Computing (ICNDC) 2012
NDN/Networking BibTeX
@article{aneasymatrixcomputationbasedsimulatorofndn2012,
  title={An Easy Matrix Computation based Simulator of NDN},
  author={Xiaoke Jiang, Jun Bi, You Wang, Pingping Lin, Zhaogeng Li},
  journal={IEEE International Conference on Networking and Distributed Computing (ICNDC) 2012},
  year={2012}
}
Mobility Support in the Internet Using Identifiers
You Wang, Jun Bi, Xiaoke Jiang
ACM International Conference on Future Internet Technologies (CFI) 2012
NDN/Networking BibTeX
@article{mobilitysupportintheinternetusingidentifiers2012,
  title={Mobility Support in the Internet Using Identifiers},
  author={You Wang, Jun Bi, Xiaoke Jiang},
  journal={ACM International Conference on Future Internet Technologies (CFI) 2012},
  year={2012}
}
The Compression of Pending Interest Table with Bloom Filter in Content Centric Network
Zhaogeng Li, Jun Bi, Sen Wang, Xiaoke Jiang
ACM International Conference on Future Internet Technologies (CFI) 2012
NDN/Networking BibTeX
@article{thecompressionofpendinginteresttablewithbloomfilterincontentcentricnetwork2012,
  title={The Compression of Pending Interest Table with Bloom Filter in Content Centric Network},
  author={Zhaogeng Li, Jun Bi, Sen Wang, Xiaoke Jiang},
  journal={ACM International Conference on Future Internet Technologies (CFI) 2012},
  year={2012}
}

2011

IPv6 Evolution, Stability and Deployment
Xiaoke Jiang, Jun Bi, Yangyang Wang, Zhijie He, Wei Zhang, Hongchen Tian
IEEE International Conference on Network Protocols (ICNP) 2011
NDN/Networking BibTeX
@article{ipv6evolution,stabilityanddeployment2011,
  title={IPv6 Evolution, Stability and Deployment},
  author={Xiaoke Jiang, Jun Bi, Yangyang Wang, Zhijie He, Wei Zhang, Hongchen Tian},
  journal={IEEE International Conference on Network Protocols (ICNP) 2011},
  year={2011}
}
Fast and Secure Probabilistic Marking Technology for IP Traceback
Fast and Secure Probabilistic Marking Technology for IP Traceback
Hongcheng Tian, Jun Bi, Xiaoke Jiang, Dekai Wang, Wei Zhang
Journal of Tsinghua University, Vol.50, No.4, 2011
NDN/Networking BibTeX
@article{fastandsecureprobabilisticmarkingtechnologyforiptraceback2011,
  title={Fast and Secure Probabilistic Marking Technology for IP Traceback},
  author={Hongcheng Tian, Jun Bi, Xiaoke Jiang, Dekai Wang, Wei Zhang},
  journal={Journal of Tsinghua University, Vol.50, No.4, 2011},
  year={2011}
}
EasyTrace: Easily-Deployable Light-Weight IP Traceback on an AS-Level Overlay Network
Hongcheng Tian, Jun Bi, Wei Zhang, Xiaoke Jiang
IEEE International Conference on Network Protocols (ICNP) 2011
NDN/Networking BibTeX
@article{easytrace2011,
  title={EasyTrace: Easily-Deployable Light-Weight IP Traceback on an AS-Level Overlay Network},
  author={Hongcheng Tian, Jun Bi, Wei Zhang, Xiaoke Jiang},
  journal={IEEE International Conference on Network Protocols (ICNP) 2011},
  year={2011}
}
A Quick Survey on Selected Approaches for Preparing Programmable Networks
A Quick Survey on Selected Approaches for Preparing Programmable Networks
Pingping Lin, Jun Bi, Hongyu Hu, Tao Feng, Xiaoke Jiang
Asian Internet Engineering Conference (AINTEC) 2011
NDN/Networking BibTeX
@article{aquicksurveyonselectedapproachesforpreparingprogrammablenetworks2011,
  title={A Quick Survey on Selected Approaches for Preparing Programmable Networks},
  author={Pingping Lin, Jun Bi, Hongyu Hu, Tao Feng, Xiaoke Jiang},
  journal={Asian Internet Engineering Conference (AINTEC) 2011},
  year={2011}
}

DINO-XGrasp

Open-Set Robotic Grasping powered by DINO-X
VLARobotOpen-Set DetectionPick&Place

Enabling robots to grasp any object accurately using consumer-level hardware (camera, GPU, robot arm), powered by open-set detection/segmentation, real-time tracking, and 3D ReID.

DINO-XGrasp at IDEA Day
▶
DINO-XGrasp at IDEA Day
Robot's working day
▶
Robot's working day
Voice-command grasping
▶
Voice-command grasping
Open-set grasping for bottles
▶
Open-set grasping for bottles

oVP: Optimized Visual Prompt

Supervised Prompt Tuning for Customized Detection
Open-Set Detection

oVP uses Supervised Prompt Tuning (SPT) to generate optimized visual prompts that customize open-set detection for specialized scenarios. Unlike text prompts that rely on category names, SPT learns domain-specific prompt embeddings from as few as 7 labeled images. Only prompt embeddings are updated during training — no model fine-tuning needed. SPT+Grounding DINO 1.6 consistently outperforms YOLOv8 across 12 industry domains (agriculture, construction, manufacturing, retail, etc.), with advantages intensifying as training data decreases.

These are small utilities I built to scratch my own itches in research and daily life — fast to use, single-purpose, and open source. I keep them lean so they stay useful instead of accreting features, and I share them in case the same itch bites someone else. 下面是我为科研和日常使用顺手做的几个小工具——专一、轻量、开源。 我刻意保持它们简单,避免功能膨胀,也希望对有同样需求的人有用。

TianYan (天眼) 天眼 TianYan

Multi-format file & dataset previewer for the web 多格式文件 / 数据集的网页预览器
WebFastAPIReactDataset PreviewerMulti-format

Why: Real research projects run on cloud GPUs, and the artifacts I actually want to look at — training datasets, model outputs, depth/mask predictions, HDF5 trajectories — live on those remote machines. SCP-ing files back is slow, Jupyter is heavyweight for "just let me see it", and OS file managers do not preview ML formats anyway. TianYan runs as a small web server next to the data: point a browser at the cloud machine and inspect any project's datasets and outputs through one unified, pluggable viewer.

Main features

  • Tree-style sidebar that recursively browses any local root directory
  • Auto-detects file type: RGB, depth (Uint16), mask, video, JSON, text, tabular (.jsonl/.parquet), pickle, .npy/.npz, .ply, USD, HDF5
  • File-tuple visualization with synced panes (RGB+depth, RGB+mask, RGB+depth+mask, RGB+bbox/mask JSON)
  • HTTP range-aware video streaming so multi-GB clips scrub instantly
  • Pluggable viewer system — add a new format with one backend route plus one React component
  • FastAPI + React/Vite stack; vite HMR for dev, single uvicorn process for prod

为什么做: 真正在跑的研究项目大多在云端 GPU 上,我想随时看一眼的训练数据、模型输出、深度 / 掩码预测、HDF5 轨迹都躺在远端机器上。 把文件 scp 回本地慢,开 Jupyter 只为"瞅一眼"太重,操作系统的文件管理器也不会预览这些 ML 格式。 天眼直接以一个小型 Web 服务的形式跑在数据旁边:浏览器打开云端机器,就能在统一可扩展的查看器里检视任意项目的数据集和输出。

主要功能

  • 树状侧边栏,递归浏览任意本地根目录
  • 自动识别文件类型:RGB、深度 (Uint16)、掩码、视频、JSON、文本、表格 (.jsonl/.parquet)、pickle、.npy/.npz、.ply、USD、HDF5
  • 文件元组联动可视化(RGB+深度、RGB+掩码、RGB+深度+掩码、RGB+bbox/掩码 JSON)
  • 支持 HTTP Range 的视频流,多 GB 视频也能秒级跳播
  • 可插拔的查看器架构——加一个后端路由 + 一个 React 组件即可支持新格式
  • FastAPI + React/Vite 技术栈;开发用 Vite HMR,生产用单进程 uvicorn

MobileEye MobileEye 移动眼

Android-hosted file-browser web server 运行在 Android 上的文件浏览 Web 服务器
AndroidWeb ServerFile BrowserNanoHTTPD no-adno-login

Why: Phones accumulate large datasets — photos, screen recordings, sensor logs — and every "easy" way to get them off the device has a catch: USB cabling is slow and needs a desktop client; mainstream file-transfer apps push you to a cloud upload, require an account login, or are smothered in ads. MobileEye flips the relationship: the phone itself runs a tiny LAN web server, and any laptop on the same Wi-Fi browses, streams, and bulk-uploads through a normal browser — no account, no ads, no cloud round-trip. It also speaks the same API as TianYan, so the same viewer UI works for both local disks and phones.

Main features

  • Runs entirely on the phone — no companion desktop app needed
  • TianYan-shaped API, so the same web frontend browses phone and laptop storage interchangeably
  • Range-aware video streaming directly from the phone's storage
  • Multi-GB chunked uploads that survive flaky Wi-Fi
  • Bundled minimal frontend ready out-of-the-box; works in any browser
  • LAN-only by default; optional password gate

为什么做: 手机里堆着越来越多的数据——照片、屏幕录像、传感器日志。可所谓"方便"的导出方案各有坑:USB 线慢、还要装桌面客户端;主流文件传输 App 要么逼你上传云、要求账号登录,要么塞满广告。 MobileEye 换了个方向:让手机自己跑一个轻量级局域网 Web 服务器,同一 Wi-Fi 下任意笔记本都能用浏览器浏览、播放、批量上传——无需账号、无广告、不绕公网。 它和天眼用同一套 API,所以同一个查看器界面既能浏览本地磁盘,也能浏览手机存储。

主要功能

  • 全部跑在手机上——桌面端无需任何客户端
  • 与天眼同一套 API,同一个网页前端既能浏览电脑磁盘也能浏览手机存储
  • 直接从手机存储进行带 Range 的视频流传输
  • 多 GB 的分片上传,能扛住不稳定 Wi-Fi
  • 自带轻量前端,开箱即用,任何浏览器都能访问
  • 默认只允许局域网访问;可选密码保护

FolderPlayer FolderPlayer 文件夹播放器

Folder-scoped Android music player 以文件夹为单位的 Android 音乐播放器
AndroidMusic PlayerMedia3ExoPlayer no-adno-login

Why: Mainstream Android music players are either ad-walled or push you into an account plus a cloud library, and they all force every track into a tag-based "library" that fights against the way I actually organize music — as folders on disk, by album / mood / workout. FolderPlayer treats a chosen folder (and its subfolders) as the playlist: pick a folder, hit play, done. No ads, no login, no library, no scan.

Main features

  • Folder = playlist: navigate the disk tree and tap to play a subtree
  • ID3 album-art rendering on the now-playing screen
  • Sleep timer for falling asleep to music
  • Shuffle within the current folder
  • Background playback with lock-screen / notification controls
  • Media3 / ExoPlayer engine: gapless playback and modern audio-focus handling

为什么做: 主流 Android 音乐播放器要么被广告淹没,要么把你推向账号 + 云端音乐库,而且统统强制把每首歌导入它们的标签化"音乐库"——这和我组织音乐的方式背道而驰:我就是按文件夹来分(按专辑、按心情、按训练歌单)。 FolderPlayer 让一个文件夹(及其子文件夹)直接就是当前播放列表:选个文件夹,按播放,完事。 没广告、不登录、没"音乐库"、不扫盘。

主要功能

  • 文件夹即播放列表:在磁盘树里点击任意子目录开始播放
  • 播放界面读取 ID3 标签,显示专辑封面
  • 睡眠定时关闭,适合靠音乐入睡
  • 在当前文件夹范围内随机播放
  • 后台播放,支持锁屏 / 通知栏控制
  • 引擎使用 Media3 / ExoPlayer:无缝衔接,符合现代音频焦点规范

Education

Aug 2010 - Jun 2016
Tsinghua University — Ph.D., Computer Science & Technology
Supervisor: Prof. Jun Bi
Jul 2014 - Jul 2015
University of California Los Angeles (UCLA) — Joint PhD Program
Supervisor: Prof. Lixia Zhang
Aug 2011 - Jul 2012
  • Served as tech leader of 2nd Award Team
Sep 2006 - Jun 2010
Tsinghua University — B.S., School of Software
Supervisor: Prof. Jun Bi and Prof. Fei He

Work Experience

2026.08 - now
Manifold AI — Partner & R&D Director
Innovative Technology Exploration & Deployment: Exploring innovative technologies and bringing frontier techniques into real products
R&D Leadership: Building and leading the algorithm team, owning the technical roadmap and productization
2022.10 - 2026.08
IDEA — Senior CV Researcher
Open-set Grasping & VLA: Developing systems to enable robots to grasp any object using consumer-level hardware, powered by open-set detection/segmentation, real-time tracking, and 3D ReID
Open-set 3D Detection: Monocular 3D Detection and RGB-D based 3D Detection
Open-Set Detection & Prompt Tuning: Customize detection targets given labeled dataset and pretrained model
Keypoint Detection: Keypoints detection in the wild
3D Reconstruction & Novel View Synthesis: LeanGaussian (CVPR'25), UniGS (ICCV'25), Coca-Splat (arXiv'25)
Inertial Navigation: IMU + GPS + wheel odometry fusion
Technology Incubation: DINO-XGrasp, oVP and related technologies incubated into the startup Visincept
2018 - 2022
SenseTime — Senior Researcher
Defect Detection: Structured light imaging, defect detection in automobile smart manufacturing
6D Pose Estimation: Uni6D (CVPR'22 Oral), Uni6Dv2 (AISTATS'23)
Person Re-identification: SSN3D (AAAI'21)
AI Automation in Transportation
Outstanding Employee Award: Received the Outstanding Employee Award in 2019
2016 - 2018
Kandao Technology — Senior Engineer
VR Video Streaming & CDN: Smart Streaming of Panoramic Video (SIGCOMM'18 VR Workshop)
Outstanding Employee Award: Received the Outstanding Employee Award in 2017

Academic Service

  • Reviewer: CVPR 2025 / CVPR 2026 / ICCV 2025 / ECCV 2026 / NeurIPS 2024

Skills

Mathematics: Linear Algebra / Lie Algebra / Rotation Group / Probability Theory / Taylor Expansion / Optimization Theory
Programming: Languages: Python, GoLang, C++ / Tools: Linux, git, make, gcc, ffmpeg / Libraries: PyTorch, mmengine, PyTorch Lightning, OpenCV, scikit-learn, matplotlib / Vibe Coding: Claude Code, Cursor

Hobbies

Badminton • Hiking • Programming to solve real-world problems • Reading novels • Philosophy

教育经历

2010.08 - 2016.06
清华大学 — 博士,计算机科学与技术系
导师:毕军教授
2014.07 - 2015.07
加州大学洛杉矶分校 (UCLA) — 联合培养博士
2011.08 - 2012.07
  • 担任二等奖团队技术负责人
2006.09 - 2010.06
清华大学 — 学士,软件学院

工作经历

2026.08 - 至今
流形空间(Manifold AI) — 技术合伙人、研发总监
创新技术探索与落地:负责创新技术的探索以及前沿技术的具体落地
研发管理:组建并带领算法团队,负责技术路线与产品化落地
2022.10 - 2026.08
IDEA 研究院 — 高级计算机视觉研究员
开放集抓取与 VLA:开发使机器人使用消费级硬件(相机、GPU、机械臂)精准抓取任意物体的系统
开放集 3D 检测:单目 3D 检测和 RGB-D 3D 检测
开放集检测与提示调优:基于标注数据集和预训练模型自定义检测目标
关键点检测:野外场景下的关键点检测
3D 重建与新视角合成:LeanGaussian (CVPR'25)、UniGS (ICCV'25)、Coca-Splat (arXiv'25)
惯性导航:IMU + GPS + 轮式里程计融合
技术孵化:DINO-XGrasp、oVP 等技术孵化创业公司视启未来(Visincept)
2018 - 2022
商汤科技 — 资深视觉研究员
缺陷检测:结构光成像、汽车智能制造缺陷检测
6D 位姿估计:Uni6D (CVPR'22 Oral)、Uni6Dv2 (AISTATS'23)
行人重识别:SSN3D (AAAI'21)
交通系统 AI 自动化
优秀员工奖:2019年获得优秀员工奖
2016 - 2018
看到科技 — 高级工程师
VR 视频流优化与 CDN:全景视频智能传输 (SIGCOMM'18 VR Workshop)
优秀员工奖:2017年获得优秀员工奖

学术服务

  • 审稿人:CVPR 2025 / CVPR 2026 / ICCV 2025 / ECCV 2026 / NeurIPS 2024

技能

数学: 线性代数 / 李代数/旋转群 / 概率论 / 泰勒展开 / 优化理论
编程: 语言:Python, GoLang, C++ / 工具:Linux, git, make, gcc, ffmpeg / 库:PyTorch, mmengine, PyTorch Lightning, OpenCV, scikit-learn, matplotlib / Vibe Coding:Claude Code, Cursor

爱好

羽毛球 • 徒步 • 编程解决实际问题 • 小说阅读 • 哲学思考

Visitors
-- Visitors
-- Page Views