DexViTac

Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation

1State Key Laboratory of Intelligent Manufacturing Equipment and Technology,
Huazhong University of Science and Technology

2Wuhan Huaweike Intelligent Technology Co., Ltd

Abstract

Large-scale, high-quality multimodal demonstrations are essential for robot learning of contact-rich dexterous manipulation. While human-centric data collection systems lower the barrier to scaling, they struggle to capture the tactile information during physical interactions. Motivated by this, we present DexViTac, a portable, human-centric data collection system tailored for contact-rich dexterous manipulation. The system enables the high-fidelity acquisition of first-person vision, high-density tactile sensing, end-effector poses, and hand kinematics within unstructured, in-the-wild environments. Building upon this hardware, we propose a kinematics-grounded tactile representation learning algorithm that effectively resolves semantic ambiguities within tactile signals. Leveraging the efficiency of DexViTac, we construct a multimodal dataset comprising over 2,400 visuo-tactile-kinematic demonstrations. Experiments demonstrate that DexViTac achieves a collection efficiency exceeding 248 demonstrations per hour and remains robust against complex visual occlusions. Real-world deployment confirms that policies trained with the proposed dataset and learning strategy achieve an average success rate exceeding 85% across four challenging tasks. This performance significantly outperforms baseline methods, thereby validating the substantial improvement the system provides for learning contact-rich dexterous manipulation.

System Design

The human demonstration interface features a decoupled design comprising a fisheye camera, motion-capture gloves, high-resolution tactile sensors, and a T265 tracking camera. The robot execution platform utilizes an isomorphic perception architecture wherein the tactile sensors remain strictly consistent with those on the human demonstration interface.

In the Wild Data Collection

We scale up data collection in diverse environments...

1 / 2

Workflow of Data Collection

To prevent frame loss and ensure tight spatiotemporal alignment across different modalities, we employ high-frequency buffering alongside a tactile-anchored synchronization strategy that involves downsampling and nearest-neighbor matching.

Dataset Visualization

The dataset contains 2,400+ visuo-tactile-kinematic demonstrations across 40+ tasks in 10+ real-world environments.

Two-Stage Learning Strategy

Stage 1: A self-supervised framework aligns high-density tactile features with visual anchors utilizing a kinematics-Grounded encoder to learn spatially anchored representations. Stage 2: The pretrained encoders are subsequently integrated into an Action Chunking with Transformers (ACT) policy to map synchronized multimodal observations to multi-step action sequences for contact-rich dexterous manipulation.

Autonomous Policy Rollouts

Deployment experiments are conducted across four representative tasks to evaluate real-world performance: pipetting, whiteboard erasing, pen insertion, and fruit collection.

Pipetting

The dexterous hand precisely regulates pinch force to operate a flexible rubber bulb, testing the high-sensitivity force control performance of the system.

Whiteboard Erasing

This requires the end-effector to maintain constant contact pressure along dynamic trajectories, evaluating force compliance and stability during dynamic interactions.

Marker Insertion

Grasping and inserting small markers, involving stable grasping and in-hand manipulation capabilities.

Fruit Collection

Operating on heterogeneous objects with varying stiffness to verify adaptability to different physical properties.

BibTeX

@misc{chen2026dexvitaccollectinghumanvisuotactilekinematic,
      title={DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation}, 
      author={Xitong Chen and Yifeng Pan and Min Li and Xiaotian Ding},
      year={2026},
      eprint={2603.17851},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.17851}, 
}
The next chapter

Extending DexViTac to bimanual visuo-tactile-kinematic demonstration collection.

DexViTac++

A Bimanual Visuo-Tactile-Kinematic
Demonstration Collection System

Xitong Chen1, Ziqiang Liu1, Min Li1,*, Zhouping Yin1
1Huazhong University of Science and Technology, Wuhan, China

* Corresponding author

From single-hand demonstrations to coordinated bimanual capture.

DexViTac++ wearable bimanual system: two tactile-equipped hand units, an EGO head unit, multi-view images, and head–hand trajectories.
Two instrumented hands. One egocentric head unit. A shared representation of vision, touch, and motion.
21 DoFHand kinematics per hand
240 gComplete hand unit
252 gEGO head unit
3 streamsStereo–inertial tracking

Abstract

Robot-free demonstration collection can broaden access to contact-rich dexterous manipulation, but bimanual acquisition requires contact, finger articulation, and wrist motion to share a consistent spatial and temporal context. We present DexViTac++, a portable system extending DexViTac to two instrumented hands and an egocentric head unit. The interface integrates magnetic–inertial hand tracking, wrist vision, fingertip tactile sensing, and three stereo–inertial tracking streams, with a complete hand unit weighing 240 g. An offline reconstruction pipeline registers independently estimated trajectories through a common calibration target and exports synchronized head-relative and inter-hand poses. The system builds on DexViTac's kinematics-grounded tactile learning formulation and provides a unified collection representation for downstream bimanual learning. This paper presents the collection system and reconstruction workflow; quantitative evaluation of the new hardware and bimanual policies is left to future work.

System Design

DexViTac++ extends DexViTac to two DexSwarm UMI hand units and a DexSwarm EGO head unit. Developed at DexSwarm AI, the integrated hardware replaces the original commercial glove and T265 modules while retaining the vision, touch, and kinematics modalities.

Wrist & Egocentric Vision

Wrist cameras capture local hand–object interactions. The EGO head unit adds first-person workspace observations and its own stereo–inertial pose stream.

Fingertip Touch

Tactile sensors on both hands record finger contact alongside images and articulation, preserving the contact context of bimanual demonstrations.

Articulated Hand Motion

A hand-dorsum electromagnetic transmitter and fingertip electromagnetic–inertial modules support 21-DoF capture per hand, with cable-free operation.

Each 240 g hand unit includes cameras, tactile sensors, and battery, and operates without an external host or power cable.

From DexViTac to DexViTac++

System-level changes from the original single-hand interface.
ComponentDexViTacDexViTac++
Collection interfaceSingle instrumented handTwo hands and an EGO head unit
Finger captureCommercial glove, 19 DoFIn-house magnetic–inertial, 21 DoF/hand
Body trackingT265-based wrist trackingThree stereo–inertial trajectories
ContactFingertip tactile measurementsFingertip tactile measurements
Spatial exportSingle-hand trajectoryRegistered head-relative and inter-hand poses
ConnectivityWired operationCable-free wireless operation

Comparison of Demonstration Interfaces

This comparison summarizes reported collection configurations rather than a controlled benchmark. These interface specifications do not establish superior robot-policy performance or tracking accuracy.

UMI is the gripper baseline. Capability indicators refer to the compared configurations; the qualifications below are part of the comparison. Scroll horizontally on smaller screens to see all columns.
SystemCapture
DoF
Weight
(g)
TrackingTac.Bi.In-the-
wild
No ext.
tracking
Link
UMI1780†VIO×✓✓✓Local
DexCapNRaNRVIO×✓✓✓Wired
DexWildNRaNRMarkerb×✓✓×NR
DexUMI6/12850†VIO✓×NR✓NR
DEXOP12/9/7>1000†FKc✓✓✓✓Wired
DexEXO6>1000†AR××✓✓NR
RealDexUMI6680dNR✓✓✓NRNR
DexViTac19<600†VIO✓×✓✓Wired
DexViTac++21240eVIO✓✓✓✓Wireless

Definitions. Capture DoF excludes wrist pose; slash-separated values denote variants. Tac.: tactile sensing. Bi.: bimanual collection. NR: not established in this comparison. In-the-wild: reported operation outside a fixed robot workspace. VIO: visual–inertial odometry. FK: forward kinematics. AR: smartphone AR tracking.

Tracking and connectivity. No external tracking means no external base station or off-hand camera observing the hand. Wired requires a host or power cable; Wireless denotes cable-free wireless communication; Local denotes onboard operation. Checkmarks and crosses refer to the compared configuration, not all possible variants.

a Keypoint representation; capture DoF unspecified. b Marker tracking with a separate camera. c Requires the AirExo-2 arm exoskeleton.

† Weights reproduced from RealDexUMI; included components may differ. d Reported by RealDexUMI. e Complete hand unit, excluding the 252 g head unit. These weights are not a controlled comparison with matched component inclusion.

Shared-Frame Reconstruction

Three independently tracked devices initially produce trajectories in unrelated local maps. DexViTac++ registers these maps to a common calibration target, then associates the streams in time to preserve the geometry between the head and both hands.

  1. 01

    Estimate local trajectories

    An offline ORB-SLAM3-based stereo–inertial estimator processes each device’s calibrated stereo images and IMU measurements.

  2. 02

    Register a shared frame

    All three devices observe a stationary 3 × 3 AprilGrid of 50 mm tags during calibration. Robust transform aggregation relates each local map to the board frame.

  3. 03

    Associate & export

    Hand poses are resampled at head timestamps, producing synchronized head-relative and inter-hand transforms alongside the head’s own motion.

Acquisition uses Tic-based wireless synchronization; spatial registration is a separate operation. DexSwarm Studio supports playback, inspection, and export, with timestamps, calibration metadata, and original measurements retained for downstream retargeting.

Temporal association uses linear translation interpolation and spherical linear interpolation of rotation. Samples are skipped when either endpoint is more than 50 ms from the reference timestamp; this threshold is an association criterion, not a measured synchronization accuracy.

Learning Foundation

DexViTac++ builds on the original DexViTac formulation: tactile features are conditioned on hand articulation, aligned with visual features through contrastive pretraining, and used by an ACT policy for action-chunk prediction. Updated hardware still requires its own hand-to-robot mapping and wrist calibration.

Original DexViTac learning pipeline, from kinematics-grounded tactile pretraining to ACT policy learning and single-arm robot execution.
Learning pipeline and illustrated executions from the original single-arm DexViTac prototype.

Evidence from the Original DexViTac

The original system collected more than 2,400 demonstrations across over 40 tasks and more than 10 environments. Evaluation on a RealMan RM-65 arm and an Inspire RH56DFTP hand used 30 trials per task.

Original single-arm results; these are not evaluations of DexViTac++.
TaskSuccess rate
Pipetting83.3%
Whiteboard erasing86.7%
Marker insertion80.0%
Fruit collection93.3%
Mean85.8%

Explore the original experiments and demonstrations →

Scope & Future Work

DexViTac++ presents the collection interface and offline reconstruction workflow. Quantitative evaluation of the new hardware and bimanual policies remains future work, including relative-pose drift, synchronization error, demonstration yield, and policy transfer across operators and tasks.

Reconstruction depends on stereo–inertial initialization, calibrated extrinsics, and a stationary calibration target. Independent drift can affect relative geometry; interpolation cannot recover missing measurements or unknown clock offsets. Human-to-robot differences in hand geometry, contact, and camera viewpoint also remain.

Original DexViTac Resources

The following resources belong to the predecessor and support the learning foundation described above.

BibTeX

@misc{chen2026dexvitacplus,
  title={DexViTac++: A Bimanual Visuo-Tactile-Kinematic Demonstration Collection System},
  author={Xitong Chen and Ziqiang Liu and Min Li and Zhouping Yin},
  year={2026},
  note={Manuscript},
  url={https://xitong-c.github.io/DexViTac/dexvitac-plus.html}
}