From Instrument-Mounted Demonstrations to In-Vivo Execution: Learning Bimanual Laparoscopic Appendectomy Without Robot-Collected Demonstrations

Most minimally invasive surgery is still performed with hand-held instruments, and the surgeon's instrument kinematics are lost when the operation ends. We present an end-to-end pipeline that captures that motion in the operating room and uses it to train a surgical robot policy: a shaft-mounted instrument-state logger, a data pipeline that measures and removes every channel’s latency, a diffusion policy selected by closed-loop rollouts in simulation, and deployment on live animals. Retrained on 849 in-vivo demonstrations, the policy completed the appendectomy in three of four live rabbits with electrosurgery armed.

Most minimally invasive surgery is still performed by hand. Robotic surgery is growing — about 3.15 million da Vinci procedures worldwide in 2025 — yet in a US national inpatient sample three of four minimally invasive abdominal operations were laparoscopic rather than robotic. On a teleoperated robot the surgeon's instrument motion is recorded as a by-product. In hand-held surgery everything except the endoscope video is lost the moment the operation ends, and that lost signal is exactly what imitation learning needs.

This work captures it in the operating room and turns it into a surgical robot policy. A logger on the shaft of an ordinary laparoscopic instrument records the surgeon's own motion; a processing chain aligns its channels; a diffusion policy is trained on those demonstrations, selected by closed-loop rollouts in simulation, retrained on in-vivo recordings, and run on two FR3 arms in live animals with electrosurgery armed. The robot never collected a demonstration — it served only as the timing reference for calibration and as the executor.

The project page collects the supplementary video, the figures, the simulator rollouts behind Table II and the links to the released data.

A logger on the surgeon's own instrument

A rigid instrument through a trocar has four degrees of freedom, not six: the remote-centre-of-motion constraint removes two. Orientation about the trocar comes from an IMU, and the remaining degree of freedom — insertion depth $d = \ell - r$, the shaft length from the board to the tip minus the board-to-trocar distance — is measured directly by a time-of-flight sensor pointed down the shaft. A linear Hall sensor on the handle reads a magnet on the moving ring and so measures jaw opening.

Nothing enters the patient that was not there before: the device mounts outside, on the shaft, and needs no external camera or tracker. The in-vivo generation streams over BLE at 50–200 Hz from a 1000 mAh cell, which at the measured 50 mA draw lasts about 20 h — longer than an operating-room day.

Against the FR3 tip pose during free motion, the latency-matched channels track shaft roll and depression to 0.42° and 0.17° RMS and insertion depth to 1.36 mm RMS, and tip displacement over a 1.6 s action chunk to 1.38 mm RMS. Video-based instrument tracking, for comparison, reaches 2.6–3.0 mm at best on public benchmarks and 5.4–12.4 mm under occlusion; a markerless fit of the shaft's silhouette edges on our own rig located the axis only to a median 10.7 mm.

Every channel is late, and by a different amount

Each channel reaches the log later than the event it describes. An FR3 drove an instrument carrying the logger through a phantom trocar along swept sines while its tip pose, published at 200 Hz, served as ground truth, and each channel was fitted with

where $L$ is a pure transport delay and $T$ the time constant of a first-order lag. The distinction matters: a channel with $T \approx 0$ can be corrected by a time shift, one with a large $T$ cannot, because part of the motion is gone.

On the ex-vivo logger the attitude channels showed pure transport delays of 24–26 ms, while insertion depth showed $L + T = 349$ ms, 93% of it first-order lag from the firmware's exponential moving average, with gain falling to 0.42 at 1 Hz. The resulting 325 ms depth–attitude skew is 3.5 policy steps at 10 Hz, and no common time shift can remove it. Rewinding each channel by its own latency (−17, +307 and +37 ms for attitude, depth and jaw) cut dynamic depth error from 2.69 to 1.36 mm (−49.5%) and shaft-roll error by 28%. The depression channel, whose 26 ms latency is below what the motion resolves, changed by +2% and serves as the control.

From logs to training pairs

The magnetometer heading of a low-cost IMU drifts in an operating room, so the observation uses only what gravity and the trocar define: the endoscope frame at 320 × 192 and, per arm, $[\cos\rho, \sin\rho, \theta, d/d_{\max}]$. The jaw aperture is deliberately not an input — with it, the policy learns "it is closed now, so keep closing". Actions are 16-step chunks at 10 Hz of per-arm tip-frame deltas relative to the chunk's anchor pose, plus absolute jaw aperture and, in vivo, an absolute electrosurgery channel owned by the cutting arm. Because every label is relative to its anchor, it is invariant to the heading zero and to where the trocar sits, so no correspondence between the demonstration trocar and the robot's trocar is needed.

Of 1,003 raw in-vivo episodes, 135 were dropped for sensor-stream quality or because they were not part of the procedure, and 19 more for a missing phase or an early-ending stream. Both corpora are released at Rosota-Research/ICRA2027-Invivo-main.

The policy, and how its configuration was chosen

The policy is a 1-D U-Net diffusion policy with DDIM sampling over 16-step chunks, with three changes. A single shared DINOv3 ViT-B/16 replaces the two ResNet-18 encoders, with attention pooling and only the last transformer block trainable (7.1 M of 85.7 M parameters). Because the observation window is 0.2 s, the policy cannot infer where in the procedure it is, so the U-Net is conditioned by FiLM on an ordinal encoding of the surgical phase plus two timing channels. And a chunk executor commits three actions per chunk and replans, with the index it starts from absorbing the observation anchor, the arm's measured execution latency and inference.

Rather than select that configuration by offline validation error, 31 candidates were compared by closed-loop rollouts in a simulator whose organs are reconstructed from depth recordings of the ex-vivo scene — 16 episodes of 15 s each, scored on stage counts and on one hard failure, a trocar violation above 10 mm.

16 episodes of 15 s per configuration; lift and full approach count violation-free episodes.

Three results come out of that table. Replacing the ResNet-18 encoders with a frozen DINOv3 cut trocar violations from 88% to 6% (Fisher $p = 7 \times 10^{-6}$) and raised lift from 12% to 69%. Training the same configuration on episodes processed without per-channel latency matching left violations unchanged but eliminated successful lifts (19% vs. 81%, $p = 0.001$) — while its offline action error was better, 0.138 against 0.148. And across the unfreezing sweep, validation MAE improves monotonically from 0.152 to 0.142 while violations rise from 6% to 69%.

That inversion is the practical lesson: in this setting offline error is not merely uninformative about closed-loop success, it is anti-correlated with safety. The least safe encoder, ResNet-18, has a better MAE than the frozen DINOv3.

On the real ex-vivo rig the selected configuration ran for about 18 minutes of policy control across 15 sessions with zero trocar violations and a peak RCM error of 0.23 mm. The vision-based phase predictor, however, saturated at the final phase within a median of 10.6 s, so in vivo the surgeon set the phase by keypress.

In-vivo training and deployment

The policy was retrained from scratch on the 849 in-vivo demonstrations, with the ex-vivo configuration and optimizer-step budget unchanged. Because offline error had already failed as a selector, the checkpoint was accepted through pre-registered event-level gates on unseen episodes: every jaw-closing event detected, energy false-fire 0.019 and missed-fire 0.032 against gates of 0.05 and 0.30, and closed-loop gain 0.86/0.82 against a 0.8 gate.

Ex-vivo pre-training turned out not to matter. Trained from random initialization and from the ex-vivo weights on 25%, 50% and 100% of the corpus, both initializations passed the same gates in all six runs; pre-training helped in offline error only while data were scarce (MAE 1.221 vs. 1.413 at 25% of the corpus, 1.157 vs. 1.173 at 100%). A policy for live tissue has to be trained on live-tissue demonstrations — which is the argument for an instrument that can collect them in the operating room.

On a single day the policy ran on four live rabbits at 5 Hz with electrosurgery armed. Inference on one RTX 4090 took a median 23.5 ms per step (95th percentile 59.7 ms), well inside the 200 ms control period. In each animal the surgeon placed the instruments, checked the hand-over state against the demonstration band, handed control to the policy, supplied the phase by keypress, and could take control back at any moment.

In three of the four animals a single rollout carried the instruments from grasp approach to the energized cut without the surgeon touching them. In the fourth the left arm was handed over in a poorly conditioned inverse-kinematics branch and the joint-limit governor aborted before the policy could act; the procedure was completed by hand. All four animals survived, with no gross abnormality during a week of observation.

On safety: across all policy-controlled time that day the RCM error never exceeded 0.84 mm against a 5 mm abort threshold, and every sample that did exceed the threshold in the day's logs occurred under human teleoperation. In every sample of cautery under policy control — 108 samples at 10 Hz — the cutting jaw was closed, with aperture at most 0.7%.

What this does not claim

Not full autonomy: the surgeon selected the phase, because the vision-based phase predictor agreed with the surgeon on only 24.6% of policy control steps under the deployment endoscope, a domain shift familiar from phase recognition across hospitals. Not a robot-free pipeline either: an FR3 is the ground truth for the latency measurement and the ex-vivo test rig, though it collected no training demonstrations. And the in-vivo study is four demonstration animals and one deployment in each of four others, on a single day.

The closed-loop screening environment used to select the configuration is described separately, in Contact-Stable Deformable Tissue Simulation.