A pose can become stale.
By the time a command is executed, the hammer has moved. Reacting to its current pose leaves the robot chasing a contact that has already passed.
PHYSICAL FUTURES → FEASIBLE CONTACT
PAF-Grasp.
Predict the contact.
Make the grasp.
Physics-Aware Object-Centric SE(3) Forecasting
for Real-Time Proactive Dynamic Grasping
1 The Hong Kong Polytechnic University · 2 Technical University of Munich
THE SHORT VERSION
The future that
actually matters.
A robot does not need to imagine an entire future scene to catch a moving object. It needs to know where the object will be, how it will be oriented, and whether contact is reachable. PAF-Grasp forecasts that compact physical future in SE(3), then selects a feasible moment to act.
15ms
SE(3) forecasting latency
67.3%
Average real-world success
+52.0pp
Gain over the reactive baseline
Reported in the supplied manuscript. Real-world average across catching, conveyor, and turntable tasks.
SEE IT IN MOTION
PAF-Grasp / Introduction01 / THE IDEA
Don't chase the present.
Reach the future.
Motion and system delay make the current pose a moving target. Predicting the contact-time state turns reactive tracking into proactive interception.
FORECAST THE CONTACTConcept animation
ObserveForecastContact
Concept illustration of forecast-guided contact. Robot geometry and motion are schematic.
Meet the hammer at the center.
Forecast the hammer's future position and orientation. Move the robot-mounted target to a reachable future contact state before the hammer arrives. The target meets the striking point at the right moment.
02 / THE METHOD
Forecast first.
Then select.
Object-centric prediction meets robot-specific feasibility. A compact physical state connects perception, dynamics, and action.
Observe & recover
Recover metric geometry and an approximate inertia prior from an anchor RGB-D image. Track the object's recent SE(3) pose history online.
RGB-D → rigid-body stateForecast in SE(3)
Encode motion history, estimate initial velocities, and integrate a Newton–Euler-structured neural ODE to predict future position and rotation.
Continuous-time dynamicsSelect & intercept
Propagate the contact frame along the forecast. Choose a future target that satisfies kinematic, workspace, and timing constraints.
Reachable contact state15 ms forecast10 Hz online replanningOne checkpoint per motion family
03 / IN THE REAL WORLD
Different motions.
One decision principle.
Catch in flight. Intercept in motion. Grasp at the right phase.
See contact-time forecasting at work.
REAL-WORLD DEMOS
REAL WORLD
Meet it in flight.
Forecast a tossed ball's contact point before the opportunity passes.
64%
grasping successvs. 0% · π₀.₅ + RTC
REAL WORLD
Intercept the motion.
Meet a sliding object at a reachable future position, rather than trailing behind.
78%
grasping successvs. 34% · π₀.₅ + RTC
REAL WORLD
Catch the right phase.
Predict position and orientation to meet a rotating cup handle at contact time.
60%
grasping successvs. 12% · π₀.₅ + RTC
Representative footage from the project introduction. Success rates come from 50 separate evaluation trials per method, per task.
THE PHYSICAL SETUP
From predicted state
to physical contact.
One experimental platform, three motion families. RGB-D sensing keeps the object's SE(3) history up to date for closed-loop interception.
AgileX Piper armRealSense D435iSE(3) pose tracking
04 / THE EVIDENCE
A physical future.
A measurable difference.
Grasping success rate
Higher is better ↑PAF-Graspπ₀.₅ + RTC
50 evaluation trials per method, per task. Same fixed RTC hyperparameters across tasks.
View task success rates +
| Setting / Method | Catch | Conveyor | Turntable | Ramp | Average |
|---|---|---|---|---|---|
| Real world · π₀.₅ + RTC | 0 | 34 | 12 | — | 15.3 |
| Real world · PAF-Grasp | 64 | 78 | 60 | — | 67.3 |
| Simulation · π₀ | 6 | 50 | 16 | 26 | 24.5 |
| Simulation · PAF-Grasp | 56 | 78 | 62 | 52 | 62.0 |
05 / WHAT SHOULD WE PREDICT?
Less scene.
More physics.
For interception, a visually plausible future is not enough. The useful prediction is a metrically accurate contact state, available in time to act.
This evaluates representation suitability, not an input-matched comparison: video models use single-image prompting; PAF-Grasp uses tracked SE(3) histories and a recovered mesh. See Section 4.2.
| Method | Mask mIoU ↑ | Time (s) ↓ |
|---|---|---|
| CogVideoX | 0.0036 | 180 |
| Wan 2.2 | 0.0225 | 41 |
| Veo 3.1 | 0.0334 | 71 |
| Kling 3.0 | 0.0466 | 30 |
| PAF-Grasp | 0.2762 | 0.015 |
Abstract +
Proactive dynamic grasping requires robots to predict the future state at which physical contact will occur, rather than reacting to the currently observed object pose. But what future must the robot predict, and at what level of abstraction? We argue that contact-relevant rigid-body state in SE(3) is a compact task-sufficient abstraction for this problem: it directly determines where and when the robot should establish contact, without requiring visually complete future scenes.
This insight leads to a lightweight object-centric forecast-then-select framework that predicts future rigid-body trajectories from short pose histories and uses a reachability-aware interception selector to choose feasible contact states under kinematic and timing constraints. Trained with randomized physics simulation over primitive shapes, the forecaster transfers to real objects once their metric geometry and SE(3) state histories are recovered.
Experiments across simulation and real-world dynamic grasping tasks show that our framework improves the average success rate from 24.5% to 62.0% in simulation, and from 15.3% to 67.3% in real-world catching, conveyor, and turntable interception. The SE(3) forecaster runs in 15 ms, orders of magnitude faster than dense visual future prediction, enabling real-time contact decision making. Our results suggest that effective proactive grasping does not require imagining the entire future scene. Instead, predicting the compact physical future that determines contact provides an effective and practical abstraction for real-time dynamic grasping.
Scope & limitations +
The framework focuses on short-horizon, continuous motion of a single rigid body. Deformable objects, coupled interactions, collisions, and contact transitions require richer dynamics representations. Performance also depends on velocity estimation, RGB-D pose tracking, approximate inertia, and the robot's reachable workspace.
Classical predictors remain competitive when motion structure is known: they match the learned forecaster on conveyor success and perform better on ramp success in the matched noisy-input study. PAF-Grasp uses a common physics-structured architecture, with one checkpoint trained for each motion family.
Citation +
Manuscript · 2026
@misc{zhou2026pafgrasp,
title={Physics-Aware Object-Centric SE(3) Forecasting
for Real-Time Proactive Dynamic Grasping},
author={Zhou, Yirui and Xu, Hongli and Zheng, Ying
and Wang, Yi and Chau, Lap-Pui},
year={2026},
note={Manuscript}
}