Under Review

LAG-Fusion

Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusion

Zihao He^*, Hongjie Fang^*, Shirun Tang, Cewu Lu^†, Hao-Shu Fang^† Shanghai Jiao Tong University · Noematrix · Shanghai Innovation Institute ^* Equal contribution ^† Corresponding authors

Hover a region to explore each policy paradigm.

LAG-Fusion preserves both modularity and high-frequency reactivity by decoupling modality-specific inference and composing guidance asynchronously.

[paper] [code coming soon]

Abstract.

Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance. However, these modalities often differ not only in information content but also in sensing rates and inference latencies. Existing multimodal diffusion policies typically rely on synchronous fusion or manually designed multi-frequency architectures, which either slow down high-frequency feedback or limit extensibility to new modality combinations.

We propose LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. LAG-Fusion allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available. To make asynchronous composition consistent, we derive a reference-frame rebasing rule for diffusion variables under relative action representations, enabling delayed guidance to be aligned before fusion. We instantiate LAG-Fusion in contact-rich manipulation by composing a low-frequency vision policy with a high-frequency force policy. Experiments under heterogeneous modality latencies show that LAG-Fusion improves policy responsiveness and task performance over synchronous fusion and specially designed force-aware baselines.

Method.

Overview of LAG-Fusion with a 5 Hz vision policy, a 25 Hz force policy, and a diffusion composition head — **LAG-Fusion** composes modality-specific diffusion policies at their native inference frequencies.

Left · Native-rate policies
A low-frequency vision policy provides long-horizon scene guidance, while a high-frequency force/torque policy provides local contact guidance.

Right · Diffusion composition head
In the diffusion composition head, the overlapping noisy action segment is rebased into the current force frame for force denoising. The corresponding vision guidance is rebased to the same frame, fused with force guidance, and mapped back to update the full-horizon noise prediction.

Reference-frame Rebase.

Delayed guidance is still expressed in the action frame at its original query time. Before fusion, LAG-Fusion transforms every diffusion variable from the old frame T₁ into the current frame T₂.

Relative action

Full affine transform

a^{'} = A_{2 \leftarrow 1} a + b_{2 \leftarrow 1}

Noisy sample

Offset scaled by the diffusion schedule

{a^{'}}_{k} = A_{2 \leftarrow 1} a_{k} + \sqrt{{\bar{α}}_{k}} b_{2 \leftarrow 1}

Predicted noise

Orthogonal linear transform only

{ε^{'}}_{k} = A_{2 \leftarrow 1} ε_{k}

Key property. Because the linear part is orthogonal, rebasing preserves the standard Gaussian diffusion distribution.

Detailed derivation From action rebase to diffusion-variable rebase · Eqs. (3)–(7)

The delayed prediction is expressed in the robot frame at query time t₁, while fresh force guidance is expressed in the current frame at t₂. The derivation below shows how to express clean actions, noisy samples, and predicted noises in the same frame without changing the Gaussian diffusion distribution.

Transform from the old frame to the current frame

Let T₁ and T₂ be the robot poses at the two policy-query times. The relative transform contains a rotation and a translation:

T_{2 \leftarrow 1} = T_{2}^{- 1} T_{1} = (R_{2 \leftarrow 1}, p_{2 \leftarrow 1})

This rigid-body transform is independent of the action-coordinate representation. The next step chooses coordinates that make its action-space rebase affine with an orthogonal linear part.

Write the 10D action rebase as an affine map

Why column-wise 6D matters

Write the relative rotation as R = [r₁, r₂, r₃]. Left-multiplying by the frame rotation transforms every column independently. Stacking the first two columns therefore gives a linear rotation rebase:

\begin{array}{l} R^{'} = R_{2 \leftarrow 1} R \Rightarrow r_{i}^{'} = R_{2 \leftarrow 1} r_{i} \\ R_{6 d}^{'} = blockdiag (R_{2 \leftarrow 1}, R_{2 \leftarrow 1}) R_{6 d} \end{array}

The specific 10D action representation and block-diagonal matrix below rely on this choice. Common Euler-angle or axis-angle coordinates do not obey the same linear rebase rule. More generally, the derivation requires a rotation representation whose frame transform has an orthogonal linear form.

Each relative action is represented by 3D translation, 6D rotation, and one gripper value:

a = {[x^{T}, R_{6 d}^{T}, g]}^{T} \in ℝ^{10}

A_{2 \leftarrow 1} = blockdiag (R_{2 \leftarrow 1}, R_{2 \leftarrow 1}, R_{2 \leftarrow 1}, 1)

b_{2 \leftarrow 1} = {[p_{2 \leftarrow 1}^{T}, 0_{3}^{T}, 0_{3}^{T}, 0]}^{T}

a^{'} = A_{2 \leftarrow 1} a + b_{2 \leftarrow 1}

Because R_2←1 is a rotation matrix, the linear part A_2←1 is orthogonal.

Apply the affine rebase to the clean-action estimate

For diffusion timestep k, the clean action estimated from noisy sample a_k and predicted noise ε_k is:

{\hat{a}}_{0} = \frac{1}{\sqrt{{\bar{α}}_{k}}} a_{k} - \frac{\sqrt{1 - {\bar{α}}_{k}}}{\sqrt{{\bar{α}}_{k}}} ε_{k}

Rebase this clean estimate with the affine action transform and regroup it into the standard diffusion-estimator form:

{\hat{a}}_{0}^{'} = A_{2 \leftarrow 1} {\hat{a}}_{0} + b_{2 \leftarrow 1} = \frac{1}{\sqrt{{\bar{α}}_{k}}} (A_{2 \leftarrow 1} a_{k} + \sqrt{{\bar{α}}_{k}} b_{2 \leftarrow 1}) - \frac{\sqrt{1 - {\bar{α}}_{k}}}{\sqrt{{\bar{α}}_{k}}} A_{2 \leftarrow 1} ε_{k}

Read off the rebase rules

Comparing the regrouped expression with the standard estimator in frame T₂ gives:

\begin{array}{l} {a^{'}}_{k} = A_{2 \leftarrow 1} a_{k} + \sqrt{{\bar{α}}_{k}} b_{2 \leftarrow 1} \\ {ε^{'}}_{k} = A_{2 \leftarrow 1} ε_{k} \end{array}

The translation offset enters the noisy sample with the schedule-dependent factor √ᾱ_k, but it is never added to the noise. Since A_2←1A_2←1^T = I, transforming ε ∼ 𝒩(0, I) by A_2←1 preserves its covariance and therefore the standard Gaussian diffusion distribution.

Latency-Aware Guidance Fusion.

Selectthe overlapping segment of the vision action horizon.
Rebasethe noisy sample and delayed vision guidance into the current force frame.
Fusealigned vision and force guidance using latency-aware weights.
Map backthe fused guidance to update the full diffusion trajectory.