Teaching robots to see in 3D

Introduction

Adapted from the master's thesis Image-aware VLA Adapter by Khalifa Fumo (Uppsala University). A walkthrough for engineers who know deep learning but are new to robot policies.

Imagine a robot arm that you can simply talk to. You say “pick up the red mug and place it on the plate,” and a single neural network turns those words, together with what the camera sees, directly into motor commands. No hand-engineered perception stage, no separate planner, no specialised controller for each task. That is the promise of Vision-Language-Action (VLA) models, and it is one of the most active frontiers in robot learning today. This article unpacks what VLAs are, the stubborn problems they still face, and a concrete architecture that tackles one of the most important of those problems: a robot's shaky sense of geometry.

What a Vision-Language-Action model actually is

VLA models aim to unify perception, language understanding, and action generation inside a single multimodal framework. By grounding natural-language instructions and visual observations (from a camera) directly into motor commands, they replace brittle, task-specific pipelines with generalist policies that can operate across many tasks and environments. The momentum behind them comes from large Vision-Language Models (VLMs): models like CLIP, Flamingo, and Prismatic, which have been trained on internet-scale image–text pairs, and have already learned rich representations of objects and spatial relationships. The core bet of the VLA paradigm is that this semantic world knowledge can be repurposed for embodied control.

The catalyst for the field was OpenVLA (Kim et al., 2024), which showed that a large pretrained VLM — a 7-billion-parameter Llama 2 backbone with a dual DINOv2 + SigLIP vision encoder — could be fine-tuned on roughly 970,000 real robot demonstrations and beat much larger closed-source systems such as the 55B RT-2-X by 16.5 percentage points across 29 real-world manipulation tasks. Impressive, but expensive: those 7B parameters demand serious GPU memory, full-model fine-tuning for each new task can take days on a multi-GPU cluster, and real-time control on consumer hardware is a stretch. For a research process that lives on rapid iteration, that is a painful bottleneck.

The efficiency wall — and how VLA-Adapter climbs it

The VLA-Adapter framework (Wang et al., 2025) rethinks adaptation entirely. Instead of fine-tuning the whole backbone, it freezes the pretrained vision-language model and trains only a lightweight policy module — the Bridge Attention Adapter — that maps the frozen model's multimodal representations into robot action space. The adapter holds only about 0.5B trainable parameters (roughly 7% of the model) yet stays competitive with full fine-tuning. The payoff is dramatic: a capable VLA trains in about 8 hours on a single RTX 4090 (a 20× reduction), and the frozen backbone enables inference exceeding 200 Hz — fast enough for reactive control.

Mechanically, Bridge Attention is a stack of cross-attention blocks in which learnable policy queries attend to the frozen VLM's outputs. In the baseline, each block draws on three sources: self-attention across the predicted action steps, an adapter + proprioception stream carrying task context and the robot's joint state, and a gated vision stream of visual patch tokens. A learnable scalar gate starts the vision stream closed and lets the model open it as training stabilizes. In the end, we have learned a projection that translates from the language of visual semantics to the language of motor control, which in this case are 8-step action chunks in a 7-DoF space (a 3D position delta, a 3D axis-angle rotation, and a gripper value).

This is the machine we set out to improve, and its blind spot is right there in the diagram. Every source feeding the action head is derived from unstructured 2D patch embeddings. They capture texture, colour, and semantics beautifully, but they encode geometry only implicitly. The model has to infer distance, orientation, and contact from flat appearance alone, and that is exactly where manipulation gets hard.

The challenge depth is meant to solve

Here is the catch that VLA-Adapter inherits from OpenVLA: it relies almost entirely on unstructured 2D patch embeddings. Those embeddings capture texture, colour, and semantics beautifully, but they encode geometry only implicitly. Transformers process an image as a flat sequence of patches, discarding explicit 3D structure. The model has to infer distance, orientation, and contact from 2D appearance alone — and that is exactly where manipulation gets hard. The thesis puts the problem vividly: a gripper positioned 5 cm versus 7 cm from an object looks nearly identical in RGB space, but that 2 cm can be the difference between a clean grasp and a collision.

Recent work confirms the geometry gap is real, not cosmetic. DepthVLA (Yuan et al., 2025) adds a dedicated depth transformer alongside the RGB-language transformer in a mixture-of-transformers design (Figure 2), initialised from a pretrained monocular depth estimator. On real-world household tasks it reached a 78.5% progress rate versus 65.0% for an RGB-only baseline — a 13.5 pp jump — and on the Simpler simulator, where ground-truth depth is available, the gap widened to 16 pp. The benefit scales with how much geometric precision a task demands. The open question this thesis takes up is not whether depth helps, but where in a frozen-backbone, adapter-based pipeline depth should be consumed to gain accuracy without sacrificing the efficiency that makes the adapter worth using in the first place.

The proposed architecture

The thesis augments VLA-Adapter with two complementary extensions, both operating entirely within the frozen-backbone constraint: a depth pathway that injects monocular-depth features directly into the action head, and a training-time segmentation teacher that sharpens where the model looks — then disappears at inference. We take them in turn. The primary benchmarks used were LIBERO (Liu et al., 2023) and CALVIN (Mees et al., 2022).

Where should depth enter? Decoupling depth from vision

The obvious move is to fuse depth with RGB at the vision-encoder level — concatenate the two before the frozen language model and let it sort them out. We tried it and unfortunately, it failed. On LIBERO-Spatial, concatenation fusion scored 70.0%, a 21-point drop from the 91.2% RGB baseline.

Why so catastrophic? The frozen backbone was pretrained on billions of RGB-text pairs to build a very specific embedding space, where visual patches map to semantic concepts. Inject depth features — which encode geometric structure, not appearance — into that space and you create a distribution mismatch. The language model processes the “corrupted” embeddings and emits degraded action-query states, and the action head is left compensating for a confused backbone while also trying to learn to act.

The clincher was an oracle experiment where we fed ground-truth simulator depth through a cross-attention module at the vision level. It reached 92.8%, barely 1.6 points above baseline. Even flawless geometry, injected in the wrong place, adds almost nothing.

The chosen path was therefore: leave the frozen model's world intact, and deliver geometry straight to the layer where actions are decided. So depth features bypass the language model entirely and are routed to the action head. Depth is estimated online from RGB by a frozen DepthAnything V3 network, so deployment needs only an ordinary camera. On the same benchmark as above (LIBERO-Spatial), we reached a 99.2% success rate, 8 points above the baseline. Concretely, here’s how we did it.

The DepthDecoder

A naïve approach — projecting raw depth features through one linear layer and feeding them to all 24 Bridge Attention blocks — trained unstably, because a single transformation cannot be simultaneously useful at every level of the stack. The fix is the DepthDecoder: a compact 6-layer transformer (about 58M parameters, roughly a fifth of the trainable budget) that refines depth features through self-attention and produces level-appropriate representations. Its outputs form a hierarchy — early layers carry raw, patch-level geometry; later layers carry globally integrated structure — and the 24 Bridge Attention blocks are grouped into sets of four, each group consuming one decoder layer so that depth abstraction is aligned with action-head depth.

Four-source Bridge Attention

The central contribution is extending Bridge Attention from three sources to four, promoting depth to a first-class input alongside self, adapter+proprioception, and vision. At each block the shared query attends to four key–value pairs jointly. The vision stream keeps its learnable gate; the depth stream is layer-normalised so its magnitude cannot overwhelm the other sources:

All four sources are concatenated along the key dimension and passed through a single scaled softmax, so they compete for a fixed attention budget rather than being combined by hand-tuned weights:

With the action chunk, proprioception, vision, and depth tokens, that is $K_{\text{total}} = 8 + 65 + 512 + 256 = 841$ key positions in the joint softmax. Notably, depth gets no explicit gate of its own. An extensive ablation found that every gating variant — per-block scalar gates, elevated gate learning rates, negative initialisation, spatial gate networks — either collapsed performance or destabilised training. Layer-normalisation on depth acts as an implicit “de-gate,” and removing the gate entirely trained more stably.

The segmentation teacher

Depth tells the policy where geometrically to act, but it does not directly shape what the VLM attends to — and depth-only models showed slight regressions on semantically demanding suites, as if depth diluted focus on the relevant objects. The remedy is an instruction-conditioned segmentation teacher. A rule-based parser turns the instruction (“pick up the orange juice and place it in the basket”) into a clean object prompt (“orange juice. basket.”). A frozen Grounding-DINO detector localises those objects in the RGB image; the boxes are downsampled to the 16×16 patch grid to form a spatial target. A KL-divergence loss then nudges the VLM's visual-token attention toward those task-relevant patches:

The crucial property is that the entire teacher — detector, parser, mask generation, KL loss — runs only during training. At inference it is discarded, so the depth+segmentation model runs at exactly the same speed as the depth-only one, and the segmentation-only variant matches RGB-only inference time. Zero deployment overhead.

Training objectives

The integrated system — frozen VLM, DepthDecoder, and four-source Bridge Attention — is trained end-to-end against a composite loss that balances three goals:

The primary term is the action prediction loss, an L1 regression over the predicted 8-step action chunk (3D position delta, 3D axis-angle rotation, and a gripper scalar):

The depth reconstruction loss anchors the DepthDecoder to geometric reality. Without it, the decoder could drift toward producing whatever features happen to help action prediction — in the extreme, ignoring depth entirely. A Smooth-L1 (Huber) loss between the decoder's per-patch depth prediction and the pooled depth map keeps its representations interpretable as real geometry; it behaves like L2 for small errors and like L1 for large ones, so occasional bad depth estimates (reflective or texture-less surfaces) do not dominate the gradient. Finally, the segmentation term $\lambda_{\text{seg}}(t)$ follows a linear warm-up so the teacher stays silent until the action head has found a stable baseline.

What it buys you

The combined depth + segmentation model achieves the best result on every LIBERO suite simultaneously. The depth-only DepthDecoder improves LIBERO-Spatial by +8 pp over the RGB baseline, and on the multi-step CALVIN ABC-D benchmark the depth variant lifts the 5-task success rate from 36% to 74% — with no inference-time cost from the segmentation teacher and only a modest +0.057 s per action chunk from depth estimation. The full results of the experiments on LIBERO are summarized in the table below:

Two honest notes live in that table. First, LIBERO-Object is already near-ceiling for everyone, so “improvement” is mostly noise — depth helps most where there is headroom and geometry is the bottleneck. Second, segmentation-only (V3) regresses on Goal by 6.8 points: when the teacher narrows attention to named objects, it can misdirect the model on goal-conditioned instructions that are about spatial relations rather than specific objects. Adding depth back (V2) recovers and exceeds baseline everywhere. The two signals are complementary, not redundant — depth grounds geometry at the action level; the teacher sharpens semantics at the backbone level.

The results on CALVIN reflect where depth really shines:

The five-in-a-row success rate roughly doubles, from 36% to 74%. Most striking is the per-task breakdown: the two lightbulb tasks, which require precise positioning to engage a switch, go from 5.5% → 100% and 1.5% → 100% — exactly the geometrically constrained motions where depth should help most. And note the last row: our depth model, fine-tuned from scratch, essentially matches a fully pre-trained reference checkpoint (4.41 vs 4.36). Depth bought back the gap that pre-training would otherwise have needed.

Opening the box

We wanted to know how the trained action head actually spends its attention across the four sources. Raw softmax weights are misleading here — they are roughly fixed by token counts. So we used a value-contribution metric that projects each source's contribution onto the total attention output, accounting for value magnitude, not just attention weight. Averaged over 500 samples, it reveals a clean and somewhat surprising picture.

Depth consistently absorbs 27–30% of the action head's budget, drawn mostly from vision (which falls from 90% to ~65%). That settles one question outright: the DepthDecoder produces features the policy genuinely uses, not just artifacts it learns to ignore.

There is also a stability story hiding in the error bars. The baseline routes attention erratically across samples (±0.087 on the adapter stream); depth-enabled models are 3–4× more stable (down to ±0.014 for V2). That consistency is a plausible mechanistic explanation for the CALVIN gains: in a five-instruction chain, erratic per-step routing compounds into failure, while stable routing survives the instruction boundaries.

The second finding is that depth acts as a parallel geometric channel, not a reshaper of vision. When we visualise where the model looks — hidden-state norm maps for the backbone, and GradientSHAP for what drives the action — depth-only models keep the same diffuse vision attribution as the baseline. Depth does not change where vision looks; it opens a separate pathway. What does visibly sharpen the maps is the segmentation teacher, whose effect persists at inference even though the teacher itself is long gone.

From simulation to a real arm

All the primary results are in simulation, but we also built the full pipeline onto a real, low-cost SO-101 6-DOF arm using the LeRobot framework — recording, training, offline evaluation, and asynchronous deployment — with two camera views and two tasks: “pick up the pen and put it in the cup,” and “grab the ball and place it in the black square.”

Two lessons came out of it, both useful. The first was an engineering bridge: LIBERO policies speak in end-effector deltas that are agnostic to the robot's kinematics, while the SO-101 accepts absolute joint positions — so a zero-shot transfer is impossible, and we fine-tuned in the robot's own joint space. Building the RLDS↔LeRobot data bridge and the two-camera routing was the bulk of the work, and once done, the DepthDecoder and segmentation teacher ported over cleanly.

The second lesson was humbling and more general. We recorded roughly 400 demonstrations of the ball task — well past the usual guidance — and the policy still failed at the moment of contact: it would reach in the right direction, then nudge the ball out of range instead of grasping it. The full VLA-Adapter did visibly better than a SmolVLA baseline, with a more top-down, demonstration-like approach and the occasional clean pick, but success stayed sparse. The bottleneck was not architecture and not data quantity — it was data consistency. Imitation learning by supervised regression averages over inconsistent demonstrations, and that averaging is fatal in contact-rich phases like grasping. More data helped the gross motion; only near-perfect, repeatable demonstrations would fix the grasp. The infrastructure is ready; the missing ingredient is curated data.

Here is a demonstration of our best result once we sped it up a bit by introducing an async pipeline that preprocesses the next chunk while the current one is executing.

Config: 35k fine tuning steps, 8Hz control frequency, 8 open loop action steps

Where this goes next

The honest footnote is that geometry is not a cure-all. Under LIBERO-PRO's distribution shifts, the model handles paraphrased language (100%) and object replacement (90%) well, but all variants fail completely on object-swap — evidence that depth and segmentation grounding do not, on their own, resolve the trajectory-memorisation tendency baked into today's VLA architectures. That is a problem for the next paper. Three directions follow naturally:

The zeros on object-swap point at affordance-based conditioning — replacing object masks with richer where-to-act signals — as the route to breaking trajectory memorisation, directly targeting both the object-swap failure and the LIBERO-Goal regression. The long-horizon ceiling on LIBERO-Long points at a compact world model in the Bridge Attention latent space, turning a reactive policy into one that can reason about the consequences of an action before committing. And the real-robot work points at the least glamorous but most immediate need: a need for demonstration data that is not reliant on cumbersome manual recording that is highly prone to error. A pathway from synthetic realistic data being used to finetune a model to perform in a real-world context is something worth exploring for faster iterations.

However, for now, as a demonstration that you can give a frozen, efficient robot policy a real sense of three-dimensional space — without breaking the budget that made it practical — the image-aware VLA adapter makes a convincing case.

If you are working on embodied AI, efficient adaptation of large models, or turning a promising research prototype into something that survives contact with the real world, that is exactly the kind of problem we like at Modulai. It starts with the right questions — get in touch.

This work was carried out by Khalifa Fumo as a master's thesis in the Image Analysis and Machine Learning programme at Uppsala University, hosted at Modulai. Supervision: Puya Sharif (Modulai), with the Modulai team; subject reader: Ewert Bengtsson (Uppsala University); examiner: Nataša Sladoje.