Embodied XAI Is All You Need

Can we trust it?

Robots that follow spoken instructions are usually built in two halves: a large vision–language model (a "VLM" — the same family of AI that answers questions about photos) decides what to do, and a small, fast controller decides how to move the legs. This split is powerful. It is also opaque: when the robot silently walks to one of two boxes, nothing tells you what in the scene drove that choice — or when the decision was made.

Two things stand in the way of trusting such a system in a home, office, or sidewalk:

A robot with two brains

The agent is a simulated Unitree Go2 quadruped in the Genesis physics engine. Its slow brain is a frozen, off-the-shelf VLM (Qwen2.5-VL-3B) that looks through the onboard camera about 3 times a second and writes a steering command — forward speed, sideways speed, and turn rate — like a slow joystick. Its fast brain is a PPO locomotion policy, a small neural network trained by reinforcement learning purely to make the legs track whatever command it's given, 50 times a second. It has never seen the camera, the task, or the VLM.

Between queries, the VLM also maintains a memory (adapted from Multi-Scale Embodied Memory): a short clip of the last 4 frames, and a running text summary — at most 600 characters — that it rewrites and compresses itself every step. Crucially, every query's exact inputs are logged, which is what makes faithful explanation possible later.

The continuous-command interface is a deliberate choice: the steering decision is realized as concrete numeric tokens in the VLM's own output — and a token you can point at is a token you can attribute. That red dashed tap is the subject of section 05.

An experiment built to be hard to fool

Two identical boxes, one red, one blue, equally far away. The instruction — "go to the friendly one" — never mentions color. If the agent consistently prefers one box, is that a color bias (always red, wherever it is) or a positional bias (always left, whatever the color)? On a single arrangement you cannot tell: "always picks red" and "always picks left" look identical whenever red happens to be on the left.

Two design elements disentangle them. Mirror testing runs every condition in both arrangements (L=red/R=blue and L=blue/R=red) — pooling separates color-tracking from side-tracking. A same-color control sets both boxes red, removing the color cue entirely to isolate pure positional bias. Try it yourself — each trial below is sampled from the real measured choice rates, and arrangements mirror automatically:

What we found

Read the pooled view like this: a RED% far from 50 with LEFT near 50 means color drove the choice; an extreme LEFT% with RED near 50 means position did. "Friendly" shows both — a real pull toward blue (RED = 30%) stacked on the big positional habit (LEFT = 74%). "Hostile" shows no color effect at all. And the control's 87% left is the loudest artifact in the study — invisible without the same-color baseline.

So the behavioral probe says the agent harbors biases. But statistics only tell you that — not where the model was looking when it decided. For that we need to open the black box.

Asking every pixel: did you matter?

Integrated gradients (IG) is a way to ask each pixel of the input image how much it influenced one specific output. The trick: fade the image up from black in steps, and at each exposure ask every pixel, "if you brightened a little right now, how much would the steering command change?" Summing those answers along the fade gives each pixel a principled share of credit.

Two details make this attribution faithful rather than decorative. It targets the exact sign-inclusive numeric tokens of the turn command the agent actually sampled while driving — not a fresh re-decode. And it runs over the same frame window and memory snapshot that conditioned that decision, replayed from the logs. Slide through the 8 steps:

Why per-frame heatmaps can't audit a robot

Now the problem of scale. The planner is re-queried at 3 Hz from a camera that walks, turns, and bobs. Each query yields its own heatmap in its own pixel coordinates — the same physical box lands in a different place in every frame. One condition of the study is thousands of these. No shared frame of reference, no aggregate signal, no way to tell a repeatable pattern from single-trial noise.

But the simulator knows the depth of every pixel and the exact camera pose of every query. So each heatmap can be unprojected: pushed back through the camera geometry onto the physical surface each pixel actually saw. In world coordinates, all those disconnected maps suddenly agree:

The 4D heat field

Accumulate those unprojected attributions over every query of a trial, and over trials of a condition, and you get one object: a heat field painted on the arena itself — floor, target faces, back wall — that grows through time. Three spatial dimensions plus time: genuinely 4D. It is the difference between auditing a robot by flipping through a shoebox of photographs and walking through the room where the decision happened.

This one is a stylized reconstruction of the "friendly" condition. The black ball is the agent; the fading trail behind it is its path so far. Drag to orbit. Scrub time and watch both targets gather heat from the very first query — unequally — until the moment of commitment, when attribution locks onto the blue cube:

Diagnostics only a world-anchored field can run

Saliency has a known failure mode: a heatmap can look convincing and mean nothing. The sanity check here is empirical — convergence. Pool one trial and you get speckle. Pool twenty and, if the hot region is real, it stops moving. Per-frame 2D maps can never run this test, because no two of them share coordinates. Slide n:

The field supports two more instruments. Outcome-split fields pool trials by what the agent did — nearer-red vs nearer-blue, reached vs drifted — isolating the attention signature of each choice, and of failure. And time-resolved accumulation videos render the field frame by frame, making the moment of target commitment — or its absence — directly visible. None of the three exist for a stack of disconnected 2D images.

Together the two halves close a loop that neither closes alone: the behavioral probe proves the agent harbors latent biases; the 4D field shows where and when its decisions were physically grounded, using the agent's own sampled actions. We argue this is the minimum transparency standard for a VLM-controlled robot operating around people — auditable by researchers and by anyone who can look at a room and see where the heat is.

What this doesn't show

— Simulation only. One Genesis scene, one 3B-parameter VLM; real-world deployment adds localization noise and effects will vary across models and scenes.

— Attention ≠ cause. Gradients show where influence concentrated — necessary, not sufficient, evidence of why. Occlusion-style interventions on the identified hot regions are the natural next step.

— IG choices are pragmatic. A black baseline and 8 Riemann steps; other variants may shift attributions.

— The memory adaptation is approximate. A frozen VLM rewriting its own summary in-context lacks MEM's trained encoder and can drift under distribution shift.