Remembering What You Don't Know

Planning with Persistent Identity Beliefs under Perceptual Aliasing

Brendon T. Murphy1,2*, Keonyoung Koh1*, Woobean Seo1,3*, Hoseong Kwak1, Chunghun Ha1, Duhyun Ro1,3†
1CONNECTEVE Co., Ltd., 2Massachusetts Institute of Technology, 3Seoul National University
*Equal contribution   †Corresponding author

When a collaborator relocates visually identical tools out of view, the robot keeps competing identity beliefs and decides whether to act or inspect based on downstream cost.

Abstract

Robots that share tools with people often face visually aliased tools whose identities become uncertain after unobserved human manipulation. The robot must then act on a belief about which tool is which rather than on a known identity. We present a belief-aware planner that chooses between acting on the current belief and inspecting a candidate when the expected cost of disambiguation is lower than the risk of a wrong handover. To give this decision the uncertainty it needs, we introduce a persistent identity belief that maintains competing joint entity–observation hypotheses instead of collapsing to a single assignment.

We evaluate the critical human-intervention event from a closed-loop Allen-key handover task over 99 replayed real episodes. Joint hypotheses and relocation modeling substantially reduce confidently wrong beliefs, while belief-aware planning reduces wrong handovers relative to maximum a posteriori (MAP) action and avoids the fixed overhead of always inspecting.

The Problem: Which Tool Is Which?

Visually aliased Allen keys relocated while unobserved

Persistent identity can become ambiguous when perceptually aliased tools get relocated while they are unobserved or occluded. The robot maintains competing identity beliefs and chooses whether to act or inspect based on downstream cost.

Collaborative robots hand tools to people, take them back, and reuse them later. This implicitly assumes the robot knows which physical tool is which. That assumption breaks when two conditions coincide: tools are visually aliased (e.g., Allen keys that differ only in size), and a collaborator relocates them while unobserved. Perception can recover where the visible candidates are, but not which tool each one is.

Existing visuomotor memories and 3D scene graphs commit to a single target, so an unobserved swap produces a confidently wrong handover. Such an error not only takes time to recover from, but also forces the collaborator to divert attention from their own task, and lowers the trust people place in the robot. Inspecting every candidate, on the other hand, adds motion and latency to every request. Our approach is a memory that remembers what it does not know, paired with a planner that inspects only when it is worth it.

Method

System overview

System overview. Anonymous per-frame observations update a persistent identity belief, from which the planner chooses to act or inspect; a VLA policy executes the choice.

Hypothesis-Conditioned Identity Memory

Rather than committing to one association, the memory keeps a weighted bank $\mathcal{H}$ of mutually consistent joint hypotheses. Each hypothesis binds every persistent entity $E_i$ to a current anonymous observation $o_j$ (or to “unobserved”) and carries a 3D Gaussian position and a relocation probability $\pi_{i,h}$:

$$h=\Big(\mathbf{a}^h,\{\mu_{i,h},\Sigma_{i,h},\pi_{i,h}\}_{i=1}^{K}\Big).$$

When a tool class is manipulated out of view, tracking is suspended. On resume, the predictive density mixes “still near the remembered pose” with “moved elsewhere”:

$$p_{i,h}^-(X) = (1-\pi_{i,h}^-)\,\mathcal{N}(X; \mu_{i,h}^-, \Sigma_{i,h}^-) + \pi_{i,h}^-\, q_{i,h}^{\mathrm{rel}}(X).$$

Each parent hypothesis branches over valid one-to-one, class-consistent assignments, scored by visibility-aware hit/miss likelihoods and a tracker continuation prior, then globally normalized. Both explanations survive until evidence resolves them.

Belief-Aware Action Planner

The planner reads the marginal identity belief $B(i,j)=\sum_{h\in\mathcal{H}}W_h\,\mathbb{I}[a_i^h=j]$ and compares acting and inspecting in units of execution time:

$$\begin{aligned} Q_{\mathrm{act}}(j)&=B(i^*,j)\,T_s(j,e)+\sum_{m\ne i^*}B(m,j)\big[T_w(j,e)+V(b_{j,m},e_{\mathrm{return}})\big],\\ Q_{\mathrm{inspect}}(j)&=T_i(j,e)+\sum_m B(m,j)\,V(b_{j,m},e_{\mathrm{inspect}}). \end{aligned}$$

Inspection is therefore triggered by the expected downstream cost of a mistake, not by a fixed confidence threshold. Because hypotheses are joint, conditioning on one candidate's identity also updates the others.

Experiments

Evaluation Scenario and Protocol

Real-world experimental setup

KUKA LBR Med 14 R820 with a Robotiq 2F-85 gripper. The wrist-mounted RealSense D405 is the only sensor used by the memory and planner; the external Orbbec Femto Mega feeds VP-VLA only.

We evaluate the identity-ambiguity event of the handover task. An original Allen key is observed in a shared bucket; while the wrist camera looks elsewhere, the collaborator returns a visually similar key and may displace the original (0.1–8.7 cm, median 4.0 cm). The robot must then identify the original. We repeat this for 100 real episodes (99 with a valid decision record).

Recorded frames and robot poses are replayed through every memory variant, so all variants receive identical sensory evidence. Policy outcomes are computed against the annotated identity: a wrong candidate is refused and identified by the collaborator, while an inspection reveals the candidate's identity. Each primitive cost is grounded in real VLA rollouts.

For planning we compare against three baselines: MAP action, which collapses the belief to the MAP identity before planning; always-inspect, which verifies identity before every action; and an identity oracle with access to the true correspondence.

Primitive durationMeasured (10 rollouts)Planner forecast
Successful handover21.37 s20.82 s
Inspection (additional)16.96 s11.24 s
Refusal / recovery15.73 s14.53 s

The deployed planner decides with a forecast calibrated from VLA training demonstrations; all policies are scored with measured durations. The forecast underestimates inspection by ~6 s.

Identity Memory: Full Ablation

Memory MAP ↑NLL ↓Brier ↓CW ↓ Time [s] ↓Wrong ↓Inspect ↓
Full (ours)0.910.330.10128.46237
HypothesesSingle hypothesis0.821.260.181828.12180
Relocationw/o relocation0.910.630.09924.7490
Always relocated0.910.450.13030.48151
Uniform relocation only0.790.420.14230.34248
Evidencew/o negative spatial update0.680.870.251733.172612
w/o observation hold†0.590.850.27635.322821

CW: confidently wrong trials (incorrect MAP identity with confidence ≥ 0.9). Time / Wrong / Inspect are the downstream behavior of the same expected-cost planner on each memory. †Without observation hold, all 100 episodes yield a valid decision; other rows use the common 99-episode subset.

  • Joint hypotheses prevent confident identity collapse. A single hypothesis drops MAP accuracy from 0.91 to 0.82, but the larger effect is calibration: NLL 0.33→1.26 and CW 1→18. The planner receives a nearly deterministic but wrong belief, never inspects, and makes 18 wrong handovers.
  • Relocation uncertainty matters even when MAP accuracy is unchanged. Without the relocation alternative, MAP stays at 0.91 but NLL nearly doubles and CW rises to 9; the planner degenerates to MAP action (9 wrong handovers). Treating every entity as relocated removes confident errors but discards useful memory, raising inspections from 37 to 51.
  • The relocation prior should retain local scene structure. A uniform-only relocation density lowers MAP accuracy to 0.79 and causes 48 inspections — tools are often only nudged from where they were.
  • Negative spatial evidence must change the state, not only its weight. Without it, misses are repeatedly scored against a stale spatial prior: MAP 0.68, 17 CW trials, 26 wrong handovers.
  • Observation holding is a perception safeguard. Without it, unstable centroids from newly emerging or truncated masks claim persistent identities too early (MAP 0.59).

Top-1 accuracy alone is insufficient for a planning-facing memory: the full model and the no-relocation variant have identical MAP accuracy, yet very different confident-error counts and downstream behavior.

Planner Behavior by Ambiguity Level

OursMAP actionAlways inspect
Displacement$n$ Time [s]WrongInspect Time [s]Wrong Time [s]Inspect
All9928.46±0.9323724.74±1.08938.33±0.0099
< 1 cm1021.370021.37038.3310
1–3 cm1821.370021.37038.3318
3–5 cm4027.3101421.37038.3340
5–9 cm3136.3522332.14938.3331

Policies acting on the full memory. Times are mean ± SEM across episodes, computed from measured primitive durations.

  • < 3 cm: the belief is near-certain and the planner acts directly in every trial, matching MAP action and avoiding the fixed cost of always inspecting.
  • 3–5 cm: the belief is underconfident (MAP action is correct in all 40 trials). Combined with the underestimated inspection forecast, this yields 14 unnecessary inspections, which account for 2.4 s of the 3.7 s mean-time gap to MAP action.
  • 5–9 cm: where identity history is most disrupted, the planner inspects 23 times and cuts wrong handovers from 9 to 2, at 4.2 s per request (36.35 vs. 32.14 s).

Scored with measured durations, MAP action is 3.7 s faster on average, while the time lost to rejected handovers is small on average. Although the planner does not dominate MAP on mean time in this cost regime, its benefit is the ability to trade execution risk against inspection cost rather than committing to either extreme. It does not inspect whenever uncertainty exists; it concentrates verification where the belief and the task cost make a wrong commitment consequential.

Time–Error Trade-off and Cost Sensitivity

Time-error trade-off

Time–error trade-off. Each point is the planner deciding with different costs, scored with measured costs. Circle / diamond: deciding with measured / forecast costs. Dotted: random mix of MAP action and always-inspect. Numbers: inspections out of 99.

Cost-ratio sweep

Cost-ratio sweep. Same costs for deciding and scoring; the ratio is varied through the inspection duration. Top: extra time per request over the oracle, in units of one inspection. Bottom: wrong handovers. The vertical line marks the measured costs of our task.

  • An efficient front. Removing 7 of the 9 wrong handovers takes 26 inspections and 1.8 s per request over MAP action, where random inspection would need 77; zero wrong handovers are reached 4.7 s faster than always-inspect.
  • Behavior adapts to the cost ratio. Below a wrong-handover/inspection ratio of about 2, the planner acts directly and matches MAP action. Above it, wrong handovers fall from 9 to 2 by a ratio of 2.7 and to 1 beyond 4.8. In time, MAP action stays faster up to a ratio of about 6.8, because the belief is underconfident (trials with MAP confidence 0.6–0.8 are correct in 29 of 31 cases).
  • Cost-model error is distinct from belief error. Deciding with measured instead of forecast durations reduces inspections from 37 to 8 and time from 28.46 to 24.98 s (within 0.3 s of MAP action), at 6 rather than 2 wrong handovers. The measured-cost decision lies on the front; the forecast-cost decision misses it by 1.9 s at the same error count.

Closed-Loop Real-Robot Demonstration

Closed-loop real-robot demonstration

(a) Shared task prefix: the robot hands over the 2 mm and 2.5 mm Allen keys, which the collaborator returns to the bucket so that two visually aliased candidates remain when the 2 mm key is requested again. (b) High-ambiguity case: MAP action commits to the wrong candidate and requires refusal and recovery, whereas our planner first performs a wrist-camera close-up inspection and then hands over the verified key. (c) Low-ambiguity case: always-inspect incurs an unnecessary verification motion, while our planner acts directly on the confident identity belief.

The replay benchmark isolates the same post-intervention ambiguity that occurs in the full interaction. Executing the complete sequence end to end shows that the identity posterior changes the executed manipulation strategy, rather than serving only as an offline association score.

Conclusion

Rather than forcing memory to resolve ambiguity into a single identity estimate, our framework exposes persistent identity uncertainty as a planning-facing state, allowing downstream decision making to determine when uncertainty is tolerable and when additional perception is worth its cost. Real-robot replay experiments show that joint hypotheses and relocation modeling reduce overconfident identity errors and enable this act-or-inspect trade-off, supporting uncertainty-aware persistent world models as an interface between perception, memory, and task-level planning.

Future work will incorporate richer priors over human interaction and object relocation by leveraging task knowledge. Evaluating more aliased candidates and imperfect inspections will more fully test generalization, joint-hypothesis resolution, and recursive planning.

Limitations

  • Value of recursive planning. In this scenario, a single inspection resolves both identities, so the decision reduces to a one-step act-vs-inspect comparison. A tuned confidence threshold traces nearly the same front (within 0.4 s), so we do not claim an advantage of recursive search over thresholding here; the cost-based rule provides an operating point derived from action costs rather than a hand-tuned threshold.
  • Belief calibration and cost-model error. Underconfidence in the identity belief and underestimation of inspection cost by the deployed forecast both contribute to unnecessary inspections in the 3–5 cm regime. Together, these effects degrade the planner's time–error trade-off relative to MAP action at moderate cost ratios.
  • Sample size and scope. 99 trials of a single scenario (two aliased Allen keys, one collaborator, displacements below 9 cm). Wrong-handover counts are in the single digits, so differences of one or two are within sampling noise. Policy outcomes are counterfactual, computed from replayed beliefs and measured primitive durations.