ESVLM-Guided Experience Replay

VLM-Guided Experience Replay

Elad Sharony1, Tom Jurgenson1, Orr Krupnik1, Dotan Di Castro2, Shie Mannor1,3

1Technion2ForSight Robotics3NVIDIA Research

arXiv Code coming soon

Overview

VLM-RB uses a frozen, pre-trained Vision-Language Model (VLM) as an automated evaluator that identifies and prioritizes promising sub-trajectories in the replay buffer of an off-policy agent. Prior work has integrated LLMs and VLMs into various components of reinforcement learning, but the replay buffer, the component that stores and reuses experience, has remained unexplored. VLM-RB is a plug-and-play prioritization layer for any off-policy algorithm that uses one.

Experience replay[2] is fundamental to off-policy RL, but which experiences should be prioritized? Prioritized Experience Replay (PER)[1] favors transitions with high temporal-difference (TD) error. TD error can correct value estimates, but it lacks semantic awareness: it cannot distinguish transitions that reflect genuine progress toward task completion from those that do not.

This limitation is acute in long-horizon, sparse-reward tasks. Critical transitions, such as unlatching a door or grasping a tool, may yield low TD errors early in training, because delayed credit assignment has not yet propagated the reward to them. Conversely, visually distinct but task-irrelevant motions can yield large TD errors despite contributing little to task completion.

Figure 1. VLM-RB learns more from the same interaction budget. Mean success across 12 algorithm–task configurations (DQN and IQN on MiniGrid DoorKey 8×8, 12×12, and 16×16; SAC and TD3 on OGBench Scene tasks 3–5), averaged across 5 seeds; shading denotes standard error, and the rose fill between the curves is VLM-RB’s gain over the best baseline. Select a legend entry to hide or show a method.

1 Method

VLM-RB is a plug-and-play framework for any off-policy algorithm that leverages a replay buffer. A VLM assigns semantic scores to sub-trajectories, and sampling is biased toward higher-scoring experiences during policy optimization.

Key insight

Pre-trained VLMs already encode rich semantic priors about what constitutes meaningful behavior. VLM-RB can therefore immediately identify semantically relevant data, enabling faster learning than approaches which must wait for the critic to converge.

Figure 2. System diagram. (1) Data is collected with the current policy $\pi_k$ interacting with multiple instances of the environment. (2) Transitions $(s,a,r,s')$ are kept in a prioritized replay buffer with a provisional VLM score $\bar{p}$. (3) Asynchronously, after each insertion, a VLM worker scores the corresponding rendered clip $\tau^O$ under a prompt $\mathsf{P}$ and writes the resulting score $p^{\text{VLM}}$ back to the replay buffer. (4) The learner samples a minibatch using the mixture distribution $q_t$, which interpolates between uniform and VLM-prioritized replay. (5) The agent policy is updated to $\pi_{k+1}$. The environment frames are from OGBench Scene, and $\lambda_t$ is drawn at $0.25$, its value a quarter of the way through training. Point at a step to play it.

Scoring

VLM-RB extracts semantic scores from collected experience with a frozen, pre-trained VLM. Let $\tau^O_i=(o_{i}, o_{i+1}, \ldots, o_{i+L-1})$ denote a visual clip of $L$ rendered frames. The VLM maps this clip and a text prompt $\mathsf{P}$ to a scalar score:

$$p^{\text{VLM}}_i = f_{\text{VLM}}(\tau^O_i, \mathsf{P}) \in \mathbb{R}$$

The prompt is goal-agnostic and domain-generic (see the environments below). Because the VLM is frozen and its evaluation depends only on static visual content, each clip is scored exactly once, whereas PER must keep updating priorities as the value function evolves.

Why clips rather than frames? A single frame is fundamentally limited by semantic ambiguity: a gripper hovering above an object looks the same at the onset of a successful grasp and in the aftermath of a failed attempt. Scoring sub-trajectories gives the VLM the temporal context to tell meaningful behavior from failure.

Value?

The shared observation: the robot arm above the table, with the cube, drawer, window, and buttons

$o_i$

Successful sub-trajectory, current frame

$\tau^O_{i,1}$ · successful grasp

The gripper grasps the cube and carries it into the drawer.

High VLM score

Stagnating sub-trajectory, current frame

$\tau^O_{i,2}$ · stagnation

The arm moves, but the cube stays where it was.

Low VLM score

Figure 3. Temporal context resolves visual ambiguity. From the initial state $o_i$ (left), multiple futures are possible. The top sub-trajectory shows a successful grasp, while the bottom shows stagnation. By scoring sub-trajectories rather than single frames, the VLM has sufficient context to distinguish meaningful progress from failure modes. Both clips are shown at their last frame; press Play clips or drag the slider to step through them together.

Prioritization

The VLM score of each clip is propagated to all transitions within that clip and combined multiplicatively with the latest TD error. This defines the priority signal, which is converted into a distribution:

$$p_i = p^{\text{VLM}}_i\,|\delta_i|, \qquad q^{\mathrm{P}}_t(i) = \frac{(p_i+\varepsilon)^\alpha}{\sum_{j\in\mathcal{D}_t}(p_j+\varepsilon)^\alpha}$$

The two factors carry complementary information: $p^{\text{VLM}}_i$ marks which experiences are semantically consistent with task success, whereas $|\delta_i|$ marks where the value function is still inaccurate.

Sampling

Because the VLM is an external prior that may score useful transitions inaccurately, VLM-RB preserves full-buffer coverage by mixing VLM-guided prioritization with uniform replay:

$$q_t(i) = \lambda_t\, q^{\mathrm{P}}_t(i) + \frac{1-\lambda_t}{|\mathcal{D}_t|}$$

Each batch draws a $\lambda_t$ fraction from $q^{\mathrm{P}}_t$ and the remainder uniformly. A linear warm-up starts at $\lambda_0=0$ (pure uniform sampling) and anneals to $\lambda_{\max}=0.5$ over the first half of training: the warm-up favors broad coverage early on, and the uniform component limits the influence of inaccurate VLM scores throughout.

Efficient implementation

The VLM scores clips asynchronously, in the background, so policy optimization is never blocked by inference latency. A transition enters the buffer with a provisional score $\bar{p}$, the running mean of the VLM scores returned so far, and the clip’s own score overwrites it once the worker returns. The scorer is a lightweight 1B-parameter Perception-LM[4].

2 Experiments

We compare VLM-RB with uniform (UER) and prioritized (PER) experience replay under four algorithms: DQN[7] and IQN[8] on MiniGrid, and SAC[9] and TD3[10] on OGBench. Agents receive state-based observations rather than pixels, isolating the effect of replay prioritization; only the VLM sees rendered frames. We report the average success rate (ASR) over 32 evaluation episodes per seed, averaged over 5 seeds, with the standard error of the mean.

Environments

MiniGrid / DoorKey[5]8x8, 12x12, 16x16

Discrete navigation with sequential subtasks: pick up the key, unlock the door, reach the goal. Rewards are sparse: zero at every intermediate step, and positive only at the goal.

Prompt: "Does this clip contain a clear instance of goal satisfaction anywhere in it? If no visible success occurs, answer No. Do not guess. Output exactly Answer: Yes or Answer: No."

OGBench / Scene[6]Task 3, Task 4, Task 5

Continuous tabletop manipulation with a UR5e arm: a cube, a drawer, a window, and two buttons that lock them. Tasks 3–5 require increasingly long compositional sequences of unlocking, placing, and relocking. All methods share a warm start of 10 demonstration episodes.

Prompt: "Is there at least one clear instance of goal satisfaction in these frames? Look for contact + displacement consistent with the goal (lift off surface, place into receptacle, open/close articulation, move to target zone). Do not guess. If not visible, answer No. Output exactly Answer: Yes or Answer: No."

Frozen VLM scores qualitatively anticipate learned value

Does the VLM offer a useful signal before any learning happens? To illustrate the semantic grounding of our prioritization, we compare the frozen VLM scores with critic value differences ($\Delta Q$) as learning progresses, on a fixed reference episode and with checkpoints from a successful training run. The critic’s estimates are initially flat and uninformative; over training, it gradually learns to assign high value to the same semantic events, such as picking up the key or opening the door, that the VLM identified from the outset.

Reference episode · MiniGrid DoorKey-16x16 · 106 steps

clip (32 steps) stand-in actions whole path

Figure 4. Frozen VLM scoring qualitatively anticipates learned value. A single reference episode in MiniGrid DoorKey-16x16 that contains both goal-directed actions and random noise; sparse reward is only received at the final goal. Left: the episode on the grid, with the agent at the selected step and, in rose, the 32-step clip centered on it. Right, top: the timeline of ground-truth events, lit as they happen. Middle: the frozen VLM score of each $L=32$-frame clip, plotted at the clip’s center; clip A captures a semantically relevant sequence, while clip B is a mostly random walk that eventually reaches the goal. Bottom: the temporal value difference $\Delta Q(t;L)=Q(t+\tfrac{L}{2})-Q(t-\tfrac{L}{2})$ for the selected critic checkpoint, taken between the first and last frames of each clip, with earlier checkpoints in lighter gray. What is recorded: the VLM scores, the critic values, and the event steps (key picked up at step 37, door opened at 45 and entered at 46, random actions from 53, goal at 106). The layout is regenerated from MiniGrid’s seed 5; the agent’s positions were not recorded and are reconstructed: the goal-directed segments with the episode generator’s planner, which reproduces its recorded plans on five other episodes, and the two random segments (steps 0–25 and 53–103, dashed) with seeded stand-in actions that reproduce every recorded event when replayed in MiniGrid; the second wanders in the upper part of the right room, where the paper’s figure draws it. Press Play episode or drag the step slider to follow the agent; hover the plot, or focus it and use the arrow keys, to move through the steps; drag the checkpoint slider or press Animate training to sweep the 20 critic checkpoints from 50K to 1M steps.

Semantic prior

Following the “Modified Game” paradigm[14], we modify only the rendered frames used for VLM scoring, leaving the MDP and the agent’s observations untouched. In the Misleading game, object sprites are swapped for semantically conflicting ones, such as keys rendered as lava; in the Abstract game, objects are replaced with noise patterns. Both slow learning and increase variance, and with abstract visuals performance degrades to the level of PER (Figure 5): the prioritization is only effective when the visual input aligns with natural semantic priors.

Figure 5. Success depends on alignment with semantic priors. We compare the performance of VLM-RB across frames rendered with Standard visuals (rose circles), Misleading swapped sprites (gray squares), and Abstract textures (light diamonds). The dashed line indicates the ASR of PER. Crucially, the agent’s actual observations and the underlying MDP are identical across all settings; only the visual input to the VLM is altered. The significant performance drop in the Misleading and Abstract settings confirms that our method relies on the VLM correctly recognizing specific semantic objects (e.g., keys and doors) to accelerate learning. Right: what the VLM sees in each setting, the rendered grid and each object’s sprite. DQN on DoorKey-16x16 with VLM-only priority; mean ± SEM over 5 seeds (PER: mean only). Point at a setting to isolate its curve.

Main results

VLM-RB matches or exceeds every baseline in success and improves sample efficiency in 23 of 24 comparisons. The largest gains appear in the most challenging settings: on DoorKey-16x16, VLM-RB achieves a +283.3% improvement over UER and +91.7% over PER; on Scene-5, the improvement is +147.2% over UER and +67.9% over PER. Even where the baselines eventually solve the task, VLM-RB reduces the number of steps needed to reach their best performance by as much as 62.0%.

Table 1. Improvement with respect to baselines. Performance denotes the mean per-seed best ASR (5 seeds). Sample efficiency tracks the steps required to reach the baseline’s best performance; unreached targets are assigned the 1M-step horizon. Values in parentheses give VLM-RB / baseline. Both metrics are averaged across the aggregated algorithms; the training curves are in Figure 6.
Algorithm Level vs. UER vs. PER
Performance ↑ Sample efficiency ↓ Performance ↑ Sample efficiency ↓
DQN + IQNDoorKey 8×8 0%(1.00/1.00) 22%(139K/178K) 0%(1.00/1.00) 26%(139K/187K)
12×12 61%(1.00/0.62) 57%(277K/640K) 22%(1.00/0.82) 37%(292K/467K)
16×16 283%(0.92/0.24) 53%(420K/893K) 92%(0.92/0.48) 51%(305K/622K)
SAC + TD3Scene 3 0%(1.00/1.00) 40%(213K/354K) 0%(1.00/1.00) 20%(213K/266K)
4 22%(1.00/0.82) 62%(295K/776K) 2%(1.00/0.98) 38%(394K/634K)
5 147%(0.89/0.36) 55%(366K/819K) 68%(0.89/0.53) 37%(508K/805K)
Figure 6. VLM-RB consistently outperforms baselines across continuous and discrete tasks. Success rates for four algorithms (TD3, SAC, DQN, IQN) on OGBench and MiniGrid, per task or averaged over the three tasks of each algorithm. Shaded regions indicate standard error across 5 seeds; curves use a 2-point moving average, and for the mean of three tasks, the tasks are averaged within each seed. Select an algorithm and a task; select a legend entry to hide or show a method.
Figure 7. Training progression on Scene task 4 (TD3). Success rate of VLM-RB, PER, and UER over training (mean across 5 seeds, shading ±SEM), and one evaluation episode of each at 300K, 500K, and 1M steps: unlock the drawer, open it, place the cube inside, and close it. All nine episodes succeed, and VLM-RB’s finishes first at every checkpoint (117 steps against 241 and 248 at 300K; 48 against 56 and 88 at 1M). The episodes come from the page’s original progression video (February 2026), recorded before the final runs behind the curves. Select a checkpoint, or point at it on the chart, to play its episodes side by side; training after it is dimmed.

Comparisons and ablations

Beyond uniform and prioritized replay, we compare VLM-RB with alternative prioritization schemes, and ask which of its design choices matter: the priority signal, the sampling mixture, and the size of the VLM.

VLM-RB is the only method to consistently solve DoorKey when compared with Attentive Experience Replay (AER)[11], Experience Replay Optimization (ERO)[12], and Reducible Loss Prioritization (ReLo)[13]. ERO and ReLo fail to prioritize task-relevant transitions because their feedback signals are either dominated by noise or remain zero throughout the long pre-success phase, while similarity-based methods like AER over-sample current states and neglect critical past dependencies such as key acquisition.

Figure 8. VLM-RB overcomes the failure modes of heuristic prioritization in sparse-reward tasks. Aggregated success rates on MiniGrid/DoorKey across grid sizes (DQN). While alternative methods (AER, ERO, ReLo) fail as they depend on dense rewards or local similarity metrics, VLM-RB successfully bridges the long-horizon dependencies. Curves show the mean across 5 seeds, with shaded standard errors; all methods share the same backbone, optimizer, and schedule.

Citation

@article{sharony2026vlm,
  title={VLM-Guided Experience Replay},
  author={Sharony, Elad and Jurgenson, Tom and Krupnik, Orr and Di Castro, Dotan and Mannor, Shie},
  journal={arXiv preprint arXiv:2602.01915},
  year={2026}
}

References

  1. Schaul, T., et al. (2015). Prioritized Experience Replay. arXiv:1511.05952.
  2. Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning.
  3. Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.
  4. Cho, J.H., et al. (2025). PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding. arXiv.
  5. Chevalier-Boisvert, M., et al. (2023). Minigrid & Miniworld: Modular & Customizable RL Environments. NeurIPS.
  6. Park, S., et al. (2024). OGBench: Benchmarking Offline Goal-Conditioned RL. arXiv.
  7. Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. AAAI.
  8. Dabney, W., et al. (2018). Implicit Quantile Networks for Distributional Reinforcement Learning. ICML.
  9. Haarnoja, T., et al. (2018). Soft Actor-Critic Algorithms and Applications. arXiv:1812.05905.
  10. Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ICML.
  11. Sun, P., Zhou, W., and Li, H. (2020). Attentive Experience Replay. AAAI.
  12. Zha, D., et al. (2019). Experience Replay Optimization. arXiv:1906.08387.
  13. Sujit, S., et al. (2023). Prioritizing Samples in Reinforcement Learning with Reducible Loss. NeurIPS.
  14. Dubey, R., et al. (2018). Investigating Human Priors for Playing Video Games. arXiv:1802.10217.