SolarWM H3 + Memory

Turn around, and the world has changed

A world model needs to remember the world it has generated. When the camera turns a corner and comes back, or leaves a room and re-enters, buildings, furniture, and objects should stay as they were. Streaming generation has a finite context window: once an earlier view leaves it, the model can no longer access it, and a revisit means generating the place again, often differently.

In the example below, the camera moves back and forth through a gallery. Watch the painting on the side wall: it first appears at 3.5 s, and the camera returns at 20 s.

with memorywithout memory

First visit 03.50 s

with memory03.50 s

without memory03.50 s

Revisit 20.00 s

with memory20.00 s

without memory20.00 s

Compare the painting on the side wall. The revisit includes camera translation, so the views are not identical.Click a frame to enlarge.

Getting there takes less than you might think. Here we share how to train Memory into a video world model simply and efficiently:

• A simple carrier. Memory is just the frames the model has already generated. We store their latents, retrieve them by camera pose, and feed them to attention as extra tokens: no 3D representation, no new memory module.
• Simple training. We train a single Memory LoRA on SolarWM’s existing Stage1 and plug it straight into Stage2: no Memory-equipped teacher, and no rerun of distillation or autoregressive training.
• Little data. A small set of revisit data built from panoramic images and videos, used solely to teach the model to read Memory when it returns to an earlier view.

Memory design: past frames as Memory

We use memory frames as the carrier. After each chunk is generated, its latents are written to a latent bank together with their camera poses. Before generating the next chunk, five historical latents most relevant to the target camera are retrieved from the bank and inserted into the sequence as Memory tokens.

The context therefore has two parts. The latest five chunks stay in the KV cache for continuous motion and chunk-to-chunk transitions, exactly as in the original Stage2. Older history is kept only as latents in the bank and retrieved on demand. The KV cache stays bounded, and neither the architecture nor the token format changes.

Memory tokens use one-way visibility in attention. The current chunk can read Memory; Memory can read shared conditions and recent context, but not the chunk being generated; shared conditions such as text and the anchor do not read Memory. History can thus inform the current generation without leaking the training target through Memory, and retrieval for future chunks cannot flow back into the shared conditions.

Memory training

Observation: Memory is relatively decoupled from stable autoregression

Streaming world models are usually trained in stages: a bidirectional teacher, then causal and few-step generation, and finally autoregressive training against accumulated error. SolarWM’s Stage0.5, Stage1, and Stage2 follow this pipeline. Mainstream world models with Memory follow the same route: train a teacher with Memory, then distill it down to a student^{[3], [4], [5], [6], [7]}. This works well but is expensive. For a model trained without Memory, adding it means going through the whole pipeline again.

We started with an experiment: on SolarWM’s trained Stage1, freeze all existing weights and train only an extra Memory LoRA with teacher forcing. Then run this Stage1 + Memory model directly in long autoregressive generation.

The kitchen example below makes this visible. Compare 4 s with 24 s: the sink, countertop, and even the blurred, distorted furnishings on the left roughly reappear on the revisit. Memory retains artifacts the model generated earlier, too.

Stage1 + Memorymbench_009

First visit04.00 s

Revisit24.00 s

Matching camera poses: 04.00 s → 24.00 s. Click a frame to enlarge.

See the same scene with Stage2 ↓

Stage1 has not been through autoregressive training, so quality degradation over a long rollout is expected. The key point is that even after degrading, the model still reproduces what it generated earlier when it revisits a place. Memory and stable autoregressive generation are relatively decoupled. The former can be learned on its own on Stage1 with teacher forcing; Stage2 already provides the latter.

Paradigm: train Memory separately, reuse the model’s other capabilities

Building on this observation, we train only the added Memory capability. Causal generation, few-step sampling, and stable autoregressive generation come from the model’s existing training. An independent LoRA learns Memory; the original model supplies the other capabilities. Adding Memory therefore does not require rerunning the full generation training pipeline.

In SolarWM, we implement this by freezing the H3 backbone and its trained Stage1 LoRA, then training a new Memory LoRA with teacher forcing. At inference, we replace the Stage1 LoRA with the trained Stage2 LoRA and keep the Memory LoRA unchanged. The two LoRAs have independent parameters and are added to the same H3 backbone. Although the Memory LoRA has never been trained on Stage2, it combines directly with it, reusing Stage2’s few-step generation and robustness to accumulated error.

Now return to the same kitchen with Stage2 + Memory. At the same 4 s and 24 s timestamps, the sofa, wall art, and kitchen countertop retain clear, recognizable structure. This illustrates the combination we want to reuse: Stage2’s generation stability with Memory learned on Stage1.

Stage2 + Memorymbench_009

First visit04.00 s

Revisit24.00 s

Matching camera poses: 04.00 s → 24.00 s. Click a frame to enlarge.

↑ Back to the Stage1 result

The two videos use different Memory training snapshots and illustrate generation behavior in the same scene.

Causal generation, few-step sampling, and handling of accumulated error are all reused from SolarWM’s existing training. The only added cost is one Memory LoRA run: no Memory-equipped teacher, and no rerun of Stage2 distillation or autoregressive training. This simple procedure alone raises SolarWM H3’s generation persistence on Orbia by 5.22 points; see the results.

Data: a small set of high-quality revisits

The model already knows how to generate; the new data only has to teach it to read Memory on a revisit. The core unit is therefore a revisit pair: observe a scene, let it leave the recent context, then return to the same or a nearby view. Only then do historical frames carry information directly useful for generation.

We build these revisits from panoramic data, which is another key design choice. Collecting revisit data with an ordinary camera means planning the departure and return route at capture time and making sure the return view lines up with the earlier one; that is costly and hard to scale. Panoramic data is different: it records the full view around a location, so camera trajectories can be designed freely after capture, and large amounts of existing panoramic images and videos can be reused directly. Concretely, panoramic images have a fixed camera center, so we can freely design turn-away, departure, and return trajectories, yielding multiple controlled revisits per scene; moving panoramic videos keep real camera translation, and viewing trajectories on top add revisits across changing viewpoints.

Below are six examples from the training data: the first three come from panoramic images, the last three from moving panoramic videos. In each, the left frame is what the camera sees when it first reaches a place, and the right frame is what it sees when it comes back after turning away and leaving the recent context; in training, the former serves as Memory and the latter as the revisit target. The curve underneath tracks how far the view has drifted from the first visit: it starts at 0°, rises, and falls back near 0°, which is one full departure and return.

Panoramic imagesFoundry workshopVictorian marketSeaside at nightMoving panoramic videosStone plazaExhibition hallFarmland

① First visit20.21 s ↗

↗

back after
32.4 s
away

② Revisit52.62 s ↗

↗

180°0°

away 32.4 s

①②

0 s60 s

Vertical axis: angle between the current view and the first-visit view.

We keep frames, cameras, and text aligned and filter out low-quality samples. Memory comes strictly from history before the query, and trajectories from the same source scene stay in the same split to prevent scene leakage between training and validation. During training, revisit data is mixed 1:1 with original high-quality H3 data.

Results

Quantitative evaluation: Orbia-Bench

We evaluate SolarWM H3 + Memory on the 720 Orbia cases against the original SolarWM H3. The overall score rises from 76.32 to 77.46. The largest gain is in generation persistence (70.28 → 75.50, +5.22), followed by 3D consistency (+2.15) and input preservation (+1.73); control, quality, and stability also improve slightly. Memory improves consistency without hurting generation quality.

Qualitative comparison: revisit consistency

All four comparisons use the same input, seed 42, and four sampling steps. SolarWM H3 + Memory is on the left, original SolarWM on the right; the videos are the two original outputs placed side by side at 1344 × 768 each. Frames from the first pass and the revisit are shown below each video.

Street · turn and returnPlaza · orbit and revisitDungeon · peek and retractRiver · turn and return

with memory SolarWM H3 + Memorywithout memory SolarWM H3

First visit11.25 s ↗

with memory11.25 s

↗

without memory11.25 s

↗

Revisit49.25 s ↗

with memory49.25 s

↗

without memory49.25 s

↗

Revisit frames follow Orbia annotations. Click a timestamp to seek; click a frame to enlarge.

References

1. [1]SolarWM. Open Data and Scalable Training for Long-Horizon Video World Models.
2. [2]MiniMax H3. Official model and inference implementation.
3. [3]WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
4. [4]Memorizon: Training World Models Beyond Their Context Window.
5. [5]Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory.
6. [6]ReWorld: An Interactive World Model with Long-Horizon Memory.
7. [7]WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon.

Citation

Citations for the underlying SolarWM paper and this blog:

@misc{huang2026solarwmopendatascalable,
  title         = {SolarWM: Open Data and Scalable Training for
                   Long-Horizon Video World Models},
  author        = {Junchao Huang and Guian Fang and Shengju Qian and
                   Xianghao Kong and Zhuoran Zhao and Wei Huang and
                   Yihua Du and Zixin Zhang and Justin Cui and
                   Yuchao Gu and Yukang Chen and Xinting Hu and
                   Tianyu He and Shaoshuai Shi and Zhuotao Tian and
                   Xin Wang and Mike Zheng Shou and Li Jiang},
  year          = {2026},
  eprint        = {2609.02886},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.02886}
}

@misc{zhang2026solarwmmemory,
  title  = {{SolarWM H3 + Memory}: Training Memory For Video World Model Is Easier Than You Think},
  author = {Zhang, Zixin and Huang, Junchao and Du, Yihua and
            Lu, Dongyue and Zhao, Zhuoran and Kong, Xianghao and
            Chen, Bohong and Fang, Guian and Yan, Dongyu and Liu, Yaoyang and Qian, Shengju},
  year   = {2026},
  month  = nov,
  note   = {SolarX-Team technical blog},
  url    = {https://solarx-team.github.io/SolarWM-H3-Memory/}
}