VideoMemory: Toward Consistent Video Generation via Memory Integration

Jinsong Zhou1,3*, Yihua Du1*, Xinli Xu1*†, Luozhou Wang1, Zijie Zhuang1, Yehang Zhang1, Shuaibo Li1, Xiaojun Hu3, Bolan Su3, Ying-Cong Chen1,2‡
1HKUST(GZ)   2HKUST   3ByteDance
*Equal contribution   †Project leader   ‡Corresponding author
arXiv Code
Teaser

From a single story prompt, VideoMemory generates coherent multi-shot videos using dynamic Character, Prop, and Background Memory Banks. This ensures exceptional entity consistency, as seen with the feather prop remaining perfectly stable across distant shots (e.g., 2, 10, and 12) despite significant scene and viewpoint variations.

Please enable sound for the best experience

The first half of this video illustrates the working mechanism of VideoMemory, while the second half showcases a complete case result.

Abstract

Maintaining consistent characters, props, and environments across multiple shots is a central challenge in narrative video generation. Existing models can produce high-quality short clips but often fail to preserve entity identity and appearance when scenes change or when entities reappear after long temporal gaps. We present VideoMemory, an entity-centric framework that integrates narrative planning with visual generation through a Dynamic Memory Bank. Given a structured script, a multi-agent system decomposes the narrative into shots, retrieves entity representations from memory, and synthesizes keyframes and videos conditioned on these retrieved states. The Dynamic Memory Bank stores explicit visual and semantic descriptors for characters, props, and backgrounds, and is updated after each shot to reflect story-driven changes while preserving identity. This retrieval–update mechanism enables consistent portrayal of entities across distant shots and supports coherent long-form generation. To evaluate this setting, we construct a 54-case multi-shot consistency benchmark covering character-, prop-, and background-persistent scenarios. Extensive experiments show that VideoMemory achieves strong entity-level coherence and high perceptual quality across diverse narrative sequences.

Contributions

  • We introduce VideoMemory, an entity-centric framework for script-to-multi-shot video generation built around a Dynamic Memory Bank that explicitly tracks and updates the visual and semantic states of narrative entities.
  • We develop a retrieval–and-update memory mechanism with dedicated slots for characters, props, and environments, enabling long-range entity consistency while allowing story-guided changes in their visual state.
  • We construct a multi-shot consistency benchmark with 54 story-driven cases covering character-, prop-, and background-persistent scenarios, providing the first structured evaluation protocol for long-range entity coherence in narrative videos.

Method & Pipeline

VideoMemory Pipeline

The framework of the proposed VideoMemory. Starting from a script synopsis, our system plans shot-level descriptions, interacts with a Dynamic Memory Bank to retrieve or create entity references, generates keyframes, and finally synthesizes a coherent multi-shot video.

Algorithm: VideoMemory Generation Pipeline

Input: Script synopsis S
Output: Multi-shot video V̂ = {V̂i}i=1N
1: {Ci}i=1N ← StoryboardAgent(S)
2: Initialize memory banks Mchar, Mprop, Mbg ← ∅
3: for shot i = 1 to N do ▷ Process shots sequentially
4: {(eij, aij, cij)}j ← MemoryAgentAnalyze(Ci)
5: for each (eij, aij, cij) do
6: (Iijref, M) ← RetrieveOrGenerate(eij, aij, cij, M)
7: end for
8: Iikey ← VisualizationAgentKeyframe(Ci, {Iijref}j)
9: V̂i ← I2VModel(Iikey, Ci)
10: end for
11: V̂ ← {V̂i}i=1N
Notation:
S: script synopsis
N: number of shots
V̂: generated multi-shot video
V̂i: video for shot i
Ci: caption for shot i
Mchar, Mprop, Mbg: memory banks
eij: name of entity j in shot i
aij: attributes of entity j in shot i
cij: entity category of entity j in shot i
Iijref: reference image for entity j in shot i
Iikey: keyframe for shot i

Qualitative Results

Qualitative Results

Qualitative comparison demonstrating superior entity consistency. Across all three subclasses (Character, Prop, Background), VideoMemory (bottom row) maintains remarkable stability where baselines fail. Note how baselines exhibit severe identity drift—changing a character's appearance (left), morphing a red kite into other objects (middle), and altering a garage's layout (right). In contrast, our method preserves the identity of all entities across distant shots, a direct result of our explicit memory management.

Quantitative Results

Multi-shot consistency results. We evaluate character, prop, and background consistency using DINOv2 similarity. Our method achieves superior performance across all metrics, especially as the number of shots increases. Best and second-best scores are marked in bold and underlined, respectively.

Character Consistency↑ Prop Consistency↑ Background Consistency↑
Shot Number 4 8 12 Avg. 4 8 12 Avg. 4 8 12 Avg.
Wan2.2 0.34 - - - 0.48 - - - 0.25 - - -
EchoShot 0.45 0.44 - - 0.59 0.51 - - 0.54 0.37 - -
IC-LoRA+Wan2.2 0.42 0.55 0.43 0.47 0.50 0.44 0.34 0.43 0.31 0.33 0.29 0.31
StoryDiffusion+Wan2.2 0.53 0.62 0.46 0.54 0.43 0.47 0.52 0.47 0.51 0.40 0.36 0.42
VGoT+Wan2.2 0.59 0.53 0.60 0.57 0.48 0.22 0.24 0.31 0.53 0.36 0.47 0.45
VideoMemory (Ours) 0.61 0.65 0.64 0.63 0.69 0.50 0.55 0.58 0.71 0.72 0.73 0.72

Video Showcase

Multi-shot video generation results demonstrating consistent entity preservation across diverse narratives.

Please enable sound for the best experience

BibTeX

@article{zhou2025videomemory,
  title={VideoMemory: Toward Consistent Video Generation via Memory Integration},
  author={Zhou, Jinsong and Du, Yihua and Xu, Xinli and Wang, Luozhou and Zhuang, Zijie and Zhang, Yehang and Li, Shuaibo and Hu, Xiaojun and Su, Bolan and Chen, Ying-cong},
  journal={arXiv preprint},
  year={2025}
}