Dataset Construction Pipeline
To build CineLong-709K, a shared entity registry anchors local presence detection and captioning; sequence-level revision reconciles descriptions, followed by speech-boundary alignment.
CineLong Consistent Long-Horizon Multi-Shot Audio-Visual Generation
Abstract
Generating coherent long-form audio-visual narratives requires preserving character and scene identities while maintaining the continuity of motion and sound across shots. This calls for training data annotations that connect local events across a narrative and a memory mechanism that retrieves useful context from an expanding, imperfect generation history. To this end, we construct CineLong-709K, a long-video dataset comprising 708,794 videos totaling approximately 20k hours, with individual sequences lasting up to 148 seconds. Its global-to-local annotations link persistent identities to local events, dialogue, and cross-chunk dependencies. Building on this dataset, we propose CineLong, a framework for long-horizon audio-visual generation that combines long-term memory for recalling characters, voices, and scenes with short-term memory for continuing ongoing actions and speech. At inference, a Goal-Driven Memory Agent (GDMA) uses the target prompt to select relevant historical chunks and the most useful keyframe within each of their shots, retaining the associated audio as context. To improve robustness, Audio-Visual Error Recycling (AVER) introduces model-derived visual and acoustic errors into memory during training. On CineLong-Bench, CineLong outperforms existing baselines in subject and background recall, cross-chunk audio continuity, and script adherence, while maintaining competitive visual quality.
Video Results
From Home to War—and Home Again
CineLong (Ours)
Opening sequence · 00:00–00:23.250. Chunks 00 and 01 are shown together, with shot timings adjusted for the trimmed video.
Chunk Prompt
Method Comparison
Method
Overall Pipeline
The generator conditions on the target prompt, long-term memory (LTM) of selected historical chunks, and short-term memory (STM) from the preceding chunk; LTM is randomly sampled in training and selected by the Goal-Driven Memory Agent (GDMA) at inference. GDMA selects chunks and per-shot keyframes according to the target prompt, together with their audio. Timestamp-Aware RoPE (TA-RoPE) positions memory and target tokens according to their timestamps, distinguishing distant evidence from recent context. During training, Audio-Visual Error Recycling (AVER) replays audio-visual prediction errors into STM.
