Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding

Holistic understanding of long-form video remains a challenge for vision-language models (VLMs). Unfortunately, current benchmarks cannot easily capture this limitation, since they mostly focus on ``needle-in-a-haystack'' details, rewarding context-insensitive retrieval over deep comprehension. Others rely on large-scale, semi-automatically generated questions (often produced by language models themselves) that are easier for models to answer but fail to reflect genuine understanding. In this paper, we address this gap by introducing MF$^2$, a new benchmark to evaluate how well models are able to comprehend, consolidate, and recall key narrative information from full-length movies (**50-170 minutes long**), requiring integration of **both** visual and language modalities. MF$^2$ includes over 50 full-length, **open-licensed** movies, each paired with manually constructed sets of claim pairs---one true (*fact*) and one plausible but false (*fib*), totalling over 850 pairs. These claims target core narrative elements such as character motivations and emotions, causal chains, and event order, and refer to **memorable moments** that humans can recall without rewatching the movie. Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance. Despite the relative ease of the task for humans who can effectively retain and reason over critical narrative information, current VLMs lack this ability and thus struggle.