Abstract:Large language models (LLMs) have revolutionized video-based computer vision applications, including action recognition, anomaly detection, and video summarization. Videos inherently pose unique challenges, combining spatial complexity with temporal dynamics that are absent in static images or textual data. Current approaches to video understanding with LLMs often rely on pretrained video encoders to extract spatiotemporal features and text encoders to capture semantic meaning. These representations are integrated within LLM frameworks, enabling multimodal reasoning across diverse video tasks. However, the critical question persists: Can LLMs truly understand the concept of time, and how effectively can they reason about temporal relationships in videos? This work critically examines the role of LLMs in video processing, with a specific focus on their temporal reasoning capabilities. We identify key limitations in the interaction between LLMs and pretrained encoders, revealing gaps in their ability to model long-term dependencies and abstract temporal concepts such as causality and event progression. Furthermore, we analyze challenges posed by existing video datasets, including biases, lack of temporal annotations, and domain-specific limitations that constrain the temporal understanding of LLMs. To address these gaps, we explore promising future directions, including the co-evolution of LLMs and encoders, the development of enriched datasets with explicit temporal labels, and innovative architectures for integrating spatial, temporal, and semantic reasoning. By addressing these challenges, we aim to advance the temporal comprehension of LLMs, unlocking their full potential in video analysis and beyond.

Language Models Still Struggle to Zero-shot Reason about Time Series

Implicit Reasoning in Deep Time Series Forecasting

A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization

Towards Time Series Reasoning with LLMs

Are Language Models Actually Useful for Time Series Forecasting?

Do Language Models Understand Time?

Large Language Models Are Zero-Shot Time Series Forecasters

Large language models can be zero-shot anomaly detectors for time series?

Are Large Language Models Useful for Time Series Data Analysis?

Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark

TimeSeriesExam: A time series understanding exam

Position: What Can Large Language Models Tell Us about Time Series Analysis

Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Timo: Towards Better Temporal Reasoning for Language Models

TableTime: Reformulating Time Series Classification as Zero-Shot Table Understanding via Large Language Models

Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks

ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning

Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities

A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Model

Can LLMs Understand Time Series Anomalies?