LEAPS: Testing Long-Context AI on Pattern Learning and State Tracking

Long-context language models now drive agents that work over very long trajectories: improving a codebase, running research experiments, or forecasting from lengthy histories. These tasks draw on three distinct capabilities. Recall means finding a fact present in the context. State tracking means computing an outcome by applying a series of transformations to a specified state. Pattern learning means making a prediction by sampling from a distribution inferred from many observations.

Most existing long-context benchmarks test only the first capability. A new benchmark called LEAPS (Long-context Evaluation of Adaptive Pattern learning and State tracking) targets the other two. It contains 12 novel datasets spanning agentic workflows, behavioral modeling, and forecasting of market and world events. By construction it resists data contamination, extends to millions of input tokens, and uses reliable metrics.

The researchers evaluated 30 API and open models at input lengths from 8K to 512K tokens. The results separate models that look identical on recall: models scoring near perfectly on recall tasks vary significantly in state tracking and pattern learning, and both abilities degrade much more sharply as input length grows. That holds even for pattern learning, where more observations should theoretically make the task easier. Thinking models often underperform their base counterparts, exhausting their generation budget or oversimplifying the task.

The practical takeaway: strong recall scores do not certify the capabilities that long-horizon agents actually need, and longer contexts can actively hurt performance on the hardest long-context skills.

This article was written by Yethikrishna R, founder and lead engineer of Mynd Labs and coordinator and founding member of Compyle (YC F25), based in Thrissur, India.

Source: https://arxiv.org/abs/2610.25919