Context Rot Quantified

Context rot is a well-known phenomenon where LLM performance degrades as context size increases. There has been some prior work in trying to quantify the performance loss but it’s been focused on simpler tasks, like needle in a haystack. We ran a study to quantify the performance degradation in challenging agentic coding tasks. We found that GPT-5.6 Sol degrades significantly whether or not the context is related to the task, while Claude Opus 5 is only affected when it is.

For performance evaluation, we use a subset of 20 tasks from terminal-bench v4. We run the eval with prefill context at 0k, 250k, and 500k tokens. Both models offer a 1M context window [1, 2] but on the harness level, Claude Code defaults to 1M context but Codex limits to 400k context by default. For this experiment, we configure Codex’s context window to 1M with the flags Tibo suggests.

Unrelated Context

In this setting, we generated unrelated prefill context by doing multi-turn Q&A with the model on the Apache Spark project, which is not related to any of the tasks that we tested the models on.

// ===== sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala =====
```scala
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
…

Each model generated its own prefill context so that we were not evaluating it on off-policy prefills. We generated prefills for each model at 250k and 500k tokens and used them across all tasks.

We see a clear drop in performance for GPT 5.6 Sol at 250k context (31% pass rate to 24% pass rate) that persists at 500k context while the performance decline for Claude Opus 5 is very limited and within evaluation noise.

Related Context

We then asked if context-rot depends on whether the earlier text is related or unrelated to the task at hand. In this setting we created the prefill with multi-turn Q&A related to the task’s own environment and asked questions about tools, library, and code the task depends on. Unlike the Spark prefill, these prefills are task-specific. So one prefill per task, per model, per context size. As before, the model being evaluated also generated the prefill, to better simulate a real user-model interaction and to avoid off-policy prefills. In this setting we made sure that the harness only took read-only action during the Q&A phase and rejected transcripts that took any write actions.

GPT-5.6 Sol loses the same amount of performance regardless of whether the context is about the task at hand or about an unrelated repo. Claude Opus 5, which was flat under unrelated context, now drops from 50% pass rate to 40% pass rate, when the context is related to task. Opus 5 is good at ignoring irrelevant context, but relevant context degrades both models.

Different Tokenizers

An important mitigating factor for the above result is that GPT and Claude models use different tokenizers, so measuring loss on fixed token thresholds would bias towards the model that has a tokenizer that encodes the same information in more tokens. Based on our data corpus, we find that 1 GPT token = ~1.7 Claude tokens or 1 Claude token = ~0.58 GPT tokens.

To account for this, we re-plot both charts with the context in MBs of UTF-8 text. For unrelated context this mitigates the result to some extent but the drop in performance is still significant in this view: at over 1 MB of data Claude’s performance degradation is minimal while GPT shows a significant regression. For related context this view doesn't change much as both models degrade, and per byte of context Opus 5 degrades at least as fast as GPT-5.6 Sol.

Momentum-induced Collapse

We looked at failure transcripts to understand why the failure rate increased at higher prefill context lengths and found an interesting failure mode for GPT 5.6 Sol that starts showing up at 250k and 500k context. In this failure mode, the model starts out on the terminal-bench task by making some tool calls, gets tool call results, and then snaps back to the earlier Q&A mode. It answers a question from earlier in the context and ends its turn. We observed this behavior in both related and unrelated mode.

Address = AddressSpace identity + byte offset

Program
├── AddressFactory → all valid spaces
└── Memory (an AddressSetView)
    └── MemoryBlock → one contiguous range and its storage

We believe this behavior could be attributed to the momentum induced by the several Q&A turns that precede the multi-turn agentic task. The model potentially has a high propensity to attend to previous assistant turns and isn’t able to adapt to a sudden change in task type at longer context lengths.

We never observe this failure mode for Claude Opus 5 across all prefill context lengths, and see it consistently for GPT 5.6 Sol at 250k context and above.

Conclusion

There is a quantifiable degradation in task performance as context size increases. Both models degrade with context, but differently. GPT-5.6 Sol loses about 25% of its solved tasks by 250k tokens of context regardless of whether the context was related to or unrelated to the task. We also see a catastrophic failures in GPT 5.6-Sol where it reverts to answering questions asked in prior turns at 250k and higher context. Claude Opus 5 is very good at ignoring unrelated context but loses 15% of solved tasks at 250k tokens of context relevant to the task at hand. In some preliminary findings, we do see that compaction can recover some of the performance degradation loss that we observe at longer context. Given these learnings, the takeaway would be to use new sessions or compact whenever possible to maintain model performance, especially when working with GPT-5.6 or if the context is related to what you are going to work on.