GitHub Copilot Refactors Context Pipeline: Same Tokens, More Work

Yesterday, GitHub published a technical post on its official blog with a straightforward title: “Getting more from each token.” The topic is also simple — Copilot has revamped its core context handling and model routing pipeline so that in each session, a higher proportion of tokens are spent on genuinely “useful work.”

This isn’t a product feature update, but an engineering-level efficiency improvement. However, looking at it in the context of June this year, the significance changes. On June 1, GitHub switched Copilot’s individual plan from “per request” billing to “per token” billing. Reddit exploded — some heavy users calculated their bills would jump from $44 to $750. Against this backdrop, GitHub talking about “how we help you save tokens” has an obvious subtext: the subsidy era is over, the platform itself has to tighten the belt, and it’s also teaching users how to spend wisely.

How Many Tokens in a Session Are Actually “Wasted”?

Let’s start with the conclusion. GitHub’s own data shows that with the new context pipeline, average token consumption for similar tasks decreased by 20%–35% (with significant variation across models and task types), while the end-to-end success rate for agent tasks (Agent Mode) actually increased by several percentage points.

How did they save these numbers? First we need to understand where Copilot’s tokens are spent in an agent session. In a typical agent task — say “fix a bug in an authentication module” — the input tokens usually include:

• System prompt
• Tool schemas (e.g., all tool signatures registered via MCP servers)
• Custom instructions (AGENTS.md, .github/copilot-instructions.md)
• Content of currently open editor tabs
• User-added files or selections
• Complete history of previous dialogue rounds (including input/output of tool calls)
• The current round’s user message

All this is repeatedly sent with each LLM call in every round. After multiple rounds, in later stages, simply “replaying history” can take up more than 60% of input tokens. This is why many developers complain that running a slightly complex task burns through thousands of credits in no time.

GitHub made changes mainly in four areas.

1. Context Pruning: From “Send Everything” to “Load on Demand”

Previously, Copilot Agent tended to send all visible context to the model every round, with the simple reasoning — the more the model sees, the fewer hallucinations. But in long sessions, this logic breaks down quickly.

The new pipeline introduces a layered pruning strategy:

• Tool schemas loaded on demand: Previously, all registered MCP tool signatures were crammed into the system message; now they are filtered dynamically according to current intent. For example, if you’re writing tests, file system tool descriptions are kept, while deployment tool descriptions are hidden and injected only when needed.
• Summarizing past rounds: For rounds beyond a certain depth, raw outputs from tool calls are compressed into structured summaries. A grep producing 200 lines might be pared down to something like “matched 12 times in 5 files, mostly in the auth/ directory” in the next round.
• Deduplication and slicing of file contents: If the same file is read in multiple rounds, only the latest version’s changes are kept; large files are sliced by AST, sending only relevant functions.

Essentially, this automates part of what users previously had to do manually via /compact or /clear. The trade-off is that pruning might cause the model to miss critical info, so GitHub also added a “pullback mechanism” — if the model’s generation shows it needs pruned-away context (e.g., repeatedly trying to read summarized content), the pipeline automatically reloads the original info.

2. Model Routing: Small Models for Small Tasks

This is something GitHub has been talking about a lot this year, and it’s a direction repeatedly emphasized by Coinbase’s Brian Armstrong — in the next 12–18 months, 80% of AI workloads will shift to models 99% cheaper.

Copilot’s new routing mechanism breaks an agent task into multiple steps, then dynamically selects a model according to each step’s difficulty:

• Intent classification, file retrieval, simple grep/find → lightweight models (GPT-4.1 mini or equivalent low-cost models)
• Code comprehension, cross-file refactoring, bug localization → mid-tier models (Claude Sonnet series)
• Complex planning, architectural decisions, long-chain reasoning → frontier models (Claude Opus, GPT-5 high reasoning mode)

The key point is that this routing is transparent to the user. Selecting Sonnet in VS Code doesn’t mean every tool call uses Sonnet — the “concierge” behind the scenes hands easy tasks to small models and only calls your chosen main model when real reasoning is needed.

This concept is already accepted industry-wide. Cursor, Cline, Windsurf are doing similar things. GitHub’s advantage is scale — they process over 71 million Actions Jobs daily, and Copilot’s backend request volume is even bigger, giving them routing model training data that others can’t match.

3. Optimizing Cache Hit Rate

This one is easy to overlook but delivers instant results.

Major model vendors (Anthropic, OpenAI, Google) now support Prompt Caching: resending the same prefix within 5–10 minutes costs only 10%–25% of the original price. Copilot takes this to the extreme:

• In multi-round sessions, keep the system prompt and tool schemas prefix stable, placing dynamic parts (current user message) at the end
• Previously, switching between models would invalidate caches; the new pipeline tries to maintain the same model continuity at the routing layer
• Reuse the same context template across subtasks during multi-step reasoning internally

GitHub’s blog data shows cache hit rates rose from about 40% before optimization to over 70%. For heavy users with long sessions, this alone could halve the bill.

4. The Agent’s Own “Discipline”

Another aspect is behavioral optimization. The new agent is more inclined to:

• Read related files in batch rather than one at a time and then decide the next
• Plan before writing code to avoid repeated “fix A, break B” cycles
• When modifying multiple files, prefer patch tools over rewriting whole files

These behavior improvements come from RLHF training on a large number of real sessions. In short, they used accumulated platform data to train an agent “personality” that saves tokens.

What This Optimization Means for Developers

To be frank, this update is technically solid, but it doesn’t change the overall trend of “AI coding getting more expensive.”

After the June billing change, the industry consensus is clear: subsidy era over, bills reflect true costs. GitHub admits it had been silently covering inference costs — charging the same for a simple question and an autonomous task running for hours — a model that’s unsustainable.

So for developers, how to use it? Here are a few practical tips I think have real value:

1. Don’t enable MCP tool sets indiscriminately. Each MCP tool’s description is a permanent token cost. GitHub’s docs specifically mention this: enable tools per workflow, not all at once.
2. Write copilot-instructions.md carefully. Clarify project structure, naming conventions, and output expectations so the agent doesn’t need to read 20 files just to “figure out what your repo looks like.”
3. Use plan mode wisely. /plan in CLI, Plan Mode in VS Code — make the agent produce a plan before acting to significantly reduce repeated changes.
4. Start new sessions between stages. When moving from design to implementation, start a fresh session. Carrying exploratory context from earlier stages only pollutes the agent’s judgment.
5. Monitor /usage and /chronicle tips. These CLI commands really save money; the former shows real-time consumption, the latter analyzes past sessions to offer optimization suggestions.

The Industry’s Other Shoe

Worth noting — GitHub’s optimization direction aligns with Hugging Face CEO Clement Delangue’s repeated talks about the “multi-model future” — but the path is completely different.

Delangue’s approach is localization, with open-source small models handling most tasks and frontier APIs as backup. Stanford data shows open-source small model accuracy in real dialogue and reasoning rose from 23.2% in 2023 to 71.3%. China’s DeepSeek V4 open-source model can now match Claude Opus in programming benchmarks at one-thirtieth the price.

GitHub’s approach is cloud routing — same goal (having 80% of work fall to cheaper models), but keeping it within their own platform instead of pushing users to local setups.

Which path wins is still unclear. But one thing is certain: the era of a single frontier model handling everything is over. For developers frequently switching between models, the value of a unified access layer is rising. What we’ve been doing at OpenAI Hub is exactly this — a single key connects GPT, Claude, Gemini, DeepSeek, etc., all supporting the OpenAI format, domestic direct connection, eliminating the need to maintain separate accounts, quotas, and networking for each vendor. In the new normal of per-token billing, having the right to choose your model is more important than anything.

Final Words

Copilot’s update is technically robust and timely in a delicate way. It offers heavy users a breather but doesn’t change the underlying reality that “compute is getting more expensive.”

Arun Chandrasekaran at Gartner put it bluntly: Copilot is just the beginning; as agent workflows drive up compute consumption, more enterprises will shift to volume-based billing. Once subsidies retreat, energy and compute, not model capabilities, become the new bottlenecks.

So in this “save tokens” battle, platform, model vendors, and developers — no one escapes.

References

• GitHub Blog: Getting more from each token — GitHub’s official details on Copilot’s context handling and model routing optimizations
• GitHub Docs: Improving agent quality to optimize AI usage — Best practices from GitHub on reducing Copilot token consumption
• Reddit Discussion: Copilot token usage increase after update — Real feedback from the developer community on consumption changes after billing switch
• Hugging Face: Open-source small model performance tracking — Reference for the open-source model ecosystem in the multi-model future trend