In 2024 the winning move was a well-crafted prompt. In 2026 the winning move is a well-composed context — everything the model sees, in the right order, at the right size.
The shift matters because prompts don't survive production. Context does. The systems shipping today aren't 'ChatGPT with a better system message.' They're four pillars stacked together: system, task, memory, retrieval — and a discipline for keeping each one small, fresh, and evaluable.
This handbook is the mental model, six techniques that pay for themselves in a week, the stack I actually reach for, and the failure modes I've learned to catch early.
Every production context stack is four things stacked together. Name them, budget them, evaluate them separately.
S
01System
The persistent frame every turn inherits.
Model, temperature, role, guardrails, output schema. Set once, referenced always. The cheapest turn is the one you don't have to redescribe.
≈ 200 – 800 tokens
T
02Task
The user's request, structured.
The message plus everything the model needs to plan a single next action — intent, constraints, examples, expected output shape. Most 'prompt eng' work moves here.
≈ 500 – 3000 tokens
M
03Memory
What the model must recall across turns.
Sliding-window history, summarized episodes, entity facts, user preferences. The art is compression — keep what changes decisions, drop what changes nothing.
≈ 1000 – 8000 tokens
R
04Retrieval
The knowledge you fetched for this turn.
Vector search, keyword search, tool outputs, MCP fetches. The 2026 stack is hybrid + reranked. Pure cosine is a 2023 anti-pattern in production.
≈ 2000 – 20000 tokens
The patterns I reach for on day one of a new build. Each pays back its implementation cost inside a week of real traffic.
01cost + quality
Progressive Context Loading
WhatLoad small, escalate only if the model asks. A router LLM decides which sub-context to fetch next; the main model never sees what it didn't need.
WhenLong documents · multi-collection RAG · agent flows with 5+ tools.
WinsCuts average tokens 40–70%. Answer quality holds flat or improves — the model isn't forced to read what doesn't help.
02retrieval
Hybrid Retrieval + Cross-Encoder Rerank
WhatBM25 for exact matches + dense vectors for semantic matches → top-50 → cross-encoder rerank → top-5. Never trust cosine similarity alone.
WhenAny RAG serving real users. Especially: legal, medical, finance, code.
WinsMean-reciprocal-rank jumps 15–30% versus single-index. Cheapest quality win in retrieval.
03memory
Sliding Window + Rolling Summary
WhatKeep last N turns verbatim. Replace older turns with a summary regenerated every K turns. Store entities in a separate structured field.
WhenChat agents, coding copilots, anything with 10+ turn conversations.
WinsFixed context budget regardless of session length. No 'forgot my name' failures at turn 30.
04tools
MCP as Context, Not Command
WhatModel Context Protocol tools are context providers first, actions second. Register read-only tools eagerly; put write-tools behind explicit confirmation.
WhenAny agent shipped to production in 2026.
WinsFewer wrong tool calls. Cleaner audit trail. Same provider works across every host that speaks MCP.
05compression
Structured Summarization
WhatDon't ask the model to summarize freeform. Ask for a JSON with the exact fields the next turn will consume. The schema is the contract.
WhenLong threads, meeting transcripts, doc pipelines, multi-agent handoffs.
WinsDeterministic downstream. Compression ratio holds even when the source is chaos.
06evals
Context Quality Evals
WhatScore every context bundle before generation — relevance, coverage, redundancy — with an LLM-as-judge. Track drift over time in a dashboard.
WhenThe moment you cross ~100 daily runs. Sooner if you're regulated.
WinsYou catch a bad retrieval config in a graph, not in a customer email.
Not exhaustive — opinionated. Everything below has shipped to production and stayed shipped.
Model
Claude Sonnet 4.6 · Opus 4.7
Best in-context reasoning; long context that actually works
Router
Haiku 4.5 or small fine-tune
Fast, cheap, deterministic routing at the edge
Vector DB
pgvector (Supabase) · Turbopuffer
Postgres-native beats bespoke DBs under ~10M vectors
Reranker
Cohere Rerank v3 · bge-reranker-v2
Cross-encoder that fits inside a 100ms budget
Observability
Langfuse · Braintrust
Traces + evals in one pane. Open-source options exist.
MCP hosts
Claude Desktop, Cursor, custom SDK
One tool implementation, every host that speaks MCP
Every one of these has cost a team I know real money or a real customer. Catch them early.
Stuffing the entire knowledge base
Attention degrades past ~40k tokens. You pay for tokens the model half-reads.
Pure cosine similarity retrieval
Semantic-only misses exact-match terms — drug names, error codes, IDs. Hybrid is table stakes.
No memory schema
"The model just remembers." Translation: the model just hallucinates. Give it a JSON to write into.
Prompts as art
If you can't diff, version, and eval your prompts, they're not engineering — they're wishes.
Three papers to internalize. Three talks to sharpen instinct. Three builds to earn the reps.
