Reference · updated daily

AI Architecture Core

The layers every production AI system is built from, how a request moves through them, and the trade-offs that decide cost, latency and quality. Read top-down for the system view; jump to a layer from the stack.

Daily note

Each day adds one short note on a core architecture idea. Notes appear here once the first one is posted.

Layer 1

Application & experience

Where people meet the system: chat surfaces, copilots inside existing tools, batch jobs, and APIs other services call. This layer owns the contract with the user: what goes in, what comes back, how long they wait, and what happens when the model is wrong.

Components

  • Chat UI, embedded copilots, batch pipelines
  • API gateway, auth, rate limiting, quotas
  • Streaming transport (SSE / WebSocket)
  • Feedback capture (thumbs, edits, escalations)

Design decisions

  • Stream tokens or return a complete answer
  • Synchronous chat vs async job with a callback
  • Human-in-the-loop checkpoints for high-impact actions
  • How citations and confidence are shown

Inside layer 1 ↓ the latency contract, designing for the wrong answer, and feedback signals.

Inside layer 1

The contract with the user: wait, trust and the wrong answer

The model decides what an answer says. Layer 1 decides how long the user waits for it, how far they trust it, and what happens when it is wrong. Those choices shape adoption more than a few points of benchmark accuracy, and none of them live in the weights.

1 · Four ways people meet the system

SurfaceWho starts itTime budgetWhen it is wrongDesign priority
Chat assistantThe user asksFirst words within about a second; the full answer can stream for longerThe user reads it and judgesStreaming, citations, easy follow-up
Embedded copilotThe user, inside a tool they already useAround a second, or the suggestion is ignoredIgnored or edited before useSmall outputs, accept or reject in one action
Background agentA trigger, schedule or delegated taskMinutes to hoursIt may act before anyone looksProgress, approvals, audit trail, undo
API / serviceAnother systemThe caller's SLO, for the whole responseBad output flows downstream silentlySchemas, validation, retries, idempotency

2 · The latency contract: streaming changes which number users feel

1 s · flow holds 10 s · attention limit Full answer no streaming blank screen 10.5 s Streamed same answer text arrives, faster than anyone reads 0.5 s 024681012 s
~238words / min, silent reading
≈
~5tokens / s a person reads
vs
40tokens / s the model writes
=
~7.5×faster than reading

Example: 0.5 s to first token, then 400 tokens at 40 tokens/s. Unstreamed, the user feels the total time and crosses the 10-second attention limit. Streamed, they feel only the time to first token, and the text outruns their reading. That is why chat surfaces optimise TTFT. Agents, API callers and structured outputs read the whole response before acting, so for them total latency is still the contract. Past about 10 seconds, switch to an async job with visible progress and a notification.

3 · Design for the wrong answer: autonomy is an interface choice

Easy to undoHard to undo High impact
High impact · easy to undoApprove the planUpdate 400 CRM records. The user approves the plan; a snapshot allows rollback.
High impact · hard to undoA person decidesIssue a refund or email every customer. The AI drafts and recommends; a named person approves.
Low impact
Low impact · easy to undoAct, then tellTag and route a support ticket. Show what changed, with one-click undo.
Low impact · hard to undoPreview, then actPost a reply in a team channel. Show the exact message before it goes.

The same model can sit in all four boxes; what changes is how much the interface lets it do before a person looks. Pair this with the scoped tokens in Identity & policy: the interface asks for approval, the token enforces it.

4 · Every interaction is a label

SignalKindWhat it usually tells you
Thumbs, ratings, reportsExplicitA direct judgment, but few users give one, and mostly when annoyed or delighted
Copy, accept, send uneditedImplicitThe output was usable as it was
Edit before useImplicitUsable but wrong somewhere, and the diff shows exactly where. The richest eval data you will get
Regenerate or rephraseImplicitThe first answer missed the intent
Stop mid-stream or abandonImplicitToo slow, too long, or heading the wrong way
Escalate to a humanImplicitThe system could not resolve it; the case is a ready-made test

Log each signal with the trace ID of the request, so it links back to the prompt, retrieved context and model version that produced it. That is how everyday use becomes eval sets and the production monitors in Governance by design.

5 · Design rules

Make stop actually stopCancel generation on the server when the user stops or leaves, or you pay for tokens nobody reads.
Limit by tokens, not requestsOne request can be 50 tokens or 200,000. Quotas and rate limits should count tokens and cost.
Design the "I don't know"An honest abstain screen with a route to a person beats a confident wrong answer.
Show where it came fromCitations let users check an answer in seconds and learn when to trust the system.
Treat every input as untrustedUploaded files, pasted text and web pages can carry prompt injection. Screen them at the edge.
Sandbox UI you did not writeTools can now ship their own interface. Render it isolated and require approval for actions it triggers.

The last rule is new. In January 2026, MCP Apps became the first official MCP extension: a tool can return an interactive interface that hosts such as Claude, ChatGPT, VS Code and Goose render in a sandboxed iframe. AG-UI standardises the event stream between agent back ends and front ends over SSE or WebSockets. Layer 1 is no longer only the screens you build; it is also the screens your tools bring with them.

Sources: J. Nielsen, Usability Engineering (1993), response-time limits of 0.1, 1 and 10 s; M. Brysbaert, “How many words do we read per minute?”, Journal of Memory and Language (2019), 238 wpm for silent reading of non-fiction; WorkOS, “MCP Apps” (27 Jan 2026); Thoughtworks Technology Radar, AG-UI protocol (Trial, Apr 2026). Tokens assume about 0.75 English words per token.
Layer 2

Orchestration & agents

The control logic between the user and the model. It assembles the prompt, decides whether to retrieve, calls tools, loops until a goal is met, and manages what the model remembers. Most product behaviour lives here rather than in the weights.

Components

  • Prompt templates and system instructions
  • Tool / function calling with JSON schemas
  • Planner–executor loops, routers, sub-agents
  • Short-term (context) and long-term (store) memory
  • Tool protocols such as MCP for connecting systems

Patterns

  • Single call: prompt in, answer out
  • Chain: fixed sequence of calls
  • Router: classify, then send to a specialist model or prompt
  • Agent loop: reason → act (tool) → observe → repeat, with a step budget
Agent cost ≈ steps × (prompt tokens + output tokens) — context grows each step, so cost rises faster than linearly without summarisation or caching.

Inside layer 2 ↓ five patterns, the cost and reliability maths of the loop, and the context budget.

Inside layer 2

Who drives the loop, and what each turn costs

Orchestration is the code between the user and the model: it builds the prompt, calls tools and decides what happens next. Its first design choice is who picks the next step, your code or the model. Everything else in this layer — cost, reliability, how much context the model has to wade through — follows from that answer.

1 · Five patterns, from code-driven to model-driven

PatternWho picks the next stepPredictabilityCalls per taskUse it when
Single callNobody: there is no next stepHighest1Classifying, extracting, or drafting from context you already have
ChainYour code, in a fixed orderHighKnown in advanceThe recipe is stable: outline, draft, check
RouterThe model picks a branch once; code runs itHigh1 decision + 1 specialistMixed traffic where a cheaper model or specialist prompt handles most cases
Orchestrator–workersThe model plans subtasks; workers run them, often in parallelMediumDepends on the planResearch or code changes whose subtasks can't be listed up front
Agent loopThe model, every turn, until it stops or hits a budgetLowestUnknown until it finishesOpen-ended work where the path depends on what the tools return

Anthropic's Building effective agents draws the line between workflows, where code fixes the path, and agents, where the model chooses it. Each row down buys flexibility and gives up predictability and cost control, so move down only when the task needs it. The router row is where a decision model like JEV fits.

2 · An agent re-reads its whole history on every turn

0 10k 20k 30k 1 5 10 15 20 turn 20 sends 32,500 tokens agent turn fixed prefix: system prompt + tools (4,000) history: +1,500 per turn
365kinput tokens across 20 turns
=
4.6×20 separate calls (80k)
→
~74ktoken-equivalents with prompt caching

Example: a 4,000-token prefix, and each turn adds about 1,500 tokens of reasoning, tool call and tool result. Because every call re-sends everything before it, total input grows with the square of the number of turns, not in step with it. Prompt caching changes the bill, not the shape: cached reads cost about a tenth of normal input on the major APIs (less on some newer models), so this run costs roughly 80% less. The context is still 32,500 tokens long by turn 20, and the model still has to read all of it. That is panel 4. Output tokens are not counted here.

3 · Reliability compounds too

Each step succeeds5 steps10 steps20 steps
99%95%90%82%
95%77%60%36%
90%59%35%12%

If every step must succeed and failures are independent, the run succeeds at the per-step rate raised to the number of steps. A step that works 95% of the time sounds solid; chain 20 of them and the run finishes cleanly about one time in three. Real agents beat this table because they notice errors and try again — that is what the loop is for. But recovery only happens if it is designed in: tool errors the model can read, steps that are safe to retry, and checks before the steps where a wrong turn gets expensive.

4 · The context window is a budget, not a bucket

In the windowWhy it growsWhat to do
System instructionsIt shouldn't; this is the stable prefixKeep it first and unchanged between turns, so it caches
Tool definitionsEvery connected tool adds its schema to every callLoad only the tools this task needs; search for tools instead of listing hundreds
Retrieved contentWhole files and search results pasted inFetch just in time; keep references (paths, IDs) and pull content when needed
Turn historyEvery turn appends to itCompact: summarise older turns and drop stale tool output
Working notesPlans and progress the model must carry forwardWrite them to a file or memory store and re-read on demand
Sub-agent outputA helper returns its full transcriptHave sub-agents return a condensed result, not their working

Bigger windows raise the ceiling; they don't make filling them free. Chroma tested 18 models in 2025 and found accuracy falls as input grows, even on simple retrieval tasks, a pattern it named context rot. On the LongMemEval benchmark, models did markedly better with a focused prompt of about 300 tokens than with the full history of about 113,000. Anthropic calls the discipline context engineering: give the model the smallest set of high-signal tokens it needs for the next step.

5 · Design rules

Start with the least autonomy that worksA chain you can test beats an agent you can't. Add a loop when the path truly depends on what the tools return.
Give every run a budgetCap steps, tokens, wall-clock time and spend. When a cap is hit, stop and report; don't carry on silently.
Keep the prefix stableCaching matches on the prefix. A timestamp at the top of the system prompt breaks the cache on every call.
Make tool errors readableReturn "customer ID not found; try search_customers", not a stack trace. The model can only recover from what it understands.
Make steps safe to retryPut idempotency keys on writes, so a retried step doesn't send the email twice.
Trace every turnLog prompts, tool calls and results under one trace ID so a bad run can be replayed, and so governance has its evidence.
Sources: E. Schluntz & B. Zhang, “Building effective agents”, Anthropic (Dec 2024); Anthropic, “Effective context engineering for AI agents” (Sep 2025); K. Hong, A. Troynikov & J. Huber, “Context Rot: How increasing input tokens impacts LLM performance”, Chroma (Jul 2025); Claude Platform and OpenAI API prompt-caching docs (accessed Oct 2026). Cost example assumes cache writes at 1.25× and reads at 0.1× base input price.
Layer 3

Knowledge & retrieval

Retrieval-augmented generation (RAG) grounds answers in your own documents without retraining. Quality is decided mostly by chunking and ranking, not by the model.

Ingest path

  • Parse (PDF, HTML, tables) and clean
  • Chunk: typically 200–800 tokens with overlap, or by document structure
  • Embed each chunk into a dense vector
  • Index in a vector store (HNSW, IVF-PQ) with metadata filters

Query path

  • Rewrite or expand the query
  • Hybrid search: dense vectors + BM25 keywords
  • Rerank top-k with a cross-encoder
  • Pack the best passages into the prompt with source IDs
similarity(q, d) = cos θ = (q · d) / (‖q‖ ‖d‖)  ·  measure with recall@k, MRR, and answer faithfulness

Inside layer 3 ↓ seven failure points, hybrid search, long context versus retrieval, and how to measure the retriever.

Inside layer 3

Most RAG failures happen before the model writes a word

When a RAG system gives a wrong answer, the instinct is to blame the model or rewrite the prompt. More often the model never saw the right passage: it was never indexed, it ranked 31st, or it was cut when the context was packed. Generation can only be as good as what retrieval hands it, so this layer is where answer quality is won or lost.

1 · Seven places a RAG system fails

Failure pointStageWhat it looks likeFirst fix
1 · Missing contentCorpusThe answer isn't in any indexed document, but the system answers anywayCoverage checks; a clear "not in the documents" response
2 · Missed the top-ranked documentsRetrieve & rankThe right chunk exists but ranks below the cut-offHybrid search, query rewriting, a reranker
3 · Not in contextAssembleRetrieved, then dropped when the prompt was packedRerank before trimming; dedupe; budget the context
4 · Not extractedGenerateThe answer is in the context, but noise or contradictions hide itFewer, better passages; flag conflicting sources
5 · Wrong formatGenerateThe model ignores the requested table, list or schemaStructured output with validation
6 · Incorrect specificityGenerateToo general or too detailed for what was askedClarifying questions; examples in the prompt
7 · IncompleteGenerateLeaves out information that was in the contextBreak multi-part questions into sub-queries

From Barnett et al. (2024), drawn from three production RAG case studies in research, education and biomedicine. The first three failures happen before generation, and an end-to-end "was the answer right?" score can't tell them apart from model errors. You only see them if you measure the retriever on its own (panel 5).

2 · Retrieval is a pipeline you can tune, step by step

Embeddings only baseline 5.7% Contextual embeddings 3.7% −35% + contextual BM25 hybrid 2.9% −49% + reranking 150 → 20 1.9% −67% share of questions whose answer chunk is missing from the top 20 (1 − recall@20)

Anthropic's 2024 Contextual Retrieval experiment, averaged across several domains. Contextual embeddings prepend 50–100 tokens of model-written context to each chunk before embedding it, so a chunk that says "revenue grew 3%" also says which company and which quarter. That one-time step cost about $1.02 per million document tokens with prompt caching. Adding keyword search (BM25) and then a reranker, which takes the top 150 candidates and keeps the best 20, cut missed retrievals by two-thirds without touching the generating model. These are a vendor's own benchmark numbers; the pattern, not the exact figures, is what transfers.

3 · Why hybrid search: meaning and exact matches fail differently

ChunkDense rankBM25 rankRRF scoreFused rank
B · "Error ERR-4012: token expired"311/63 + 1/61 = 0.03231
A · "Troubleshooting sign-in problems"191/61 + 1/69 = 0.03092
C · "Resetting your password"2—1/62 = 0.01613

Query: "what does ERR-4012 mean?" Dense embeddings match meaning, so they rank general sign-in pages highly but blur exact strings like error codes, part numbers and names. BM25 matches the exact token. Reciprocal rank fusion combines the two lists without tuning: each list contributes 1 / (60 + rank), and the sums are re-sorted. The chunk that both methods rate well wins. Here, the exact error page rises to the top.

4 · A million-token window doesn't retire retrieval

~150ktokens: a 300-page manual, every query
vs
4ktokens: 5 retrieved chunks of 800
=
~37×fewer input tokens per query
QuestionPut it all in the windowRetrieve
How big is the corpus?Fits in the window, and stays that sizeLarger than any window, or growing
Who may read what?Everyone can see all of itPermissions differ by document or user
How often does it change?Rarely; cache the prefix and reuse itDaily or faster; re-index only what changed
Must answers cite a source?Possible, but harder to traceEach chunk carries its source ID
How does the question relate to the text?It needs the whole document: summarise, compare, find conflictsIt needs a few specific facts

Long context is the right call for small, stable, whole-document tasks, especially with prompt caching. For everything else, retrieval is cheaper per query, enforces permissions at the chunk level, and keeps the window small, which matters because accuracy falls as input grows (see context rot). The 150k figure assumes about 500 tokens per page.

5 · Measure the retriever on its own

MetricQuestion it answersCatches
Recall@kDid the right chunk come back in the top k at all?Failure 2
MRR / nDCGHow high did it rank?Failures 2 and 3
Context precisionHow much of what we packed was relevant?Failures 3 and 4
FaithfulnessIs every claim in the answer supported by the context?Hallucination on top of good retrieval
Answer correctnessWas the final answer right?Everything, but can't say where it broke

Start with a golden set of 50–200 real questions, each labelled with the passage that answers it, plus some questions the corpus can't answer. Run it on every change to chunking, embeddings or ranking. When answer correctness drops, the retrieval metrics tell you whether to fix the index or the prompt, which is the traceability Governance by design asks for.

6 · Design rules

Chunk by structure, not by countSplit on headings, sections and table boundaries. A chunk that cuts a table in half answers nothing.
Hybrid by defaultDense plus keyword search, fused. It costs little and fixes the exact-match blind spot.
Retrieve wide, rerank narrowPull 50–150 candidates cheaply, then let a reranker choose the few that enter the prompt.
Filter permissions before rankingStore access rights with every chunk and filter on them in the query, so restricted text never reaches the prompt.
Design for "not in the documents"When retrieval scores are low, say so and route to a person rather than letting the model fill the gap.
Version the indexChanging the embedding model means re-embedding everything. Record which model built each index, and re-index changed documents promptly.
Sources: S. Barnett et al., “Seven Failure Points When Engineering a Retrieval Augmented Generation System”, CAIN 2024 (arXiv:2401.05856); Anthropic, “Introducing Contextual Retrieval” (Sep 2024); G. Cormack, C. Clarke & S. Büttcher, “Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods”, SIGIR 2009 (k = 60); K. Hong et al., “Context Rot”, Chroma (Jul 2025). RRF example is illustrative.
Layer 4

Foundation model

Almost every current large language model is a decoder-only transformer: a stack of identical blocks that turns a sequence of tokens into a probability distribution over the next token. Variants change the attention pattern, the feed-forward layer, or add encoders for images and audio.

Building blocks

  • Tokenizer (BPE / SentencePiece), vocab ~32k–256k
  • Token embeddings + rotary positions (RoPE)
  • Self-attention: multi-head, grouped-query (GQA)
  • Feed-forward (SwiGLU) or Mixture-of-Experts
  • RMSNorm and residual connections

Variants to know

  • Dense: every parameter used per token
  • MoE: router picks top-k experts; large total, small active params
  • Multimodal: vision/audio encoder projected into token space
  • Long context: RoPE scaling, sliding-window or sparse attention

Inside layer 4 ↓ mixture-of-experts: why a trillion-parameter model can cost less per token than a 405B dense one.

Inside layer 4

One transformer block

Each block reads the running representation (the residual stream), lets every token look back at earlier tokens through attention, then transforms each position independently with the feed-forward layer. Residual connections (dashed) add each sub-layer's output back to its input, which is what lets very deep stacks train.

Attention(Q, K, V) = softmax( Q·Kᵀ / √dk + mask ) · V
  • Attention mixes information across positions; cost grows with sequence length squared in prefill.
  • Feed-forward holds most parameters (about two-thirds in a dense model) and much of the stored knowledge.
  • GQA shares key/value heads across query heads to shrink the KV cache.
  • MoE swaps the single feed-forward for many experts; only top-k run per token.
Inside layer 4

A trillion parameters, 32 billion at work

The block diagram above has a small note on the feed-forward layer: "or MoE experts". That one swap explains how open models grew past a trillion parameters in 2024–25 without getting proportionally more expensive to run. Kimi K2 holds about 1,040 billion parameters, yet each token passes through only 32 billion of them, about 3%. In a mixture-of-experts (MoE) model, "how big is it?" has two answers: total parameters set the memory bill, active parameters set the compute bill.

1 · How a token picks its experts

Token hidden state Router keeps top 2 Expert 1 · idle Expert 2 · idle Expert 3 Expert 4 · idle Expert 5 · idle Expert 6 Expert 7 · idle Expert 8 · idle × 0.62 × 0.38 + Output to next layer Next token, same layer: the router may pick a different pair

An MoE layer replaces one large feed-forward network with many smaller ones, the experts. A small learned router scores every expert for each token at each MoE layer, keeps the top k, and mixes their outputs by the router's weights. Attention, embeddings and norms stay shared. The idea is old: Shazeer et al. made it work at scale in 2017, and Google's Switch Transformer (2021) sent each token to one expert and pre-trained up to 7× faster than T5-Base on the same compute. Mixtral 8x7B (Dec 2023) uses 2 of 8 experts and, in Mistral's words, runs "at the same speed and for the same cost as a 12.9B model". Despite the name, experts rarely map to subjects: Mistral's analysis of Mixtral found no clear topic specialisation, with routing following syntax more than domain.

2 · Big in memory, small per token

parameters held in memory (outline) vs used for each token (filled), same scale Llama 3.1 405B · dense 405B of 405B · 100% Kimi K2 32B of 1,040B · 3.1% DeepSeek-V3 37B of 671B · 5.5% Llama 4 Maverick 17B of ~400B · 4.3% gpt-oss-120b 5.1B of 117B · 4.4% Llama 4 Scout 17B of 109B · 15.6% Mixtral 8x7B 12.9B of 46.7B · 27.6%
ModelReleasedExperts per MoE layerUsed per token
Mixtral 8x7BDec 202382
DeepSeek-V3Dec 2024256 routed + 1 shared8 routed + the shared one
Llama 4 MaverickApr 2025128 routed + 1 shared1 routed + the shared one; MoE on alternate layers
Kimi K2Jul 20253848
gpt-oss-120bAug 20251284

The trend runs one way: more, smaller experts, and a smaller share of the model awake for each token, from 2 of 8 in Mixtral to 8 of 384 in Kimi K2. A shared expert sees every token and holds common knowledge, so the routed experts can specialise. Closed frontier labs rarely publish their architectures, so open-weight models are where the numbers are.

3 · The bill: compute follows active, memory follows total

671Bparameters held (DeepSeek-V3)
→
37Bused for each token
=
~11×less compute per token than dense 405B
ModelWeights in memoryCompute per token ≈ 2 × activevs dense 405B
Llama 3.1 405B (dense)810 GB at BF16810 GFLOP1×
DeepSeek-V3~671 GB at FP874 GFLOP~11× less
Kimi K2~1.04 TB at FP864 GFLOP~13× less
Mixtral 8x7B~93 GB at BF1626 GFLOP~31× less
gpt-oss-120bfits one 80 GB GPU (4-bit MXFP4 experts)10 GFLOP~79× less

Compute uses the rule of thumb of about 2 FLOPs per active parameter per token (Kaplan et al. 2020) and ignores attention over the context. Memory is weights only, before the KV cache and activations. The catch is in the second column: at FP8, DeepSeek-V3's 671 GB of weights don't fit on one 8 × 80 GB H100 server (640 GB) and need something like 8 × H200 (1,128 GB). MoE moves the bottleneck from compute to memory capacity and the links between GPUs. In return you get the quality of a much larger dense model: Mistral reported that Mixtral, with 12.9B active parameters, beat Llama 2 70B on most benchmarks.

4 · What MoE costs you

ProblemWhy it happensWhat teams do
Memory capacityEvery expert must sit in GPU memory, even though only a few run per tokenSize the cluster by total parameters; quantise expert weights (gpt-oss ships them at 4 bits)
Load imbalanceThe router learns to favour a few experts; they become hot spots while others starve, or "collapse" and stop learningAdd a balancing loss (Switch); DeepSeek-V3 instead nudges a per-expert bias, avoiding an extra loss term; cap tokens per expert
CommunicationWith experts spread across GPUs, every MoE layer sends tokens out to their experts and back (all-to-all)Keep expert groups inside the fastest interconnect; DeepSeek-V3 limits each token to experts on at most 4 nodes; overlap compute with transfers
Small batchesAt low traffic each expert gets a handful of tokens, so its matrix multiplies are small and GPUs sit idleMoE pays off at high, steady throughput; for quiet self-hosted workloads, a smaller dense model can be cheaper
Training stabilityRouter logits can grow and destabilise training; fine-tuning can shift routing and starve rare expertsRouter z-loss (ST-MoE, 2022); evaluate per domain after fine-tuning; adapt shared layers first

None of these show up in a benchmark table. They show up in the serving bill and the p95 latency, which is why an architecture review should ask for both parameter counts and the batch size behind any throughput claim.

5 · Dense or MoE? It depends on where you sit

Your situationLean towardsWhy
Calling a hosted APIWhatever wins your evalsThe provider carries the memory cost. You see price per token and latency, so judge on those and on quality
One GPU or a laptopSmall dense, or a small MoE that fitsMemory capacity is the hard limit; gpt-oss-20b (21B total, 3.6B active) runs in 16 GB
High, steady traffic on a GPU clusterMoELow compute per token, and full batches keep every expert busy
Low or spiky traffic, self-hostedSmaller dense, or a shared MoE endpointIdle experts still occupy memory you pay for around the clock
Heavy domain fine-tuningDense is simplerFewer moving parts; MoE fine-tuning needs care with the router
Single-user latency, fits in fast memoryMoE often winsGenerating a token is limited by memory bandwidth, and only the active experts' weights are read per token

For most enterprise teams the first row applies, and the architecture matters indirectly: MoE is one reason open models got cheaper to serve as they got stronger. It matters directly once you self-host, where the hardware and the traffic pattern decide.

6 · Design rules

Size memory by total, cost by activeTotal parameters decide how many GPUs you need. Active parameters decide compute per token. Plan for both.
Always quote both numbersWrite "37B active / 671B total". Comparing an MoE's total with a dense model's size misleads in both directions.
Benchmark at your batch sizePublished MoE throughput assumes full batches. Measure at your real traffic before you commit hardware.
Keep experts on the fastest linksPlace expert-parallel groups inside one server's interconnect where possible; cross-node all-to-all is where latency hides.
Watch expert load in productionLog each expert's share of tokens. Imbalance shows up as tail latency before it shows up as quality.
Choose by evals, not parameter countTotal parameters no longer stand in for capability or cost. Run your own tasks and price the result.
Sources: N. Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR 2017; W. Fedus, B. Zoph & N. Shazeer, “Switch Transformers”, JMLR 2022 (arXiv:2101.03961); B. Zoph et al., “ST-MoE”, 2022; Mistral AI, “Mixtral of experts” (Dec 2023); DeepSeek-AI, “DeepSeek-V3 Technical Report” (arXiv:2412.19437); Meta, Llama 4 Scout and Maverick (Apr 2025); Kimi Team, “Kimi K2: Open Agentic Intelligence” (arXiv:2507.20534); OpenAI, gpt-oss-120b and gpt-oss-20b model card (Aug 2025); J. Kaplan et al., “Scaling Laws for Neural Language Models”, 2020. Routing weights in panel 1 are illustrative; memory figures are weights only.
Layer 5

Training & adaptation

A model passes through stages, each with different data and objectives. For most enterprise work the choice is not whether to pretrain but which adaptation to apply on top of an existing model.

Stages

  • Pretraining: next-token prediction on trillions of tokens
  • Supervised fine-tuning: instruction–response pairs
  • Preference tuning: RLHF (reward model + PPO) or DPO
  • Distillation: a small model learns from a larger one

Adaptation ladder (cheapest first)

  • Prompting and few-shot examples
  • RAG for knowledge that changes
  • Parameter-efficient tuning: LoRA / QLoRA adapters
  • Full fine-tune for behaviour and format at scale
Training compute ≈ 6 · N · D FLOPs (N params, D tokens)  ·  compute-optimal ≈ 20 tokens per parameter (Chinchilla)

Inside layer 5 ↓ why a 1.3B model beat a 175B one, how LoRA makes adaptation cheap, and the adaptation ladder.

Inside layer 5

A 1.3B model beat a 175B one

In OpenAI's InstructGPT study, people preferred answers from a 1.3-billion-parameter model to answers from the 175-billion-parameter GPT-3, a model with over 100 times more parameters. The architecture was the same. The difference was post-training: a few tens of thousands of prompts of human demonstrations and rankings, on top of a model pre-trained on hundreds of billions of tokens. Pre-training gives a model its knowledge; post-training decides how it uses it. For enterprise teams, the useful corollary is that adapting a model costs a tiny fraction of building one.

1 · Where a model's behaviour comes from

StageDataHow muchWhat it changes
Pre-trainingRaw text and code, next-token predictionTrillions of tokens (Llama 3: 15T)Knowledge, language, reasoning patterns
Supervised fine-tuningHuman-written example answers~13k prompts in InstructGPT; 1,000 in LIMAFollowing instructions, answer format
Reward modelPeople rank several answers to the same prompt~33k promptsA learned score for "what people prefer"
RL from human feedbackThe model's own answers, scored by the reward model (PPO); DPO skips the separate reward model~31k promptsHelpfulness, tone, when to refuse
Your adaptationYour tasks, formats and documentsHundreds to thousands of examplesDomain style, output schema, narrow skills

Prompt counts are from the InstructGPT paper. Every stage after pre-training uses a sliver of its data, which is why post-training is where most teams, and most of a model's personality, actually live. LIMA (Meta, 2023) pushed this furthest: a 65B model fine-tuned on just 1,000 carefully curated examples gave answers judged equivalent or better than GPT-4's in 43% of cases. The authors' reading: knowledge comes from pre-training, and fine-tuning mostly teaches format.

2 · 100× fewer parameters, and still preferred

GPT-3 · 175B InstructGPT · 1.3B preferred by human raters area ∝ parameters 175B InstructGPT preferred over GPT-385% … over GPT-3 with few-shot prompts71% made-up facts on closed-domain tasks 41% GPT-3 21% InstructGPT

From Ouyang et al. (2022). Squares are drawn to scale by area. The method: fine-tune GPT-3 on demonstrations, train a reward model on human rankings, then optimise against it with reinforcement learning. It halved made-up facts on closed-domain tasks such as summarisation. It wasn't free: the paper reports an "alignment tax", small drops on some academic benchmarks, which they reduced by mixing pre-training data back into the RL step. Any adaptation can trade away skills you didn't test, which is why panel 6 starts with evals.

3 · Why adapting is cheap: LoRA

W frozen 12,288 × 12,288 + B 12,288 × 4 × A 4 × 12,288 frozen: 150,994,944 trainable: 98,304 ~1,536× fewer for this one matrix, at rank 4
350 GB → 35 MBsaved checkpoint per task
1.2 TB → 350 GBGPU memory to train
~10,000×fewer trainable parameters

Low-rank adaptation (Hu et al., 2021) freezes the original weights and learns a small update, W + B·A, where B and A are thin matrices of rank r. On GPT-3 175B, adapting only the query and value projections at rank 4 matched or beat full fine-tuning on the paper's tasks, trained about 25% faster, and adds no latency at serving time once B·A is merged into W. QLoRA (Dettmers et al., 2023) went further: it fine-tuned a 65B model on a single 48 GB GPU by keeping the frozen weights in 4-bit, and its Guanaco model reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours of training.

4 · The adaptation ladder: climb one rung at a time

RungChanges weights?Good forWrong tool for
Prompting & few-shotNoFormat, tone, quick tasks; testing whether the model can do the job at allLarge or private knowledge
RAGNoFacts that change, citations, per-user permissions (Inside L3)Changing behaviour or style
LoRA / QLoRA adaptersA small add-onConsistent output schema, domain style, narrow tasks; letting a small model replace a large oneTeaching facts that change often
Full fine-tuneAll of themDeep behaviour change, when adapters measurably fall shortMost enterprise needs; expensive to repeat on each new base model
Continued pre-trainingAll of themA new language or a domain with its own vocabulary, given billions of tokensAnything a lower rung solves

Each rung costs more to build, more to evaluate and more to keep current. The usual pattern in production is a hosted or open model, plus RAG for knowledge, plus at most a LoRA adapter for format and style. Training from scratch is a separate decision, made by very few organisations.

5 · Fun fact: Llama 3 was trained "too long", on purpose

~200Btokens: Chinchilla-optimal for an 8B model
vs
15Ttokens: what Llama 3 8B saw
=
75×past the "optimal" point

The Chinchilla rule (about 20 tokens per parameter, in the Layer 5 box above) gives the best model for a fixed training budget. But training happens once, and serving goes on for the model's whole life. A smaller model trained far longer costs more to build and less on every request after that, the same serving maths as the KV cache and MoE. Meta reported both Llama 3 8B and 70B "continued to improve log-linearly" up to 15T tokens. By the 6·N·D rule, the 8B run took about 7.2 × 10²³ FLOPs.

6 · Design rules

Build the eval set firstTask examples plus a regression suite for skills you need to keep. Without it you can't tell gains from the alignment tax.
Climb the ladder one rung at a timePrompt, then RAG, then adapters. Fine-tune fully only when an eval shows the lower rungs can't reach the bar.
Fine-tune behaviour, retrieve factsFacts baked into weights go stale and can't cite a source. Keep them in the index; teach the model how to answer.
Curate before you collectA thousand reviewed examples can beat a hundred thousand scraped ones. Spend the budget on quality and review.
One base, many adaptersKeep the base model fixed and version a small adapter per task. At tens of megabytes each, they are cheap to store, swap and roll back.
Plan for the next base modelAdapters are tied to their base. Keep training data and pipelines reproducible so you can re-run them on the next release.
Sources: L. Ouyang et al., “Training language models to follow instructions with human feedback”, NeurIPS 2022 (arXiv:2203.02155); E. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, 2021 (arXiv:2106.09685); T. Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs”, NeurIPS 2023 (arXiv:2305.14314); C. Zhou et al., “LIMA: Less Is More for Alignment”, NeurIPS 2023 (arXiv:2305.11206); J. Hoffmann et al., “Training Compute-Optimal Large Language Models” (Chinchilla), 2022; Meta, “Introducing Meta Llama 3” (Apr 2024). Benchmark figures are the authors' own.
Layer 6

Inference & serving

Serving splits into two phases with different bottlenecks. Prefill processes the whole prompt in parallel and is compute-bound; it sets time-to-first-token. Decode produces one token at a time, rereads all weights and the KV cache each step, and is memory-bandwidth-bound; it sets time-per-output-token.

Techniques

  • KV cache, with paged allocation (PagedAttention)
  • Continuous (in-flight) batching
  • Quantization: FP8, INT8, INT4 weights
  • Speculative decoding with a draft model
  • Prefix / prompt caching for shared system prompts

Metrics

  • TTFT: time to first token (ms)
  • TPOT: time per output token (ms)
  • Throughput: tokens/s per GPU
  • Goodput: requests/s that meet the latency SLO
  • Cost per 1M input / output tokens
Weights memory = params × bytes  →  70B × 2 B (FP16) = 140 GB; at INT4 ≈ 35 GB
KV cache per token = 2 × layers × kv_heads × head_dim × bytes
  e.g. 2 × 32 × 8 × 128 × 2 B = 128 KiB/token → 8,192-token context ≈ 1 GiB per sequence
Inference compute ≈ 2 · N FLOPs per generated token
Inside layer 6

The KV cache, explained

A language model writes one token at a time. To pick each new token, attention compares it with every earlier token using their keys (what each token offers to be matched on) and values (what it contributes once matched). Those keys and values never change once computed, so serving engines keep them in GPU memory instead of recomputing them. That store is the KV cache. It makes generation fast, and it is usually what limits how many people one GPU can serve.

1 · What the cache saves

Without a cache t1 t2 t3 t4 t5 step 1 Step 1, token 1: K and V computed this step Step 1, token 2: not generated yet Step 1, token 3: not generated yet Step 1, token 4: not generated yet Step 1, token 5: not generated yet step 2 Step 2, token 1: K and V computed this step Step 2, token 2: K and V computed this step Step 2, token 3: not generated yet Step 2, token 4: not generated yet Step 2, token 5: not generated yet step 3 Step 3, token 1: K and V computed this step Step 3, token 2: K and V computed this step Step 3, token 3: K and V computed this step Step 3, token 4: not generated yet Step 3, token 5: not generated yet step 4 Step 4, token 1: K and V computed this step Step 4, token 2: K and V computed this step Step 4, token 3: K and V computed this step Step 4, token 4: K and V computed this step Step 4, token 5: not generated yet step 5 Step 5, token 1: K and V computed this step Step 5, token 2: K and V computed this step Step 5, token 3: K and V computed this step Step 5, token 4: K and V computed this step Step 5, token 5: K and V computed this step 15 K/V computations With a KV cache t1 t2 t3 t4 t5 step 1 Step 1, token 1: K and V computed this step Step 1, token 2: not generated yet Step 1, token 3: not generated yet Step 1, token 4: not generated yet Step 1, token 5: not generated yet step 2 Step 2, token 1: K and V read from cache Step 2, token 2: K and V computed this step Step 2, token 3: not generated yet Step 2, token 4: not generated yet Step 2, token 5: not generated yet step 3 Step 3, token 1: K and V read from cache Step 3, token 2: K and V read from cache Step 3, token 3: K and V computed this step Step 3, token 4: not generated yet Step 3, token 5: not generated yet step 4 Step 4, token 1: K and V read from cache Step 4, token 2: K and V read from cache Step 4, token 3: K and V read from cache Step 4, token 4: K and V computed this step Step 4, token 5: not generated yet step 5 Step 5, token 1: K and V read from cache Step 5, token 2: K and V read from cache Step 5, token 3: K and V read from cache Step 5, token 4: K and V read from cache Step 5, token 5: K and V computed this step 5 K/V computations + 10 reads computed this step read from cache not generated yet

To produce token 5, the model needs keys and values for tokens 1 to 5. Without a cache it recomputes all of them at every step. With a cache it computes only the newest one and reads the rest. Over a 1,000-token answer that is 500,500 key/value computations per layer without a cache, against 1,000 with one.

2 · What one token costs in memory

2K and V
×
32layers
×
8KV heads
×
128head dim
×
2 BFP16
=
128 KiBper token

Every token in every active conversation holds this much, in every layer, for as long as the conversation is live. The shape shown is typical of an 8B-parameter model.

3 · How many conversations fit on one 80 GB GPU

weights 16 GBKV cache budget 64 GB (≈ 59.6 GiB)
2K context0.25 GiB per conversation
238
8K context1 GiB per conversation
59
32K context4 GiB per conversation
14
128K context16 GiB per conversation
3

Same GPU, same model. Only the context length changes, and capacity falls from 238 conversations to 3. Long context is a memory decision before it is a model decision.

Assumes an 8B model in FP16 (32 layers, 8 KV heads, head dim 128) and every conversation at full length. Ignores activations and runtime overhead, so real capacity is somewhat lower.

4 · How architects shrink it

Multi-head attention32 KV heads · FP16
512 KiB
Grouped-query attention8 KV heads · FP16
128 KiB
GQA + FP8 cache8 KV heads · 1 byte
64 KiB

Make each token smaller

  • Grouped-query attention: several query heads share one KV head
  • FP8 / INT8 KV cache: halve the bytes per value
  • Latent attention (MLA): store a compressed latent and expand on use

Store fewer tokens, waste less

  • PagedAttention: allocate in small blocks, so little memory sits reserved but unused
  • Prefix caching: a shared system prompt is stored once for all users
  • Sliding-window attention: keep only the most recent N tokens in some layers
  • Offload: move idle conversations to CPU memory or SSD
GPU memory needed ≈ weights + (concurrent users × context tokens × KV bytes per token) + headroom for activations and runtime
Layer 7

Data foundation

Every layer above depends on data that is clean, permissioned and traceable. The same pipelines feed pretraining corpora, fine-tuning sets, retrieval indexes and evaluation sets.

Components

  • Ingestion from lakes, warehouses, SaaS sources
  • Deduplication (exact + MinHash near-dup)
  • Quality and toxicity filtering, PII redaction
  • Labelling and synthetic data generation

Controls

  • Lineage: which data trained or grounded which output
  • Access control carried into the retrieval index
  • Versioned datasets and eval sets
  • Retention and consent rules
Layer 8

Compute & infrastructure

Accelerators, memory and the network between them. For large models, memory capacity and bandwidth matter as much as raw FLOPs, and the interconnect decides how well work splits across devices.

Hardware

  • GPUs / TPUs / custom accelerators
  • HBM capacity and bandwidth per device
  • Intra-node links (NVLink) and inter-node fabric (InfiniBand, RoCE)
  • Managed platforms: Azure AI Foundry, Amazon Bedrock / SageMaker, Vertex AI

Parallelism

  • Data: copies of the model, split the batch
  • Tensor: split each matrix across devices
  • Pipeline: split layers into stages
  • Expert: spread MoE experts across devices
  • Sharded optimizers (ZeRO / FSDP)
Across all layers

Cross-cutting concerns

Evaluation

  • Offline golden sets per use case
  • LLM-as-judge with human calibration
  • RAG: retrieval recall, faithfulness, answer relevance
  • Online A/B tests and regression gates in CI

Safety & guardrails

  • Input checks: prompt injection, PII, policy
  • Output checks: grounding, toxicity, data leakage
  • Tool permissions and least privilege for agents
  • Red-teaming before each release

Observability

  • Traces per request across retrieval, tools, model
  • Token usage and cost attribution
  • Latency (TTFT, TPOT) and error rates
  • Drift in inputs and quality scores

Governance

  • Model and prompt registry with versions
  • Risk tiering and approvals per use case
  • Audit logs and explainability records
  • Regulatory mapping (e.g. EU AI Act, NIST AI RMF)
Across all layers · added from reader feedback

Identity & policy: on whose authority?

Compute and data platforms settled access control years ago with IAM: every call carries a principal, and policy decides what it may touch. AI systems reopen the question, because the caller is increasingly an agent acting for a person. Every layer, from retrieval to tool execution, now has to answer one question: on whose authority is this running, and with what limits?

1 · Authority should narrow at every hop

Usersigned in via SSO
Everything this person is allowed to see and do
↓ delegates a scoped token
Applicationregistered client
Read documents and calendar, on this user's behalf
↓ token exchange for one task
Agentone task, short-lived
Read documents in Project X, for 30 minutes
↓ per-tool scope
Tool callsingle action
Search Project X, read only

The agent never holds more access than the person it acts for, and each step down gets only what that step needs. If a token leaks or a prompt injection hijacks the agent, the damage stays inside the narrowest scope.

2 · The question each layer must answer

LayerQuestionControl
L1 ApplicationWho is the user?SSO with OIDC, MFA, session lifetime
L2 OrchestrationWhat may this agent do for them?Delegated, short-lived tokens (OAuth on-behalf-of, token exchange), step budgets, human approval for high-impact actions
L2 ToolsWhat can this call change?Per-tool scopes and allow-lists, OAuth on MCP servers, read-only by default, dry runs
L3 RetrievalWhich documents may this user see?Source ACLs carried into the index and filtered at query time, before anything reaches the prompt
L6 GatewayWhich tenant, quota and model?API gateway, tenant isolation, rate limits, data-residency routing
L7 DataMay this data be used for this purpose?Purpose-based access, consent flags, lineage
AuditWho did what, for whom?Log the full chain per action: user → agent → tool → resource

3 · Design rules

Enforce outside the modelPrompts can be injected; policy checks in the gateway and tools cannot be talked out of a decision.
Deny by defaultA tool or document is reachable only when a policy explicitly allows it.
Short-lived, scoped credentialsMinutes, not months. No shared service accounts acting for everyone.
Filter before generationTrim retrieval results by permission before the model sees them, never after it answers.
Policy as codeExpress rules in an engine such as OPA or Cedar so they are versioned, tested and reviewed.
Watch for confused deputiesAn agent with broad access can be tricked into using it for a user who lacks it. Carry the user's identity on every call.
Across all layers · added from reader feedback

Governance by design: proving the system does what it should

Governance added at review time produces documents about a system. Governance built in at design time produces evidence of how it behaves. The rule is simple: every intended behaviour becomes a testable requirement, and every requirement leaves a trail that a reviewer or auditor can follow from intent to production.

1 · The traceability chain

  1. Intended useWhat the system is for, who uses it, what it must never do, and its risk tier.
  2. RequirementEach "must" and "must not" written so it can be measured.
  3. Design controlThe mechanism that enforces it: retrieval scope, guardrail, human approval, tool permission.
  4. Eval with a thresholdA test set and pass mark, run in CI. A failed gate blocks the release.
  5. Production monitorThe same check sampled on live traffic, with an alert and an owner.
  6. EvidenceVersioned eval reports, approvals, logs and incidents, produced by the pipeline itself.

Worked example, an HR policy assistant. Intent: answer employee questions from the approved handbook only. Requirement: no answer without a citation to the handbook. Design control: retrieval limited to the handbook index, and the response is rejected if it has no citation. Eval: 300 test questions, at least 95% grounded and zero uncited answers. Monitor: 2% of live answers scored for groundedness each week. Evidence: eval report per release, weekly monitor results, sign-off record.

2 · The evidence pack

EvidenceProduced byWhenWhat frameworks ask for
Intended use & risk tierProduct owner with riskDesignNIST AI RMF Map; EU AI Act risk classification
Requirements as eval setsEngineeringDesign and buildNIST AI RMF Measure; accuracy and robustness obligations
Model, data & prompt cardsEngineeringEvery releaseTechnical documentation; data governance
Eval reports & gate decisionsCI pipelineEvery releaseISO/IEC 42001 performance evaluation
Human-oversight design & approvalsAccountable ownerBefore deployHuman oversight obligations
Logs, monitors & incidentsPlatform teamContinuouslyRecord-keeping; NIST AI RMF Manage
ISO/IEC 42001, the NIST AI RMF and the EU AI Act overlap heavily. Design the evidence once and map it to each, rather than running three separate compliance efforts.

3 · The AI engineering governance skill set

Technical

  • Eval design and metrics
  • RAG, agents and guardrails
  • CI/CD and MLOps pipelines
  • Observability and logging

AI engineering governance

  • Turn policy into testable requirements
  • Set release gates and thresholds
  • Design evidence that is generated, not assembled
  • Explain model behaviour to auditors and the board

Governance

  • Risk assessment and tiering
  • Regulatory mapping
  • Accountability and approvals
  • Audit and assurance

The middle column is the scarce role: someone who can read a model card and a control framework with equal confidence.

Fun fact · AI engineering

The model is the smallest box in the system.

In 2015, a team of Google engineers published a paper called Hidden Technical Debt in Machine Learning Systems. Its most famous diagram shows a real-world ML system as a large set of boxes, with the ML code as a small black box in the middle. The paper warned that a mature system might end up being at most 5% machine learning code and at least 95% everything around it.

At most 5 squares in 100

5%ML code95%data, configuration, serving, monitoring and glue code

A decade later, the boxes have new names

The 95% in 2015The 95% in LLM systems today
Data collection & verification→Data pipelines, RAG ingestion
Feature extraction→Chunking & embeddings
Configuration→Prompts, model routing
Serving infrastructure→Inference engines, KV cache
Monitoring & analysis tools→Evals, LLM observability
Process management→Agent orchestration, tool permissions
ML code→Often a single API call

With foundation models the ratio has only grown. The model can now be one call to a hosted endpoint, while the retrieval, evals, guardrails, identity and serving around it are what you actually build and run. That is why this blog maps eight layers, and only one of them is the model.

Source: D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 2015. The 5% figure is the paper's estimate for glue-code-heavy systems, not a measured average.
Layer 2 · a new kind of model

JEV: the model that decides instead of writes

JEV, from TypeSafe AI (early access since 15 September 2026), never writes a sentence. You hand it messy input plus a set of typed questions, and it returns typed answers, each with a calibrated probability. TypeSafe calls it the first System One model, after Kahneman's fast, intuitive System 1. Today's LLMs are System 2 machines: slow, deliberate, and expensive to ask for every small choice an agent makes.

1 · Where it sits: a decision gate in front of the model

Agent step L2 orchestration JEV answer + confidence Confident? Act directly route · tool · block Strong LLM L4 · reason or write yes no, or text needed LLM result

Every agent step asks small, bounded questions: which tool, which model, retry or stop, allow or block. JEV answers those in one fast pass. When its confidence falls below a threshold, or the step needs text or free-form arguments, the call escalates to the foundation model. JEV does not replace the LLM; it decides when the LLM is worth calling.

2 · How it works

Typed questions, not promptsYou declare the answer space up front: a yes/no, one of up to 255 labels, a score. JEV can only answer inside it, unlike LLM structured output, which generates first and validates after.
One parallel passNo token-by-token decoding. All questions are answered at once, so many decisions cost roughly one call. TypeSafe quotes 70–500 ms end to end.
Calibrated confidenceTrained with RLCD (Reinforcement Learning for Calibrated Decisions). A 0.9 should be right about 90% of the time, which is what makes a confidence threshold safe to automate on.
Always typed, not always rightJEV cannot hallucinate a paragraph, but it can still pick the wrong label. Evaluate it like any classifier, against your own labelled set.

3 · Claims vs caveats

Vendor claimFigureRead it as
End-to-end latency70–500 msPlausible: removing autoregression explains most of it
Speed vs frontier LLMs40–200×Only on bounded decisions, not reasoning
Best single workflow193.6× faster, 444.6× cheaperOne showcase workflow, not an average
Input price$0.042 per million tokensEarly-access pricing; may change

No independent evaluation accompanied the launch. TypeSafe's own documentation lists weak spots: counting and maths, date ordering, numeric proximity, double negation, long context, adversarial inputs and CJK languages. The first academic study, REFLEX (Wu & Lim, NTU), found that JEV-style selective control saves compute only when a task actually contains bounded decisions to offload.

Sources: TypeSafe AI launch materials (15 Sep 2026) as reported by DigitalOcean, DataCamp and RuntimeWire; T. Wu and W. Y. B. Lim, “REFLEX with Jev for Efficient Selective Control in LLM Agents”, arXiv 2609.26532. All performance figures are vendor-reported.
Fun fact · Layers 5 & 8

Every modern LLM traces back to 8 GPUs and 3.5 days.

In June 2017, eight Google researchers published Attention Is All You Need. They were not building a chatbot; they wanted better translation between English, German and French. Their larger Transformer trained on one machine with 8 NVIDIA P100 GPUs for 300,000 steps, about 3.5 days. Seven years later, Meta's Llama 3.1 405B used essentially the same decoder design, with refinements such as grouped-query attention and rotary positions, and trained on 16,384 H100s.

1 · The same idea, at a different scale

Each dot is one GPU. The 8 highlighted (top left) trained the Transformer. All 16,384 trained Llama 3.1 405B.
2017 Transformer2024 Llama 3.1Increase
GPUs8 × P10016,384 × H100~2,000×
Parameters213 million405 billion~1,900×
GPU-hours~67230.84 million~46,000×
Training data36M sentences15.6T tokens—
Compute—3.8 × 10²⁵ FLOPs—

Each H100 is also several times faster than a P100, so the real increase in compute is larger than the GPU-hours suggest. The 2017 model reached state-of-the-art translation (28.4 BLEU English–German, 41.8 English–French) at what its authors called a small fraction of earlier training costs.

2 · At scale, failure is routine

During a 54-day stretch of Llama 3 training, Meta recorded 419 unexpected interruptions, about one every three hours.

148 faulty GPUs72 GPU memory (HBM3)199 other causes

Automatic failure detection, fast checkpointing and restarts still kept effective training time above 90%. On 8 GPUs, a failure is an annoyance. On 16,384 it is a design requirement.

3 · What actually changed: the engineering

ParallelismOne model split across thousands of GPUs with data, tensor and pipeline parallelism (Layer 8).
Data at scaleTrillions of tokens cleaned, deduplicated and streamed fast enough to keep every GPU busy (Layer 7).
ReliabilitySurviving a hardware failure every few hours without losing progress (Layer 8).
ServingKV caches, batching and quantization to make a 405B model affordable to run (Layer 6).
Sources: A. Vaswani et al., “Attention Is All You Need”, NeurIPS 2017; Meta, “The Llama 3 Herd of Models” (2024) and Llama 3.1 model card. 672 GPU-hours = 8 GPUs × 3.5 days.
How the layers connect

Lifecycle of one request

  1. Receive and authenticateGateway checks identity, quota and tenant (L1).
    I/O
  2. Input guardrailsScreen for prompt injection, PII and policy violations.
    I/O
  3. Plan and retrieveOrchestrator decides on retrieval; hybrid search + rerank return passages (L2, L3).
    I/O
  4. Assemble the promptSystem instructions, tools, retrieved context, history; cached prefix reused.
    I/O
  5. PrefillAll prompt tokens processed in parallel; KV cache written. Sets TTFT (L6).
    compute-bound
  6. DecodeOne token per step, streamed to the client. Sets TPOT (L6).
    memory-bound
  7. Tool calls (if any)Model emits a call, orchestrator runs it, result is appended, decode resumes.
    loop
  8. Output guardrailsCheck grounding, leakage and policy before release.
    I/O
  9. Log, trace and evaluateTrace, token counts, cost and feedback flow to observability and eval sets.
    async
Decisions architects make

Key trade-offs

DecisionOptionsChoose based on
KnowledgeRAG vs fine-tuningRAG for facts that change or need citations; fine-tuning for style, format and narrow skills.
Model sizeFrontier vs small / distilledTask difficulty, latency SLO and cost per request. Route easy traffic to small models.
HostingManaged API vs self-hosted open weightsData residency, volume (self-hosting pays off at high steady load), team capacity.
PrecisionFP16 vs FP8 / INT8 / INT4Memory and speed gains against measured quality loss on your eval set.
Control flowFixed chain vs autonomous agentPredictability and auditability vs flexibility. Start with chains, add agency where it earns its cost.
ContextLong context vs retrievalLong context is simpler but costs more per call and can dilute attention; retrieval scales to large corpora.
BatchingLatency vs throughputBigger batches raise tokens/s per GPU but increase per-request latency.