Daily note
Each day adds one short note on a core architecture idea. Notes appear here once the first one is posted.
Application & experience
Where people meet the system: chat surfaces, copilots inside existing tools, batch jobs, and APIs other services call. This layer owns the contract with the user: what goes in, what comes back, how long they wait, and what happens when the model is wrong.
Components
- Chat UI, embedded copilots, batch pipelines
- API gateway, auth, rate limiting, quotas
- Streaming transport (SSE / WebSocket)
- Feedback capture (thumbs, edits, escalations)
Design decisions
- Stream tokens or return a complete answer
- Synchronous chat vs async job with a callback
- Human-in-the-loop checkpoints for high-impact actions
- How citations and confidence are shown
Inside layer 1 ↓ the latency contract, designing for the wrong answer, and feedback signals.
The contract with the user: wait, trust and the wrong answer
The model decides what an answer says. Layer 1 decides how long the user waits for it, how far they trust it, and what happens when it is wrong. Those choices shape adoption more than a few points of benchmark accuracy, and none of them live in the weights.
1 · Four ways people meet the system
| Surface | Who starts it | Time budget | When it is wrong | Design priority |
|---|---|---|---|---|
| Chat assistant | The user asks | First words within about a second; the full answer can stream for longer | The user reads it and judges | Streaming, citations, easy follow-up |
| Embedded copilot | The user, inside a tool they already use | Around a second, or the suggestion is ignored | Ignored or edited before use | Small outputs, accept or reject in one action |
| Background agent | A trigger, schedule or delegated task | Minutes to hours | It may act before anyone looks | Progress, approvals, audit trail, undo |
| API / service | Another system | The caller's SLO, for the whole response | Bad output flows downstream silently | Schemas, validation, retries, idempotency |
2 · The latency contract: streaming changes which number users feel
Example: 0.5 s to first token, then 400 tokens at 40 tokens/s. Unstreamed, the user feels the total time and crosses the 10-second attention limit. Streamed, they feel only the time to first token, and the text outruns their reading. That is why chat surfaces optimise TTFT. Agents, API callers and structured outputs read the whole response before acting, so for them total latency is still the contract. Past about 10 seconds, switch to an async job with visible progress and a notification.
3 · Design for the wrong answer: autonomy is an interface choice
The same model can sit in all four boxes; what changes is how much the interface lets it do before a person looks. Pair this with the scoped tokens in Identity & policy: the interface asks for approval, the token enforces it.
4 · Every interaction is a label
| Signal | Kind | What it usually tells you |
|---|---|---|
| Thumbs, ratings, reports | Explicit | A direct judgment, but few users give one, and mostly when annoyed or delighted |
| Copy, accept, send unedited | Implicit | The output was usable as it was |
| Edit before use | Implicit | Usable but wrong somewhere, and the diff shows exactly where. The richest eval data you will get |
| Regenerate or rephrase | Implicit | The first answer missed the intent |
| Stop mid-stream or abandon | Implicit | Too slow, too long, or heading the wrong way |
| Escalate to a human | Implicit | The system could not resolve it; the case is a ready-made test |
Log each signal with the trace ID of the request, so it links back to the prompt, retrieved context and model version that produced it. That is how everyday use becomes eval sets and the production monitors in Governance by design.
5 · Design rules
The last rule is new. In January 2026, MCP Apps became the first official MCP extension: a tool can return an interactive interface that hosts such as Claude, ChatGPT, VS Code and Goose render in a sandboxed iframe. AG-UI standardises the event stream between agent back ends and front ends over SSE or WebSockets. Layer 1 is no longer only the screens you build; it is also the screens your tools bring with them.
Sources: J. Nielsen, Usability Engineering (1993), response-time limits of 0.1, 1 and 10 s; M. Brysbaert, “How many words do we read per minute?”, Journal of Memory and Language (2019), 238 wpm for silent reading of non-fiction; WorkOS, “MCP Apps” (27 Jan 2026); Thoughtworks Technology Radar, AG-UI protocol (Trial, Apr 2026). Tokens assume about 0.75 English words per token.Orchestration & agents
The control logic between the user and the model. It assembles the prompt, decides whether to retrieve, calls tools, loops until a goal is met, and manages what the model remembers. Most product behaviour lives here rather than in the weights.
Components
- Prompt templates and system instructions
- Tool / function calling with JSON schemas
- Planner–executor loops, routers, sub-agents
- Short-term (context) and long-term (store) memory
- Tool protocols such as MCP for connecting systems
Patterns
- Single call: prompt in, answer out
- Chain: fixed sequence of calls
- Router: classify, then send to a specialist model or prompt
- Agent loop: reason → act (tool) → observe → repeat, with a step budget
Inside layer 2 ↓ five patterns, the cost and reliability maths of the loop, and the context budget.
Who drives the loop, and what each turn costs
Orchestration is the code between the user and the model: it builds the prompt, calls tools and decides what happens next. Its first design choice is who picks the next step, your code or the model. Everything else in this layer — cost, reliability, how much context the model has to wade through — follows from that answer.
1 · Five patterns, from code-driven to model-driven
| Pattern | Who picks the next step | Predictability | Calls per task | Use it when |
|---|---|---|---|---|
| Single call | Nobody: there is no next step | Highest | 1 | Classifying, extracting, or drafting from context you already have |
| Chain | Your code, in a fixed order | High | Known in advance | The recipe is stable: outline, draft, check |
| Router | The model picks a branch once; code runs it | High | 1 decision + 1 specialist | Mixed traffic where a cheaper model or specialist prompt handles most cases |
| Orchestrator–workers | The model plans subtasks; workers run them, often in parallel | Medium | Depends on the plan | Research or code changes whose subtasks can't be listed up front |
| Agent loop | The model, every turn, until it stops or hits a budget | Lowest | Unknown until it finishes | Open-ended work where the path depends on what the tools return |
Anthropic's Building effective agents draws the line between workflows, where code fixes the path, and agents, where the model chooses it. Each row down buys flexibility and gives up predictability and cost control, so move down only when the task needs it. The router row is where a decision model like JEV fits.
2 · An agent re-reads its whole history on every turn
Example: a 4,000-token prefix, and each turn adds about 1,500 tokens of reasoning, tool call and tool result. Because every call re-sends everything before it, total input grows with the square of the number of turns, not in step with it. Prompt caching changes the bill, not the shape: cached reads cost about a tenth of normal input on the major APIs (less on some newer models), so this run costs roughly 80% less. The context is still 32,500 tokens long by turn 20, and the model still has to read all of it. That is panel 4. Output tokens are not counted here.
3 · Reliability compounds too
| Each step succeeds | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 99% | 95% | 90% | 82% |
| 95% | 77% | 60% | 36% |
| 90% | 59% | 35% | 12% |
If every step must succeed and failures are independent, the run succeeds at the per-step rate raised to the number of steps. A step that works 95% of the time sounds solid; chain 20 of them and the run finishes cleanly about one time in three. Real agents beat this table because they notice errors and try again — that is what the loop is for. But recovery only happens if it is designed in: tool errors the model can read, steps that are safe to retry, and checks before the steps where a wrong turn gets expensive.
4 · The context window is a budget, not a bucket
| In the window | Why it grows | What to do |
|---|---|---|
| System instructions | It shouldn't; this is the stable prefix | Keep it first and unchanged between turns, so it caches |
| Tool definitions | Every connected tool adds its schema to every call | Load only the tools this task needs; search for tools instead of listing hundreds |
| Retrieved content | Whole files and search results pasted in | Fetch just in time; keep references (paths, IDs) and pull content when needed |
| Turn history | Every turn appends to it | Compact: summarise older turns and drop stale tool output |
| Working notes | Plans and progress the model must carry forward | Write them to a file or memory store and re-read on demand |
| Sub-agent output | A helper returns its full transcript | Have sub-agents return a condensed result, not their working |
Bigger windows raise the ceiling; they don't make filling them free. Chroma tested 18 models in 2025 and found accuracy falls as input grows, even on simple retrieval tasks, a pattern it named context rot. On the LongMemEval benchmark, models did markedly better with a focused prompt of about 300 tokens than with the full history of about 113,000. Anthropic calls the discipline context engineering: give the model the smallest set of high-signal tokens it needs for the next step.
5 · Design rules
Knowledge & retrieval
Retrieval-augmented generation (RAG) grounds answers in your own documents without retraining. Quality is decided mostly by chunking and ranking, not by the model.
Ingest path
- Parse (PDF, HTML, tables) and clean
- Chunk: typically 200–800 tokens with overlap, or by document structure
- Embed each chunk into a dense vector
- Index in a vector store (HNSW, IVF-PQ) with metadata filters
Query path
- Rewrite or expand the query
- Hybrid search: dense vectors + BM25 keywords
- Rerank top-k with a cross-encoder
- Pack the best passages into the prompt with source IDs
Inside layer 3 ↓ seven failure points, hybrid search, long context versus retrieval, and how to measure the retriever.
Most RAG failures happen before the model writes a word
When a RAG system gives a wrong answer, the instinct is to blame the model or rewrite the prompt. More often the model never saw the right passage: it was never indexed, it ranked 31st, or it was cut when the context was packed. Generation can only be as good as what retrieval hands it, so this layer is where answer quality is won or lost.
1 · Seven places a RAG system fails
| Failure point | Stage | What it looks like | First fix |
|---|---|---|---|
| 1 · Missing content | Corpus | The answer isn't in any indexed document, but the system answers anyway | Coverage checks; a clear "not in the documents" response |
| 2 · Missed the top-ranked documents | Retrieve & rank | The right chunk exists but ranks below the cut-off | Hybrid search, query rewriting, a reranker |
| 3 · Not in context | Assemble | Retrieved, then dropped when the prompt was packed | Rerank before trimming; dedupe; budget the context |
| 4 · Not extracted | Generate | The answer is in the context, but noise or contradictions hide it | Fewer, better passages; flag conflicting sources |
| 5 · Wrong format | Generate | The model ignores the requested table, list or schema | Structured output with validation |
| 6 · Incorrect specificity | Generate | Too general or too detailed for what was asked | Clarifying questions; examples in the prompt |
| 7 · Incomplete | Generate | Leaves out information that was in the context | Break multi-part questions into sub-queries |
From Barnett et al. (2024), drawn from three production RAG case studies in research, education and biomedicine. The first three failures happen before generation, and an end-to-end "was the answer right?" score can't tell them apart from model errors. You only see them if you measure the retriever on its own (panel 5).
2 · Retrieval is a pipeline you can tune, step by step
Anthropic's 2024 Contextual Retrieval experiment, averaged across several domains. Contextual embeddings prepend 50–100 tokens of model-written context to each chunk before embedding it, so a chunk that says "revenue grew 3%" also says which company and which quarter. That one-time step cost about $1.02 per million document tokens with prompt caching. Adding keyword search (BM25) and then a reranker, which takes the top 150 candidates and keeps the best 20, cut missed retrievals by two-thirds without touching the generating model. These are a vendor's own benchmark numbers; the pattern, not the exact figures, is what transfers.
3 · Why hybrid search: meaning and exact matches fail differently
| Chunk | Dense rank | BM25 rank | RRF score | Fused rank |
|---|---|---|---|---|
| B · "Error ERR-4012: token expired" | 3 | 1 | 1/63 + 1/61 = 0.0323 | 1 |
| A · "Troubleshooting sign-in problems" | 1 | 9 | 1/61 + 1/69 = 0.0309 | 2 |
| C · "Resetting your password" | 2 | — | 1/62 = 0.0161 | 3 |
Query: "what does ERR-4012 mean?" Dense embeddings match meaning, so they rank general sign-in pages highly but blur exact strings like error codes, part numbers and names. BM25 matches the exact token. Reciprocal rank fusion combines the two lists without tuning: each list contributes 1 / (60 + rank), and the sums are re-sorted. The chunk that both methods rate well wins. Here, the exact error page rises to the top.
4 · A million-token window doesn't retire retrieval
| Question | Put it all in the window | Retrieve |
|---|---|---|
| How big is the corpus? | Fits in the window, and stays that size | Larger than any window, or growing |
| Who may read what? | Everyone can see all of it | Permissions differ by document or user |
| How often does it change? | Rarely; cache the prefix and reuse it | Daily or faster; re-index only what changed |
| Must answers cite a source? | Possible, but harder to trace | Each chunk carries its source ID |
| How does the question relate to the text? | It needs the whole document: summarise, compare, find conflicts | It needs a few specific facts |
Long context is the right call for small, stable, whole-document tasks, especially with prompt caching. For everything else, retrieval is cheaper per query, enforces permissions at the chunk level, and keeps the window small, which matters because accuracy falls as input grows (see context rot). The 150k figure assumes about 500 tokens per page.
5 · Measure the retriever on its own
| Metric | Question it answers | Catches |
|---|---|---|
| Recall@k | Did the right chunk come back in the top k at all? | Failure 2 |
| MRR / nDCG | How high did it rank? | Failures 2 and 3 |
| Context precision | How much of what we packed was relevant? | Failures 3 and 4 |
| Faithfulness | Is every claim in the answer supported by the context? | Hallucination on top of good retrieval |
| Answer correctness | Was the final answer right? | Everything, but can't say where it broke |
Start with a golden set of 50–200 real questions, each labelled with the passage that answers it, plus some questions the corpus can't answer. Run it on every change to chunking, embeddings or ranking. When answer correctness drops, the retrieval metrics tell you whether to fix the index or the prompt, which is the traceability Governance by design asks for.
6 · Design rules
Foundation model
Almost every current large language model is a decoder-only transformer: a stack of identical blocks that turns a sequence of tokens into a probability distribution over the next token. Variants change the attention pattern, the feed-forward layer, or add encoders for images and audio.
Building blocks
- Tokenizer (BPE / SentencePiece), vocab ~32k–256k
- Token embeddings + rotary positions (RoPE)
- Self-attention: multi-head, grouped-query (GQA)
- Feed-forward (SwiGLU) or Mixture-of-Experts
- RMSNorm and residual connections
Variants to know
- Dense: every parameter used per token
- MoE: router picks top-k experts; large total, small active params
- Multimodal: vision/audio encoder projected into token space
- Long context: RoPE scaling, sliding-window or sparse attention
Inside layer 4 ↓ mixture-of-experts: why a trillion-parameter model can cost less per token than a 405B dense one.
One transformer block
Each block reads the running representation (the residual stream), lets every token look back at earlier tokens through attention, then transforms each position independently with the feed-forward layer. Residual connections (dashed) add each sub-layer's output back to its input, which is what lets very deep stacks train.
- Attention mixes information across positions; cost grows with sequence length squared in prefill.
- Feed-forward holds most parameters (about two-thirds in a dense model) and much of the stored knowledge.
- GQA shares key/value heads across query heads to shrink the KV cache.
- MoE swaps the single feed-forward for many experts; only top-k run per token.
A trillion parameters, 32 billion at work
The block diagram above has a small note on the feed-forward layer: "or MoE experts". That one swap explains how open models grew past a trillion parameters in 2024–25 without getting proportionally more expensive to run. Kimi K2 holds about 1,040 billion parameters, yet each token passes through only 32 billion of them, about 3%. In a mixture-of-experts (MoE) model, "how big is it?" has two answers: total parameters set the memory bill, active parameters set the compute bill.
1 · How a token picks its experts
An MoE layer replaces one large feed-forward network with many smaller ones, the experts. A small learned router scores every expert for each token at each MoE layer, keeps the top k, and mixes their outputs by the router's weights. Attention, embeddings and norms stay shared. The idea is old: Shazeer et al. made it work at scale in 2017, and Google's Switch Transformer (2021) sent each token to one expert and pre-trained up to 7× faster than T5-Base on the same compute. Mixtral 8x7B (Dec 2023) uses 2 of 8 experts and, in Mistral's words, runs "at the same speed and for the same cost as a 12.9B model". Despite the name, experts rarely map to subjects: Mistral's analysis of Mixtral found no clear topic specialisation, with routing following syntax more than domain.
2 · Big in memory, small per token
| Model | Released | Experts per MoE layer | Used per token |
|---|---|---|---|
| Mixtral 8x7B | Dec 2023 | 8 | 2 |
| DeepSeek-V3 | Dec 2024 | 256 routed + 1 shared | 8 routed + the shared one |
| Llama 4 Maverick | Apr 2025 | 128 routed + 1 shared | 1 routed + the shared one; MoE on alternate layers |
| Kimi K2 | Jul 2025 | 384 | 8 |
| gpt-oss-120b | Aug 2025 | 128 | 4 |
The trend runs one way: more, smaller experts, and a smaller share of the model awake for each token, from 2 of 8 in Mixtral to 8 of 384 in Kimi K2. A shared expert sees every token and holds common knowledge, so the routed experts can specialise. Closed frontier labs rarely publish their architectures, so open-weight models are where the numbers are.
3 · The bill: compute follows active, memory follows total
| Model | Weights in memory | Compute per token ≈ 2 × active | vs dense 405B |
|---|---|---|---|
| Llama 3.1 405B (dense) | 810 GB at BF16 | 810 GFLOP | 1× |
| DeepSeek-V3 | ~671 GB at FP8 | 74 GFLOP | ~11× less |
| Kimi K2 | ~1.04 TB at FP8 | 64 GFLOP | ~13× less |
| Mixtral 8x7B | ~93 GB at BF16 | 26 GFLOP | ~31× less |
| gpt-oss-120b | fits one 80 GB GPU (4-bit MXFP4 experts) | 10 GFLOP | ~79× less |
Compute uses the rule of thumb of about 2 FLOPs per active parameter per token (Kaplan et al. 2020) and ignores attention over the context. Memory is weights only, before the KV cache and activations. The catch is in the second column: at FP8, DeepSeek-V3's 671 GB of weights don't fit on one 8 × 80 GB H100 server (640 GB) and need something like 8 × H200 (1,128 GB). MoE moves the bottleneck from compute to memory capacity and the links between GPUs. In return you get the quality of a much larger dense model: Mistral reported that Mixtral, with 12.9B active parameters, beat Llama 2 70B on most benchmarks.
4 · What MoE costs you
| Problem | Why it happens | What teams do |
|---|---|---|
| Memory capacity | Every expert must sit in GPU memory, even though only a few run per token | Size the cluster by total parameters; quantise expert weights (gpt-oss ships them at 4 bits) |
| Load imbalance | The router learns to favour a few experts; they become hot spots while others starve, or "collapse" and stop learning | Add a balancing loss (Switch); DeepSeek-V3 instead nudges a per-expert bias, avoiding an extra loss term; cap tokens per expert |
| Communication | With experts spread across GPUs, every MoE layer sends tokens out to their experts and back (all-to-all) | Keep expert groups inside the fastest interconnect; DeepSeek-V3 limits each token to experts on at most 4 nodes; overlap compute with transfers |
| Small batches | At low traffic each expert gets a handful of tokens, so its matrix multiplies are small and GPUs sit idle | MoE pays off at high, steady throughput; for quiet self-hosted workloads, a smaller dense model can be cheaper |
| Training stability | Router logits can grow and destabilise training; fine-tuning can shift routing and starve rare experts | Router z-loss (ST-MoE, 2022); evaluate per domain after fine-tuning; adapt shared layers first |
None of these show up in a benchmark table. They show up in the serving bill and the p95 latency, which is why an architecture review should ask for both parameter counts and the batch size behind any throughput claim.
5 · Dense or MoE? It depends on where you sit
| Your situation | Lean towards | Why |
|---|---|---|
| Calling a hosted API | Whatever wins your evals | The provider carries the memory cost. You see price per token and latency, so judge on those and on quality |
| One GPU or a laptop | Small dense, or a small MoE that fits | Memory capacity is the hard limit; gpt-oss-20b (21B total, 3.6B active) runs in 16 GB |
| High, steady traffic on a GPU cluster | MoE | Low compute per token, and full batches keep every expert busy |
| Low or spiky traffic, self-hosted | Smaller dense, or a shared MoE endpoint | Idle experts still occupy memory you pay for around the clock |
| Heavy domain fine-tuning | Dense is simpler | Fewer moving parts; MoE fine-tuning needs care with the router |
| Single-user latency, fits in fast memory | MoE often wins | Generating a token is limited by memory bandwidth, and only the active experts' weights are read per token |
For most enterprise teams the first row applies, and the architecture matters indirectly: MoE is one reason open models got cheaper to serve as they got stronger. It matters directly once you self-host, where the hardware and the traffic pattern decide.
6 · Design rules
Training & adaptation
A model passes through stages, each with different data and objectives. For most enterprise work the choice is not whether to pretrain but which adaptation to apply on top of an existing model.
Stages
- Pretraining: next-token prediction on trillions of tokens
- Supervised fine-tuning: instruction–response pairs
- Preference tuning: RLHF (reward model + PPO) or DPO
- Distillation: a small model learns from a larger one
Adaptation ladder (cheapest first)
- Prompting and few-shot examples
- RAG for knowledge that changes
- Parameter-efficient tuning: LoRA / QLoRA adapters
- Full fine-tune for behaviour and format at scale
Inside layer 5 ↓ why a 1.3B model beat a 175B one, how LoRA makes adaptation cheap, and the adaptation ladder.
A 1.3B model beat a 175B one
In OpenAI's InstructGPT study, people preferred answers from a 1.3-billion-parameter model to answers from the 175-billion-parameter GPT-3, a model with over 100 times more parameters. The architecture was the same. The difference was post-training: a few tens of thousands of prompts of human demonstrations and rankings, on top of a model pre-trained on hundreds of billions of tokens. Pre-training gives a model its knowledge; post-training decides how it uses it. For enterprise teams, the useful corollary is that adapting a model costs a tiny fraction of building one.
1 · Where a model's behaviour comes from
| Stage | Data | How much | What it changes |
|---|---|---|---|
| Pre-training | Raw text and code, next-token prediction | Trillions of tokens (Llama 3: 15T) | Knowledge, language, reasoning patterns |
| Supervised fine-tuning | Human-written example answers | ~13k prompts in InstructGPT; 1,000 in LIMA | Following instructions, answer format |
| Reward model | People rank several answers to the same prompt | ~33k prompts | A learned score for "what people prefer" |
| RL from human feedback | The model's own answers, scored by the reward model (PPO); DPO skips the separate reward model | ~31k prompts | Helpfulness, tone, when to refuse |
| Your adaptation | Your tasks, formats and documents | Hundreds to thousands of examples | Domain style, output schema, narrow skills |
Prompt counts are from the InstructGPT paper. Every stage after pre-training uses a sliver of its data, which is why post-training is where most teams, and most of a model's personality, actually live. LIMA (Meta, 2023) pushed this furthest: a 65B model fine-tuned on just 1,000 carefully curated examples gave answers judged equivalent or better than GPT-4's in 43% of cases. The authors' reading: knowledge comes from pre-training, and fine-tuning mostly teaches format.
2 · 100× fewer parameters, and still preferred
From Ouyang et al. (2022). Squares are drawn to scale by area. The method: fine-tune GPT-3 on demonstrations, train a reward model on human rankings, then optimise against it with reinforcement learning. It halved made-up facts on closed-domain tasks such as summarisation. It wasn't free: the paper reports an "alignment tax", small drops on some academic benchmarks, which they reduced by mixing pre-training data back into the RL step. Any adaptation can trade away skills you didn't test, which is why panel 6 starts with evals.
3 · Why adapting is cheap: LoRA
Low-rank adaptation (Hu et al., 2021) freezes the original weights and learns a small update, W + B·A, where B and A are thin matrices of rank r. On GPT-3 175B, adapting only the query and value projections at rank 4 matched or beat full fine-tuning on the paper's tasks, trained about 25% faster, and adds no latency at serving time once B·A is merged into W. QLoRA (Dettmers et al., 2023) went further: it fine-tuned a 65B model on a single 48 GB GPU by keeping the frozen weights in 4-bit, and its Guanaco model reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours of training.
4 · The adaptation ladder: climb one rung at a time
| Rung | Changes weights? | Good for | Wrong tool for |
|---|---|---|---|
| Prompting & few-shot | No | Format, tone, quick tasks; testing whether the model can do the job at all | Large or private knowledge |
| RAG | No | Facts that change, citations, per-user permissions (Inside L3) | Changing behaviour or style |
| LoRA / QLoRA adapters | A small add-on | Consistent output schema, domain style, narrow tasks; letting a small model replace a large one | Teaching facts that change often |
| Full fine-tune | All of them | Deep behaviour change, when adapters measurably fall short | Most enterprise needs; expensive to repeat on each new base model |
| Continued pre-training | All of them | A new language or a domain with its own vocabulary, given billions of tokens | Anything a lower rung solves |
Each rung costs more to build, more to evaluate and more to keep current. The usual pattern in production is a hosted or open model, plus RAG for knowledge, plus at most a LoRA adapter for format and style. Training from scratch is a separate decision, made by very few organisations.
5 · Fun fact: Llama 3 was trained "too long", on purpose
The Chinchilla rule (about 20 tokens per parameter, in the Layer 5 box above) gives the best model for a fixed training budget. But training happens once, and serving goes on for the model's whole life. A smaller model trained far longer costs more to build and less on every request after that, the same serving maths as the KV cache and MoE. Meta reported both Llama 3 8B and 70B "continued to improve log-linearly" up to 15T tokens. By the 6·N·D rule, the 8B run took about 7.2 × 10²³ FLOPs.
6 · Design rules
Inference & serving
Serving splits into two phases with different bottlenecks. Prefill processes the whole prompt in parallel and is compute-bound; it sets time-to-first-token. Decode produces one token at a time, rereads all weights and the KV cache each step, and is memory-bandwidth-bound; it sets time-per-output-token.
Techniques
- KV cache, with paged allocation (PagedAttention)
- Continuous (in-flight) batching
- Quantization: FP8, INT8, INT4 weights
- Speculative decoding with a draft model
- Prefix / prompt caching for shared system prompts
Metrics
- TTFT: time to first token (ms)
- TPOT: time per output token (ms)
- Throughput: tokens/s per GPU
- Goodput: requests/s that meet the latency SLO
- Cost per 1M input / output tokens
KV cache per token = 2 × layers × kv_heads × head_dim × bytes
e.g. 2 × 32 × 8 × 128 × 2 B = 128 KiB/token → 8,192-token context ≈ 1 GiB per sequence
Inference compute ≈ 2 · N FLOPs per generated token
The KV cache, explained
A language model writes one token at a time. To pick each new token, attention compares it with every earlier token using their keys (what each token offers to be matched on) and values (what it contributes once matched). Those keys and values never change once computed, so serving engines keep them in GPU memory instead of recomputing them. That store is the KV cache. It makes generation fast, and it is usually what limits how many people one GPU can serve.
1 · What the cache saves
To produce token 5, the model needs keys and values for tokens 1 to 5. Without a cache it recomputes all of them at every step. With a cache it computes only the newest one and reads the rest. Over a 1,000-token answer that is 500,500 key/value computations per layer without a cache, against 1,000 with one.
2 · What one token costs in memory
Every token in every active conversation holds this much, in every layer, for as long as the conversation is live. The shape shown is typical of an 8B-parameter model.
3 · How many conversations fit on one 80 GB GPU
Same GPU, same model. Only the context length changes, and capacity falls from 238 conversations to 3. Long context is a memory decision before it is a model decision.
Assumes an 8B model in FP16 (32 layers, 8 KV heads, head dim 128) and every conversation at full length. Ignores activations and runtime overhead, so real capacity is somewhat lower.4 · How architects shrink it
Make each token smaller
- Grouped-query attention: several query heads share one KV head
- FP8 / INT8 KV cache: halve the bytes per value
- Latent attention (MLA): store a compressed latent and expand on use
Store fewer tokens, waste less
- PagedAttention: allocate in small blocks, so little memory sits reserved but unused
- Prefix caching: a shared system prompt is stored once for all users
- Sliding-window attention: keep only the most recent N tokens in some layers
- Offload: move idle conversations to CPU memory or SSD
Data foundation
Every layer above depends on data that is clean, permissioned and traceable. The same pipelines feed pretraining corpora, fine-tuning sets, retrieval indexes and evaluation sets.
Components
- Ingestion from lakes, warehouses, SaaS sources
- Deduplication (exact + MinHash near-dup)
- Quality and toxicity filtering, PII redaction
- Labelling and synthetic data generation
Controls
- Lineage: which data trained or grounded which output
- Access control carried into the retrieval index
- Versioned datasets and eval sets
- Retention and consent rules
Compute & infrastructure
Accelerators, memory and the network between them. For large models, memory capacity and bandwidth matter as much as raw FLOPs, and the interconnect decides how well work splits across devices.
Hardware
- GPUs / TPUs / custom accelerators
- HBM capacity and bandwidth per device
- Intra-node links (NVLink) and inter-node fabric (InfiniBand, RoCE)
- Managed platforms: Azure AI Foundry, Amazon Bedrock / SageMaker, Vertex AI
Parallelism
- Data: copies of the model, split the batch
- Tensor: split each matrix across devices
- Pipeline: split layers into stages
- Expert: spread MoE experts across devices
- Sharded optimizers (ZeRO / FSDP)
Cross-cutting concerns
Evaluation
- Offline golden sets per use case
- LLM-as-judge with human calibration
- RAG: retrieval recall, faithfulness, answer relevance
- Online A/B tests and regression gates in CI
Safety & guardrails
- Input checks: prompt injection, PII, policy
- Output checks: grounding, toxicity, data leakage
- Tool permissions and least privilege for agents
- Red-teaming before each release
Observability
- Traces per request across retrieval, tools, model
- Token usage and cost attribution
- Latency (TTFT, TPOT) and error rates
- Drift in inputs and quality scores
Governance
- Model and prompt registry with versions
- Risk tiering and approvals per use case
- Audit logs and explainability records
- Regulatory mapping (e.g. EU AI Act, NIST AI RMF)
Identity & policy: on whose authority?
Compute and data platforms settled access control years ago with IAM: every call carries a principal, and policy decides what it may touch. AI systems reopen the question, because the caller is increasingly an agent acting for a person. Every layer, from retrieval to tool execution, now has to answer one question: on whose authority is this running, and with what limits?
1 · Authority should narrow at every hop
The agent never holds more access than the person it acts for, and each step down gets only what that step needs. If a token leaks or a prompt injection hijacks the agent, the damage stays inside the narrowest scope.
2 · The question each layer must answer
| Layer | Question | Control |
|---|---|---|
| L1 Application | Who is the user? | SSO with OIDC, MFA, session lifetime |
| L2 Orchestration | What may this agent do for them? | Delegated, short-lived tokens (OAuth on-behalf-of, token exchange), step budgets, human approval for high-impact actions |
| L2 Tools | What can this call change? | Per-tool scopes and allow-lists, OAuth on MCP servers, read-only by default, dry runs |
| L3 Retrieval | Which documents may this user see? | Source ACLs carried into the index and filtered at query time, before anything reaches the prompt |
| L6 Gateway | Which tenant, quota and model? | API gateway, tenant isolation, rate limits, data-residency routing |
| L7 Data | May this data be used for this purpose? | Purpose-based access, consent flags, lineage |
| Audit | Who did what, for whom? | Log the full chain per action: user → agent → tool → resource |
3 · Design rules
Governance by design: proving the system does what it should
Governance added at review time produces documents about a system. Governance built in at design time produces evidence of how it behaves. The rule is simple: every intended behaviour becomes a testable requirement, and every requirement leaves a trail that a reviewer or auditor can follow from intent to production.
1 · The traceability chain
- Intended useWhat the system is for, who uses it, what it must never do, and its risk tier.
- RequirementEach "must" and "must not" written so it can be measured.
- Design controlThe mechanism that enforces it: retrieval scope, guardrail, human approval, tool permission.
- Eval with a thresholdA test set and pass mark, run in CI. A failed gate blocks the release.
- Production monitorThe same check sampled on live traffic, with an alert and an owner.
- EvidenceVersioned eval reports, approvals, logs and incidents, produced by the pipeline itself.
Worked example, an HR policy assistant. Intent: answer employee questions from the approved handbook only. Requirement: no answer without a citation to the handbook. Design control: retrieval limited to the handbook index, and the response is rejected if it has no citation. Eval: 300 test questions, at least 95% grounded and zero uncited answers. Monitor: 2% of live answers scored for groundedness each week. Evidence: eval report per release, weekly monitor results, sign-off record.
2 · The evidence pack
| Evidence | Produced by | When | What frameworks ask for |
|---|---|---|---|
| Intended use & risk tier | Product owner with risk | Design | NIST AI RMF Map; EU AI Act risk classification |
| Requirements as eval sets | Engineering | Design and build | NIST AI RMF Measure; accuracy and robustness obligations |
| Model, data & prompt cards | Engineering | Every release | Technical documentation; data governance |
| Eval reports & gate decisions | CI pipeline | Every release | ISO/IEC 42001 performance evaluation |
| Human-oversight design & approvals | Accountable owner | Before deploy | Human oversight obligations |
| Logs, monitors & incidents | Platform team | Continuously | Record-keeping; NIST AI RMF Manage |
3 · The AI engineering governance skill set
Technical
- Eval design and metrics
- RAG, agents and guardrails
- CI/CD and MLOps pipelines
- Observability and logging
AI engineering governance
- Turn policy into testable requirements
- Set release gates and thresholds
- Design evidence that is generated, not assembled
- Explain model behaviour to auditors and the board
Governance
- Risk assessment and tiering
- Regulatory mapping
- Accountability and approvals
- Audit and assurance
The middle column is the scarce role: someone who can read a model card and a control framework with equal confidence.
The model is the smallest box in the system.
In 2015, a team of Google engineers published a paper called Hidden Technical Debt in Machine Learning Systems. Its most famous diagram shows a real-world ML system as a large set of boxes, with the ML code as a small black box in the middle. The paper warned that a mature system might end up being at most 5% machine learning code and at least 95% everything around it.
At most 5 squares in 100
A decade later, the boxes have new names
With foundation models the ratio has only grown. The model can now be one call to a hosted endpoint, while the retrieval, evals, guardrails, identity and serving around it are what you actually build and run. That is why this blog maps eight layers, and only one of them is the model.
Source: D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 2015. The 5% figure is the paper's estimate for glue-code-heavy systems, not a measured average.JEV: the model that decides instead of writes
JEV, from TypeSafe AI (early access since 15 September 2026), never writes a sentence. You hand it messy input plus a set of typed questions, and it returns typed answers, each with a calibrated probability. TypeSafe calls it the first System One model, after Kahneman's fast, intuitive System 1. Today's LLMs are System 2 machines: slow, deliberate, and expensive to ask for every small choice an agent makes.
1 · Where it sits: a decision gate in front of the model
Every agent step asks small, bounded questions: which tool, which model, retry or stop, allow or block. JEV answers those in one fast pass. When its confidence falls below a threshold, or the step needs text or free-form arguments, the call escalates to the foundation model. JEV does not replace the LLM; it decides when the LLM is worth calling.
2 · How it works
3 · Claims vs caveats
| Vendor claim | Figure | Read it as |
|---|---|---|
| End-to-end latency | 70–500 ms | Plausible: removing autoregression explains most of it |
| Speed vs frontier LLMs | 40–200× | Only on bounded decisions, not reasoning |
| Best single workflow | 193.6× faster, 444.6× cheaper | One showcase workflow, not an average |
| Input price | $0.042 per million tokens | Early-access pricing; may change |
No independent evaluation accompanied the launch. TypeSafe's own documentation lists weak spots: counting and maths, date ordering, numeric proximity, double negation, long context, adversarial inputs and CJK languages. The first academic study, REFLEX (Wu & Lim, NTU), found that JEV-style selective control saves compute only when a task actually contains bounded decisions to offload.
Sources: TypeSafe AI launch materials (15 Sep 2026) as reported by DigitalOcean, DataCamp and RuntimeWire; T. Wu and W. Y. B. Lim, “REFLEX with Jev for Efficient Selective Control in LLM Agents”, arXiv 2609.26532. All performance figures are vendor-reported.Every modern LLM traces back to 8 GPUs and 3.5 days.
In June 2017, eight Google researchers published Attention Is All You Need. They were not building a chatbot; they wanted better translation between English, German and French. Their larger Transformer trained on one machine with 8 NVIDIA P100 GPUs for 300,000 steps, about 3.5 days. Seven years later, Meta's Llama 3.1 405B used essentially the same decoder design, with refinements such as grouped-query attention and rotary positions, and trained on 16,384 H100s.
1 · The same idea, at a different scale
| 2017 Transformer | 2024 Llama 3.1 | Increase | |
|---|---|---|---|
| GPUs | 8 × P100 | 16,384 × H100 | ~2,000× |
| Parameters | 213 million | 405 billion | ~1,900× |
| GPU-hours | ~672 | 30.84 million | ~46,000× |
| Training data | 36M sentences | 15.6T tokens | — |
| Compute | — | 3.8 × 10²⁵ FLOPs | — |
Each H100 is also several times faster than a P100, so the real increase in compute is larger than the GPU-hours suggest. The 2017 model reached state-of-the-art translation (28.4 BLEU English–German, 41.8 English–French) at what its authors called a small fraction of earlier training costs.
2 · At scale, failure is routine
During a 54-day stretch of Llama 3 training, Meta recorded 419 unexpected interruptions, about one every three hours.
Automatic failure detection, fast checkpointing and restarts still kept effective training time above 90%. On 8 GPUs, a failure is an annoyance. On 16,384 it is a design requirement.
3 · What actually changed: the engineering
Lifecycle of one request
- Receive and authenticateGateway checks identity, quota and tenant (L1).I/O
- Input guardrailsScreen for prompt injection, PII and policy violations.I/O
- Plan and retrieveOrchestrator decides on retrieval; hybrid search + rerank return passages (L2, L3).I/O
- Assemble the promptSystem instructions, tools, retrieved context, history; cached prefix reused.I/O
- PrefillAll prompt tokens processed in parallel; KV cache written. Sets TTFT (L6).compute-bound
- DecodeOne token per step, streamed to the client. Sets TPOT (L6).memory-bound
- Tool calls (if any)Model emits a call, orchestrator runs it, result is appended, decode resumes.loop
- Output guardrailsCheck grounding, leakage and policy before release.I/O
- Log, trace and evaluateTrace, token counts, cost and feedback flow to observability and eval sets.async
Key trade-offs
| Decision | Options | Choose based on |
|---|---|---|
| Knowledge | RAG vs fine-tuning | RAG for facts that change or need citations; fine-tuning for style, format and narrow skills. |
| Model size | Frontier vs small / distilled | Task difficulty, latency SLO and cost per request. Route easy traffic to small models. |
| Hosting | Managed API vs self-hosted open weights | Data residency, volume (self-hosting pays off at high steady load), team capacity. |
| Precision | FP16 vs FP8 / INT8 / INT4 | Memory and speed gains against measured quality loss on your eval set. |
| Control flow | Fixed chain vs autonomous agent | Predictability and auditability vs flexibility. Start with chains, add agency where it earns its cost. |
| Context | Long context vs retrieval | Long context is simpler but costs more per call and can dilute attention; retrieval scales to large corpora. |
| Batching | Latency vs throughput | Bigger batches raise tokens/s per GPU but increase per-request latency. |