“With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” — John von Neumann
You are living through a shouting match about AI. Half the world says it is a mind being born, the other half says it is an overgrown autocomplete, and almost nobody on either side can tell you what the machine in front of them actually does. This essay shows you the machine.
It is just math; one equation, applied over and over, and this essay builds every tool it uses, starting from a thrown ball. Stay to the end and you will know what these models actually compute, why they cost what they cost, which of their limits more scale will cure and which it cannot, and roughly when the thing that today fills a warehouse will fit on your lap. Nothing here asks for trust. Every claim is checkable, and that is the point.
Strip away the word “intelligence” and here is what remains.
A large language model is a function approximator. Nothing more. It takes a state in, it puts a state out, and everything between is arithmetic. There is no ghost in it, and, despite the name, barely a neuron. That name is worth a moment, because nearly everyone has heard it and almost no one has been shown what it means. A brain cell adds up the signals from the cells wired into it and fires when the total runs high enough; the artificial “neuron” is a stripped-down cartoon of that, a node that adds up numbers from other nodes and passes one number on.
Wire enough of these nodes together and you have a “neural network.” Strip the biology, though, and what remains is plain arithmetic on a graph: nodes, weighted connections, and a rule for what each node forwards. So “neural network” is only one way to build a function approximator, a particular stack of multiplications and nonlinearities (not a straight line), and that stack is a choice, not a law. For every neural network, there exists a simple matrix (think Excel spreadsheet) whose elements map to the connections between “neurons” in a neural network. This is what we call an adjacency matrix: what there is, underneath, is a matrix representation of a very large computational graph; write that representation down honestly and it stops looking like magic and starts looking like something physicists have handled before.
I studied computational physics, so when I first looked at a transformer I did the thing that training makes reflexive: I tried to write it as an evolution equation, a state pushed forward by an operator. Once I did, most of the machine-learning vocabulary fell away, replaced by things numerical physicists have worked with for a century. I want to give you that representation. Not the engineering jargon; the linear algebra jargon. If you have ever solved a system by stepping a field forward in time, you already have the intuition; the vocabulary the field uses is what hides it from you.
If you have not, here is the whole idea from scratch, in a single thrown ball. At any instant the ball’s situation is just a few numbers: where it is and how fast it is going. A short, ordered list of numbers like that is all a vector is; call this one the state. One tick of the clock later the state has changed by a fixed rule: the new position is the old position plus the velocity, and the new velocity is the old one adjusted by forces that depend on where the ball is and how fast it is going. Look at the shape of that rule. Each new number is a weighted blend of the old numbers, a little recipe of the form so much of this plus so much of that. Line those recipes up, one row for each number you want out, and the table you get is all a matrix is. Applying it, running every recipe on the old numbers to produce the new ones, is exactly what “multiplying the state by the matrix” means:
That table M is the operator: the law of motion, written so that one step forward in time is one multiplication.
To make it concrete, throw a ball straight up. Its state is two numbers, its height h and its velocity v, and one tick Δ t later the height has gained the velocity while the velocity has lost a fixed bite of gravity, gΔ t. That fixed bite is the one piece that is not a blend of the old numbers, so you carry a constant 1 alongside them to hold it, and the whole step becomes a single multiplication:
Reading it is exactly the recipe idea from before: to get each new number, run along the matching row of the table, multiply each entry by the state number it pairs with (the first by h, the second by v, the third by 1), and add them up. The top row, [1, Δ t, 0], gives 1· h + Δ t· v + 0· 1 = h + vΔ t, the old height plus a tick’s worth of velocity. The middle row, [0, 1, -gΔ t], gives v - gΔ t, the velocity minus a tick’s worth of gravity. The bottom row just copies the 1 forward. That is all a matrix multiplication is, one row at a time; and that three-by-three table is the operator M, which you apply tick after tick to trace the arc of the throw.
To follow the ball for a hundred ticks, you apply M a hundred times. Richer physics, more forces or another moving part, only means longer lists and a bigger table of recipes; the form does not change.
One more thing the table buys you, worth planting now because training will grow out of it later. Throw the ball a hair differently — a little more height, a little more speed — and call the difference δ, the physicist’s symbol for a small variation you chose to make. The nudge is itself a tiny state, and the same multiplication that carries the ball forward carries the nudge forward: after one tick the perturbed throw is M(state + δ) = Mstate + Mδ. States and the nudges riding on them obey the same algebra. Hold onto that.
Everything below is this picture, scaled up. The state stops being a handful of numbers and becomes thousands. The operator stops being fixed and is rebuilt at every step, out of the very state it is about to move. But the equation, new state equals operator times old state, is the one that runs the whole essay. Where the notation thickens from here, that one line is what it is always saying; a reader who is after the ideas more than the algebra can watch the picture and let the symbols wash past.
One general claim first, because it is what licenses everything after. Recall the adjacency matrix: number a network’s neurons 1 through N, and W is the table of their connection weights, Wᵢⱼ the weight from neuron j into neuron i, zero where there is no edge.
Figure 1. From wiring to operator: a feedforward network, its adjacency matrix W (entry Wᵢⱼ: the weight from neuron j into neuron i; strictly triangular because every edge points forward), and the resolvent (I-W)⁻¹ = I + W + W², which sums every path through the graph at once. The series stops by itself: this graph has no path longer than two edges.
Watch it become an operator. Run the network once and freeze the picture: let a be the vector holding every neuron’s activation in that run, all of them at once, and let input be the vector carrying the externally supplied values at the input nodes and zero everywhere else. Each neuron’s rule — output equals a nonlinearity of the weighted sum of its inputs — is one equation, and stacked together they read a = σ(W a + input). Read that equation carefully, because it does not compute a; it constrains it. It is a self-consistency condition, and a is defined as the vector that satisfies it. Whether such a vector exists, uniquely, is exactly where the wiring matters, and we will collect that debt in a moment. Set the nonlinearity aside first and the condition is linear, a = W a + input, with solution
Because a feedforward network’s edges only point forward, W is nilpotent and that series is finite: it is precisely the sum over every directed path through the graph, each path weighted by the product of its edges. And there is the debt paid: nilpotent W means det(I - W) = 1, so the self-consistency condition has exactly one solution; a is well-defined because the graph has no cycles. Order the neurons so every edge points from a lower index to a higher one and W is strictly triangular, which makes the familiar forward pass — compute each neuron once its inputs are ready — nothing more than forward substitution. A forward pass is a triangular solve. Wire in a feedback loop and all of this evaporates: the same equation becomes a genuine fixed-point problem that may have no solution or many, which is one more reason recurrent networks are harder animals. A physicist knows (I - W)⁻¹ by name. It is the resolvent, the Green’s function of the graph, the propagator that carries a signal from input to output. The whole linear network is one operator, read straight off its wiring.
The nonlinearity does not break this; it only makes the operator depend on the state. Write σ(z) = D(z)z with D the diagonal of gains from the feed-forward block, and the same algebra gives an operator of the same shape, (I - D W)⁻¹ D, whose gates D depend on the activations passing through them. The wiring W is fixed; the operator it induces is rebuilt for every input. There is the thesis in one line: a neural network is its adjacency matrix, turned into an operator and made data-dependent by its nonlinearities. The transformer is this for one particular, enormous, cleverly wired W, with the twist that its sharpest edges, the attention weights, are themselves computed from the data. The rest of the essay is what that wiring looks like, and what it costs.
Start with the field.
Every token becomes a vector. To the model the word “king” is not a symbol; it is a point in a high-dimensional space, something like [0.7,-0.2,0.5,…] with a few thousand coordinates. Those coordinates are not arbitrary labels. They are positions in a space where direction carries meaning, which is why the famous arithmetic works: the vector for king, minus the vector for man, plus the vector for woman, lands near the vector for queen. “Royalty” is a direction. “Female” is a direction. You can add them. A token embedding is a coordinate in a space of meaning.
And nobody chooses the coordinates. The embedding is a lookup table — one row of d numbers per token in the vocabulary — and the rows begin as random noise and are trained like every other weight in the machine; when training runs its corrections backward, they flow all the way down into this table. The geometry is not designed. It emerges, because arranging meanings so that nearby things are nearby and directions carry analogies turns out to be what minimizes prediction error. The map of meaning is not the cartographer’s; it is the residue of a trillion corrections.
A sentence is a sequence of these vectors, one per token. Stack them nose to tail into a single tall column, and that column is the state; call it X. If each token vector has d coordinates and the sentence is T tokens long, then X is a column of length dT: the first d entries are the first token, the next d are the second, and so on down the sentence. This is not a metaphor. It is the object in memory, and folding the whole sentence into one column is exactly what lets a full layer be written as a single multiplication.
Figure 2. The state X: each token’s embedding vector, stacked nose to tail into one tall column of length dT.
You already accept that anything can be written as a function of something. Good. Then accept the next line, because the whole architecture lives in it.
Here Mₙ(Xₙ) is a dT × dT matrix. It takes the entire stacked state and returns the next one in a single left-multiplication, and because it spans the whole column it can move information both within a token and between tokens at once. That is the whole game.
The subscript on Mₙ is not decoration. Each layer carries its own weight matrices, so the rule that builds the operator is itself different at every step; in the language of dynamics, the system is non-autonomous, like a force that changes with time. There is no separate step size to write down either: the Δ t of the analogy has been absorbed into the weights and fixed at one. The revealing exception is a model that shares one set of weights across all its layers, so that Mₙ = M for every n. Then the equation really is a single operator applied over and over, and the depth becomes literal recursion.
Figure 3. One layer as a single matrix–vector product: the dT × dT operator Mₙ(Xₙ) acting on the stacked column Xₙ to produce Xₙ₊₁.
A layer takes the current state, builds an operator M from it, and applies that operator to produce the next state. (In practice a normalization sits in front of the operator: pre-LN transformers center and rescale each token onto a sphere before M acts, a mild nonlinearity that is itself one of the conditioning devices below.) That is one step. Stack a hundred of them and you have a “deep” network; the depth is just the number of steps. “Deep learning” is a marketing term for a long integration.
And because every step is a left-multiplication, a stack of them is just a product of operators, ML ⋯ M₁, applied to the input. For any given input, then, the whole network is one data-dependent operator acting on that input; a physicist would call such a product a propagator. Depth adds no new kind of operation. It only deepens how much that single operator depends on the input.
The load-bearing symbol is the parenthesis. Mₙ(Xₙ). The operator is not fixed. It is manufactured on the spot, out of the state it is about to act on. The softmaxes and nonlinearities that people gesture at when they say “neural network” are nothing but the recipe for filling in the entries of M from the current X. The network does not have an operator. It grows one, uses it once, discards it, and grows another at the next step.
Say that plainly, because it is the whole point. A transformer is a solver that rebuilds its own operator at every step of the integration.
And the moment the operator is written explicitly, everything we know about operators is on the table. Spectra. Bases. Rank. Conditioning. Adjoints. Pseudospectra. Nothing here is a new model; it is the same computation the engineers already run, rewritten in a different basis and mathematically identical to theirs. But changing notation earns its keep only when it reveals structure, and this change reveals a great deal: written as an input-dependent operator, the transformer exposes a century of numerical linear algebra that the interpretability toolbox has almost entirely ignored. It is the move a physicist makes when Hamiltonians and Lagrangians and path integrals turn out to be one mechanics in different clothes, each cut to make a different structure obvious. This basis makes the operator obvious, and hands you the tools built for exactly that. That is what the jargon costs you.
The operator carries the state from the first layer to the last; one step is left, and it is the mirror of the first. A token entered as a vector, its embedding; to turn the final state back into a token you run that map in reverse. Take the last token’s vector at the top layer and compare it, by a dot product, against every token’s embedding. Tokens whose embedding the final state points toward score high, and a softmax turns those scores into a probability across the whole vocabulary. Usually the same embedding table is reused for this, transposed, so the model’s last act is to ask which token its final state most resembles.
That distribution is the prediction, and it is why there is no single correct next token. There is a spread of plausible ones, and the model commits to a continuation only by sampling from the spread. Append the sampled token and the sequence is one longer; run the whole operator again for the one after. That outer loop, generate-append-repeat, is the autoregression the later sections lean on, and the reason generation is sequential even though training is not.
The loop ends the way everything else in the machine works: one of the tokens in the vocabulary means stop, and when the model samples it, generation halts. Knowing when to finish is not wired in anywhere; it is one more thing the distribution learned.
And none of it is forced. Predicting one token at a time is a choice; you could train the operator to emit a whole answer in a single pass, as non-autoregressive and diffusion models do. The one-token-at-a-time objective simply happens to be cheap to train and strong in quality, at the cost of a slow sequential decode. Even the output is a design decision, not a law.
The celebrated ingredient is called “attention.” The name suggests focus, a mind choosing what matters. Drop that picture. It is not what happens.
What happens is an all-to-all comparison. Every token is turned into three vectors: a query, a key, a value. Every query is compared against every key. Every one, against every other. The result is a dense table of couplings, and that table is how the off-diagonal entries of M get filled: it sets how strongly the value at each coordinate pulls on the value at every other coordinate.
In the engineer’s own notation this is three matrices, and the cleanest way to see them is to lay the tokens back out as columns for a moment: the d × T layout, before we stacked everything into one column. There, WQ, WK, and WV turn the state into queries, keys, and values, Q = WQ Xₙ, K = WK Xₙ, V = WV Xₙ, and the coupling table is Aₙ(Xₙ) = softmax(Qᵀ K / √d), where the softmax turns each row of raw scores into positive weights that sum to one: a T × T grid whose row i says how much token i draws from each of the others. This object deserves a better name than the literature gives it, and it will keep earning the name for the rest of the essay: call it the routing matrix, because that is its entire job. It says which tokens feed which, and how much.
Read one entry at a time. To fill αᵢⱼ, take token i’s query and token j’s key, two d-vectors that are fixed linear images of the two tokens’ states, and dot them, giving qᵢ · kⱼ, a single raw score that runs large when the two vectors point the same way and small when they do not. That dot product is the entire comparison, and it is the only reason Q and K exist. Divide by √d so the scores do not grow with the dimension, then send the row through the softmax, αᵢⱼ = eqᵢ · kⱼ / √d / Σj' eqᵢ · kj' / √d, which exponentiates each score and rescales the row to sum to one, letting the largest come to dominate; softmax is a smooth stand-in for picking the single best match.
Each αᵢⱼ is then just a normalized similarity between two tokens, a number between zero and one, and the grid Aₙ is every token scored against every other. Both marks on Aₙ(Xₙ) matter: the attention matrix depends on the step n, through that layer’s own WQ and WK, and on the state Xₙ it is looking at. I will write Aₙ for short.
Stop and ask the question a physicist should: why project at all? The state already has a dot product; why not score tokens by raw similarity, xᵢ · xⱼ, and skip Q and K entirely? Because raw similarity is the wrong relation, twice over. It is symmetric, and the relations that matter in language are not: the adjective seeks its noun in a way the noun does not seek back. Fold the projections together and the score is sᵢⱼ = xᵢᵀ (WQᵀ WK) xⱼ — a learned bilinear form, a metric on meaning-space that training gets to choose, with nothing forcing it symmetric. Q asks what am I looking for; K answers what do I advertise; the projections exist so those can be different questions asked of the same vector. And each head’s form is low-rank, probing a few directions, so different heads learn different notions of relevance instead of sharing the embedding’s one frozen geometry. Could a big enough generic network learn all of this with no such machinery? In the universal-approximation limit, surely. But the same two thin matrices are reused at every position, on sequences of any length, handing gradient descent a cheap, low-dimensional knob labeled which relationships matter; a generic net would have to rediscover pairwise routing from scratch, with parameters that grow with the sequence instead of riding along it. The projections are not decoration on the network. They are the prior that makes the routing learnable.
The attention step is then two ordinary multiplications, one on each axis:
Left-multiplying by WV mixes the channels; right-multiplying by Aₙᵀ mixes the tokens. Two factors, two axes, and the softmax is the only nonlinearity. This is the form that is pleasant to compute with, and it is why implementations keep the state as a d × T matrix.
But two-sided multiplication is not a single operator, and the single operator is the thing we came for; you cannot take the spectrum of “multiply on the left and on the right.” So restack the columns into the one tall dT vector and the two-sided step collapses into a single left-multiply. The rule that translates one form into the other, true for any Xₙ, is the Kronecker product:
(On the left, Xₙ is the d × T matrix; vec stacks it into the tall column.) In words, replace each entry αᵢⱼ of the attention matrix with a whole copy of WV scaled by that entry. Add the identity for the residual connection (the standard trick of adding a layer’s input back to its output, so the operator carries an I), and the entire attention layer is a single operator, Mₙ = I + Aₙ ⊗ WV, written out as a grid of d × d blocks:
Each entry is a d × d block, not a number: αᵢⱼ WV is all of WV scaled by the single attention weight αᵢⱼ, so the whole of WV sits in every block, and the I on the diagonal is the d × d identity, which is the residual connection. (The transpose on Aₙ is only bookkeeping about which index is which; it does not change the spectrum.)
Now every name you know has a home. Q and K never appear in the blocks at all; their only job was to produce the scalars αᵢⱼ = softmax(Qᵀ K / √d) that fill the grid. And V never appears as its own symbol, because the operator does not precompute it. WV sits inside every block; when Mₙ multiplies X, each block applies WV to one token’s slice, and that application is the value projection V = WV X, built on the fly and immediately scaled by αᵢⱼ. Carry out the multiply and token i emerges as Σⱼ αᵢⱼ WV Xⱼ = Σⱼ αᵢⱼ vⱼ, the attention-weighted sum of value vectors, exactly as before. The engineer computes V once and caches it; the operator rebuilds it inside the product. Same values, different bookkeeping. The softmax is where all of the operator’s nonlinearity lives.
In real transformers the block is not one term but a sum of several, one per what engineers call an attention head: each head builds its own coupling from a low-dimensional slice of the state, and an output projection WO glues them back together, so the block is Σₘ α⁽ᵐ⁾ WO⁽ᵐ⁾ WV⁽ᵐ⁾. The name suggests separate mechanisms, but in the operator they are only terms of a sum, one way of writing a single coupling block as several low-rank pieces. A head is a unit of bookkeeping, not a unit of computation, and everything said about the block still holds. And the reason the operator is dense, the reason it costs O(T²), is now something you can see: no block is empty. Every token pair gets one.
And now the thesis is literal, not a figure of speech. Stack the state into its tall column and one whole layer is vec(Xₙ₊₁) = Mₙvec(Xₙ): a single matrix times a single vector, Xₙ₊₁ = Mₙ Xₙ, with attention, values, and the residual all folded into one operator. That is the entire point of writing it this way. You never multiply this matrix by hand; the factorization below is how it is computed. But the operator is the object you reason about, and its spectrum, later, is the only place a layer’s work on this particular sentence becomes visible. Compute with the factors; think with the operator.
The operator itself factors, and the split is the most revealing move in the essay. Because WV is the same for every input, pull the coupling into two operators applied in turn:
The content operator I ⊗ WV applies the same fixed map to every token’s slice: it mixes channels, it is what turns each token into its value vector, and it never looks at a token’s neighbors. It is frozen, identical for every input; you could diagonalize it once, offline, and never look again. The routing operator Aₙ ⊗ I does the opposite: it leaves every channel untouched and mixes the tokens, blending each token’s values into the others by the pure numbers in Aₙ. Content transforms in place; routing moves between sites. (Unstack the tall vector back into the d × T matrix and this pair is the two-sided multiply Xₙ₊₁ = Xₙ + WV Xₙ Aₙᵀ that implementations run: the same operator, cheaper arithmetic.)
This is the split that matters for everything after. Of the two operators, only one changes with the input. The content operator is frozen; the entire per-sentence character of the layer lives in the routing operator, in Aₙ, which is only T × T, a small matrix of numbers, not the dT × dT monster. Whatever the model is doing to this sentence, it is doing it there.
Every transformer layer pairs attention with a second operator: the feed-forward network, or MLP (multi-layer perceptron), a small network W₂σ(W₁ X) with σ a pointwise nonlinearity, applied to each token on its own. In the operator picture it is the same block matrix with the grid emptied out. Its operator is block-diagonal; only the (i, i) blocks are nonzero, because it works on each token alone and never mixes them. And like attention it is still an input-dependent linear operator: write the pointwise nonlinearity as a gating diagonal, σ(z) = D(z) z with D(z) = diag(σ(z)/z), and each block is W₂ D(W₁ X) W₁, fixed matrices wrapped around a diagonal that depends on the input. Do not mistake local for minor: those diagonal blocks hold most of the model’s parameters, and much of what it factually knows is stored there.
In the physics picture the two operators split cleanly. Attention is the nonlocal coupling, the term that moves information between sites. The MLP is the on-site potential, the term that acts on each site in place. A full layer is one of each: couple, then act locally.
Figure 4. Building the operator from the state: Q, K, V are projected from X; Q and K form the T× T routing matrix Aₙ = softmax(Qᵀ K/√d), and Aₙ then weights the values V into the attention output.
Physics has a name for this kind of operator, and it is worth a moment to earn it. Some rules are local: to update a point, you look only at its neighbors. Simulating heat through a metal bar works this way — each spot warms or cools according to the spots touching it — and the little neighbors-only template the solver stamps down at every point is called a stencil. Other rules are global: every point affects every other, with a strength set by some formula, the way every mass in the solar system pulls on every other mass at once. A rule like that — a table saying how strongly each point feeds each other point, so that every output is a weighted sum over all the inputs — is called a kernel. The routing matrix is exactly one: a nonlocal kernel, reaching across the entire domain at once, the opposite of a stencil. Attention is not a lens. It is a field in which everything acts on everything, by default, because it was never told it was allowed to ignore anything.
And notice what the construction never reads: the slot index. Every weight matrix acts identically on every token’s slice, and the couplings are dot products of contents alone, so the operator as built is permutation-equivariant; shuffle the tokens and the outputs shuffle with them, none the wiser. Word order, which is most of syntax, is invisible to it. So position has to be injected somewhere, and every deployed remedy is one more design decision: add a position-dependent vector into each token’s content, as the original transformers did; add index-dependent terms directly into the coupling scores, as ALiBi and T5’s relative bias do; or rotate each query and key by an angle set by its position, as the RoPE scheme in current frontier models does, which makes the similarity itself depend on the distance j-i. Position is not something attention knows. It is something the designer smuggles in, by one door or another.
One more thing the picture needs before it is honest. In the models that generate text, the grid is not full: token i is allowed to draw only on tokens j le i, so the routing matrix is triangular. Nothing about attention requires this; it is the shadow of the training objective. All T next-token predictions are scored in one parallel pass, so each position must be blinded to the future it is graded on; and at generation time the mask costs nothing, because the future tokens do not exist yet. An encoder model like BERT, trained on a different objective, runs the full dense grid. The triangle is not architecture. It is the one-token-at-a-time choice, showing up in the operator’s shape.
None of this is new physics and none of it is new engineering. It is one object under two vocabularies, and the dictionary is short.
The dictionary is honest but not complete. I have said nothing about how text becomes tokens in the first place (a learned subword vocabulary, its own nontrivial map), the caches that make generation affordable, the temperature knob on the sampling, the variants that skip layers or vary depth per token, or the second round of training that shapes a raw predictor into an assistant. Every one of them matters to an engineer. None of them changes the picture: state in, operator built, operator applied. That invariance is why the simplifications are safe.
Everything after this point is a statement about that matrix. The jargon hides it. The matrix does not.
Where do the numbers in M come from? They are learned. The weights that build every operator, the WQ, WK, WV and the rest, begin as random noise, and training tunes them by gradient descent: define an error, meaning how wrong the model’s next-word guess was, and nudge every weight a little in the direction that lowers it. Repeat until the error stops falling. That is the whole of the idea; the hard part is the bookkeeping. With billions of weights, how do you find which way is downhill for all of them at once? Computing each separately, by nudging one weight and re-running the whole network, would take billions of forward passes. Backpropagation gets them all in a single backward pass, and in the operator picture it is one idea: run the propagator in reverse.
Here is the whole thing, concretely. Run the forward pass and the model produces a final state XL, and from it a guess at the next token. That guess is never exact; the truth is the guess plus some error. Write that error as ε, the correction that would have to be added to the output to set it right. The loss measures how large ε is, and the whole of training is a single objective: drive ε toward zero, averaged over every example the model ever sees. What follows is how one example is spent toward that end.
Take εL for that correction at the output, the small nudge to XL that would shrink the gap. Only the output has a target; the layers in the middle do not. But you can still ask what correction at each earlier layer would have produced the output’s correction, and answering that, layer by layer, is the whole of backpropagation.
First watch a nudge move forward, because that is where the machinery comes from; it is the thrown ball’s δ again, with one new wrinkle. Give a layer a nudged input, Xₙ₋₁ + δ Xₙ₋₁, and expand what comes out. For the ball the answer was just Mδ: the operator carries the nudge. Here that happens too; but the transformer’s operator is built from the state it acts on, and the nudge rides into the construction. The queries and keys are cut from the state, so they shift with it, WQδ X and WKδ X; the shifted queries and keys shift the scores; and the softmax turns shifted scores into shifted couplings. The operator itself is rebuilt a little, and that change in Mₙ, applied to the old state, contributes a second shift. Collect both pieces and, for a small variation, the output’s shift is still one linear map acting on the input’s shift:
where Jₙ is Mₙ plus the rebuild term; the softmax’s derivative lives in that second piece. This Jₙ is the layer’s Jacobian, and it is the law of motion for nudges: throw a hair higher and the whole arc shifts, and the shift itself follows a rule, tick after tick. Variations ride the stack forward the same way states do, one multiplication per layer.
Training asks the reverse question, and here the two symbols earn their difference. δ was a variation: hypothetical, yours to choose, pushed forward to see what would happen — a creature of calculus, honest only to first order. ε is not chosen; it is measured. The data says the next token was this; the model said that; εL is the gap between them — a plain subtraction of two concrete vectors, with nothing infinitesimal about it. Keep the taxonomy, because the symbols will keep it for you: δ is calculus, a first-order variation; ε is arithmetic, a finite measured difference; and Δ, when it lands on the weights below, is a finite change actually applied. Discrete objects for a discrete machine. You know the shift you want at the output because the training data hands it to you; what you need is the shift at each earlier layer that would have produced it. So pull the correction backward one layer at a time, through the transpose:
If that derivative looks exotic — a matrix differentiated by a matrix — remember the stacking: Xₙ here is the tall column, so Jₙ is an ordinary Jacobian, output vector by input vector, a dT × dT matrix and nothing stranger. And if you ever ran backpropagation by hand, you already own all of this. Your εL was prediction minus target, and that is literally the loss gradient, for squared error and for softmax cross-entropy alike. Your layer rule — multiply the error column by Wᵀ, gate it elementwise by σ' — is exactly Jₙᵀ εₙ for a plain layer, where Jₙ = DW. The Jacobian is not new machinery. It is the name for the recipe you already know, general enough to survive the transformer’s one twist: here the operator moves with the state, so Jₙ carries the rebuild term along with Mₙ.
Start with εL and apply this down the stack. The forward pass ran the state through ML ⋯ M₁; this runs the correction back through J₁ᵀ ⋯ JLᵀ, the same product of operators, transposed and reversed; the same Jacobians that carry the forward nudges, run the other way.
Now the part you were after: how the error reaches the weights. Layer n produced Xₙ = Mₙ Xₙ₋₁, and you want it to have produced Xₙ + εₙ. But you never reach in and edit Mₙ to make that happen; you can’t, really. Mₙ is the huge dT × dT object, assembled from a handful of small weight matrices (WQ, WK, WV, and the rest) and the data, and it is thrown away and rebuilt on every pass. What you actually adjust are those small matrices, and εₙ is the term that shows up in the gradient of every one of them. Each gradient is a compact outer product of εₙ with the activation that weight acted on. For the value weights,
a d × d update: the correction, routed by the attention pattern, times the input.
The queries and keys are the subtler case, worth spelling out, because they never touch the output directly; they only set the couplings. Their updates arrive through the rebuild term. Dot the layer’s error against the values and you learn how each coupling should have differed — which tokens should have drawn more from which, a desired nudge to every αᵢⱼ. The softmax’s derivative converts those into corrections on the raw scores; and each score is a dot product qᵢ · kⱼ, so its correction splits two ways: nudge qᵢ toward (or away from) kⱼ, and kⱼ toward qᵢ. Outer products with the input then turn those nudges into Δ WQ and Δ WK. Same εₙ, one chain-rule step deeper: the error first says how the traffic should have flowed; the traffic then tells the projections how to move. So the one error signal is baked in all over the layer, but it lands on the parameters, not on Mₙ’s cells: you move a few small matrices, and on the next pass Mₙ reassembles from them and leans the way the correction asked. That is the phantom term you remembered, and it is why the update is cheap: one shared εₙ, a few small outer products, never the dT × dT monster.
Repeat over trillions of examples and that is the whole of training: what falls out is a set of weights whose operators fit the data they were trained on. What makes it affordable is the reuse: εₙ is computed once per layer, and every weight in that layer reads from it, so one backward sweep produces every update in the model at once. Working one weight at a time would cost a forward pass apiece; carrying ε back costs one pass for all of them.
Notice what that makes the data. Every ε in training is measured against an example some pipeline chose, so the weights end up a compression of those choices: the parameter count sets how much the machine can hold, and the data decides what was worth holding. The two are equally load-bearing, and the second is the murkier one, because nobody has a clean definition of good training data. That is why an entire industry — Scale AI and a raft of labeling and curation companies — exists to manufacture it: humans ranking answers, labeling examples, writing down the truth that ε gets measured against. The model is arithmetic all the way down. But the target the arithmetic was bent toward was chosen by people, one label at a time.
That is also why a physicist recognizes it. What we just derived is the adjoint method, the same reverse-time computation used for sensitivities in seismic imaging, optimal control, and weather assimilation; the recurrence εₙ₋₁ = Jₙᵀ εₙ is its adjoint equation, and everything above is that method worked for this particular flow. Backpropagation is not a machine-learning invention; it is the adjoint of a flow, rediscovered.
And once backprop is a product of Jacobians, its failure modes are the failure modes of any long product. Multiply L matrices whose singular values sit a little above one and the result explodes; a little below one and it vanishes. That is the exploding and vanishing gradient, and it is not a quirk of neural networks; it is what products of operators do. The cure is already in the formula. The identity in Mₙ = I + ⋯, the residual connection, keeps every Jacobian close to I, so the product stays near the identity and the gradient neither dies nor detonates. It is forward Euler kept stable by never straying far from the identity. Layer normalization and the 1/√d inside the softmax are the same instinct: scale controls that keep the operator’s entries from saturating. When a large model’s loss spikes mid-training, it is usually one of these conditioning devices losing its grip.
Even when the product is stable it is badly conditioned. The curvature of the loss spans many orders of magnitude, a few stiff directions among vast flat ones, and plain gradient descent is forward Euler on a stiff system: the step must be small enough for the stiffest direction, so it crawls along all the others. That is the real reason training is slow, and it is why second-order methods, K-FAC and Shampoo and natural gradient, help at all. They are preconditioners, rescaling the step per direction, exactly as a physicist preconditions a stiff solve.
That refusal has a price, and the operator makes it plain. If every coordinate couples to every other coordinate, the number of couplings is the number of tokens squared. Double the length of the text and you quadruple the work. This is the quadratic wall, O(T²), and it is not an implementation accident. It is the direct cost of building a dense operator with no assumed structure. Clever bookkeeping has kept the machine from holding the whole table in memory at once, but bookkeeping does not touch the arithmetic. The couplings still happen. The square is still a square.
We pay it because of an assumption no one wrote down: that M must be dense. That there is no shorter description of how these words relate than the full table of every pair. Sometimes that is true. Usually it is not, and the operator has no way to know the difference, so it pays in full, every layer, every sentence.
To be fair to the design, that assumption is why it works at all. Earlier architectures hard-wired locality, so a word only spoke to its neighbors, and that prior was a cage: it could not learn the long-range relationships nobody thought to wire in. Density is the refusal to impose structure in advance, and refusing was right when the structure was unknown; you cannot hand-design couplings you have not discovered yet. The cost bought the discovery. The waste is that the operator keeps paying it, dense at every layer and every input, long after it could read the real structure off the state and act on the far smaller thing it found.
But density is the shallow assumption. And here the ground shifts: everything up to this point has been reformulation, the same transformer rewritten so its operator is visible, and none of it is in dispute. What follows is argument. There are three deeper assumptions, all claims about the geometry of meaning; some the evidence already backs, some are mine, and I will say which.
The first is that meaning-space is flat. The comparisons inside M are inner products; Euclidean dot products in a high-dimensional space. That choice quietly asserts that meaning lives on a flat manifold, where the distance between two ideas is a straight line. But meaning is largely hierarchical, and hierarchies do not fit in flat space. A tree’s node count grows exponentially with depth, and Euclidean volume does not; you run out of room. Hyperbolic space has exactly the exponential room a tree needs, which is why embedding hierarchies into curved space works so much better than forcing them flat. The architecture assumes the wrong curvature and then spends capacity papering over the mismatch.
The second is that meaning-space is smooth. Everything M is built from — the softmax, the GELU-family nonlinearities — is infinitely differentiable. The one household exception is the ReLU, and its kink at a single point is measure zero; a corner is still not a cliff. So the function the model computes is a smooth function of its input: nudge the state a little, the output moves a little. This is wonderful for training, which slides downhill on smooth slopes. It is also a claim about the world. It says meaning has no cliffs.
Meaning has cliffs. A proof is valid or it is not. A parity bit is zero or one. A conditional branches or it does not, and the answer flips hard at the boundary with nothing sensible in between. A smooth operator can lean toward a step, steeper and steeper, but it can never contain one; it approximates the cliff with a very fast hill, and the approximation gets worse exactly where the meaning gets sharp. This is not an argument about expressivity; universal approximation still holds, and a deep enough network will build a startlingly good imitation of a step on the data it has seen. It is an argument about inductive bias and generalization. The imitation is a steep hill, not a cliff, and it slips the moment the input crosses into ground the training never covered. Whether that gap matters is, I think, the most important open question in the field. The architecture is built as though the answer is no.
The third assumption is the interesting one, because the models themselves have already refuted it. The operator is constructed to be full-rank and dense, as if every direction in meaning-space carried independent information. It does not. When you look at the trained weights, their spectra are heavy-tailed; a handful of directions dominate and the rest are nearly flat. When you adapt one of these models to a new task, the change you need is low-rank; it lives in a tiny subspace, which is the entire reason low-rank fine-tuning works at all. The machine assumes full rank and then learns something low rank inside it.
Flat, smooth, full-rank. Three geometric assumptions, none of them true, all of them baked into how M is built.
This is, I think, where the famous depth of these models comes from. A single application of M is a weak move: one flat, smooth, mostly-linear step. It cannot bend space and it cannot cut it. But you are not applying it once. You are applying it again, and again, a hundred times, each layer feeding its output into the next as a fresh X; and composition is how weak steps become strong maps. Enough small flat steps trace a curved path, the way a polygon with enough sides becomes a circle. Enough small smooth steps, stacked in the right order, approximate a sharp turn. The model buys back the curvature and the sharpness it was denied per step by taking a great many steps.
This is the same thing a fluid solver does when it takes thousands of tiny time steps to evolve one complicated flow. No single step is the answer. The trajectory is. And it says something uncomfortable: much of the depth is not there because reasoning has a hundred stages. It is there to compensate for an operator too simple, one step at a time, to represent what it is chasing. Depth is partly power, and partly penance for the geometry.
Hold those three together and a genuine puzzle resolves.
It should not be obvious that gradient descent works. You are minimizing a wildly non-convex function of billions of parameters; there is no general reason simple downhill stepping should land anywhere good. The reassurances usually offered are real but partial: in this regime the bad local minima are rare and saddles dominate, and in the wide-network limit the training dynamics linearize. Fine. But the cleaner reason is sitting in the third assumption.
Gradient descent works because the problem is secretly low-dimensional. The loss landscape has a few stiff directions and vast flat basins; the effective dimension you actually have to navigate is small. That is the same low-rank structure the weight spectra show and the same one that makes fine-tuning cheap. The geometry the machine assumes is high-dimensional and isotropic. The geometry it lives in is low-dimensional and anisotropic. Gradient descent gets away with being crude because there is far less to search than the parameter count suggests.
So the surprise is not that gradient descent works. The surprise is that we keep describing the system in the flat, full-rank language its own spectrum contradicts.
Which brings me to the thing I find most striking, and most neglected.
If you knew the right basis, optimization would be nearly free. The optimal basis is not exotic; it is the eigenbasis of the curvature. Rotate into it, rescale by the eigenvalues, and the landscape becomes an isotropic bowl you descend in essentially one step. We know this. Newton’s method is this. The reason we do not use it on these models is brutally practical: the curvature matrix is far too large to form, and it will not hold still — it changes as you train. So we approximate it, badly, and fall back on crude gradient steps in whatever basis the parameters happened to arrive in. We know the optimal basis exists. We cannot afford to stand in it, and it moves when we try.
People have mined the static weights for structure. The heavy-tailed spectra predict how well a model generalizes. The eigenvalues of certain weight products tell you which internal circuits copy and which compare, so you can read a network’s mechanism off its spectrum. That work is real and it is good.
But M is not static. It is rebuilt every input. Its spectrum changes from one sentence to the next, because the operator is a function of the state. People have noticed that the routing matrix is often low-rank for a given input; that observation is exactly what the efficient approximations of attention are built on.
But rank is a headline number; the structure underneath is the content, and here the operator has a property that breaks the usual tools. For well-behaved matrices — the normal ones, whose eigenvectors stand mutually perpendicular — eigenvalues are trustworthy: perturb the matrix a little and they move a little, and each eigenvector reports something independent. The attention operator is not that. It is badly non-normal; its eigenvectors crowd nearly parallel, and the eigenvalues of such a matrix are fragile and easily fooled. The generating models make the failure vivid. Their routing is triangular, and a triangular matrix’s eigenvalues are simply its diagonal — here, each token’s attention to itself — while everything attention actually does, every coupling from one token to another, sits off the diagonal where eigenvalues cannot see it. An operator can shift every token one slot down the sequence, rearranging the entire state, and have every eigenvalue exactly zero. The spectrum reports a machine at rest inside a machine at work.
The numerical analyst’s repair is the pseudospectrum: do not ask what the eigenvalues are; ask what they could become under a small perturbation — the set of values the operator is within a hair of having. For normal matrices the two questions give the same answer and nobody bothers asking the second. For operators like this one they part company completely, and the pseudospectrum is the one still telling the truth.
Read any of it as a picture of how the model reasoned about this particular sentence and you are doing something almost no one does. The static weights have a spectral structure; the data-dependent operator has a different one for every input it sees, and that per-input basis is where the reasoning would show up, if we bothered to resolve it. The sharper question, and the one I think is genuinely new, is whether those input-dependent coordinates are reusable: whether the same computational directions recur across prompts, a basis the model already thinks in that today’s representation-centric analyses miss entirely. The formulation hands you the object. Nobody is reading it.
I do not have that result to give you. I have the claim that it is worth looking, and the reason: the operator is explicit, it is data-dependent, and its spectral structure is unexamined.
Gradient descent searches inside the space of functions the operator can express; it does not enlarge it. And the shape of that space is set by M: by the flat inner products, the smooth nonlinearities, the dense couplings. A wide enough network can approximate almost anything in the limit, so the barrier is rarely flat impossibility. It is that a smooth operator pays for a discontinuity exactly where the discontinuity matters. It can fit the cliff on the data it trained on by raising a steep enough hill, then fail the moment it has to generalize across the edge, because the edge was never something the geometry could hold. It was only something the parameters memorized.
This is why I do not think scaling gets us all the way. Some behaviors are not a data problem or a compute problem; they are an M problem. You can pour more points near the corner and the smooth operator will hug them more tightly, and it will still round the corner off the moment you step past the training set. To change what the operator generalizes, you change the operator; and changing the operator is not something gradient descent does. It is something a designer does.
That is the uncomfortable part. The levers we have gotten good at pulling all move the model around inside the operator’s reach; more data, more parameters, more training, and none of them changes the reach itself. Whatever sits outside it stays outside, no matter how hard you pull. The only thing that helps is the one thing scaling cannot do for you: build a different operator.
The field has started conceding this without saying so. The move to sparse, routed operators is an admission that dense was wrong. The state-space models that are quietly matching transformers on long sequences work by diagonalizing their state transition; by finding a basis in which the operator is simple and stepping the field forward there instead. That is the physicist’s instinct exactly: do not fight the operator in the basis it came in; find the basis where it goes quiet.
I have a private reason for trusting that instinct. Years ago, doing numerical work on nonlinear simulations, I found I could get much better answers by letting my operators take complex values; what looked like a merely linear method kept capturing behavior that was supposed to require the nonlinearity. I did not have the vocabulary for it then, but the license has a name: Koopman. Every nonlinear dynamical system is exactly linear in the right coordinates — lift from the state to the space of observables and the dynamics advance by a single linear operator, infinite-dimensional, with a complex spectrum whose phases are the system’s oscillations and whose magnitudes are its growth and decay. Complex numbers are the native algebra of rotation; a real matrix must impersonate an oscillation with a two-by-two block, while a complex entry simply multiplies by eiθ. And the field is already sneaking them in without ceremony: the diagonal state matrices of the S4 line began life complex, and RoPE’s positional rotations are unit complex numbers wearing a two-real-coordinates disguise. The lesson lands on the essay’s own diagnosis. Linearity was never the cage. The cage is demanding that the linearity live over the reals, in a flat basis, and then paying in depth for everything that basis cannot say.
Nor does the ladder stop at ℂ. One rung up sit the quaternions, whose multiplication is three-dimensional rotation; networks built from them exist, buying the same thing complex entries buy — symmetry expressed by the arithmetic itself rather than learned by brute force — at the price of commutativity, which roughs up the spectral tools this essay leans on. And algebra hands you the complete menu: by Frobenius’s theorem, ℝ, ℂ, and ℍ are the only associative division algebras over the reals, and the field’s newest work skips to the general case, Clifford algebras, choosing the products that encode whatever geometry the data actually has. The number system is one more operator decision. Today’s machines made it by default: real, always, everywhere.
The factorization tempts you toward a shortcut. If Aₙ is the expensive, data-dependent part and the values are cheap and fixed, why compute Aₙ the hard way at all? Train a small network to predict the routing straight from the input, skip the queries and keys and the softmax, and just apply the predicted Aₙ to the values. It is a clean idea, and pieces of it are already deployed: a mixture-of-experts router is exactly a cheap network predicting routing instead of computing it, and linear attention predicts the routing’s action without ever forming it.
But there is a wall, and it is worth seeing plainly. Aₙ is not a function of the input X₀. It is a function of Xₙ, the state n layers deep, and that state is what sets the routing. To predict Aₙ from the raw input, a network would have to reproduce internally whatever those n layers computed. A lossless predictor of the routing is therefore the forward pass again in disguise; you would have moved the cost, not removed it. And if a genuinely cheap predictor did suffice, it would be a better model than the transformer, and you would train it directly and throw the transformer away. This is computational irreducibility in one line: you cannot skip the steps of a computation whose intermediate states carry the information.
What keeps the idea alive is that much of the routing is not irreducible at all. Attention is heavily positional, local, low-rank, and repetitive, and the same patterns recur across inputs and even across models. That predictable skeleton is exactly what the cheap approximations already cash in. The remainder, the content-specific routing that genuinely depends on the deep intermediate state, is the part no shortcut can reach.
And the split between the two is measurable. Train a cheap predictor of Aₙ and watch where it succeeds and where it fails. Where it succeeds, attention was doing compressible, near-universal work. Where it fails, attention was doing something genuinely irreducible for that input, and the failure is not a defect of the predictor; it is a fingerprint of computation. You would get a per-layer, per-input map of where the model is actually thinking and where it is only routing on habit, which is the interpretability prize of the per-input spectrum reached from the other side.
Step back from the architecture and ask what the whole enterprise is reaching for: a machine you feed machines (programs, proofs, specifications, the outputs of other minds) and that hands you back machines you can open up and examine. That is what a programmer does, what a scientist does, what a mind does when it reads someone else’s work and produces its own. It is the grail, and it is worth being clear that nothing gets there.
Start with where the transformer sits. One forward pass is a fixed-depth transducer; in the formal hierarchy, a bounded-depth threshold circuit no larger than TC⁰. A warning about that word: a circuit, in the computer scientist’s vocabulary and against its own etymology, is a thing with no loops at all — gates wired strictly forward, evaluated once from inputs to outputs, exactly the feedforward graph this essay opened with. That is the point of the classification. A forward pass cannot loop, cannot hold unbounded state, and cannot run the machine you handed it. So per pass it maps program to program the way it maps sentence to sentence: by resemblance to transformations it has seen, not by understanding what either machine does. The underdetermination you noticed is real, and it goes deep: there is no single right machine to return, because “right” depends on the analysis you want, and the model picks one by conditioning on the prompt. The prompt is the only thing telling it which transformation you meant.
The loop buys back a great deal. Feed the output in as new input and the fixed-depth circuit gains a scratchpad; with enough steps it is Turing-complete, and each additional step of reasoning climbs the hierarchy. This is the recurrence again: one pass is a circuit, the loop is a computer. Most of what looks like a transformer reasoning about code is the loop, not the pass.
But the grail has a wall the loop does not breach, and neither does any successor architecture, and neither do you. To examine an arbitrary machine and certify what it computes is Rice’s theorem: undecidable. Constructing a machine that carries its own certificate is possible — certified compilers do exactly that — but certifying one you were merely handed is not. Halting is the classic instance. This is not hardness, not expense, not an M-problem you fix with a better operator; it is undecidability, and no amount of depth, width, scale, or looping decides an undecidable question. The general act of reading an arbitrary machine and saying what it does cannot be performed by any machine, silicon or wet.
So the reachable version of the grail is bounded, and worth naming honestly. Transformations that are common, or decidable, or cheap to check by simply running them: those a machine can learn and return, and examine after the fact by execution. The general case, an arbitrary machine in and a fully understood machine out, is not waiting on a better architecture. It is closed by computability itself. We can build something that reads many machines and writes many machines and is often right. We cannot build the one that always is. Nothing will.
One scenario hangs over all of this, so take it apart with the tools already on the table. The fear, or the hope, is a machine that improves itself: a model trains its successor, the successor is smarter, repeat, and the curve leaves us behind. Whether that loop runs is not a question about intelligence. It is a stability question, and we already have the symbol for it.
Self-improvement is an iterated map. Generation k carries an error εₖ: the gap between what its weights encode and what is true. It generates the data, curates the examples, and does the judging that its successor trains on. So ask the numericist’s question, the same one training already forced on us: what happens to ε under repeated application of the loop? If each pass contracts the error, εₖ → 0 and the loop ascends. If each pass amplifies it, the loop compounds its own mistakes and the ascent is an illusion that degrades.
Contractions exist, and that is the tantalizing part. Solve a linear system by iteration, xₖ₊₁ = G xₖ + c, and when the iteration operator’s spectral radius sits below one the error shrinks geometrically: εₖ = Gᵏ ε₀ → 0, convergence guaranteed; a badly non-normal G may bulge before the decay wins — the operator theme again — but the limit is not in doubt. But notice what the solver has that a self-improving model does not. For Ax = b you can compute the residual b - Axₖ exactly, at every step, without knowing the answer. The problem carries its own ruler. The geometric descent is purchased with an external, incorruptible measurement of how wrong you still are.
The machines have their own existence proof of this. Self-play took Go from nothing to superhuman with no human data at all; the loop ran, and it contracted. It contracted because the rules of the game are the residual: who won is computed by the board, not by the model’s opinion of itself. The same trick works wherever a verifier exists that is cheaper than the thing it checks: code that must compile and pass its tests, proofs a checker accepts, answers a calculation can grade. Where verification is external and exact, self-improvement is real, and you should expect machines to climb in exactly those domains.
Now remove the verifier, which is what most of the open-ended world is. The loop must grade its own outputs with itself, and ε is measured with a ruler made of the same material as the error. A model’s systematic mistakes are invisible to it by construction; they are what it thinks correct looks like. And systematic is the operative word: random label noise averages away over a trillion examples, but a judge’s bias is correlated, pointed the same direction on every sample, and no amount of averaging removes it. Averaging kills noise, never bias; a crooked ruler does not straighten because you measure with it a million times. Run the loop anyway and you get the pathologies that have already been measured: the distribution’s tails vanish first, variance dissipates generation over generation, and what remains is a model ever more confidently shaped like its own habits. In the solver’s language, that is not convergence. It is a scheme going quietly unstable, dissipating exactly the fine structure that mattered. The known stabilizer is the numericist’s stabilizer: keep injecting fresh real data, a source term coupling the loop back to the world it is supposed to describe. Which is why the labs, whatever they say about synthetic data, are still buying the real thing.
This is also the honest reading of the present acceleration. Discoveries are coming faster with these models in the room, and it is worth looking at where they actually come from: mostly, from people mining the weights. Training compressed an enormous fraction of everything humans have written into that low-rank structure, and researchers are now reading the compression out; querying it, recombining it, interpolating across fields no single human has read. That is genuinely valuable, and it will continue. But it is extraction, not ascent. The library does not grow by being read, and the model’s ceiling is still set by what its ε was measured against; which is to say, by the data, which is to say, by us.
And this outermost loop does not stand alone; it is the top of a nest. Underneath it, faster loops are already running stability problems of their own. The forward pass is a product of a hundred operators whose gain must be held near one, or signals die before reaching the output; that is what the residual connections and the normalizations are for — criticality, engineered by hand. The backward pass is the same product transposed, and its gain straying from one is exactly the vanishing or exploding gradient. And generation itself is a loop — generate, append, repeat — in which an early wrong token perturbs everything after it, and the perturbation either damps or compounds; a model doubling down on a mistake for a thousand tokens is that loop running supercritical. The generational loop is built out of all three. Its data is manufactured by the generation loop, its learning is carried by the backward pass, and every inner instability is exported upward: transients in the token stream become bias in the training set become drift in the next model. The gains compose. You cannot build a contracting outer loop from exploding inner ones, and the criticality question at the top inherits every criticality below it.
Say plainly what is known and what is not. The two extremes have been measured. With an exact external referee, the loop contracts; the game players proved it. Feeding purely on itself, it dissipates; the collapse experiments proved that. But the loop actually running today is neither. It is a mixture: real data and synthetic, partial verifiers over slices of the work, models grading models with humans auditing samples of the grading. That loop has an effective gain — one number, whether a generation’s error comes out larger or smaller than it went in — and nobody has measured it. This is a criticality question, subcritical against supercritical, an enormous consequence hanging on whether a single number sits above or below one; and the number is unknown, probably different in every domain, and probably moving year by year. I do not know which regime we are in. Neither, as far as I can tell, does anyone else.
So the takeoff question has a precise answer, and it is conditional rather than mystical. Self-improvement runs where the world itself referees: games, code, proof, anything a simulation or an experiment can grade. It stalls where the only judge available is the thing being judged. Unless someone finds a way to make repeated ε contract without an external oracle — and nothing in the mathematics on the table does it — the explosion stays confined to the refereed domains, and everywhere else the machine remains what it is today: a very large, very fast compression of us, improving only when we hand it something true.
Bring the same ε down to the workbench, because it explains something every programmer working with these machines has felt and few have named.
Rapid prototyping with a model is genuinely helpful; I want that said plainly before the knife comes out. Ask for a demo, a scaffold, a first draft of a thing that does not exist yet, and the machine is a marvel. The trouble starts when the code has to be correct, and the shape of the trouble is the shape of everything above.
A prompt is a boundary condition. It pins the solution at a handful of points — the requirements you thought to state, the examples you gave, the tests you mentioned — and leaves the interior of the problem unpinned. The essay already said what fills the interior: the operator returns the typical completion, the prior’s best guess, not your intent. Your intent has detail at every scale; edge cases, and edge cases of edge cases, a series your finite prompt truncates. So the generated code satisfies the boundary conditions given and leaks everywhere else, at every scale you did not pin. Fix a bug and you have pinned one more point; the repair is a fresh generation, filling its neighborhood with fresh typical-guess, and the leak moves. Debugging it feels like zooming in on a fractal because it is: self-similar error at every scale of specification, revealed one level at a time. And there is a classical theorem for the whack-a-mole: Runge’s phenomenon. Pin a global polynomial through more and more equally spaced points and the fit between the points does not improve; it oscillates harder, the error near the edges growing exponentially, because you added pins. Exact at the constraints, wild in between, worse as the constraints accumulate — anyone who has watched a prompt grow clause by clause has seen that curve before. The curve-fitting world’s escape was to go local: splines, Bézier segments, low-degree pieces that never let one constraint reach across the whole domain. A global smooth operator has no such escape. Approximation theory sharpens the picture further. A best approximation, in the standard sense, has equioscillating error; spread thin and everywhere, by construction. A human author’s bugs cluster where the author was confused, and the author knows where that was. The machine’s bugs are diffuse, and correlated with nobody’s suspicion.
Now the physicist’s objection, which is the heart of it. Physics runs on approximation; nobody integrates the full problem. But a physics approximation is licensed: sinθ ≈ θ comes with its remainder term, θ³/6, and a validity regime — at ten degrees the error is about half a percent, and the theory itself tells you so. Perturbation theory ships with the size of the first neglected term. An approximation with an error bound and a domain of validity is knowledge. An approximation with neither is a guess wearing knowledge’s clothes. The machine’s output is the second kind: it is an approximation of the answer you wanted, arriving with no remainder term, no validity regime, and no way to ask for one; the ε is unknown, unbounded, and, per the paragraph above, located nowhere in particular. Close only counts in horseshoes and hand grenades, and nothing in engineering is horseshoes.
That asymmetry is a cost law, and it is not about AI at all; AI is only its newest and largest customer. I have not seen it named, so I will commit the small vanity of naming it after myself. Chancellor’s Law: an almost-correct artifact costs more to fix than a correct one costs to produce. Verification effort grows as the error shrinks. A crude counterfeit is caught at a glance by anyone; a good forgery takes an expert, instruments, and time, and the best are only ever caught by accident. Anyone who has proofread documents knows it in their hands: the thing obviously in the wrong category flags itself, and the thing almost right sails through. The reviewer’s cost diverges exactly as the artifact approaches correctness; and in a cliffed domain like code, the harm does not shrink with the error at all. Small ε: expensive to find, fully wrong.
For generated code the law compounds twice. The author of a piece of code gets its causal model free, as a byproduct of writing it; the debugger of generated code must reconstruct that model from the outside, which costs about what writing costs. And the errors still in the code are, by selection, exactly the ones that survived review — invisible to the checks you knew to make. Then notice what the training objective was: maximize the likelihood of looking like the corpus. The machine is optimized, directly and by construction, to make its errors plausible; when it is wrong, it produces the most convincing available forgery of being right. It mass-produces the precise class of artifact that verification finds dearest.
This is measurable, and it has been measured. A randomized trial by METR put experienced open-source developers on real tasks in mature repositories — over a million lines, dense with the implicit invariants no prompt states — and found them 19% slower with AI assistance, while they themselves estimated they had been 20% faster. Both numbers matter. The slowdown is the law above; the misestimate is why the law goes unnoticed. (The study is a snapshot of early-2025 tools, and the balance will move; the mechanism it measured will not.)
None of this contradicts the refereed-domain optimism of the last section; it locates its edge. The compiler and the tests are real referees, but they referee only what you formalized. The loop contracts ε onto the constraint surface you built, and the component of intent orthogonal to it drifts on the prior. Prototyping thrives because there, satisfying the stated boundary is the whole job. Mature codebases suffer because there, the unstated interior is nearly everything.
Notice what this does to the craft. Programming does not go away; it changes what it is made of. The old job was specifying behavior, instruction by instruction; the new one is specifying assertions — tests, types, invariants, checks — the pins that define the constraint surface the operator gets contracted onto. You used to write the interior. Now you write the boundary, and the quality of what fills the interior is set by exactly one thing: how much of your intent you managed to turn into something checkable. Programming is still programming. It has moved from writing the answer to writing the ruler.
It also says which seat the machine should sit in. The property that makes it a treacherous author — attention spread thin and everywhere, decorrelated from any human’s suspicion — is exactly the property you want in a reviewer. So turn it around: let the human write, holding the causal model that authorship hands out free, and let the machine read, tirelessly, sweeping the places nobody thought to look. The failure modes flip with the seats. A reviewer’s false alarm costs a minute to dismiss, and its miss leaves you no worse than you were; an author’s error is embedded in the artifact, camouflaged, compounding. Same operator, same ε: as a generator, its errors are yours to find; as a filter, its errors are nearly free.
Used as the author anyway, it returns what it always returns: an approximation of the answer you wanted, never the answer itself. Physics forgives its approximations because they arrive carrying their own error bars. This one arrives carrying confidence.
A natural question: how does this compare to the thing between your ears? The honest comparison is not neuron to neuron; it is synapse to parameter. A weight in one of these networks plays the role a synapse plays in tissue, a single tunable connection strength, so that is the unit to count. A neuron is closer to a whole row of the operator than to any one weight.
The brain has about 86 billion neurons, but most of them are not doing what an LLM does. Roughly 69 billion sit in the cerebellum, running the fine timing of movement; the cerebral cortex, where language and deliberate reasoning live, holds about 16 billion. Each cortical neuron carries on the order of several thousand synapses, so the cortex runs to well over a hundred trillion connections, and the whole brain to somewhere between a few hundred trillion and a quadrillion. Language and reasoning are only a slice of even the cortex; take the tenth or so that handles words and logic rather than faces, edges, and limbs, and you land near ten to fifteen trillion synapses for the part that argues with you.
Now the machines. GPT-3, the last one whose size was stated plainly, held 175 billion parameters. The current frontier, GPT-4-class and Claude among them, is disclosed by no one; the credible estimates put it in the one-to-two-trillion range, usually as a mixture of experts where only a fraction of the weights fire on any given token. So the frontier already sits within an order of magnitude of the synapse count for the reasoning-and-language cortex, two orders below the whole cortex, and between two and three below the whole brain.
Read that carefully before it is oversold. A synapse is not a scalar; it has dynamics, chemistry, timing, and neighbors that modulate it, so counting it as one number almost certainly undercounts what the tissue can do. And a model sees more text in training than a person could read in a thousand lifetimes, so equal parameter counts do not mean equal anything. The comparison sets a scale; it does not declare a winner.
The scale still costs what scale costs. At two bytes a weight, a two-trillion-parameter model is four terabytes just to hold still, before a single word comes out. A top datacenter accelerator carries eighty gigabytes, so the weights alone want dozens of them wired together, and training the thing burned tens of thousands of such chips for months. This is why the frontier lives in a warehouse.
It does not stay there. Two curves are closing on each other. Hardware memory grows slowly, maybe a doubling every two or three years; and the software side, quantization down to four bits and below, distillation of a large model’s behavior into a small one, sparsity that skips the weights that do not matter, improves the capability you get per parameter faster than that. The consequence is already visible: a phone now runs a three-billion-parameter assistant on its own silicon, and a good desktop runs a seventy-billion one quantized. That is one to three orders of magnitude below the frontier, depending on the device, the same range as the gap we just measured against the cortex.
If those curves hold, frontier-grade capability, not the literal trillion-weight file but a distilled thing that answers as well, reaches a high-end desktop in something like five to eight years, and a phone a few years after that. The machine that today needs a building will, on that trajectory, need a lap.
The forms are not only a diagnosis. Writing a layer as one matrix on one vector is why the machine runs at all: dense linear algebra is the one thing a GPU does better than anything else, and a transformer is dense linear algebra almost end to end. That is much of why it beat the recurrent networks before it, which had a sequential chain to wait on; the transformer has only matrix multiplies, the exact operation the hardware was built to accelerate. The same move that makes the operator legible on paper makes it fast in silicon. And once the computation is written in that language, the rest of the language comes with it: eigenvalues and conditioning, factorizations and preconditioners, spectral methods and reduced-order models, a century of numerical linear algebra that already has a name for most of what these networks quietly do. The perspective does not hand you the next architecture; it hands you the toolbox.
None of this says the machine is broken. It works, and that is not in dispute. It says the machine is built on three geometric assumptions we have stopped noticing because the language we use to describe it never names them. Flat, smooth, full-rank. Meaning is probably none of the three.
The engineer’s vocabulary hides that. It talks about attention and heads and parameters, and it scales. The physicist’s vocabulary makes it plain. An operator, rebuilt each step, of the wrong curvature, assuming a density and a smoothness the data does not have, searched by a method that only works because the real structure is smaller than the operator admits.
That understanding does not fix anything. It does not tell you which operator to build instead. It only tells you where to stop scaling and start looking.
We built a machine that assumes meaning is flat, smooth, and full. It is none of those.
We are only now getting the invoice.
Further reading, for anyone who wants the primary sources behind the claims above: on heavy-tailed weight spectra and generalization, Martin and Mahoney’s work on implicit self-regularization; on low-rank adaptation, the LoRA paper (Hu et al.) and the intrinsic-dimensionality results (Aghajanyan et al.); on reading circuits off spectra, the transformer-circuits framework (Elhage et al.); on hierarchy and curvature, Nickel and Kiela’s Poincaré embeddings; on diagonalized state transitions, the S4 and Mamba line of work (Gu et al.); on curvature-aware optimization, natural gradient and its approximations, K-FAC and Shampoo; and on backpropagation as the adjoint method, the adjoint-state method from optimal control and its continuous form in Chen et al.’s Neural ODEs.
Read and subscribe on Substack