Entropy

Disorder, counted

Leave a room alone and it gets messy. Drop cream in coffee and it swirls out until it's evenly brown. Let a hot cup sit and it cools to room temperature — never the reverse. Every one of these is the same law wearing a different costume: things drift, on their own, from ordered toward disordered. Physicists gave that drift a name in the 1800s, decades before anyone understood machines that learn. They called it entropy.

Here is the crucial move — the one that makes entropy a mental model and not just a physics term. Entropy is not "messiness" as a vibe. It is a count. A tidy room has exactly one arrangement that counts as tidy: everything in its place. A messy room has billions of arrangements that all count as messy. There are simply vastly more ways to be disordered than ordered — so if arrangements are shuffled at random, you land in "messy" almost every time. Entropy measures how many ways a situation can be arranged while still looking the same from the outside.

figure generating…

Hold onto that reframing. It sounds like it belongs only to gas molecules and thermodynamics. By the end of this lecture, the exact same count will explain the loss function you have used in every previous lecture, the "temperature" knob on a chatbot, and the confidence of a model's next guess.

Heat is just jiggling

Temperature feels like a fundamental thing — hot, cold, a number on a thermometer. But underneath, temperature is nothing more than the average speed of the molecules. In a cold solid, atoms barely quiver, locked into a neat lattice — few arrangements, low entropy. Add heat and they jiggle harder, break their bonds, and start wandering; the solid melts to liquid, then boils to gas, molecules ricocheting freely through the whole container. Same atoms — but now astronomically more arrangements. Heat is disorder you can feel with the back of your hand.

Drag the slider. Cold: the molecules settle and pile at the bottom into an orderly crystal — the entropy reading is low, because there are few ways to arrange a settled crystal. Hot: they fill the whole box, fast and evenly spread, glowing red as they speed up — the entropy reading climbs, because a filled box has overwhelmingly more arrangements. You are watching the count go up. The colour is the speed: blue is slow, red is fast. Temperature is that colour.

figure generating…

The arrow of time

Drop a bead of ink into a glass of water and it spreads until the whole glass is faintly, evenly tinted. You will wait a very long time to see the reverse — the ink spontaneously gathering itself back into a single crisp drop. The physics of a single molecule is perfectly reversible; run the film backward and nothing breaks. Yet the film of the whole glass has an obvious direction. Why?

Because of the count, again. There is exactly one "ink in a tight drop" arrangement and an unfathomable number of "ink spread everywhere" arrangements. Random jostling doesn't prefer disorder — it just stumbles into the vastly more common state. This is the Second Law of Thermodynamics: in a closed system, entropy tends to increase. It is the closest thing physics has to an arrow of time. The future is simply the direction in which the count is larger.

figure generating…

Boltzmann counts the ways

In the 1870s Ludwig Boltzmann wrote down the equation that turned this intuition into mathematics — the one now carved on his tombstone in Vienna:

Read it slowly. S is the entropy. W is the number of microscopic arrangements — microstates — that all produce the same visible, large-scale state. k is just a constant fixing the units. The whole content is in one word: entropy is the logarithm of the number of ways. More ways, more entropy. And the logarithm matters — it means entropy adds up: two systems together have entropies you can sum, even though their arrangements multiply.

figure generating…

To see "microstates" concretely, roll two dice. The visible outcome — the sum — is the macrostate. A sum of 2 has exactly one microstate (1 and 1). A sum of 7 has six microstates (1-6, 2-5, 3-4, and their mirrors). That is precisely why 7 comes up most: it has the highest entropy. The most likely thing you observe is almost always the thing with the most ways to happen.

figure generating…

Shannon steals the fire

Jump to 1948. A young engineer at Bell Labs, Claude Shannon, is thinking about a completely different problem: how to send messages down a noisy telephone wire without losing information. He needs to measure something slippery — how much information a message actually carries. And he arrives, by a totally separate road, at the very same formula as Boltzmann. The legend goes that John von Neumann told him to call it "entropy," partly because it was the right analogy and partly because "no one really knows what entropy is, so you'll always win the argument."

Shannon's reframing is the key that unlocks the whole rest of AI. Information, he said, is surprise. A message that tells you something you already knew carries almost no information. A message that tells you something shocking carries a lot. If an event has probability p, the surprise it delivers when it happens is:

figure generating…

Check the extremes. An event that is certain (p = 1) delivers −log 1 = 0 surprise — "the sun rose this morning" tells you nothing. A rare event (p tiny) delivers a large number — "it snowed in the desert today" tells you a lot. Measure that surprise in base-2 logarithms and the unit is the bit: the amount of information in one fair coin flip.

figure generating…

Entropy = expected surprise

Surprise is defined per outcome. Entropy is just the average surprise you expect across all outcomes, each weighted by how often it happens. That single sentence collapses Boltzmann and Shannon into one equation — the one you should tattoo on the inside of your eyelids:

Take a coin. If it is fair — heads and tails equally likely — you are maximally uncertain about each flip, and the entropy is exactly 1 bit. Bend the coin so it lands heads 90% of the time and the entropy drops: most flips are unsurprising, so the average surprise is low. A two-headed coin that always lands heads has entropy zero — no surprise, no information, nothing to learn. Uncertainty peaks in the middle.

Notice the shape: a single smooth arch, dead flat at both ends and peaking at p = 0.5. Certainty — either kind — is the boring floor. Maximum uncertainty is the summit, and it sits exactly where you have no idea what will happen. Entropy is the mathematics of "how much don't I know."

figure generating…

figure generating…

The loss behind everything

Now the payoff. Every classifier, every language model, nearly every deep network you have met in this series was trained by minimizing one thing: cross-entropy. You used it without me naming it. Here is what it actually does.

The true answer is a distribution too — a spiky one, with all its probability on the correct class ("this image is a cat: 100%"). The model outputs its own distribution, spread across the possibilities. Cross-entropy measures the average surprise the true answer causes the model. If the model was confident and right, the correct answer is unsurprising to it, and the loss is tiny. If it was confident and wrong, the correct answer is a shock, and the loss is enormous. Training is just: repeatedly reduce your surprise at the truth.

figure generating…

There is a sister measure you will meet everywhere in language modelling: perplexity. It is just entropy, exponentiated back out of log-space, and it has a beautiful plain reading — how many equally-likely options does the model feel it is choosing between? A perplexity of 2 means the model is as unsure as a fair coin flip at every word. A perplexity of 50 means it feels like it's picking from fifty doors. Lower perplexity, more confident model. It is the industry's favourite way to say "how surprised is this model, on average, by real text."

figure generating…

The temperature knob

When you use a chatbot, there is often a setting called temperature. Turn it down and the model is precise, repetitive, safe. Turn it up and it gets creative, surprising, sometimes unhinged. That word is not a metaphor. It is literally the same temperature as the molecules in the box from Section Two — the same mathematics, borrowed wholesale from physics.

A model scores each possible next word, then passes those scores through softmax to turn them into probabilities — and softmax has a temperature T baked in, exactly as Boltzmann's distribution p ∝ e^−E/T does for molecules. Low T: probability piles onto the single best word — a peaked, low-entropy, "frozen crystal" of a distribution. High T: probability spreads evenly across many words — a flat, high-entropy, "hot gas" distribution. Temperature is a direct dial on the entropy of the model's choice.

Watch the bars as you drag. At low temperature one word towers over the rest and the model will say the same safe thing every time — the entropy reading falls toward zero. At high temperature the bars flatten out, the model reaches for rare words, and the entropy climbs. Creativity and hallucination are the same phenomenon seen from two sides: both are just a model sampling from a higher-entropy distribution. When people say a model "runs hot," they mean it precisely.

figure generating…

figure generating…

Distance between beliefs

One more tool falls straight out of entropy, and it is everywhere in modern AI. Suppose the world follows one distribution and your model believes another. How far apart are they? The answer is KL divergence — relative entropy — and it reads as: the extra surprise you suffer by using the wrong distribution instead of the true one. If your beliefs match reality, the extra surprise is zero. The more wrong you are, the more bits you waste being caught off guard.

figure generating…

This one quantity is the quiet engine under a startling number of methods. It is the leash in RLHF and PPO that keeps a fine-tuned model from drifting too far from its original self. It is the objective inside variational autoencoders. It is how diffusion models (Lecture Seven) are derived, and how knowledge is distilled from a big model into a small one. And the same "reduce entropy" instinct explains why a decision tree splits where it does — it cuts the data at whichever question removes the most uncertainty, a quantity literally called information gain. Once you can measure surprise, you can minimize it, match it, or spend it deliberately.

figure generating…

From heat to hallucination

Step back and look at the span of a single idea. A concept invented to explain steam engines — why heat flows one way, why a cup cools, why molecules fill a room — turned out to be the exact mathematics for measuring information, for training every classifier, for tuning a chatbot's creativity, and for measuring the distance between what a model believes and what is true. One equation, H = − Σ p log p, quietly runs underneath thermodynamics and machine learning alike.

figure generating…

That is the deepest reason to keep this mental model close. When you next meet a loss function, a "temperature," a perplexity score, or a KL term, don't treat them as four unrelated pieces of jargon. They are four faces of one physical idea: the count of ways, the measure of surprise, the currency of uncertainty. Entropy is what you are always paying, spending, or trying to reduce.