I studied neural networks in Lisbon in the early 1990s, as coursework inside a master’s in statistics and operations research. The networks had a few dozen units and we trained them on problems small enough to check by hand. Thirty years later the networks have billions of parameters, and what surprises me is how much of the old course still applies.

Training is still optimization. You define a loss, a number that says how wrong the model is, and you adjust the parameters in the direction that makes it smaller. That is gradient descent, and operations research students met it long before anyone called it machine learning. The chain rule that makes it work on a deep network, backpropagation, was already in the textbooks I used.

The loss itself is statistics. A language model is trained to assign high probability to the word that actually comes next. Minimizing that loss is maximum likelihood estimation, the first method any statistics student learns for fitting a model to data. The vocabulary changed. The principle did not.

So did the central worry. A model with enough parameters can memorize its training data and still fail on anything new. We called that overfitting in 1992 and controlled it with held-out data and penalties on complexity. Today’s models are far larger than the old rules of thumb said was safe, and they generalize anyway, which is one of the open questions in the field. But the test is the same one: judge the model on data it has not seen.

What did change is scale and architecture. The amount of text, the compute to process it, and the transformer design introduced in 2017 turned a method that worked on toy problems into one that writes code. None of that was visible from a classroom in Lisbon. I would not have predicted it.

The background still helps me in a practical way. When someone says a model “knows” or “decides” something, I translate it back: a function was fitted to data by minimizing a loss, and it is producing a probability distribution. That translation deflates a lot of hype and a fair amount of fear. It also points at where the real risks are, which is in the data the function was fitted to and in what we connect its outputs to.

I spent the decades in between running technology and security for large organizations. Coming back to this field, I find I am not starting over. I am picking up a thread I put down in 1994.