In the last post, we put together a neural network: A set of neurons, each of which has a whole bunch of weighted inputs and one simple function that sums up and transforms those inputs into a single output number. But right now, it doesn’t do anything or mean anything. Even if you poke it with a stick. We can make a neural network, but we can’t yet design one. Getting there is going to require three tasks:

1. Decide what we want the network to do
2. Decide what structure the network needs
3. Train the network

Task 1: What does it do?

The first task is to figure out what our inputs and outputs are. Neural networks work with numbers; it’s up to human beings to pick those numbers and decide what they mean.

The thing is, there are numbers and there are numbers. We’re surrounded by and described by numbers. I have an age, a height, a ZIP code. My blood pressure is two numbers, and my IP address is four¹. More things can be converted to numbers: letters have ASCII or Unicode representations, while pixels in an image are stored as color values describing how much red, green, and blue are in each place. But not all numbers are used in the same way, and so they’re not all equally suited to neural networks.

Broadly speaking, we use numbers to measure and to label. My age, height, blood pressure—those are all measurements or counts.

Recall what those simple little neurons actually do with numbers: they add them, multiply them, and check if and how much one number is bigger than another. We can do that to measurements. It makes sense to talk about someone being twice my age, or two inches taller than me, or how much my systolic blood pressure is above 120.

My ZIP code, on the other hand, is a label that happens to be expressed as a number. I can calculate “half my ZIP code” but it has no meaning in the realm of ZIP codes. ZIP codes could have been letters. I learn nothing from checking if a phone number is above 8,675,309. (Sorry)

House numbers are potentially interesting, because they’re implicitly a count. Adding or subtracting one or two or ½ might tell me something if I know they’re on the same street and exist. Multiplying them, though, likely won’t. Sometimes numbers-as-labels are cleverly assigned, as when you can add 32 to the ASCII value to get its lowercase equivalent, but those are incidental. The more you add and multiply the numeric values of labels, the quicker you get away from any real meaning.

For neural networks we need to work as much as possible with measurements. Where we must work with labels, we need to find ways to turn them into measurements.

Nothing says we have to use data just as it was given to us. In the map above, the street numbers go from 216 to 234 ½ to 308. Sure, that’s a label, but it also feels like a measurement, doesn’t it? When we as humans think about these things, we’re mapping them onto physical space. As a pre-processing step, we can convert addresses to latitude and longitude, and use those as our inputs. Maybe we add altitude for a three-dimensional space. This adds the kind of meaning that we’re instinctively looking for when we want to use address as an input: we want a measurement where things that are related are close to each other.

Of course, the physical map is sometimes inconvenient. Sometimes what I really want is some way to group things the way I think of them, where things are close because they’re kinda like each other rather than because they’re physical neighbors. Let’s put a bookmark there, though, because there’s a lot we can do with that thought. We’ll tackle it in the next post. For now, let’s try something a little more straightforward.

A common tutorial problem for neural networks is handwriting recognition, and in fact one of the longest-running real-world uses is automatically reading handwritten numbers, like ZIP codes on envelopes in the United States Postal Service, or numbers in Census forms. An early series of operational neural networks designed at Bell Labs were named LeNet^2[1]after Yann LeCun, who’s still active in the field (I actually saw him give a talk last week, about how LLMs are a dead end). Neural nets aren’t the only way this is handled (maybe not even the best way) but they are a decent fit for the problem: it’s easy to describe; it’s mostly easy for human beings to label; and it’s not feasible to simply list all of the possible forms a given numeral might take, especially when there might be stray marks. It’s the kind of thing that humans learn by rote memorization, rather than by learning and applying a rule. And it can all be distilled to measurements.

One reason this is a common tutorial is that it’s very easy to interpret pixels as measurements. A 3 x 3 grid of black and white pixels is just 9 numbers. Shades of gray just turn that number into a number between 0 and 1. It makes sense to talk about a pixel being half as gray, or add 0.1 grayness, or to ask if a pixel’s value is over 0.25. Even if the original pixels are only black and white, it still makes sense to talk about values in between 0 and 1.

A neural network designed to recognize handwritten numerals is going to have pixels as its inputs, but what are its outputs? Conceivably, you might say, “Well, the output of a neuron is one number, and I want one number, so there we go.” But if you think about it, the numeral “7” isn’t being used as a measurement, it’s a label. There exists a number 7.5, but it’s very difficult to come up with a principled description for what you’re doing when you add 0.5 to the label 7. If I have an image like this:

That could be a 7 or it could be a 9³. It makes no sense to try to split the difference—it’s definitely not an 8! But I could say it has a certain amount of sevenness but also a certain amount of nineness, and if push came to shove, I think it has more nineness than sevenness. Maybe 60% nineness, 35% sevenness. Sevenness might not be a terribly useful measurement in day-to-day life, but it is a measurement, and I can work with that!

So, what if we had ten outputs: one for each of the numbers 0 through 9. Each one measures one particular numberness. One neuron measures zeroness, another oneness, etc. Ideally, we’d want our network to be very certain: this handwritten “6” has full sixness and no twoness or anything else. But we’d know what it means if it weren’t quite so certain: we could interpret a messy set of outputs naturally in terms of unclear handwriting, and in a pinch go with the neuron with the highest measurement.

This gives us a bunch of options for tinkering. We could add an 11^th output for “not a number”. We could add letters or mathematical symbols or smiley faces. Each new label is just a neuron, right? (What can one neuron cost, ten dollars?) These decisions are going to drive the structure of the network and determine what I need out of my training data.

I need one last thing out of task 1: I can look at “70% eightness, 30% zeroness” for an example that really is an 8 and understand it’s mostly right but I need to tone down the zeroness. I need to write an error or loss function to measure how right or wrong an output is, so if next time that example get an “80% eightness” then I can describe exactly how much the error was reduced. A common loss function called “mean square error” basically just adds up for all the outputs how different they are from the output I wanted.⁴ Error is a measurement, and a very handy one too, because it’s going to let me automate things.

Task 2: Structure

Now we know what we want the network to do. We defined the input and output numbers as measurements that a neural network can make sense of, and figured out a way to measure output error. Let’s get into the network itself. In between those input and outputs we’ll need layers of neurons, and they give us our first set of constraints: The first layer will be made up of neurons with exactly 784 inputs apiece (one for each pixel in a 28×28 image); the last layer will need to consist of ten neurons, each corresponding to a single numeral.

We can of course satisfy these constraints with a single layer consisting of ten neurons. That would be equivalent to adding up all the black pixels, with some worth more than others, and trying to find a number above or below which one neuron calls it a “0”, another calls it a “1”, etc. That very simple network honestly works surprisingly well⁵ for this particular simple dataset, but it’s a bit fragile and tends to top out at about 90% accuracy.

Typically, you’ll see a multi-layer network for an example problem like this. But why? How does the designer of a neural network have the first clue how to go about putting together layers and figuring out how big they need to be? Let’s think about what each layer sees.

The first layer of my number network has direct access to the original pixels. Each neuron gets a full copy of the image. Any one of them can do whatever it wants, but it’s essentially limited to adding up individual weighted pixels and checking if the sum is above a value. That’s all. None of them can do anything more complicated than that.

The second layer doesn’t see the original pixels. Each neuron in the second layer sees only what the first layer comes up with. A third layer only sees the second layer’s output, etc.

Notice that a layer doesn’t have to have as many neurons as there are inputs. If you think about that, you might realize: that layer must throw away information. Whatever comes out of the layer’s outputs, it cannot simply pass along what was given to it in the first place. If it’s going to ever be able to solve the problem, it needs to learn a way to compact the information it was given. The greater the difference between the number of inputs and the number of outputs, the more ruthlessly it needs to discard information.

For example, let’s say I’m given the ages of two people, but I can only write down one number. I’ll need to home in on the most salient simple thing about those two ages for the problem at hand: maybe it’s most important to know whether a particular person is at least a year older than the other. Maybe their combined age is the most important thing, or some weird combination of twice person A’s age plus half person B’s. Maybe it’s more important to know if person A is over 50 (and we just have to ignore person B). A single neuron just can’t get that complicated: there’s no capacity for ratios or “if either person is over 50”. Even if there are other useful conclusions that can be drawn from these two pieces of information, the structure requires that only the most useful simple conclusion be kept.

Of course, except possibly at the final output layer, we’re not going to just have one neuron in a layer. Having fewer neurons than inputs just means losing some information. But any given input’s information will be taken into account by multiple neurons. For one neuron it might be a positive; for another negative. Some pixels might be ignored entirely; others effectively grouped together.

There’s no reason a layer can’t have more neurons than inputs, though! What, conceptually, is happening in that case? In the prior example, now we can test multiple things about the same inputs and pass along multiple conclusions. Is person A at least a year older than person B? That’s one neuron. Is person B at least three years older than person A? Is person B over 25? What’s their combined age? The more additional neurons in the layer, compared to the inputs, the more little factoids can be passed along for the next layer to work with.

There’s another way to interpret what’s going on here. The process of going from more numbers to fewer can be thought of as compression or encoding, just like zipping a text document or organizing geographic regions by numbered postal code. The other way around can be thought of, then, as decompression or decoding.

Let’s try to see this from the perspective of detecting a seven. If you were to try to sit down and work through this conceptually, you might think through it like this:

“OK, I’m kind of looking at multiple regions in the picture. There’s going to be a kind of a cross bar up at the top, but maybe not right at the very top. And there’s going to be a down slash, kind of on the right hand side. I can divide and conquer, too, to take advantage of multiple numerals using that slanting vertical, and look for those two things separately. Some people put a little crossbar in the middle, too. I can also look for things that shouldn’t be there. Like, there probably shouldn’t be anything in the lower left…”

I could even draw a picture of what each neuron is looking for in the image. If it sees a pixel where I’ve drawn red, that’s really interesting to that neuron. It gets excited, and the more pixels it sees in red zones, the more likely it is to output a higher number: a higher “I’m seeing what I’m looking for!” measurement. If it sees a pixel where there’s blue, that’s a let-down; the more pixels a neuron sees in blue areas make it more likely to output a lower number. The darker the pixel and the darker the red or blue, the stronger the effect. So kind of like this, where the third neuron is basically looking for not-sevenness, and its output would tend to discourage a conclusion that it’s a picture of a seven:

What’s being described here are features of the image, and you can see how those features correspond to neuron weights. One neuron might learn a vague line shape. The main body of the resulting rectangle is really important, so heavy weights there. Off toward the corners or bottom less weight or negative weight.

Maybe three or four neurons have their own takes on this approach. Each one produces an output that corresponds to basically, “I think this image has a horizontal line thing going on,” or, “the top of the image is pretty empty”. Others focus on different patterns or regions, and draw their own conclusions. Subsequent layers have access not to the original inputs, but to the conclusions of the prior layer. In this case, though, I think we have our answer: we probably only need two layers here. The first layer can pay attention to specific parts of the image, looking for lines or circles or empty space. I don’t know quite how many of these features there might be, but I do know that I want it to try to compress information from the original image, so I’m going to use fewer neurons than there are pixels. The second layer, of ten neurons (for the numerals 0 through 9) can draw conclusions from those identified features of the image and directly guess the drawn numeral. This mirrors pretty well our own instincts about how to identify written numerals– how we might explain it to a child, say.

This is all terribly anthropomorphized, all this talk of watching for lines and such. Anthropomorphizing helps me think through and convey some of the ideas, but it is a mistake when talking about neural networks. To show why this is a mistake, let’s think through what a human-designed neural network would look like.

Consider a simple neural network designed to work with the MNIST dataset—that 1990s collection of 28×28 pixel images of handwritten digits. My network is two layers: the first with 64 neurons, and the second 10 neurons. The first layer is going to be paying attention to the image itself, with different neurons activating as they see specific patterns directly in the image. A 28×28 image is 784 inputs, so the first layer, being smaller, will need to compress information. Then the second layer is going to be working with the compressed information to directly signal sevenness, eightness, etc.

Spend a moment picturing what you think a neuron in the first layer might be doing to recognize parts of numerals, like I did above. Maybe write a few numbers on a paper and look at them like a designer might. Each neuron needs a weight for every pixel position, and that weight is going to be a positive number (red in my drawing), a negative number (blue), or zero (white).

For example, a pixel in the upper left corner. What do you want to happen if your network sees a pixel there? If a neuron has a positive weight for that position, then if an image has that pixel filled in, it will “excite” the neuron and push it toward activating. If a negative weight, it will “inhibit” the neuron, pulling it away from activating. A zero weight just does nothing. The terms “excite” and “inhibit” are useful, but they are a bit biomorphizing, perhaps borrowing too much from biology: it means nothing more than “if you add a number, you’re getting closer to the threshold; if you subtract a number you get further away.”

Recall how we thought about setting a neuron’s weights from scratch, where you might have a swath of red where the top edge of a 7 might be, with some blue above and below it. You might reuse that for a 2 or 5. You’d be thinking about what the parts of a written numeral would be. Think for a moment about what you’re doing as you think through what the parts would be. That’s an anthropomorphized network.

The machine is not doing what you’re doing. It’s doing this:

The image above is a visualization of 32 neurons⁶. Red pixels still represent a positive weight; blue negative. That is… not how a human being would design these neurons, I think it’s fair to say. If I stare at these long enough, I can tell myself a story about what’s going on. That fifth neuron at the top, that looks like static, is probably a dead neuron: for whatever reason, its output is always zero, so there’s no pressure for it to develop any real pattern. The others, I can make out handwriting-like swathes. But this ultimately is just a story I’m telling myself⁷.

This is not intentionally designed, but it’s not purely random, either. The work of a neural network is done little by little, scattered across the network and using only what was produced by the prior section of the network. When we architect our network, we use our intuition about the problem to set up a scaffold that will be used in the training, but we just won’t know how it will be used. This is why anthropomorphizing a neural network, even when we designed its structure according to our human intuitions, is a mistake.

Anthropomorphized or not, we can still use this thinking to at least make a first stab at structure. My instinct is that two layers will do the job, because my personal approach is to pay attention to certain parts of an image, and build up a numeral according to lines and circles and such that I find, in two steps. I can imagine it being useful to have another layer in between the first layer and the output, though, to compress things a little further and maybe require fewer neurons overall. The first two layers, whose outputs I never directly see, are called hidden layers. They’re called hidden because I only see their activations as processed by later layers. And in fact, a network of this type has been shown⁸ to only ever need one hidden layer, like I’m using, to approximate any continuous function you might come up with. It is a universal approximator, to whatever degree of accuracy needed, with enough neurons. There are similar proofs for other structures, and it doesn’t tell me how many neurons is “enough” (so I don’t know how big those layers need to be), but it means I’ll always have a starting point.

One final note about structure, specifically the output. The more mathematically-inclined reader might have gotten a little distracted up above when I talked in terms of percentages. After all, it seems a bit much to expect the simple math here to reliably produce a valid probability between 0 and 1, especially when there are multiple outputs. And it is! Typically what will happen is that the output of the neurons, called in this case “logits” (after the statistical term, a portmanteau of “logistic unit”), will be taken up and together adjusted by a mathematical function to turn the collected output into a neat probability distribution where all of the possible outputs add up to 100%. That’s going to prove helpful in training since our error functions like mean square error work better when we know each neuron’s output. But it’ll really come into its own later, when we start talking about how generative language models select each output word.

Task 3: Training

This is the main show.

Training a neural network refers to a process of repeatedly seeing what the network outputs for a set of known inputs and then making tiny adjustments to its parameters so that the next time it encounters that input, the output is closer to what I want. The adjustments are so small that I might have to do that thousands, millions, or billions of times, but that’s all it is.

Getting a useful network out, then, requires me to have example inputs—lots of them—for which I can already tell the model what output I want. The example inputs should ideally cover a wide selection: anything I might consider a valid 7, etc. but what’s most important is that for each input, I know what output I want the network to return. The whole collection of input and output pairs is called a training set.

For each example in our training set, we just try it in the neural network as it exists right then, and we check what comes out. Remember that error function from task 1? My goal in training is to get that error as low as I can. I’ll try to alter the network a tiny bit so that next time the error is a little smaller: If it happened to give the answer I wanted, then I want the network to do more of whatever it’s doing. If it gave a wrong answer, I want the network to do less of whatever it was doing.

So I’m going to start with the pattern I wanted to see at the output. If the image was supposed to be a “7”, then the output neuron representing 7 should activate, and none of the others should. I can then go to that neuron’s inputs and see what it was working with.

A neuron’s output is determined by its inputs times its weights. Heavier weights (larger positive or negative numbers) times a large input contribute the most to what this neuron did, and I can trace their influence all the way back to the input pixels. For example, consider this gentleman’s budget:

We’re going to adjust everything, but it’ll be according to how much each element drove the error from my error function (which here might be the difference between actual expense and the monthly budget). Candles are the biggest expense, so that’ll get adjusted the most, but what’s also important is what isn’t there. Even though future months’ budgets might be blown on taking taxis everywhere, transportation had no impact on the error for this particular month (example) so we’re not going to do anything with it.

When running one drawn “7” through my network, it turned out that neuron 55 was the “candles” of my network: it had the biggest impact on the decision to call this a 7. Neuron 55 has weights for every pixel, but only the ones directly involved contribute to neuron 55’s decision to activate, so those are the only weights I know how to update.

This gets involved as the network gets more complex, and especially as more layers are added, but the math is not complicated⁹. The trick is to only make very small adjustments each time, wherever you have an actual contribution, proportional to the contribution.

This is the “learning” in “deep learning.” This is how example data becomes a giant mathematical equation that interprets an image as a numeral. It takes inspiration from biology, but I think it has more in common with geological processes like erosion and deposition, or architectural concepts like desire paths. Any individual example has a tiny effect, like a drop of flowing water or a grain of carried sand, but many examples will weather and build up a logical landscape.

This process—working my way backward through the network from one example, tweaking parameters according to whether and how they contributed to the answer I wanted—is called backward propagation or backpropagation. Each training example (an input and desired-output pair) run through the network causes me to solve a series of calculus problems that tell me exactly how every single parameter in the network contributed to or suppressed the answer I wanted. That tells me how much to adjust each weight in consequence.

Now, I could just run that one image through the network however many times until it starts saying, “Yeah, that’s a 7.” It probably wouldn’t even take that long. The weights for the pixels in that image of a 7 would slowly all become larger positive numbers.. The other weights wouldn’t change from their original random numbers; they’d all be multiplied by 0 and contribute nothing to whether the sevenness output neuron activated.

But I want to produce a network that tends to match what I as a human being would recognize as a 7. There are a whole lot of ways to draw a recognizable 7, even in such a small image. I cannot possibly produce them all, and anyway I want it to recognize 0s, 1s, 2s, etc. too. My hope is that I can put together a training set of so many varied, good examples that the training process will not only learn the examples I give it, but also other numerals that I didn’t, but that I would personally recognize.

If I do this many times with good examples, I’m approximating a process called gradient descent in which the network will essentially embody a kind of mathematical landscape shaped by my examples that will also allow it to infer what should happen for similar inputs that it wasn’t shown.¹⁰ The end result of that gradient descent is not guaranteed or even likely to be understandable: that is what produces those very alien patterns of which pixels each neuron is paying attention to.

History has shown that for some important problems, this is possible. There really is a mathematical landscape for which a set of 784 numbers between 0 and 1 reliably mimic the ability to discern the intended handwritten numeral. In fact, there are probably millions of different ways to do it, and which one you actually get depends on a lot of random factors. The result is not perfect, but this isn’t a job that human beings do perfectly. It will fail in predictable ways, and I’m sure we can all come up with examples that such a system would probably get wrong.

But it can also fail in unpredictable and sometimes alien ways. Some biological concepts of training do help in thinking through failure modes. A very large network is capable of simply memorizing the examples or parts of them (called overfitting), or noticing unintended commonalities to your examples. If you’ve ever tried to teach a cat to stay off the table, you probably realized that what the cat actually learns is, “don’t get on the table when this human can see you.” If all your pictures of sheep are in sunny green pastures, your network may be learning to simply guess “sheep” when shown a sunny green pasture.

One problem with working with the MNIST data is that they are all positive examples: every single image is an actual handwritten numeral, nicely centered so some pixels in the corners and edges aren’t ever filled in. Any weights involving those unused pixels will still be random, so if it ever encounters a filled-in corner in the wild, its behavior could be baffling. For example, if you shift a numeral it recognizes readily so that some of it is in that normally-unused space, accuracy plummets. We see this as “the same numeral” but the network never saw numerals there and so can’t deal well with shifting.

It’s worth asking: why use a neural net for this? If I have to put together a big list of examples anyway, why not just match against those known patterns? If we could make a complete set of all the possible handwritten numerals, we could do that! That’s essentially the overfitting case, but if the training set is truly comprehensive, then that’s fine. Even better would be if we knew how to write a rule; that would also be faster and more reliable.

What neural nets are good for are when we can provide a lot of good “know it when you see it” examples that adequately cover the range of what you expect to encounter, but where you can’t cover everything and a rule would be hard to write. The kind of examples matter, and again, we need to be working with numbers representing measurements, but we mostly need a lot of them.

The Unreasonable Effectiveness of Data

One of the key insights behind neural networks is the idea that with enough well-chosen labeled data, it’s possible to fine-tune the values of a very large mathematical equation that tends to return the desired values even for examples not in the training data. This is kind of an alien way to interact with computers. There is an art to designing the structure of the network, and experienced programmers can gain considerable intuition into what will work. You can quantify how good it is at solving the problem, but it is difficult or impossible to understand how the computer is solving the problem. It is in that respect much like training an animal.

Analogies to biology only get you so far, though: real brains are adapted to learn and generalize very efficiently from sparse data. You don’t get ten million meals in your lifetime, let alone ten million examples of which berries taste good. Human beings also benefit from elegant learning approaches like analogy and reason. The designer of a machine learning system essentially gives up on an elegant and understandable solution in favor of brute force with sometimes millions or billions of examples, but training neural networks can be startlingly (even unreasonably¹¹) effective.

I’ve avoided an important biology-like question, though: how can you tell the network learned how to solve a problem, versus simply memorizing enough examples?

Some of that is answered by the structure of the network: we already saw that constraining the number of neurons in a layer forces it to throw away information. Keeping the layers in the network as small as possible, including bottleneck layers, can help ensure that memorization is simply infeasible. More of the process of avoiding memorization is a matter of art: training processes designed explicitly to disrupt memorization. But we can test how successful we are.

Memorization can be tested by giving a trained network a pop quiz. By holding back some labeled data for testing or validation, it’s possible to evaluate the network on examples it has never seen to determine how well it’s learning real patterns. Typically, you’ll track improvement by “epochs” or training passes, in which a single epoch represents the whole training set having been used in random order once, and there will be nice graphs that hopefully show continued improvement for both the training data and the validation data. Here’s a very early example, from LeCun, et al’s 1989 paper “Backpropagation Applied to Handwritten Zip Code Recognition”, showing error rate and the natural log of the mean square error over 30 training passes. Basically: the first few epochs show the starkest improvements for both the training data (solid line) and the validation data (dotted line), after which improvements are both gradual and can diverge.

Here’s a similar graph for my simple 64 neuron network¹² using a larger dataset than LeCun, et al had access to:

Often, the improvement for the validation set will plateau like the dotted lines: the network continues getting better on the training data (LeCun gets down to a 0.14% error rate on their dataset), but continued training produces no further gain (or worse!) when it’s tested against examples it hasn’t seen before (that early network never does better than 95% accuracy; mine tops out at 97% on the larger dataset). That’s a sign that the networks are increasingly leaning on memorization rather than patterns.

The next question, of course, is: where do you get all that labeled data? Someone has to do a lot of work, and often that work is drudgery. Data doesn’t just appear, let alone high-quality labeled data, and the provenance can be remarkably haphazard. It’s not like data labels itself, right?¹³

The MNIST dataset¹⁴ of handwritten labeled digits is a standard dataset from the 1990s. It was derived from multiple sources, including a 1980s-era collection by the National Institute of Standards and Technology (the folks who funded my PhD research, in fact), the US Postal Service (specifically in Buffalo, New York) and the Census Bureau. I think student test data’s in there too. Putting it together was no easy task, requiring a lot of collaboration and is the result of a lot of judgement calls and arguments. It and its successors have been used now for decades, but the students whose numerals were used are probably still kicking around out there.

In Practice

As you can imagine, the chances of getting this all right on the first try can be slim. It is not at all uncommon to train multiple networks with different structures and hyperparameters before getting it to work well enough for the task. There are a lot of things to tweak, and it’s not always clear what will help, because it’s not always clear why something might be working poorly. Insight based on experience can get you pretty far, but trial and error play a role.

So too does randomness. The parameters of a network have to start somewhere, and randomly generated weights are common as a starting point. We saw that earlier with those neurons that accidentally got cut out of the training. The training data might all be used (every example is used once per epoch) but the order in which they’re used is generally randomized. As a result, even otherwise identical trainings can deviate quite a bit.

There’s also the nuts and bolts of how a neural network is actually made in computer code, which is tied up in the culture of programming. While the really high-end models probably use some custom code¹⁵, a lot of the programming is done using a set of tools that are developed and released for free. Some of the tools are written by academics; some by hobbyists; some still by multibillion dollar companies. Often it’s a mix of all three.

The cultural aspects here are worth exploring: most of the tools being used by people working with neural networks are given away for free, having been written and tested by volunteers. Many of those volunteers are academics (a lot of this work came out of academia) and hobbyists, but large companies sometimes do contribute to important tools. A lot of common datasets are likewise free, often having been compiled by government agencies, academics, or hobbyists. For better or worse, someone working on neural networks is often justified in assuming that whatever they need, if it exists, will be available free online. Researchers and hackers alike have a deeply-ingrained ethos of sharing¹⁶. If a programmer finds a software package or dataset that fits their needs, they may not look too closely at it.

For the code I wrote above, I used a program called VSCode, which is written by Microsoft and provided free. It’s very good, and in very common use. They actively update it, and many of the updates are actually useful. It probably costs them millions of dollars a year to keep this going. If like me you’re paying through the nose for Microsoft Word, that might surprise you. Microsoft’s reasons for releasing and maintaining this free software are probably complex, but they’ve decided that it benefits them to have this tool out there, provided (and controlled) by Microsoft.

I used a programming language called Python to write the code. Python has been around for about 35 years, originally a hobby (named by its designer Guido van Rossum after Monty Python) and then developed with the help of volunteers and funding from a variety of sources.

Programming languages like Python are extended with libraries: packaged code that implements functions that aren’t built into the original language; these can be for making graphics, reading different file types, etc. I used the free PyTorch library. It’s based on Torch, which was originally developed 24 years ago by the Idiap Research Institute in Switzerland. PyTorch is a very common set of tools for working with neural networks and other machine learning techniques. The Python version of that tool was adapted by Facebook engineers, and likewise released for free. I also used common free libraries for graphing outputs, and those libraries in turn incorporated additional libraries for shared tasks like mathematics. And the dataset, as you know, was compiled and provided by a sequence of researchers including the US government and Bell Labs.

So far, this has all been numbers. You can do a lot with numbers, and ultimately everything needs to be expressed as a number. We’re used to thinking of pictures as numbers thanks to how readily most people understand the concepts behind pixels. But chances are, the deep learning systems you write about will involve inputs that aren’t quite so obvious.  Next time we’ll talk about how language enters into it all.

How might this look in fiction?

Let’s take the tasks in order.

The first task is probably going to be inverted in your story planning: Instead of having a problem and choosing neural networks from among any number of machine learning techniques as the right tool for the job, you’ve probably decided that you want your characters to be building a neural network. You need to pick a problem that makes sense. Sometimes in the real world it’s as simple as “This is the buzzword. We’re going to do the buzzword.” And honestly, that “sometimes” probably isn’t necessary. A while back the big thing was fuzzy logic.¹⁷ Researchers in industry and academia absolutely used it because sure it might solve their problem, but it would absolutely get attention and funding. Deep learning is not different in this regard.

But let’s say that you want that part of the story to make sense, or at least not be quite so cynical. You’ll need to identify a problem where there is plenty of data, and where the inputs and outputs can be expressed numerically. That said, there are lots of ways to express even non-numeric things as numbers. In the numeral example we assigned each output to a numeral, so that if one output neuron activated it meant “1” or “7”; you can do something similar with inputs.

At its most simplistic, you can use what’s called a “one-hot encoding” in which you set aside individual inputs to each correspond to a specific value. Say, four inputs for each of North, South, East, and West. To signal East, you set the input for that neuron to 1, and the other three to 0¹⁸. But these could really be anything, and they don’t have to be constrained to 0 and 1. Or assign each input to every possible ingredient in a soup, where the value is the quantity used. You could even just list every single word in your vocabulary, however many thousands you know, and have an input for each one, for each position in a sentence. That would be a whole lot of inputs and a whole lot of weights, but nothing about that scheme wouldn’t function.

Next task: Structure. This’ll probably be less of an issue for most fiction, which isn’t going to go into such detail, but it’s good to have a sense that the layers or substructures of a network are working in sequence and transforming inputs in often-unrecognizable ways. A neural network isn’t a completely opaque box that your characters will have no insight into, but they’re also not going to have a good sense of what’s going on. Neural networks are weird, and they learn to do familiar tasks in ways we find utterly alien. As the networks get larger and deeper (more structure steps between input and output) it becomes increasingly difficult to interpret what’s going on. It’s not impossible, and there are a variety of explainable AI (XAI) techniques, so you can have your characters investigate. Just be aware that it’ll be a painstaking and time-consuming process using a lot of math, and the results will probably be weird like the neuron images above. If your characters are seeing problems like memorization, overfitting, or failing to improve past a certain point, they’ll likely try to first address those problems by a combination of adjusting the structure of the network and trying out different training techniques.

Finally: Training. Let’s start with the training data.

Training data can be a rich source of story material. You can add stress or conflict to your story even just at the level of collecting data in the first place. Where do your characters get labeled training data? How do they enough of it? Are they getting it ethically? Who is labeling it, and are the labelers trustworthy?

Sometimes your answer might be: Your characters simply can’t get the data they need, or it’s poor quality¹⁹, or there’s not enough, or it doesn’t cover the ground, etc. They may or may not understand that they don’t have what they need. The danger of a network overfitting to training data, or built on insufficient data can lead to interesting narrative developments: something that seems like it’s working very well can prove to have dangerous blind spots. Neural networks fail in weird ways sometimes, and those failures very often can be traced to defects in the training data.

Training data needs to cover as much ground as possible, and humans are the ones judging that ground, so it’s easy to see how bias creeps in. Remember, it’s not just the training that’s automated, it’s the validation too. Famous examples in real life include face recognition systems undertrained on darker skin, with predictable results. Or, more accurately, unpredictable results: you just don’t know what it will do in cases it wasn’t trained on. It might have picked up on cues that would help it extrapolate to faces of all colors, or it might not. Or it might fail seemingly at random. In fact, they fail in such weird ways that your real work as a writer might be in convincing readers that real failure modes are realistic.

There have been famous examples where the training data was selected without noticing other features: training a network to recognize sheep through a set of images that always showed grassy fields, might just train for grassy field recognition. Counter-examples (grassy fields without sheep; sheep loitering on street corners) are just as important as a wider set of positive examples. Did your characters remember that? It’s difficult to know what features of the input are being used to come to a conclusion, but once failures do become evident, humans are very good pattern matchers.

It’s important to understand unpredictability in cases not covered or only sparsely covered by training data. It’s not that it will necessarily arrive at wrong answers, it’s that the answer space will become chaotic. Sometimes it will seem to get things right, and then sometimes fail wildly. It might limp along on a secondary assumption in many cases, like all human beings are lit by fluorescent lights, or the difference between an adult and a child is that only adults wear glasses. Readers may well expect more consistency from a fictional neural network than is realistic, but it should be possible to make any desired failure mode feel reasonable.

There are other fragilities to look for, such as data format. My 28×28 handwritten numeral detector just won’t work for a 25×25 or 30×30 image: it’s not that it’s confusing, it’s that my network needs exactly 784 input numbers. Your characters might say, “well, it’s obvious, just add pixels and pad it out”. OK, but remember what just a little shifting did? Worse, it might be done automatically by some image processing stage in a way you don’t expect, like using gray pixels to pad, or trying to awkwardly scale it, or padding on different sides.

Your character’s data stages are worth thinking through. Not just “where do they get the data?”, but “how do they make the data uniform?”, and “how do they label the data and ensure uniform labeling?” Preprocessing data can be a problem, like that example of converting house numbers into latitude and longitude: what’s the failure mode if it can’t find an address? Is there, say, a default location it always chooses?²⁰ Data is a chore, and is rarely usable without a ton of work. It’s a very rich source of problems in fiction, just as it is in real life.

Lastly, let’s talk about the training itself. This is the most expensive part of making a neural network in pretty much every sense of the word: it takes the most time, it takes the most planning, and it takes the most computing power. When large companies use billions of dollars worth of compute²¹ much of that goes toward training. What they would dearly love to do is train once and never again. But I think you can tell by now that it is very difficult to get this right on the first try, and randomness plays an uncomfortably large role.

If you need a ticking clock or character conflict in your story, model training has some interesting potential. The biggest training gains come at the beginning. Every successive epoch (a complete run-through of the training data) tends to bring diminishing gains. Every additional epoch costs time and money. The training is interruptible; it can be stopped and resumed. Your characters may very well argue: is the model good enough to stop now? Do we start over with a different structure or risk throwing good money after bad? Likewise, there’s no reason you can’t change up your data mid-training if your characters find problems in it or need to add cases… but they don’t really know what it’s already learned, do they?

In a fantasy setting, the mechanisms of training might provide some useful story fodder. How do your characters design a system whose components change in response to having produced a correct or incorrect output? In the gold wire example, perhaps it can be placed in a chemical bath that will either deposit or remove metal according to electrical (or magical) signals. Maybe literal erosion and sedimentation are used. Fungal networks might alternate nutrients and poisons. Crystals might grow opaque or clear under different color lights, a little like how microchips are etched.

However the training is done, the training data may be more of a problem in fantasy than in a science fiction setting. Managing a large dataset without computers is tough; potentially a much larger challenge than building the network itself. Data might be unreasonably effective, but not if you can’t get it, organize it, and use it. In a fantasy world, then, your characters may be strongly motivated to put their cleverness into the design of the network so that they don’t need as much training data. Think back to the Drosophila connectome: fruit fly neural anatomy works through cleverly evolved structure rather than prolonged learning. We ourselves somehow manage to speak and read without first consuming the entire Internet. But as the training set size shrinks, the potential for blind spots, bias, and simple memorization grows.

Even if your story takes place after the neural network is trained and shipped, the training process looms over the network it produces. Having a sense of how that might have gone, and where it might have gone badly, can give you options for your story’s plot and help make it feel more real.

---

1. Look, if IPv6 was going to happen, it would have happened by now. ↩︎
2. The LeNet series of neural networks used a more complex form called convolutional neural networks (CNN), which are common in visual applications. In particular they address two problems: pixel counts (and thus weight counts) get big even for modest-size images, and the kind of neural networks I’m talking about just throw away the knowledge of which pixels are next to each other. Convolutional networks do some cool things focusing neurons on portions of the full input. There are a whole lot of LeSomething networks, and he claims he tries to talk people out of them. ↩︎
3. Seriously. I forget what I was going for there. ↩︎
4. Little bit of math for a treat: What I’m doing here is comparing two vectors, the expected output and the actual output. The expected output is [0,0,…,1,0] and the actual is [0.3,0,…,0.7,0]. Mean Squared Error (MSE) squares the individual differences, sums them, and divides by the number of output neurons. So, ((-0.3)² +(0.3)²)/10 = 0.018. An error of 0 is only possible if the network gives us exactly the expected output. ↩︎
5. This is called a linear classifier, and here’s an example that accomplishes just that: https://github.com/david-lorow/mnist-linear-classifier ↩︎
6. This is actually a selection of 32 neurons from a network of over 500 in the first layer, to make it more likely I’d get a dead neuron. Also, there are standard techniques I would have ordinarily been using to prevent that occurrence. ↩︎
7. There are sub-fields of deep learning such as “explainable AI (XAI)” and “mechanistic interpretability” that seek to do better than just telling a story. Some of it is really wild, like pinning one neuron to always activate and then seeing what it does. Doesn’t sound wild? Check this out: https://talkieneurons.vercel.app/ ↩︎
8. By George Cybenko in 1989. I’m only partly bringing this up because he was my postdoc advisor and just generally a nice guy; I’ve always thought it was a really neat insight. ↩︎
9. Grant Sanderson’s Deep Learning series has a fantastic video on this process in the context of exactly this problem, and you should be equipped now to get the gist of it. (https://www.youtube.com/watch?v=Ilg3gGewQ5U) ↩︎
10. And in fact, the process of running an input through a neural network and getting an output is sometimes called inference. ↩︎
11. Referring to a 2009 article entitled “The unreasonable effectiveness of data.” and itself a reference to a 1960 lecture “The unreasonable effectiveness of mathematics in the natural sciences”. My read of the 2009 article is that until then, approaches to machine learning tended to assume that the real smarts come in at the level of architecture and design, whereas you get better results by just using a ton of data than you’d have any right to expect. Philosophically, I think it’s fair to say that this approach is still the dominant one, but that it’s increasingly an unexamined belief. ↩︎
12. On my MacBook Air laptop, this takes maybe three minutes. ↩︎
13. Yes, this is foreshadowing. ↩︎
14. The full history is convoluted but interesting. https://en.wikipedia.org/wiki/MNIST_database ↩︎
15. I’m not familiar with any specific system, but I would expect portions to be written in more specialized programming languages. The bulk is probably still Python like I used. ↩︎
16. There’s a lot that can go into this footnote, but I’m trying to stick to what I promised in the intro in terms of coming to conclusions for you. I’ll only say that within the academic and hacker communities, this sense that information, work, and tools should be shared is part of what it means to belong to those communities. ↩︎
17. I liked fuzzy logic, damn it. I took a whole class on it in the 90s. Fuzzy logic makes for good rice cookers. There are even combination neuro-fuzzy networks where neural networks incorporate fuzzy logic. It’s great. Mark my words, it will return. But it was also a fad. ↩︎
18. Often there will be systems in place outside the neural network to prevent nonsensical values. Even if setting both North and East hot might make intuitive sense to a human, that’s just as inherently legal as setting both North and South. Program constraints and what’s present in the training data make a lot of difference! ↩︎
19. The Abby Normal dataset? ↩︎
20. Just ask James and Theresa Arnold, whose Kansas farm smack in the middle of the country was the default result for IP address lookups. https://www.theguardian.com/technology/2016/aug/09/maxmind-mapping-lawsuit-kansas-farm-ip-address ↩︎
21. I know the term is barbaric, but there’s no escaping it at this point. ↩︎