A neural network from scratch in NumPy

After working through the theory underlying neural networks, I built a network to identify handwritten digits. The objective was to apply the concepts in practice and understand how the design choices affected performance. I avoided frameworks such as TensorFlow, PyTorch and Keras because they hide much of the implementation. NumPy was used for the matrix operations, but the forward pass, backpropagation and gradient descent were written from scratch.

The same saved network is used in the interactive demo on the front page; it has not been retrained for the website. The training code, experiments and weights are available on GitHub.

Design and structure

The input data consists of 28 by 28 pixel grayscale images. This gives the network 784 input neurons, one for each pixel. These feed into two hidden layers containing 20 and 10 neurons, followed by 10 output neurons representing the digits zero to nine. ReLU is used in the hidden layers, with softmax and cross-entropy loss at the output.

Weight initialisation initially appeared to be a minor concern, but it had a serious effect during training. I first selected weights randomly between -0.5 and 0.5. Changing to Kaiming He initialisation improved convergence and accuracy because it is designed for non-linear activation functions such as ReLU. Poor initialisation can make the gradients too small or too large as they pass through the network.

Backpropagation

Training requires the network to determine how much each weight and bias contributed to the total error. These gradients are calculated from the output layer backwards using the chain rule. Although the derivatives of softmax and cross entropy are complicated separately, their product simplifies to:

dC_dz3 = a3 - y

In other words, the gradient at the output is the model's prediction minus the target value. Deriving this result made it clear why softmax and cross entropy are normally used together for classification.

The error is then passed backwards through each transposed weight matrix and multiplied by the derivative of ReLU. Each gradient is averaged across the batch. The base network uses full-batch gradient descent with a learning rate of 0.1.

Data preparation

The dataset is a CSV file containing 42,000 labelled images. I shuffle the data, reserve 1,000 images for testing and train on the remaining 41,000. The pixel values are divided by 255 to normalise them to a range between zero and one, and the labels are converted using one-hot encoding.

The separate test set measures how well the network generalises to unseen images. A high training accuracy alone can be misleading because the network may have memorised features in the training data instead of learning patterns that transfer.

Experiments and results

To assess how design features affected performance, each experiment changed one feature while keeping the rest of the base network constant. The experiments covered the number of hidden layers and neurons, sigmoid and tanh activation functions, different learning rates, a smaller training set, removing biases and mini-batch gradient descent.

Reducing the amount of training data provided the clearest example of overfitting. Training accuracy continued to increase while test accuracy flattened, showing that the network was becoming better at its training examples without improving on unseen data. More layers and neurons could improve accuracy, but they also increased training time and the risk of overfitting. ReLU performed similarly to tanh but trained faster, while sigmoid was less accurate because of the vanishing-gradient problem.

Running the network in a browser

For the website demo, I exported the six weight and bias arrays as base64-encoded 32-bit floats and implemented the forward pass in JavaScript. This only requires three matrix multiplications and the activation functions.

A hand-drawn canvas image does not initially have the same format as an MNIST image. MNIST digits are scaled to fit within a 20 pixel box and positioned by their centre of mass inside a 28 pixel frame. Before classification, the browser therefore crops the image to its drawn pixels, scales its longest side to 20 pixels and shifts its centre of mass to the middle.

The saved weights achieve 95.8% accuracy across all 42,000 labelled images and 96.8% on the 1,000-image test set. These are the results reproduced with the weights used by the demo, rather than the highest result observed during experimentation.

What I learned

Building the network without a framework made the relationship between the theory and the implementation much clearer. I now use PyTorch for this type of work, but writing the calculations myself first gave me a better understanding of what functions such as loss.backward() are doing.