IB Computer Science Topic A4.3.9: Design of Cnns Notes

Convolutional Neural Networks: Building Image Features From the Ground Up

1. A plain multi-layer perceptron can be pointed at an image, but it handles one badly for two reasons.
2. Connecting every pixel of a 200 by 200 colour image to every node in the first hidden layer needs 120,000 weights per node, which is unaffordable.
3. An MLP also treats each pixel position as unrelated to its neighbours, so a cat learned in the top-left corner is unrecognised in the bottom-right.
4. A CNN fixes both by applying the same small filter across the whole image, which cuts the weight count enormously and makes the detected features work anywhere in the frame.

The Six Stages of a CNN

1. Input layer
   1. Receives the raw pixel values, arranged as a grid with one channel for greyscale or three for colour.
2. Convolutional layers
   1. Slide learned filters, also called kernels, across the input to produce feature maps that mark where each feature occurs.
3. Activation functions
   1. Apply a non-linear function, almost always ReLU, to every value in the feature map.
4. Pooling layers
   1. Shrink each feature map, keeping the strongest responses and discarding exact positions.
5. Fully connected layers
   1. Combine the features found anywhere in the image into evidence for each class.
6. Output layer
   1. Produces the final prediction, usually through a softmax function that turns the scores into probabilities.

Convolution: Sliding a Filter Over the Image

1. A filter is a small grid of weights, commonly 3 by 3, whose values are learned during training rather than set by hand.
2. At each position the filter's weights are multiplied by the pixel values underneath and the products are added, which is a dot product.
3. The resulting number goes into the feature map at the corresponding position, so a high value marks a position where the filter's pattern is present.
4. Each convolutional layer applies many filters at once, producing one feature map per filter.

Pooling: Shrinking the Feature Map

1. Max pooling takes the largest value in each small window, typically 2 by 2, and discards the rest.
2. Halving the width and height cuts the number of values by four, which reduces computation in every later layer.
3. Keeping only the maximum records that a feature was present in the region while forgetting its exact position, which is why a shape shifted a few pixels still registers.
4. The cost is deliberate information loss, so pooling too aggressively throws away detail the classification needed.

From Feature Maps to a Decision

1. After the last pooling layer the feature maps are flattened into a single list of values.
2. The fully connected layers then work like an ordinary MLP, with every value connected to every node.
3. These layers do the reasoning, weighing up which combinations of detected features point to which class.
4. The output layer carries one node per class, and softmax scales its values so they are all positive and sum to 1.

The Four Design Choices That Shape a CNN

Number of Layers

1. More convolutional layers let the network build features from features, so it can represent more abstract structure.
2. Depth is what allows a network to move from edges, to textures, to object parts, to whole objects.
3. Deeper networks need more training data and more computation, and they overfit more readily on a small data set.
4. Depth also makes training harder, because the error signal weakens as it is propagated back through many layers.

Kernel Size and Stride

1. Kernel size sets how much of the image each filter sees at once.
2. A small kernel such as 3 by 3 captures fine local detail and uses few weights.
3. A large kernel such as 7 by 7 captures broader structure in one step but costs far more weights per filter.
4. Stride sets how far the filter moves between positions.
5. A stride of 1 evaluates every position and gives a detailed feature map the same size as the input.
6. A stride of 2 skips every other position, which halves the feature map's width and height and speeds the network up at the cost of resolution.

Activation Function Selection

1. The role of an activation function and the shapes of ReLU, sigmoid and tanh are covered under artificial neural networks.
2. ReLU is the default in CNN hidden layers because it costs one comparison to compute and keeps the error signal strong through many layers.
3. Leaky ReLU is used when too many nodes have stopped responding, since it lets a small signal through for negative inputs instead of zero.
4. Softmax belongs in the output layer of a multi-class classifier, and sigmoid in the output layer of a two-class one.

The Loss Function

1. The loss function measures how wrong a prediction was and is the quantity training minimises.
2. Cross-entropy loss is the standard choice for classification, and it compares the predicted probabilities against the correct label.
3. It punishes confident mistakes heavily, so predicting 0.99 for the wrong class costs far more than predicting 0.6 for it.
4. Mean squared error is used instead when a CNN predicts a number, such as the position of a bounding box.
5. Choosing the wrong loss function undermines everything else, since the network will optimise faithfully towards the wrong target.

Spatial Hierarchies and Where CNNs Are Used

1. A spatial hierarchy is the layered progression of features the network learns, from local to global.
2. Early layers respond to edges and colour gradients, middle layers to textures and corners, and late layers to object parts.
3. The word adaptively in the syllabus statement points at the key fact: the filters are learned from the data, not designed by an engineer.
4. Image classification assigns one label to a whole photograph.
5. Object detection finds and labels several objects and returns their positions.
6. Medical imaging flags anomalies in X-rays and MRI scans, where a CNN can be trained on more labelled scans than a radiologist sees in a career.
7. Autonomous vehicles use CNNs to read lane markings, signs and pedestrians from camera feeds.