A4.3.8 Structure and Function of ANNs (HL only) Notes

What Are Artificial Neural Networks?

1. Artificial Neural Networks (ANNs) are computational models inspired by the human brain.
2. They consist of interconnected nodes or "neurons" that work together to process and learn from input data.

Key Components of ANNs

Nodes (Neurons)

1. Role: Fundamental units of the network.
2. Each neuron:
   1. Receives inputs.
   2. Applies weights + bias.
   3. Passes the result through an activation function.
3. Produces an output that feeds into the next layer.

Weights

1. Numerical values that determine the strength/importance of each input connection.
2. Adjusted during training to minimize errors.

Bias

1. A constant term added to the weighted sum.
2. Helps the network shift activation functions to better fit data.
3. Without bias, some patterns may be impossible to learn.

Activation Function

1. Decides whether a neuron “fires” and introduces non-linearity.
2. Common examples:
   1. Sigmoid: Output between 0 and 1.
   2. ReLU (Rectified Linear Unit): Outputs positive values or 0.
   3. Tanh: Output between –1 and 1.
3. Without activation functions, ANNs would act like simple linear models.

Layers of an ANN

1. Input Layer: Receives raw data (features).
2. Hidden Layers: Perform computations and extract patterns.
   1. Multiple hidden layers = deep learning.
3. Output Layer: Produces final prediction (e.g., class label, probability, number).

How ANNs Work

1. Forward Propagation:
   1. Data moves from the input layer through hidden layers to the output layer.
2. Activation Functions:
   1. Introduce non-linearity, allowing the network to learn complex patterns.
3. Backpropagation:
   1. The network adjusts weights and biases based on the error between predicted and actual outputs.

The Perceptron: Building Block of ANNs

What Is a Perceptron?

1. A perceptron is the simplest form of a neural network, consisting of a single node.
2. It can classify linearly separable data by applying a weighted sum of inputs and an activation function.

Structure of a Perceptron

1. Inputs:
   1. Data features (e.g., $I_1$, $I_2$).
2. Weights:
   1. $W_1$, $W_2$, etc.
3. Bias:
   1. A constant value added to the weighted sum.
4. Activation Function:
   1. Transforms the weighted sum into an output.

Activation Functions

1. Step Function:
   1. Outputs 1 if the weighted sum exceeds a threshold, otherwise 0.
2. Sigmoid Function:
   1. Maps the output between 0 and 1, useful for probabilities.
3. ReLU (Rectified Linear Unit):
   1. Outputs the input if positive, otherwise 0.

Multi-Layer Perceptrons (MLPs)

What Is a Multi-Layer Perceptron?

A Multi-Layer Perceptron (MLP) is a type of ANN with one or more hidden layers. It can model complex, non-linear patterns in data.

Structure of an MLP

1. Input Layer:
   1. Receives raw data.
2. Hidden Layers:
   1. Perform computations to extract features.
3. Output Layer:
   1. Produces the final prediction.

How MLPs Work

1. Forward Propagation:
   1. Data flows through the network, with each layer applying weights, biases, and activation functions.
2. Backpropagation:
   1. The network adjusts weights based on the error, using techniques like gradient descent.

Modeling Complex Patterns with MLPs

Why Use MLPs?

1. MLPs are powerful because they can model non-linear relationships in data.
2. This makes them ideal for tasks like image recognition, natural language processing, and more.