Lesson 10.3: Forward Propagation & Backpropagation (Conceptual)

🔹 Forward Propagation

Forward propagation is the process where input data passes through the network to produce an output.

1. Inputs are multiplied by weights.

2. Bias is added.

3. The result passes through an activation function.

4. Output is generated at the final layer.

Mathematical Representation:

a[l]=activation(W[l]⋅a[l−1]+b[l])a^{[l]} = activation(W^{[l]} \cdot a^{[l-1]} + b^{[l]})a[l]=activation(W[l]⋅a[l−1]+b[l])

• a[l−1]a^{[l-1]}a[l−1] → Activation from previous layer

• W[l]W^{[l]}W[l], b[l]b^{[l]}b[l] → Weights and bias for current layer

🔹 Loss Function

• Measures difference between predicted output and actual target.

• Examples: Mean Squared Error (MSE) for regression, Cross-Entropy Loss for classification.

🔹 Backpropagation

Backpropagation is the process of updating weights to minimize the loss.

1. Compute gradient of loss w.r.t weights using chain rule.

2. Propagate errors backward through the network.

3. Update weights using gradient descent:

W=W−η⋅∂Loss∂WW = W – \eta \cdot \frac{\partial Loss}{\partial W}W=W−η⋅∂W∂Loss​

• $\eta$ → Learning rate

🔹 Example (Conceptual)

• Forward → Compute predicted output ypredy_{pred}ypred​

• Loss → Loss(ytrue,ypred)Loss(y_{true}, y_{pred})Loss(ytrue​,ypred​)

• Backward → Adjust weights to reduce loss

🔹 Advantages

• Enables neural networks to learn complex patterns.

• Fundamental for training deep networks.

🔹 Limitations

• Can be computationally intensive for deep networks.

• Requires careful learning rate tuning.

✅ Quick Recap:

• Forward Propagation → Compute output from input.

• Loss → Measure prediction error.

• Backpropagation → Update weights to minimize error.