AI from scratch

Module 03 · Deep Learning

The same ideas — stacked until they get smart

Neural networks end-to-end: neurons, activations, loss, backpropagation, optimizers, regularisation, a taste of CNNs — and why this is the engine under modern AI.

From classic ML to deep nets

Module 2 ended with a map: tabular data → classic ML; images, text, audio → deep learning. A neural network is not a new species of magic — it is logistic regression stacked on logistic regression, with a non-linear twist that lets it draw curves classic models cannot.

Industry vocabulary you’ll hear:

parameters     the weights & biases being learned (can be billions)
forward pass   compute predictions from inputs
loss           scalar “how wrong am I?”
backward pass  compute gradients of the loss w.r.t. each parameter
update         nudge parameters a step downhill (optimizer)

One neuron, then a layer

A single neuron does three things:

1. take inputs x₁, x₂, …
2. weighted sum   z = w₁x₁ + w₂x₂ + … + b
3. activation     a = f(z)     e.g. ReLU: max(0, z)

One neuron alone is basically linear/logistic regression. The leap is stacking many neurons into a layer, then stacking layers into a multilayer perceptron (MLP). Each layer’s outputs become the next layer’s inputs.

Width = neurons per layer. Depth = number of layers. Early layers tend to learn simple pieces; later layers compose them — edges → shapes → objects, or characters → words → ideas. That compositional hierarchy is why deep nets crush unstructured data.

Without non-linear activations, a stack of linear layers collapses into one linear layer — no matter how deep. The bend is what makes depth useful.

Activations (the bend that makes depth useful)

ReLU      max(0, z)     default for hidden layers; sparse, fast
Sigmoid   1/(1+e^(−z))  classic; saturates → vanishing gradients
Tanh      similar S-curve, zero-centered
Softmax   multi-class probabilities at the output (sum to 1)
GELU/SiLU smoother ReLU cousins — common in transformers

Interview habit: say what you use where — ReLU (or GELU) in hidden layers, sigmoid/softmax at the output depending on binary vs multi-class.

Loss, gradients, backprop, optimizers

Loss — the score you minimise

Gradient descent — nudge downhill

The gradient says which way the loss increases fastest. You step the opposite way. Step size = learning rate.

too high  → loss explodes or oscillates
too low   → training crawls; may stall
practice  → try a range; use schedules; watch curves

Minibatch SGD and Adam

Batch = how many examples per update. Full-batch is rare. Minibatch SGD estimates the gradient from a small batch — noisier, but faster and often generalises better. An epoch = one full pass through the training set.

Adam (and friends) adapt the step per-parameter using momentum-like statistics. Default starting point for many deep-learning projects; SGD-with-momentum still wins in some vision recipes. Know both names.

Backpropagation

Backprop is just an efficient application of the chain rule: start from the loss, walk backward through each layer, and ask how much each weight contributed to the error. Frameworks (PyTorch, JAX) do this with autograd — you write the forward pass; they build the backward graph.

GPT-scale training is still this loop: forward → loss → backward → optimizer step — on mountains of text, with fancy layers (attention) and enormous compute. Same skeleton you just learned.

Regularisation & training hygiene

Vanishing & exploding gradients

In very deep stacks, gradients can shrink toward zero (early layers stop learning) or explode. Mitigations you’ll cite in interviews: ReLU, careful init, normalisation, residual connections (skip connections — the trick behind ResNets and used inside transformers).

Experiment: logistic vs MLP on interlocking rings

Classic linear models draw straight boundaries. A neural net can curve. Two interlocking rings of points (make_circles): no straight line separates them.

model                  train acc   test acc
Logistic Regression      0.607      0.563   ← barely above a coin flip
MLP (32 → 16 hidden)     0.959      0.907   ← the curve-drawer wins
▶ Run this yourself in Colab

Synthetic data, fixed seed — same two accuracy numbers.

That ~34-point gap is deep learning in one screenshot: when the pattern is non-linear, stacked activations invent the curves. Logistic regression cannot.

On tabular data, remember Module 2 — gradient boosting often still wins. Neural nets earn their keep on unstructured data and compositional/curved problems. Pick the tool for the shape of the problem.

CNNs in one page (vision intuition)

Convolutional neural nets slide small filters over an image to detect local patterns (edges, textures) while sharing the same weights across positions — fewer parameters than a giant MLP on raw pixels, and translation-aware.

Practice project (roadmap week 6): classify Fashion-MNIST or CIFAR-10 in PyTorch. You’ll reuse the same training loop — only the layer types change. Transformers later replace a lot of this for language (and increasingly for vision), but CNN intuition still shows up in interviews and hybrid systems.

Why this unlocks GenAI

An LLM is a very deep neural network with a special layer called attention (Module 4). Everything else still applies: weights, activations, a loss (next-token cross-entropy), gradients, backprop, optimizers, regularisation.

Once you can say “a transformer is a deep net trained with backprop to predict the next token,” the mystique drops — and interview answers get sharper.

Try it yourself

from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = make_circles(n_samples=1000, noise=0.15, factor=0.5, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

for name, m in [
    ("logistic", LogisticRegression(max_iter=1000, random_state=42)),
    ("mlp", MLPClassifier(hidden_layer_sizes=(32, 16), max_iter=1000, random_state=42)),
]:
    pipe = make_pipeline(StandardScaler(), m)
    pipe.fit(Xtr, ytr)
    print(name, round(pipe.score(Xtr, ytr), 3), round(pipe.score(Xte, yte), 3))

Interview questions

Also in the full interview bank.

What does an activation function do, and why do we need one?

It adds a non-linearity after the weighted sum. Without it, stacked layers collapse into a single linear transform and cannot model curved boundaries. ReLU is the common default for hidden layers.

Explain backpropagation in one breath.

Compute the loss, then use the chain rule to push gradients backward through each layer so every weight learns how much it contributed to the error — then take a small step downhill (optimizer / gradient descent).

SGD vs Adam?

SGD (often with momentum) takes steps from minibatch gradients. Adam adapts step sizes per parameter using running averages of gradients — usually converges faster out of the box; SGD still preferred in some vision training recipes.

When would you still prefer gradient boosting over a neural net?

Structured/tabular data, smaller datasets, tight latency budgets, or strong out-of-the-box baselines. Deep nets shine on unstructured, high-dimensional, compositional inputs.

What is the vanishing gradient problem?

Gradients shrink toward zero as they flow backward through deep stacks, so early layers stop learning. Mitigations: ReLU, residual connections, better init, normalisation — all common in modern transformers.

Batch norm vs dropout — what problem does each attack?

Batch/layer norm stabilises activations (faster, steadier training). Dropout randomly zeros units during training to reduce co-adaptation (regularisation). Complementary, not substitutes.

Why share weights in a CNN?

The same local pattern (an edge, a texture) can appear anywhere in the image. Shared filters detect it efficiently with far fewer parameters than a dense MLP on pixels, and buy translation awareness.