The workhorses — and how close simple gets

• Why GenAI engineers still need these
• Linear & logistic regression (the DNA of neural nets)
• The tree family: single tree → forest → boosting
• Other workhorses you’ll be asked about
• Bake-off: four models on NYC taxi fares
• When to use what (+ feature importance)
• Try it yourself
• Interview questions

Why GenAI engineers still need these

Structured data is everywhere (billing, clicks, risk, ops). Interviews test these relentlessly. And every idea here — loss functions, gradient steps, softmax outputs — gets inherited by neural networks and LLMs. Master these and deep learning becomes “the same ideas, scaled up.”

Linear & logistic regression (the DNA of neural nets)

Linear regression — predict a number

Fit the straight line (or hyperplane) of best fit: ŷ = w₁x₁ + w₂x₂ + … + b. “Given square footage, predict price.”

Training usually minimises mean squared error (MSE) — the average of (prediction − truth)². You can solve it in closed form for small problems, or use gradient descent: nudge weights a little downhill on the loss surface until MSE stops improving. That nudge-the-weights loop is exactly what deep nets do — just with millions more weights and a fancier architecture.

Logistic regression — predict a category

Despite the name, it’s a classifier. Take the same linear score, then squash it through a sigmoid into a probability between 0 and 1:

score  z = w·x + b
prob   p = 1 / (1 + e^(−z))
label  ŷ = 1 if p ≥ 0.5 (threshold is tunable)

Training minimises log loss / cross-entropy — heavily punish confident wrong probabilities. The threshold 0.5 is not sacred: for rare fraud you may flag at 0.1 to catch more cases (higher recall, lower precision).

Regularisation (L1 / L2)

Add a penalty on large weights so the model can’t rely on wild coefficients. L2 (ridge) shrinks weights smoothly. L1 (lasso) can drive some weights to zero (built-in feature selection). Elastic net mixes both. In neural nets, “weight decay” is usually L2.

The tree family: single tree → forest → boosting

Decision tree

A flowchart of yes/no questions learned from data: “distance > 3 miles? passengers > 2?” At each split, the algorithm picks the question that most purifies the labels — measured with Gini impurity or entropy (classification) or variance reduction (regression).

Superpower: interpretability — you can read why it decided. Weakness: alone, a deep tree memorises. That’s why Module 1’s deep trees overfit.

Random forest — bagging

Train hundreds of slightly different trees (random rows + random feature subsets) and let them vote or average. This is bagging (bootstrap aggregating): individuals overfit in different directions; averaging cancels noise. Forests reduce variance. Harder to overfit than one tree; still strong on tabular data.

Gradient boosting — fix mistakes in sequence

Build trees sequentially: each new tree fits the residual errors of the current ensemble. This is boosting — usually higher accuracy, more sensitive to hyperparameters (learning rate, depth, number of trees). XGBoost, LightGBM, and CatBoost are the industry defaults for tabular competitions and many production rankers.

Other workhorses you’ll be asked about

You don’t need to master all of these today — but you should recognise them in one line:

• k-Nearest Neighbors (k-NN) — predict from the k closest training points. Simple, no training step, slow at prediction, sensitive to scaling.
• Naive Bayes — fast probabilistic classifier assuming features are independent. Classic for text baselines; shockingly strong for the simplicity.
• SVM — find the maximum-margin boundary (kernels for non-linear). Less dominant now than boosting/nets, still appears in interviews and older systems.

Bake-off: four models on NYC taxi fares

Same real data as Module 1 — predict fare from distance + passenger count. R² on rides each model never saw:

model               train R²   test R²
Linear Regression    0.846     0.848
Decision Tree        0.932     0.769   ← overfit — worst on new data
Random Forest        0.922     0.849
Gradient Boosting    0.903     0.883   ← winner

Real data, fixed seed — you’ll get the same four numbers.

Gradient boosting did win — but look how close: linear regression landed at 0.848 vs 0.883 (~3.5 points). The simple baseline tied the random forest and nearly matched the best model, for a fraction of the complexity. The lone tree overfit — classic Module 1 failure mode.

When to use what (+ feature importance)

Structured / tabular data (rows & columns, business metrics) → gradient boosting usually wins. Classic ML is not dead here; it dominates. (Our taxi run is a live example.)

Unstructured data (images, text, audio) → deep learning (Module 3+).

Need explanations / regulated domains → linear models or shallow trees first; keep a glass-box baseline even if a black box scores slightly higher.

Feature importance

Tree ensembles can rank how much each input drove predictions (e.g. “distance mattered far more than passenger count for fare”). That’s gold for debugging and stakeholder trust. Neural nets mostly sacrifice this unless you add tools like SHAP. Interviews ask: how would you explain this model’s decision?

Try it yourself

import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.tree import DecisionTreeRegressor

df = sns.load_dataset("taxis").dropna(subset=["distance", "passengers", "fare"])
X, y = df[["distance", "passengers"]], df["fare"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

for name, m in [
    ("linear", LinearRegression()),
    ("tree",   DecisionTreeRegressor(random_state=42)),
    ("forest", RandomForestRegressor(n_estimators=200, random_state=42)),
    ("boost",  GradientBoostingRegressor(random_state=42)),
]:
    m.fit(Xtr, ytr)
    print(name, round(m.score(Xtr, ytr), 3), round(m.score(Xte, yte), 3))

Interview questions

Also in the full interview bank.