Messy data beats clever models

• The data pipeline
• Leakage & honest splits
• Feature engineering that matters
• Imbalance & rare events
• Labels are a product
• Data for GenAI / RAG
• Interview questions

The data pipeline

sources → extract → validate → transform/features
       → split → train/eval → store artifacts → monitor inputs in prod

Industry habit: fit transforms on train only, then apply to val/test/prod with the same statistics. Scalers, imputers, target encoders — all leak if fitted on the full dataset before the split.

Validate early: null rates, ranges, sudden schema changes, duplicate keys. A model can’t fix a broken join.

Leakage & honest splits

Leakage = information unavailable at prediction time helped training. Metrics look heroic; production falls over.

• Scaling/encoding on train+test together
• Post-outcome features (filled after the event you’re predicting)
• Random splits on time-series / drifting worlds
• Same user/patient/device in train and test (entity leakage)

Split recipes: random (i.i.d.), time-based (train past / test future), grouped (keep entities together), stratified (preserve class rates — critical for fraud).

Feature engineering that matters

• Numeric — scaling, log transforms for heavy tails, winsorising outliers
• Categorical — one-hot (low cardinality), target/ordinal encodings (careful with leakage), embeddings for high cardinality
• Time — lags, rolling stats, calendar features; always aligned to what was known at decision time
• Text — classic TF-IDF baselines still strong; embeddings for semantic tasks

Tree models often need less scaling; linear models and neural nets care more. Always ask: would this feature exist in the live request path?

Feature importance & explainability

Tree gain/permutation importance and SHAP-style attributions help debug and satisfy “why?” questions in regulated domains. Keep a glass-box baseline even if a black box scores slightly higher.

Imbalance & rare events

Fraud, defaults, failures — rare positives. Recipe:

1. Pick the right metric (PR-AUC, precision@k, recall at fixed precision)
2. Class weights / calibrated probabilities
3. Threshold for the business cost matrix
4. Only then consider resampling tricks

Don’t jump to SMOTE before a weighted baseline and a clear cost story — interviewers notice.

Labels are a product

Garbage labels → garbage models. Budget for:

• Clear labelling guidelines and examples
• Inter-annotator agreement checks
• Handling delayed labels (fraud often confirmed weeks later)
• Feedback loops that don’t silently bias the next train set

Data-centric AI in one sentence: improving labels and slices often beats another model architecture.

Data for GenAI / RAG

• Chunking strategy (size, overlap, respect headers/tables)
• Metadata filters (product, date, access control)
• Eval questions with gold passages
• PII redaction before indexing
• Freshness SLA for the index

Interview questions

Also in the interview bank.