Data Scientist

What this role actually does

The data scientist’s job is to build models that generalise: that make accurate predictions on data they have never seen, explain the patterns they find in terms a human can act on, and quantify how uncertain those predictions are. Not to explore data — that was Stage 1. Not to serve models at scale — that is Stage 3. The distinct skill here is the full modelling cycle, done rigorously.

Day-to-day: a problem framing conversation in the morning, feature engineering and EDA through the early afternoon, model comparison and cross-validation by 3pm, a SHAP waterfall chart explaining the biggest prediction to a stakeholder by 4. The skill is not any single algorithm. It is knowing which model assumptions hold for this dataset, evaluating without leakage, and explaining the result without misrepresentation.

Where this role sits relative to its neighbours. The data scientist is not a data analyst. You are building predictive and generative models, not answering “what happened” questions with statistical tests. The inference mindset of Stage 1 — p-values, confidence intervals, hypothesis tests — does not go away; it is subsumed. Every model evaluation is a statistical question. Every uncertainty interval is inference. Stage 2 uses the same mathematical machinery as Stage 1, applied to a different problem.

The data scientist is also not a machine learning engineer. You are responsible for the model’s correctness and interpretability. The MLE is responsible for its reliability and scale in production. In small teams these roles merge; in mature ML organisations they do not. This stage ends at the point where you hand off a well-specified, well-evaluated model to engineering — not at the point where it is serving traffic.

The most important single concept here is leakage. Data leakage means information from outside the training window contaminating the model — future data, target-correlated preprocessing fitted on the full dataset, labels leaking into features. It is the reason models that look incredible in development fail silently in production. Everything in this stage is downstream of understanding why leakage happens and how to prevent it.

Core concepts to master

Depth tags: Competent = can do independently, Production = reliable under pressure, Expert = can teach it and debug edge cases.

Unfamiliar with a term? The glossary defines every concept used across the map.

• Problem framing — supervised, unsupervised, and reinforcementProduction

  Choosing the wrong paradigm poisons every step that follows. A churn-prediction problem framed as clustering instead of supervised classification produces a model that cannot be evaluated or deployed. Frame first; code second.

• The bias-variance tradeoffExpert

  Every model decision — regularisation strength, tree depth, number of features — is a point on the bias-variance curve. Without this concept, hyperparameter tuning is superstition.

• Train/val/test splits, cross-validation, and data leakageExpert

  A model that performs well on training data is not an achievement. A model that performs well on held-out data, without any information from the future, is. Data leakage is the single most common source of false wins in published ML benchmarks.

• Feature engineering and feature selectionProduction

  On tabular data, the features matter more than the model. The best XGBoost with mediocre features loses to a logistic regression with excellent features. Feature work is the highest-return activity in most real projects.

• Linear and logistic regressionExpert

  Not just as baselines — as models whose coefficients have meaning. A logistic regression coefficient is an interpretable odds ratio. A linear regression residual plot tells you whether your model has systematic errors. These are not the simple models you skip; they are the models you understand deeply.

• Regularisation — ridge (L2), lasso (L1), elastic netProduction

  Overfit a model badly enough and it memorises training data; regularisation is the penalty that prevents this. L1 gives sparsity (some weights go to zero); L2 gives stability. Knowing which to choose and why is a fundamentals question in every ML interview.

• Decision trees and random forestsProduction

  The workhorse of tabular ML. Random forests are robust, require minimal hyperparameter tuning, and provide feature importances that are interpretable at first pass. Understanding how they work — variance reduction through bagging, feature subsampling — is not optional.

• Gradient boosting — XGBoost and LightGBMProduction

  The algorithm that wins structured/tabular competitions. XGBoost changed applied ML when it appeared; LightGBM made it practical at scale. On tabular data, boosted trees are still the default serious approach before deep learning.

• Clustering — k-means, hierarchical, DBSCANCompetent

  The entry to unsupervised learning. Clustering is frequently misused — k-means on high-dimensional data without preprocessing produces meaningless results. Knowing the assumptions of each algorithm (and when they break) is the practical skill.

• Dimensionality reduction — PCA, t-SNE, UMAPCompetent

  Visualising a 100-dimensional dataset requires collapsing it to 2D without destroying structure. PCA is linear and fast; t-SNE preserves local structure but distorts global; UMAP is faster than t-SNE and often better. More importantly, PCA has genuine predictive uses as preprocessing.

• Model evaluation — ROC/AUC, calibration, precision/recallExpert

  Accuracy is almost never the right metric. A model predicting 'no fraud' on a 0.1% fraud dataset has 99.9% accuracy and catches zero fraud. ROC/AUC, precision-recall curves, and calibration curves are what honest evaluation looks like.

• Uncertainty quantification and conformal predictionProduction

  A prediction without a confidence interval is an opinion. Conformal prediction provides coverage guarantees that are valid under minimal assumptions — no distributional assumptions, no model-correctness assumptions. This is the modern standard for honest uncertainty.

• Interpretability — SHAP, permutation importanceProduction

  A model you cannot explain is a model you cannot debug, defend, or improve. SHAP values give each feature a locally consistent, globally coherent contribution to every prediction. The business stakeholder wants to know why; SHAP gives you the answer.

• Causal inference introduction — potential outcomes, selection biasCompetent

  ML models predict; causal models explain. A model that predicts churn accurately from 'received a support call' does not tell you whether the support call caused the churn or whether unhappy customers happen to call support. Getting this wrong leads to interventions that make things worse.

• Time series basics — stationarity, autocorrelation, decompositionCompetent

  A huge fraction of real business data is time-indexed. Applying a standard train/test split to time series data gives a future-leaking evaluation. Time series has its own set of assumptions, its own train/val split protocol, and its own failure modes.

Mathematics required

Minimum — what you must understand to use the algorithms correctly

Linear algebra — the geometry of data. Vectors as points in space; matrices as linear transformations. Matrix multiplication: what it means geometrically (a composition of transformations), not just how to compute it. The dot product as a measure of alignment. Rank as the dimensionality of the column space. Eigenvalues and eigenvectors at the conceptual level: the directions a transformation stretches, and by how much. You need these for PCA, for understanding gradient descent on the loss surface, and for reading any ML paper that uses matrix notation (all of them).

Calculus — the mathematics of change. Derivatives as rates of change. Partial derivatives: how a function changes as one input varies while others are held fixed. The gradient: the vector of partial derivatives, pointing in the direction of steepest ascent. The chain rule: how derivatives compose through layers of functions — this is what backpropagation computes. You do not need to prove the chain rule; you need to apply it fluently to composite functions.

Probability and statistics — carried forward from Stage 1, extended. Joint distributions, marginal distributions, and conditional distributions. Maximum likelihood estimation (MLE): the parameter values that make the observed data most probable. The maximum a posteriori (MAP) estimate: MLE plus a prior, which turns out to be equivalent to regularisation. Bayes’ theorem not just as a formula but as an update rule: prior belief + evidence = posterior belief.

Optimisation — why gradient descent works. A loss function is a surface in parameter space. Gradient descent moves downhill by following the negative gradient. Learning rate controls step size — too large and you oscillate or diverge, too small and you crawl. The intuition for why this finds a minimum for convex losses and a local minimum for non-convex ones. Stochastic gradient descent: using a noisy estimate of the gradient computed on a mini-batch, which is often faster and sometimes regularises the solution.

Research-grade — where the algorithms become legible

Derive the bias-variance decomposition. The expected test error of a model decomposes into irreducible noise, bias squared (how far the average model prediction is from the truth), and variance (how much the model varies across different training sets). This derivation is not hard — it is an expectation calculation using the definition of variance — but it makes the tradeoff precise and explains why averaging models (bagging) reduces variance without increasing bias.

The geometry of L1 vs L2 regularisation. L2 regularisation (ridge) adds a spherical constraint to the parameter space; the optimum lives somewhere on the sphere. L1 regularisation (lasso) adds a diamond-shaped constraint. The corners of the diamond lie on the axes — where many parameters are exactly zero. This geometric picture explains why L1 gives sparse solutions and L2 does not. It is one of the most illuminating geometric arguments in all of applied mathematics, and it takes about twenty minutes to understand once you have the picture in front of you.

Information theory — entropy and KL divergence. Shannon entropy H(X) = −Σ p(x) log p(x) measures the average uncertainty in a random variable. The KL divergence D_KL(P ‖ Q) measures how much information is lost when Q is used to approximate P. These appear everywhere: cross-entropy loss is KL divergence to a one-hot distribution; information gain in decision trees is a difference in entropy; variational inference minimises KL divergence. One chapter of any information theory textbook is sufficient.

SVD and its connection to PCA. The singular value decomposition A = UΣVᵀ decomposes any matrix into rotations and scaling. PCA is SVD applied to the centred data matrix. The principal components are the right singular vectors; the explained variance is the squared singular values. Understanding this makes PCA a tool rather than a black box, and it opens the door to the matrix factorisation methods used in recommender systems.

The EM algorithm for mixture models. The expectation-maximisation algorithm alternates between assigning data points to clusters (E step) and updating cluster parameters given those assignments (M step). Gaussian mixture models are the classic case. Understanding EM transforms k-means from a heuristic into a special case of a principled probabilistic algorithm, and explains why k-means is sensitive to initialisation.

→ Linear algebra · Calculus and optimisation · Probability and statistics — each section has worked derivations and a depth ladder.

Tools and engineering skills

scikit-learn — deeply, not just the fit/predict API. Pipelines (Pipeline and ColumnTransformer) so that preprocessing is part of the model object and cannot leak between train and test. Cross-validation with cross_val_score and StratifiedKFold — understanding why you pass the full unfitted pipeline, not a pre-fitted transformer. GridSearchCV and RandomizedSearchCV for hyperparameter search with nested cross-validation when the dataset is small. The calibration_curveutility for checking whether your model’s predicted probabilities are meaningful.

XGBoost and LightGBM. Both are gradient boosted trees with different implementation strategies — LightGBM uses leaf-wise growth and histogram binning, making it faster on large datasets; XGBoost uses level-wise growth and is often more stable. In practice: try both, tune the learning rate and number of rounds carefully (these interact), and use early stopping on a validation set. Understand the regularisation parameters (lambda, alpha, min_child_weight) — they are not knobs to turn until the model improves; they are controls on the bias-variance tradeoff with interpretable effects.

SHAP. The standard for model explanations. SHAP values have a theoretical basis (they are the unique attribution satisfying four fairness axioms from cooperative game theory) and a practical implementation that is fast enough for production. Know the difference between global explanations (feature importance plots, beeswarm plots) and local explanations (waterfall plots, force plots for a single prediction). Know when SHAP explanations are misleading: highly correlated features split the attribution between them in ways that can be unintuitive.

MLflow or Weights & Biases. Experiment tracking is not optional once you have run more than ten experiments. The discipline of logging every run — parameters, metrics, artifact paths — so you can reproduce any previous result is the difference between a scientist and a notebook tinkerer. Pick one, use it from the first experiment of every project, and never run an unlogged experiment again.

Conformal prediction libraries. The modern standard for uncertainty quantification requires minimal code — the MAPIE library wraps scikit-learn models and provides coverage-guaranteed prediction intervals in a few lines. The key insight: calibrate on a held-out calibration set after training, not on the training set, to get valid coverage guarantees.

Pandas and NumPy at scale. Not just the basics — efficient aggregation (groupby with transform for group-level features), memory management (dtype selection, chunked reading for large files), and vectorised operations instead of Python loops. A pandas operation that takes 10 minutes to run is usually a loop in disguise; vectorise it and it takes 10 seconds.

The project

Three projects — an end-to-end prediction system, a feature engineering deep-dive, and a paper reproduction — together prove the full Stage 2 competency. The first shows you can build and evaluate correctly. The second shows you can squeeze performance from data rather than just from model choice. The third shows you can engage with the research literature and understand what reproducible results look like.

Expected total time: 12–18 weeks alongside the curriculum. Build them in order — each one uses skills developed in the previous.

Project 1 — the end-to-end prediction system (6–8 weeks)

What it proves: you can build a production-credible predictive model — correctly evaluated, properly calibrated, uncertainty-quantified, and explained.

Choose a tabular classification or regression problem with genuine stakes: credit risk, customer churn prediction, medical outcome prediction, equipment failure forecasting, or any domain where the prediction has a real cost if it is wrong. The dataset must have at least 10,000 rows, at least 15 features of mixed types (numeric, categorical, at minimum one date), and a meaningful class imbalance (for classification) or non-trivial noise (for regression). Public sources: Kaggle, the UCI ML Repository, government open data portals.

Execute the full pipeline in order, using a scikit-learn Pipeline that wraps every step from raw features to final predictions:

1. Problem framing and baseline. State the prediction task formally: what is the input, what is the output, what is the evaluation metric, and why. Build a baseline — a dummy classifier or the mean prediction for regression — and record its performance. Every model you build for the rest of the project must beat the baseline or it is not progress.
2. EDA and leakage audit. For every feature, ask whether it can be known at prediction time in production. A feature computed from future data, a feature that directly encodes the target, or a feature that is only available after the event you are predicting — all are leaks. Document your audit. List every feature you excluded and why.
3. Feature engineering.Build at least five derived features that your initial EDA suggested are meaningful. Document the hypothesis behind each: “users who contacted support more than twice in the 30 days before their renewal date churn at 3× the base rate — so I am creating a binary flag for this.” Test whether each feature improves performance in isolation before adding it to the full model.
4. Model comparison under proper cross-validation. Compare at minimum: logistic regression (your interpretable baseline), random forest, XGBoost, and LightGBM. Use stratified 5-fold or 10-fold cross-validation with the full Pipeline fitted inside each fold — never fit a preprocessor on the full dataset before splitting. Report mean ± standard deviation of your primary metric across folds. The standard deviation tells you whether your result is stable.
5. Calibration. For classification: plot the calibration curve of your best model. Is the predicted probability of 0.7 associated with about 70% of positive outcomes? If not, apply Platt scaling or isotonic regression calibration and re-plot. A model whose probabilities are not calibrated should not be used to make decisions that depend on those probabilities (most real decisions do).
6. Conformal prediction intervals.Wrap your best model in a conformal predictor (MAPIE or equivalent). Set a coverage level of 90%. Report what fraction of held-out test examples fall within the predicted interval — it should be close to 90%. This is the first place you will encounter what “coverage-guaranteed uncertainty” actually means in practice.
7. Interpretability.Compute SHAP values for your best model. Produce: (1) a global feature importance beeswarm plot, (2) a local explanation for the three most interesting individual predictions (the highest-confidence correct, the highest-confidence wrong, and the most uncertain). For each local explanation, write one sentence describing what the model “saw” in the data and whether the explanation makes domain sense.

The deliverable is a technical report, not a notebook. A 10–15 page document: problem definition, data description (including the leakage audit), methods (every preprocessing and modelling step), results (all cross-validation numbers in a table, calibration plots, conformal coverage result), interpretation (SHAP plots + prose explanation), limitations (what the model cannot do and why), and a one-page executive summary written for a non-technical stakeholder. The notebook is the appendix. The report is the deliverable.

Project 2 — the feature engineering deep-dive (2–3 weeks)

What it proves: you understand that features matter more than models, and you can engineer them systematically rather than by intuition alone.

Take a Kaggle competition dataset (any structured competition, active or historical) with a public leaderboard. Start with a baseline: raw features, minimal preprocessing, XGBoost with default hyperparameters. Record the cross-validation score and leaderboard position. Then, over two weeks, engineer features systematically:

• Interaction features: products and ratios of numeric features you hypothesise interact. Test each one: does adding it improve cross-validation score?
• Aggregation features: for grouped data (users, products, time windows), compute group statistics (mean, std, max, min, percentiles) and join them back. These are often the highest-value features in practice.
• Temporal features: if there is a date column, extract day of week, month, quarter, time since a reference event, and any domain-relevant seasonal signals.
• Encoding decisions: for high-cardinality categoricals, compare target encoding vs ordinal encoding vs leave-one-out encoding. Document which works better and why.

The deliverable is a notebook documenting every feature you tried, the hypothesis behind it, and whether it improved performance. Expected outcome: a 5–15% improvement over baseline from features alone, before any hyperparameter tuning. Write a half-page reflection: which features helped most, which surprised you, and what you would try with more time.

Project 3 — reproduce and extend a published result (3–4 weeks)

What it proves: you can engage with the research literature and understand what rigorous evaluation looks like.

Choose a paper that presents a machine learning algorithm or method with benchmark results on a public dataset. A strong default: the XGBoost paper (Chen & Guestrin, 2016 — linked in the Resources section above) on any of its public benchmarks. Alternatively, find a recent tabular-ML paper on Papers With Code that reports results on a UCI or Kaggle dataset you can download. The criterion is simple: the paper must describe its experimental setup clearly enough that you can reproduce it from scratch.

1. Reproduce the number. Implement or use the library implementation, reproduce the exact dataset split and preprocessing the paper describes, and run the evaluation. How close did you get to the reported number? Any discrepancy is worth investigating: is it a random seed difference, a preprocessing difference, a library version difference?
2. Characterise the sensitivity. Vary two hyperparameters around the values the paper reports. How sensitive is performance to each? Papers often report results at a single tuned point without showing the sensitivity landscape — this step reveals whether the reported result is robust or fragile.
3. Write a reproduction report. One to two pages: what you reproduced (or failed to reproduce and why), what the sensitivity analysis showed, and one observation that surprised you. This is the foundation of the skill of reading research papers critically — not accepting numbers, but checking them.

Senior extension — model monitoring design

For your Project 1 model, design (but do not implement) a monitoring strategy: which metrics would you track in production, how would you detect data drift (PSI, KS test, or population stability analysis), and what threshold would trigger a retraining run? Write a one-page monitoring specification as if you were handing it to an ML engineer for implementation. This is the bridge to Stage 3 thinking: “my model is not done when it is trained; it is done when it is maintained.”

Research extension — novel evaluation

Identify a failure mode of your Project 1 model that is not captured by your primary metric — a subgroup where performance is systematically worse, a region of the input space where calibration breaks down, or a tail scenario where the conformal intervals are unexpectedly wide. Propose and implement a secondary evaluation that surfaces this failure. Write it up as a one-page appendix to your technical report. The ability to identify the failure modes your primary metric misses is the research-grade version of model evaluation.

Resources

Start with ISLR — it is the canonical entry, free, and available in both R and Python editions. Pair it with MML for the mathematics and StatQuest for intuition on each algorithm. Hands-On ML is the practical companion you work through when you want code. ESLis the rigorous big brother — consult it when ISLR’s answer is “it’s more complicated than that.” The mathematics resources (3Blue1Brown, MIT 18.06, Stat 110) cover the linear algebra, calculus, and probability this stage requires — use them in parallel with ISLR, not after.

How you know you’re done

Exit criteria — you can answer “yes” to all of these:

• Given a tabular dataset and a prediction task, you can choose between linear regression, logistic regression, random forest, and XGBoost — and justify the choice based on the number of samples, the presence of interactions, the interpretability requirement, and the training time budget.
• You can implement a scikit-learn Pipeline that wraps preprocessing and a model, run stratified k-fold cross-validation with the full pipeline, and explain why fitting the preprocessor inside each fold matters.
• You can explain what a SHAP value is — not just how to compute it, but what it means (the marginal contribution of a feature value, averaged over all possible feature orderings) — and describe a scenario where the SHAP explanation could be misleading.
• You can compute a conformal prediction interval and state its coverage guarantee precisely: “with probability at least 1 − α over the calibration set, the interval contains the true label.” You understand that this guarantee is marginal, not conditional.
• An interviewer says “your model has 94% accuracy on the test set — how confident are you in this number?” You can walk through: class imbalance, the evaluation protocol (single split vs. cross-validation), the possibility of leakage, calibration, and whether accuracy is even the right metric for this task.
• You understand the difference between correlation and causation at the operational level: you can give a concrete example from your own project where a predictive feature is not a causal factor, and explain the implication for any intervention based on that feature.

Self-test questions

1. You train a random forest on a dataset with 50 features. Feature importance (mean decrease in impurity) shows Feature A as most important. You remove Feature A, retrain, and performance drops by less than 0.5%. Explain what likely happened. How would you properly evaluate whether Feature A is genuinely important?
2. Your logistic regression classifier has ROC-AUC = 0.91 on the test set. Your stakeholder asks: “If the model outputs a probability of 0.7 for a customer, does that mean there is really a 70% chance they will churn?” How do you answer? What would you check?
3. Explain the bias-variance tradeoff to a non-technical product manager in two sentences. Then explain it formally: write the expected mean-squared error decomposition and identify each term.
4. You are building a model to predict customer lifetime value (CLV) using features including the number of support tickets a customer has filed. Should you include this feature? What question would you ask to determine whether it is a leak?
5. Your XGBoost model achieves 0.88 AUC in cross-validation but 0.72 AUC on a held-out test set from three months later. List three possible explanations in decreasing order of likelihood, and describe how you would diagnose each.

Bridge to the next stage

The data scientist asks: can I build a model that works? The machine learning engineer asks: can I make a model work reliably, at scale, for months? These are different questions, and they require different skills.

Stage 3 is the transition from “this model works in my notebook” to “this model works in production at 10,000 requests per second, with monitoring, retraining pipelines, and a team of engineers depending on it.” The mathematics of Stage 2 continues — you need it for deep learning — but it is joined by software engineering, systems thinking, and the discipline of making things fail gracefully rather than just making them work once.

Deep learning enters properly at Stage 3. The neural network is not a mystery if you understand the chain rule, gradient descent, and the building blocks from Stage 2. It is a very large, very expressive model trained by a very general optimisation algorithm. The concepts are the same; the scale and the failure modes are different.

What to take into Stage 3: the modelling discipline (never evaluate without a proper protocol, never deploy without monitoring), Python fluency with pandas and scikit-learn, Git and experiment tracking as reflexes, and a working mental model of the bias-variance tradeoff. The research branch is available now — if Stage 3 production work is not your direction, Stage 2 is the point to consider branching toward research.