Mathematics

Algebra and functions

Algebraic manipulation, function composition, and the behaviour of logarithms and exponentials. The key identity every ML practitioner uses daily is the relationship between multiplication and addition of logs (which turns products of probabilities into sums):

The natural logarithm $\ln x$ and the exponential $e^x$ are inverses: $e^{\ln x} = x$. The softmax function — the gating mechanism of neural networks and the output of every classifier — is built entirely from this pair. Logarithms also appear in entropy, KL divergence, and information theory (all Tier 3).

Set theory, logic, and notation

Probability theory is built on set theory. The symbols $\in, \subseteq, \cap, \cup, \setminus$ appear in almost every probability definition. Propositional logic (implication, equivalence, contrapositive) is needed for reading proofs. Mathematical quantifiers $\forall, \exists$ appear throughout.

Summation and product notation

Compact notation for sums and products. The training loss of nearly every model is a sum over data points; likelihoods are products. These notations let you read and write such expressions without confusion:

Proof techniques — the research seed

You do not need to write proofs at Stage 0. But learning to read them — especially induction and proof by contradiction — is the research-seed investment that pays off at Stage 2 and beyond. Induction is the standard tool for recurrences and complexity arguments; contradiction is how most impossibility results are established.

Vectors, matrices, and operations

A vector $\mathbf{x} \in \mathbb{R}^n$ is a point in $n$-dimensional space. A matrix $A \in \mathbb{R}^{m \times n}$ is a linear map from $\mathbb{R}^n$ to $\mathbb{R}^m$. Matrix multiplication $AB$ composes two linear maps. These are not just notational conventions — they are the geometric content.

The Euclidean norm of a vector:

The dot product $\mathbf{x}^\top \mathbf{y} = \sum_i x_i y_i = \|\mathbf{x}\|_2 \|\mathbf{y}\|_2 \cos\theta$ measures alignment — the foundation of cosine similarity, attention scores, and the kernel trick.

Eigenvalues, eigenvectors, and eigendecomposition

A vector $\mathbf{q}$ is an eigenvector of $A$ with eigenvalue $\lambda$ if:

For a symmetric positive-definite matrix — the type that appears in covariance matrices, kernel matrices, and Hessians — the eigendecomposition factors $A$ as:

where $Q$ is an orthonormal matrix of eigenvectors and $\Lambda$ is diagonal with the eigenvalues. This decomposition is behind PCA, kernel methods, spectral clustering, and the analysis of gradient descent convergence (the condition number of $A$ is the ratio $\lambda_{\max}/\lambda_{\min}$).

Singular Value Decomposition (SVD)

The SVD generalises eigendecomposition to non-square matrices — which is essentially everything in ML, since data matrices are $n \times d$ with $n \neq d$:

$U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthonormal; $\Sigma$ is diagonal with the singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$. Truncating to the top $k$ singular values gives the best rank-$k$ approximation to $A$ in the Frobenius norm — this is the mathematics behind PCA, dimensionality reduction, and collaborative filtering. LoRA (the fine-tuning technique in Stage 4) exploits exactly this low-rank structure.

The chain rule and backpropagation

The scalar chain rule:

Backpropagation is just the chain rule applied recursively through a computational graph. Every gradient in a neural network is computed this way. The key insight is that the chain rule decomposes the derivative of the loss $L$ with respect to early-layer weights into a product of local Jacobians, each computable from the forward pass.

Gradients and gradient descent

The gradient $\nabla_{\mathbf{w}} L$ is the vector of partial derivatives of the loss with respect to every parameter — the direction of steepest ascent. Gradient descent steps in the opposite direction:

where $\alpha > 0$ is the learning rate. The learning rate is the most consequential hyperparameter in training — too large and the loss diverges, too small and training stalls. Adaptive methods (Adam, RMSProp) effectively adjust the per-parameter learning rate from curvature information.

Matrix calculus

Matrix calculus extends scalar calculus to vector and matrix arguments. The two most common identities you will use:

These identities appear in the derivation of linear regression (where the normal equations emerge from $\nabla_{\mathbf{w}} \|\mathbf{y} - X\mathbf{w}\|^2 = 0$), in the analysis of quadratic forms in attention, and in second-order optimisation methods.

Convexity and why it matters

A function $f$ is convex if, for all $\mathbf{x}, \mathbf{y}$ and $t \in [0,1]$:

For convex functions, every local minimum is a global minimum — gradient descent is guaranteed to converge. Deep network loss landscapes are non-convex, which is why understanding when convexity fails and what the alternatives are (saddle-point theory, loss-landscape analysis) matters for serious research.

Bayes' theorem

The fundamental rule for updating beliefs given evidence:

In ML: posterior = likelihood × prior / evidence. Naive Bayes, Bayesian networks, variational inference, and the theoretical underpinning of regularisation (L2 = Gaussian prior; L1 = Laplace prior) all flow from this single equation. Be sure you can derive it from the definition of conditional probability.

Maximum Likelihood Estimation and MAP

MLE finds the parameter $\hat{\theta}$ that makes the observed data most probable under the model:

The log-sum form (equivalent to the product-of-likelihoods by the Tier 1 log identity) is numerically stable and separates into per-sample terms. Minimising cross-entropy loss is exactly MLE under a categorical model. MAP extends MLE by adding a prior, which introduces the regularisation term.

The Central Limit Theorem

For i.i.d. random variables $X_1, \ldots, X_n$ with mean $\mu$ and variance $\sigma^2$, the sample mean converges in distribution to a Gaussian:

This is why confidence intervals around test-set performance have the form “mean ± z·σ/√n”, why statistical tests use t or z statistics, and why batch gradients in SGD can be used as unbiased estimates of the full-data gradient. The CLT is the mathematical licence for sampling.

Hypothesis testing and p-values

A p-value is the probability of observing data at least as extreme as the data you actually observed, assuming the null hypothesis is true. It is not the probability the null is true. This distinction matters enormously in practice: a p = 0.03 result means the data is surprising under the null, not that the effect is large or real. Know the one-sample $t$-test, the two-sample $t$-test, ANOVA, chi-square, and when each applies. Know the Bonferroni and Benjamini–Hochberg corrections for multiple comparisons.

Books

Courses & Videos