This is the first part of a series on the mathematics that makes machine learning work.
I have been trying to understand machine learning at a level that actually satisfies me. I wasn't fine with the "neural networks are like the brain" talk. Nor was I fine with the "gradient descent finds the minimum" level without actually understanding what that means. I wanted to get to the level where I could look at what is happening inside a model and say I know why that works...Well maybe not everything but you get what I mean.
After going over 3Blue1Brown's Essence of Linear Algebra series, I can say I have a much better understanding of the mathematics that makes machine learning work...again, maybe not everything but you get what I mean. This is a series, remember? I plan to go over multiple series and books and videos and document my understanding.
Most introductions to ML treat the math as a tax you pay to get to the interesting part. I want to try something different. I want to show you that the math is the interesting part. That every major concept in linear algebra has a direct, non-metaphorical counterpart in how models are built and trained.
Like I said earlier, I started with 3Blue1Brown's Essence of Linear Algebra series. If you have not watched it, stop reading and go watch it. Then come back.
Vectors: Your Data Has Always Been This
The first thing 3Blue1Brown does is reframe what a vector is. A vector is a point in space, described by how far you travel in each direction to reach it.
A house with three features is a vector:
house = [1400, 3, 250000]
size beds price
A 28x28 pixel image is a vector of 784 numbers. A word in any modern language model is a vector of hundreds or thousands of numbers called an embedding. Every data point you will ever feed into a machine learning model is a vector.
When you add two vectors, you are combining two data points component by component. When you scale a vector by a number, you are amplifying or shrinking every feature proportionally:
2 × [1400, 3, 250000] = [2800, 6, 500000]
The reason vector arithmetic is so important is that everything that comes after depends on it.
Span: The Boundary of What a Model Can Learn
Take two vectors:
v = [1, 2]
w = [3, 1]
Scale each one by any real number and add the results. For example:
2v + 1w = 2[1,2] + 1[3,1] = [2,4] + [3,1] = [5, 5]
-1v + 3w = -1[1,2] + 3[3,1] = [-1,-2] + [9,3] = [8, 1]
You can reach [5,5]. You can reach [8,1]. In fact, because v and w point in genuinely different directions, you can reach every point in the 2D plane by choosing the right scalars. That reachable set is called the span of v and w.
Now look at these two vectors instead:
v = [1, 2]
w = [2, 4]
w is just 2 times v. They point in the same direction. Try to reach [3, 1]:
av + bw = a[1,2] + b[2,4] = [a + 2b, 2a + 4b]
For this to equal [3, 1], you need a + 2b = 3 and 2a + 4b = 1. The second equation is just 2 times the first, which means it would need 2(3) = 1, which is impossible. You cannot reach [3, 1]. The span collapsed to a single line and [3, 1] is not on it.
This is the geometric explanation for why redundant features break your model. If two of your input features are perfectly correlated, say income and income-in-thousands, they are v and w above. They span a line. Your model is trying to find patterns in a space that is smaller than you think it is.
This is also the foundation of Principal Component Analysis. PCA finds the directions of maximum spread in your data and rewrites everything in terms of those directions. It is finding the best possible set of vectors whose span captures the most useful structure in your dataset.
Linear Transformations: Every Layer in a Neural Network
A linear transformation takes a vector and produces a new vector, under two rules: lines stay straight, and the origin stays fixed.
The key insight is that you only need to know where î-hat [1, 0] and ĵ-hat [0, 1] land to know where everything lands. Because every vector is built from î-hat and ĵ-hat:
[3, 2] = 3 × [1,0] + 2 × [0,1] = 3î + 2ĵ
If î lands at [1, -2] and ĵ lands at [3, 0], then [3, 2] lands at:
3 × [1,-2] + 2 × [3,0] = [3,-6] + [6,0] = [9, -6]
You write î's landing spot and ĵ's landing spot as the columns of a matrix:
M = | 1 3 |
| -2 0 |
Multiplying M by [3, 2] gives:
| 1 3 | × | 3 | = | (1×3) + (3×2) | = | 9 |
| -2 0 | | 2 | | (-2×3) + (0×2) | | -6 |
Same answer. The matrix is a compact way of storing and applying the transformation.
A matrix is a linear transformation. They are the same thing.
Now look at a neural network layer. Your input is a vector x. The layer multiplies it by a weight matrix W and adds a bias:
x = [0.5, 0.8, 0.3] (your input features)
W = | 0.2 0.5 -0.1 |
| 0.4 -0.3 0.7 |
Wx = | (0.2×0.5) + (0.5×0.8) + (-0.1×0.3) | = | 0.10 + 0.40 - 0.03 | = | 0.47 |
| (0.4×0.5) + (-0.3×0.8) + (0.7×0.3) | = | 0.20 - 0.24 + 0.21 | = | 0.17 |
The weight matrix W just transformed your 3-dimensional input into a 2-dimensional output. It reshaped your data in space. That's what a layer does.
This is also why activation functions exist. If you stack two linear layers:
Layer 1: W₁x
Layer 2: W₂(W₁x) = (W₂W₁)x
Two linear transformations compose into one. No matter how many purely linear layers you stack, you get exactly one linear transformation. A 100-layer linear network is no more expressive than a 1-layer linear network. Activation functions like ReLU break this collapse by introducing non-linearity between layers.
Matrix Multiplication: Stacking Transformations
When you apply transformation M₁ and then M₂, the combined result is M₂M₁. Let us work through a real example.
Say you have two layers in a network:
M₁ = | 1 0 | (first layer: some shear)
| 1 1 |
M₂ = | 0 -1 | (second layer: 90° rotation)
| 1 0 |
The combined transformation M₂M₁:
M₂M₁ = | 0 -1 | × | 1 0 |
| 1 0 | | 1 1 |
Row 1 of M₂ · Col 1 of M₁: (0×1) + (-1×1) = -1
Row 1 of M₂ · Col 2 of M₁: (0×0) + (-1×1) = -1
Row 2 of M₂ · Col 1 of M₁: (1×1) + (0×1) = 1
Row 2 of M₂ · Col 2 of M₁: (1×0) + (0×1) = 0
M₂M₁ = | -1 -1 |
| 1 0 |
Now pass a data point [3, 2] through both layers. You can do it step by step:
Step 1 (through M₁): | 1 0 | × | 3 | = | 3 |
| 1 1 | | 2 | | 5 |
Step 2 (through M₂): | 0 -1 | × | 3 | = | -5 |
| 1 0 | | 5 | | 3 |
Or apply the combined matrix directly:
M₂M₁ × [3,2] = | -1 -1 | × | 3 | = | (-1×3)+(-1×2) | = | -5 |
| 1 0 | | 2 | | (1×3)+(0×2) | | 3 |
Same result. This is what a neural network is doing when data flows through multiple layers. Order matters here. M₂M₁ is not the same as M₁M₂.
The determinant property follows directly from this. When you compose transformations:
det(M₂M₁) = det(M₂) × det(M₁)
det(M₁) = (1×1) - (0×1) = 1
det(M₂) = (0×0) - (-1×1) = 1
det(M₂M₁) = 1 × 1 = 1
The combined transformation preserved area exactly. In training terms, information was not destroyed by passing through these two layers.
The Determinant: What Happens to Information
The determinant of a 2×2 matrix:
M = | a b | det(M) = ad − bc
| c d |
Let us compute a few to build intuition.
M = | 3 1 | det = (3×2) - (1×4) = 6 - 4 = 2
| 4 2 |
This transformation doubles area. Every region of space gets stretched by a factor of 2.
M = | 1 2 | det = (1×4) - (2×2) = 4 - 4 = 0
| 2 4 |
Determinant is zero. This matrix squishes the 2D plane onto a line. To see why, notice that column 2 is just 2 times column 1. î-hat lands at [1,2] and ĵ-hat also lands at [2,4] = 2×[1,2]. Both basis vectors land on the same line. Everything collapses.
What does this mean for a neural network weight matrix with determinant zero? It means that layer is destroying information. Inputs that were different before that layer become identical after it. That is irreversible. You cannot recover the original signal downstream, no matter what later layers do.
This is why vanishing gradients happen. If a weight matrix has very small values, its determinant is very small. Multiply many small determinants together across many layers and the product approaches zero. The gradient signal, which has to travel backward through every layer, shrinks to nothing. Early layers stop updating. Training stalls.
The fix, used in almost all modern networks, is careful weight initialisation. You initialise weights so that the determinant of each layer's matrix starts close to 1. Information is preserved and gradients can flow.
Inverse Matrices, Column Space, and Null Space
These three ideas come as a set. The question connecting them: given a transformation M and an output v, can I find the original input x?
Mx = v → x = M⁻¹v
The inverse matrix undoes M. Multiplying M⁻¹ by M gives the identity matrix, which changes nothing:
M = | 2 1 | M⁻¹ = | 3 -1 |
| 5 3 | | -5 2 |
M⁻¹M = | 3 -1 | × | 2 1 | = | (6-5) (3-3) | = | 1 0 |
| -5 2 | | 5 3 | | (-10+10) (-5+6) | | 0 1 |
The inverse exists only when the determinant is not zero. When det(M) = 0, M squished space and you cannot un-squish it, which means information was lost
The column space is every output M can possibly produce. Look at the columns of M:
M = | 1 2 |
| 2 4 |
Column 1 is [1,2]. Column 2 is [2,4] = 2×[1,2]. They point in the same direction. Every output of this transformation is some multiple of [1,2]. The column space is the line through [1,2]. This matrix cannot produce the output [1,0] no matter what you feed in.
In a neural network, this tells you the limit of what a layer can express. If your weight matrix has rank 1 (all columns pointing in the same direction), that layer can only produce outputs along a single line. No amount of training data or training time will make it represent anything richer. The architecture itself is the constraint.
The null space is every input that produces zero output:
M = | 1 2 | × | x | = | 0 |
| 2 4 | | y | | 0 |
This gives x + 2y = 0, so x = -2y. Every vector of the form [-2y, y] = y[-2,1] maps to zero. The null space is the entire line through [-2, 1].
What this means for a neural network is that any component of your input that lies in the null space of a weight matrix contributes nothing to the output. It passes through the layer and disappears. If you can identify which input directions are in the null space, you can remove them without changing anything the network produces. This is the mathematical basis for model compression. Modern large models are often compressed by 50% or more by identifying and removing the null space directions in their weight matrices.
Now the inverse matrix connects to the most concrete thing in supervised learning... solving for optimal weights.
In linear regression you have a data matrix X and known targets y. You want weights w such that Xw = y. This is Mx = v with different letters. The closed-form solution is:
w = (XᵀX)⁻¹Xᵀy
Let us work through a small example. Three houses:
X = | 1 1000 | y = | 200 |
| 1 1500 | | 280 |
| 1 2000 | | 350 |
The first column of X is all ones, representing the bias term. The second column is house size. We want w = [bias, price_per_sqft].
First compute XᵀX:
Xᵀ = | 1 1 1 |
| 1000 1500 2000 |
XᵀX = | 1 1 1 | × | 1 1000 | = | 3 4500 |
| 1000 1500 2000 | | 1 1500 | | 4500 7750000 |
| 1 2000 |
Then Xᵀy:
Xᵀy = | 1 1 1 | × | 200 | = | 830 |
| 1000 1500 2000 | | 280 | | 1,145,000 |
| 350 |
Then compute (XᵀX)⁻¹ and multiply by Xᵀy. The result:
w ≈ | 50 | → bias = 50 (base price $50k)
| 0.15 | → $150 per square foot
Prediction for a 1800 sqft house: 50 + 0.15 × 1800 = 50 + 270 = $320k.
That is the normal equation, just algebra giving you the optimal weights directly.
The reason deep learning does not use this is the cost. Inverting (XᵀX) scales as O(n³) where n is the number of features. With millions of parameters, this is computationally impossible. Gradient descent is slower in principle but tractable at scale. But the normal equation is the exact thing gradient descent is approximating, and knowing that makes training feel less like magic.
What This Part Was About
Every concept we covered has a direct and computable home in machine learning.
Vectors are your data, represented as arrays of numbers. The span of your feature vectors determines what patterns your model can detect. A linear transformation is what every network layer computes when it multiplies your data by a weight matrix. Stacking layers is composing transformations, which is matrix multiplication. The determinant tells you whether information is being preserved or destroyed as data passes through a layer. Column space is the limit of what a layer can express, and rank tells you how large that limit is. The null space contains the input directions that contribute nothing to any output, which is why models can be compressed. And the inverse matrix is the direct solution to finding optimal weights in linear regression.
In part two, we will go deeper on eigenvectors and eigenvalues, dot products, and how they show up in attention mechanisms and dimensionality reduction.