http://parrt.cs.usfca.edu/doc/matrix-calculus/index.html
(We teach in University of San Francisco's
MS in Data Science program and have other nefarious projects underway. You might know Terence as the creator of the
ANTLR parser generator. For more material, see Jeremy's
fast.ai courses and University of San Francisco's Data Institute
in-person version of the deep learning course.)
Printable version (This HTML was generated from markup using
bookish)
Abstract
This paper is an attempt to explain all the matrix calculus you
need in order to understand the training of deep neural networks. We
assume no math knowledge beyond what you learned in calculus 1, and
provide links to help you refresh the necessary math where needed. Note
that you do
not need to understand this material before you start
learning to train and use deep learning in practice; rather, this
material is for those who are already familiar with the basics of neural
networks, and wish to deepen their understanding of the underlying
math. Don't worry if you get stuck at some point along the way---just go
back and reread the previous section, and try writing down and working
through some examples. And if you're still stuck, we're happy to answer
your questions in the
Theory category at forums.fast.ai.
Note: There is a
reference section at the end of the paper summarizing all the key matrix calculus rules and terminology discussed here.
Introduction
Most of us last saw calculus in school, but derivatives are a
critical part of machine learning, particularly deep neural networks,
which are trained by optimizing a loss function. Pick up a machine
learning paper or the documentation of a library such as
PyTorch
and calculus comes screeching back into your life like distant
relatives around the holidays. And it's not just any old scalar
calculus that pops up---you need differential
matrix calculus, the shotgun wedding of
linear algebra and
multivariate calculus.
Well... maybe
need isn't the right word; Jeremy's courses show
how to become a world-class deep learning practitioner with only a
minimal level of scalar calculus, thanks to leveraging the automatic
differentiation built in to modern deep learning libraries. But if you
really want to really understand what's going on under the hood of these
libraries, and grok academic papers discussing the latest advances in
model training techniques, you'll need to understand certain bits of the
field of matrix calculus.
For example, the activation of a single computation unit in a neural
network is typically calculated using the dot product (from linear
algebra) of an edge weight vector
w with an input vector
x plus a scalar bias (threshold):
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EEDCFA4252D0992243A283CE0EB777A6-depth003.31.svg)
. Function
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C599D931407509E0FA08F8686B205B6D-depth003.25.svg)
is called the unit's
affine function and is followed by a
rectified linear unit, which clips negative values to zero:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-66258AA93A4746DA10D306190271DE4B-depth003.25.svg)
. Such a computational unit is sometimes referred to as an “artificial neuron” and looks like:
Neural networks consist of many of these units, organized into multiple collections of neurons called
layers.
The activation of one layer's units become the input to the next
layer's units. The activation of the unit or units in the final layer is
called the network output.
Training this neuron means choosing weights
w and bias
b so that we get the desired output for all
N inputs
x. To do that, we minimize a
loss function that compares the network's final
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-7971D42A6C6C6A28D6443F0645E4A036-depth003.25.svg)
with the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E146B831A1E53B95E4C63775285D62CF-depth003.25.svg)
(desired output of
x) for all input
x vectors. To minimize the loss, we use some variation on gradient descent, such as plain
stochastic gradient descent (SGD), SGD with momentum, or
Adam. All of those require the partial derivative (the gradient) of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-7971D42A6C6C6A28D6443F0645E4A036-depth003.25.svg)
with respect to the model parameters
w and
b. Our goal is to gradually tweak
w and
b so that the overall loss function keeps getting smaller across all
x inputs.
If we're careful, we can derive the gradient by differentiating the
scalar version of a common loss function (mean squared error):
But this is just one neuron, and neural networks must train the
weights and biases of all neurons in all layers simultaneously. Because
there are multiple inputs and (potentially) multiple network outputs,
we really need general rules for the derivative of a function with
respect to a vector and even rules for the derivative of a vector-valued
function with respect to a vector.
This article walks through the derivation of some important rules for
computing partial derivatives with respect to vectors, particularly
those useful for training neural networks. This field is known as
matrix calculus,
and the good news is, we only need a small subset of that field, which
we introduce here. While there is a lot of online material on
multivariate calculus and linear algebra, they are typically taught as
two separate undergraduate courses so most material treats them in
isolation. The pages that do discuss matrix calculus often are really
just lists of rules with minimal explanation or are just pieces of the
story. They also tend to be quite obscure to all but a narrow audience
of mathematicians, thanks to their use of dense notation and minimal
discussion of foundational concepts. (See the annotated list of
resources at the end.)
In contrast, we're going to rederive and rediscover some key matrix
calculus rules in an effort to explain them. It turns out that matrix
calculus is really not that hard! There aren't dozens of new rules to
learn; just a couple of key concepts. Our hope is that this short paper
will get you started quickly in the world of matrix calculus as it
relates to training neural networks. We're assuming you're already
familiar with the basics of neural network architecture and training. If
you're not, head over to
Jeremy's course
and complete part 1 of that, then we'll see you back here when you're
done. (Note that, unlike many more academic approaches, we strongly
suggest
first learning to train and use neural networks in practice and
then
study the underlying math. The math will be much more understandable
with the context in place; besides, it's not necessary to grok all this
calculus to become an effective practitioner.)
A note on notation: Jeremy's course exclusively uses code,
instead of math notation, to explain concepts since unfamiliar functions
in code are easy to search for and experiment with. In this paper, we
do the opposite: there is a lot of math notation because one of the
goals of this paper is to help you understand the notation that you'll
see in deep learning papers and books. At the
end of the paper, you'll find a brief table of the notation used, including a word or phrase you can use to search for more details.
Review: Scalar derivative rules
Hopefully you remember some of these main scalar derivative rules. If your memory is a bit fuzzy on this, have a look at
Khan academy vid on scalar derivative rules.
Rule | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-50BBD36E1FD2333108437A2CA378BE62-depth003.25.svg) | Scalar derivative notation with respect to x | Example |
Constant | c | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-CFCD208495D565EF66E7DFF9F98764DA-depth000.20.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-64F478FC4F8CA7284B3A00A378371DD9-depth004.58.svg) |
Multiplication by constant | cf | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6F4437952F8A3645EDFDBA34800939C7-depth004.58.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E6BBA7095B8356839BB31CBAA81BEB2D-depth004.58.svg) |
Power Rule | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B41952E9DFED8E1ED562FDDAFECA7C70-depth000.14.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-778B829130B1FD49AB222C57E43AA511-depth000.14.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-0E35EE43AF43B99B9DEC113EF495C9B1-depth004.58.svg) |
Sum Rule | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F90CB0F11C73CF039948C790CB958167-depth002.72.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-272CCFD170B42C2D5FC34B8F6B59B919-depth004.58.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-30CBA4C53B1DA339D30A685E5F750181-depth004.58.svg) |
Difference Rule | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6BF1771953527A02849B0FD62C669A81-depth002.72.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-94A32391D8AF942F5DD6DF56F3A6B9D7-depth004.58.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A5BC3AF74DAB50898B00534C3A67F8E7-depth004.58.svg) |
Product Rule | fg | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BEE9614EE8574DD7D3739CB7B896CF87-depth004.58.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-80B75BFF0AE78CDC1377C9DA385E5733-depth004.58.svg) |
Chain Rule | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8BC3A7E80988236E8F017205F413461C-depth003.25.svg) | , let ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F1EA68A4A1E70E96ADA7592EF8752C30-depth003.25.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-951A8B94D11F280E3A440D55B3D520D3-depth004.61.svg) |
There are other rules for trigonometry, exponentials, etc., which you can find at
Khan Academy differential calculus course.
When a function has a single parameter,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-50BBD36E1FD2333108437A2CA378BE62-depth003.25.svg)
, you'll often see
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6BDA8AF54C40BC23ED858E9E9F5C11D2-depth002.72.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-74CAF4D1EC90D3A36EA7C7BBFE65B516-depth003.25.svg)
used as shorthands for
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5BA9D16419154B1BDBECA39D99E8E809-depth004.58.svg)
. We recommend against this notation as it does not make clear the variable we're taking the derivative with respect to.
You can think of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-624FEBB9A49A3FC96353C861D175C806-depth004.58.svg)
as an operator that maps a function of one parameter to another function. That means that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EC0CEC5F9488EC510F8D688E7003222D-depth004.58.svg)
maps
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-50BBD36E1FD2333108437A2CA378BE62-depth003.25.svg)
to its derivative with respect to
x, which is the same thing as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-07A5EA519C4CEA1A3539E3A7FC289163-depth004.58.svg)
. Also, if
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FD91C508F91C2C84498680BD337C1D7A-depth003.25.svg)
, then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B1ED3CF9BA4D6F25A5A4F481C45EC658-depth004.58.svg)
.
Thinking of the derivative as an operator helps to simplify complicated
derivatives because the operator is distributive and lets us pull out
constants. For example, in the following equation, we can pull out the
constant 9 and distribute the derivative operator across the elements
within the parentheses.
That procedure reduced the derivative of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FDFD125C741CD062B2CA779DDE0524BE-depth003.25.svg)
to a bit of arithmetic and the derivatives of
x and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-32F5240D0DBF2CCBE75EF7F8EF2015E0-depth000.14.svg)
, which are much easier to solve than the original derivative.
Introduction to vector calculus and partial derivatives
Neural network layers are not single functions of a single parameter,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-50BBD36E1FD2333108437A2CA378BE62-depth003.25.svg)
. So, let's move on to functions of multiple parameters such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3BAF1600AE50930A155F58AE172B51BD-depth003.25.svg)
. For example, what is the derivative of
xy (i.e., the multiplication of
x and
y)? In other words, how does the product
xy change when we wiggle the variables? Well, it depends on whether we are changing
x or
y. We compute derivatives with respect to one variable (parameter) at a time, giving us two different
partial derivatives for this two-parameter function (one for
x and one for
y). Instead of using operator
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-624FEBB9A49A3FC96353C861D175C806-depth004.58.svg)
, the partial derivative operator is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4D6C379F66675645B3FFE28A15306857-depth004.67.svg)
(a stylized
d and not the Greek letter
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-77A3B715842B45E440A5BEE15357AD29-depth000.22.svg)
). So,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-08DCCFBE629A14FCCD9FB9A20F2E367C-depth004.67.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1FDCF7A9F137AE48FA25EE34A69F8201-depth006.34.svg)
are the partial derivatives of
xy; often, these are just called the
partials. For functions of a single parameter, operator
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4D6C379F66675645B3FFE28A15306857-depth004.67.svg)
is equivalent to
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-624FEBB9A49A3FC96353C861D175C806-depth004.58.svg)
(for sufficiently smooth functions). However, it's better to use
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-624FEBB9A49A3FC96353C861D175C806-depth004.58.svg)
to make it clear you're referring to a scalar derivative.
The partial derivative with respect to
x is just the usual scalar derivative, simply treating any other variable in the equation as a constant. Consider function
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D6DEAE7E403381C2C425D4B40CCA936E-depth003.25.svg)
. The partial derivative with respect to
x is written
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F063B8EC812DF3D204F9327F5D094073-depth004.67.svg)
. There are three constants from the perspective of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4D6C379F66675645B3FFE28A15306857-depth004.67.svg)
: 3, 2, and
y. Therefore,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D981BA4BD14AC44C43A4E4E0EC750B4A-depth004.67.svg)
. The partial derivative with respect to
y treats
x like a constant:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-55A3A400FAD3326FEF1BB9DDD2658383-depth006.34.svg)
. It's a good idea to derive these yourself before continuing otherwise the rest of the article won't make sense. Here's the
Khan Academy video on partials if you need help.
To make it clear we are doing vector calculus and not just
multivariate calculus, let's consider what we do with the partial
derivatives
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-84D799755A7F73945BD58B2E057121AB-depth004.67.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A2DE2EC029172B84A0A0E8A8D00F5A6F-depth006.34.svg)
(another way to say
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-75645E70B6C95F7466C353E9C2306FE0-depth004.67.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B20FF70C037320C2D0B710F4B592927E-depth006.34.svg)
) that we computed for
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D6DEAE7E403381C2C425D4B40CCA936E-depth003.25.svg)
.
Instead of having them just floating around and not organized in any
way, let's organize them into a horizontal vector. We call this vector
the
gradient of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3BAF1600AE50930A155F58AE172B51BD-depth003.25.svg)
and write it as:
So the gradient of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3BAF1600AE50930A155F58AE172B51BD-depth003.25.svg)
is simply a vector of its partials. Gradients are part of the vector calculus world, which deals with functions that map
n scalar parameters to a single scalar. Now, let's get crazy and consider derivatives of multiple functions simultaneously.
Matrix calculus
When we move from derivatives of one function to derivatives of many
functions, we move from the world of vector calculus to matrix calculus.
Let's compute partial derivatives for two functions, both of which take
two parameters. We can keep the same
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D6DEAE7E403381C2C425D4B40CCA936E-depth003.25.svg)
from the last section, but let's also bring in
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D182AD2135D1E887AFFCA045F432B2CA-depth003.25.svg)
. The gradient for
g has two entries, a partial derivative for each parameter:
and
giving us gradient
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5E1F4483B06DEA88CD339E02F7980A74-depth003.25.svg)
.
Gradient vectors organize all of the partial derivatives for a
specific scalar function. If we have two functions, we can also organize
their gradients into a matrix by stacking the gradients. When we do so,
we get the
Jacobian matrix (or just the
Jacobian) where the gradients are rows:
Welcome to matrix calculus!
Note that there are multiple ways to represent the Jacobian. We are using the so-called
numerator layout but many papers and software will use the
denominator layout. This is just transpose of the numerator layout Jacobian (flip it around its diagonal):
Generalization of the Jacobian
So far, we've looked at a specific example of a Jacobian matrix. To
define the Jacobian matrix more generally, let's combine multiple
parameters into a single vector argument:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8C56090E55CDB76D1CD0E738EBA7F164-depth003.25.svg)
. (You will sometimes see notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B07B5ABD67EE0B72F4136C82C68A0C48-depth000.14.svg)
for vectors in the literature as well.) Lowercase letters in bold font such as
x are vectors and those in italics font like
x are scalars.
xi is the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-97361F12A3555FC4FC4E2FFCE1799AC3-depth000.14.svg)
element of vector
x and is in italics because a single vector element is a scalar. We also have to define an orientation for vector
x. We'll assume that all vectors are vertical by default of size
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-88512AB12706879FEC83C0C3AA79931F-depth001.08.svg)
:
With multiple scalar-valued functions, we can combine them all into a vector just like we did with the parameters. Let
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A16203A31AD7C6CC63FD297D522170F1-depth003.25.svg)
be a vector of
m scalar-valued functions that each take a vector
x of length
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E3114C625CDDDC18ED29BA629242BD65-depth003.25.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DEA8E196A572D082201CD5ABF2FA82DE-depth003.25.svg)
is the cardinality (count) of elements in
x. Each
fi function within
f returns a scalar just as in the previous section:
For instance, we'd represent
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D6DEAE7E403381C2C425D4B40CCA936E-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D182AD2135D1E887AFFCA045F432B2CA-depth003.25.svg)
from the last section as
It's very often the case that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A193EBC083B4745370F6F1343383D9CC-depth000.14.svg)
because we will have a scalar function result for each element of the
x vector. For example, consider the identity function
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-23225C9E5521B6A9777579BE4B92245C-depth003.25.svg)
:
So we have
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A193EBC083B4745370F6F1343383D9CC-depth000.14.svg)
functions and parameters, in this case. Generally speaking, though, the Jacobian matrix is the collection of all
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FBFEB9C8459FEE5A2BD529C07B881153-depth001.08.svg)
possible partial derivatives (
m rows and
n columns), which is the stack of
m gradients with respect to
x:
Each
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-98F52E62B85A836D750F1CEDF32E1D68-depth004.67.svg)
is a horizontal
n-vector because the partial derivative is with respect to a vector,
x, whose length is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A921357AFE67E47CC9D1DB575BCE1B77-depth003.25.svg)
. The width of the Jacobian is
n if we're taking the partial derivative with respect to
x because there are
n parameters we can wiggle, each potentially changing the function's value. Therefore, the Jacobian is always
m rows for
m equations. It helps to think about the possible Jacobian shapes visually:
The Jacobian of the identity function
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3ABE4A0471143ABFC180C9FA485E5F0A-depth003.25.svg)
, with
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E314355F5E2135483279531C62D7E8EC-depth003.25.svg)
, has
n functions and each function has
n parameters held in a single vector
x. The Jacobian is, therefore, a square matrix since
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A193EBC083B4745370F6F1343383D9CC-depth000.14.svg)
:
Make sure that you can derive each step above before moving on. If
you get stuck, just consider each element of the matrix in isolation and
apply the usual scalar derivative rules. That is a generally useful
trick: Reduce vector expressions down to a set of scalar expressions and
then take all of the partials, combining the results appropriately into
vectors and matrices at the end.
Also be careful to track whether a matrix is vertical,
x, or horizontal,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FAAEE783424BC0E27E9AA2F56A7B50B8-depth000.00.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FAAEE783424BC0E27E9AA2F56A7B50B8-depth000.00.svg)
means
x transpose. Also make sure you pay attention to whether something is a scalar-valued function,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D3172B9A65679CB6EF09F17BE0918890-depth002.65.svg)
, or a vector of functions (or a vector-valued function),
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BCAEE673D10908E8197A79E9D4FB6249-depth002.33.svg)
.
Derivatives of vector element-wise binary operators
Element-wise binary operations on vectors, such as vector addition
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F26AC9FE4C8C9B953164428DCE00BE9C-depth001.06.svg)
,
are important because we can express many common vector operations,
such as the multiplication of a vector by a scalar, as element-wise
binary operations. By “element-wise binary operations” we simply mean
applying an operator to the first item of each vector to get the first
item of the output, then to the second items of the inputs for the
second item of the output, and so forth. This is how all the basic math
operators are applied by default in numpy or tensorflow, for example.
Examples that often crop up in deep learning are
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E50C0CB908AF7566CCC6D4585634EDC2-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C65762AB635D8E08F759F1203D223C29-depth000.51.svg)
(returns a vector of ones and zeros).
We can generalize the element-wise binary operations with notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E2E59AE84EE7A5B1C905E50FA7753A31-depth003.25.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5D19E7D8CDFE53DEB40F29D8936E6C89-depth003.25.svg)
. (Reminder:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-CF513DECF6E4ACE0E25CB1C932AAA049-depth003.25.svg)
is the number of items in
x.) The
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1A74909B6CBAA4532A76D83B72C12DE0-depth002.52.svg)
symbol represents any element-wise operator (such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-26B17225B626FB9238849FD60EABDF60-depth001.06.svg)
) and not the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1B3C1A40F9CB094D47E8C6F9B0DF773F-depth000.00.svg)
function composition operator. Here's what equation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E2E59AE84EE7A5B1C905E50FA7753A31-depth003.25.svg)
looks like when we zoom in to examine the scalar equations:
where we write
n (not
m) equations vertically to emphasize the fact that the result of element-wise operators give
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A193EBC083B4745370F6F1343383D9CC-depth000.14.svg)
sized vector results.
Using the ideas from the last section, we can see that the general case for the Jacobian with respect to
w is the square matrix:
and the Jacobian with respect to
x is:
That's quite a furball, but fortunately the Jacobian is very often a
diagonal matrix, a matrix that is zero everywhere but the diagonal.
Because this greatly simplifies the Jacobian, let's examine in detail
when the Jacobian reduces to a diagonal matrix for element-wise
operations.
In a diagonal Jacobian, all elements off the diagonal are zero,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8B14469E98630C19F16578F90C45F62E-depth007.21.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B064F8555EC660F2F8BDC927D9636A06-depth002.72.svg)
. (Notice that we are taking the partial derivative with respect to
wj not
wi.) Under what conditions are those off-diagonal elements zero? Precisely when
fi and
gi are contants with respect to
wj,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-44C0281EC00A2C0E93E4E3863EE9083D-depth007.21.svg)
. Regardless of the operator, if those partial derivatives go to zero, the operation goes to zero,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A180A4C31CAC188626034423680B71E0-depth002.52.svg)
no matter what, and the partial derivative of a constant is zero.
Those partials go to zero when
fi and
gi are not functions of
wj. We know that element-wise operations imply that
fi is purely a function of
wi and
gi is purely a function of
xi. For example,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-2361C23CB78AC04537D7D642DF065EF5-depth001.06.svg)
sums
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4AF62C4A6B74D712D3FFD3FA4A0062BD-depth002.05.svg)
. Consequently,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-45553270D20A27EBD4AAE84292606CDD-depth003.25.svg)
reduces to
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5F2A3B3A730ABED47918785C5EBF5039-depth003.25.svg)
and the goal becomes
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-767EBC7C8A1785557E38FD32A10FB123-depth007.21.svg)
.
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DB590A5BF8B0ACC05B1FEEDC07929CD7-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1BF8B4ED9F2C1D993DC0A1E547BDF5CB-depth003.25.svg)
look like constants to the partial differentiation operator with respect to
wj when
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B064F8555EC660F2F8BDC927D9636A06-depth002.72.svg)
so the partials are zero off the diagonal. (Notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DB590A5BF8B0ACC05B1FEEDC07929CD7-depth003.25.svg)
is technically an abuse of our notation because
fi and
gi are functions of vectors not individual elements. We should really write something like
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-616C179024B43E6C340B1EE24D414DE8-depth003.25.svg)
,
but that would muddy the equations further, and programmers are
comfortable overloading functions, so we'll proceed with the notation
anyway.)
We'll take advantage of this simplification later and refer to the constraint that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD5A63074F44F11CB2ED06325816582A-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5E597DAA4E5D9263DCBFB6AB02BDB67F-depth003.25.svg)
access at most
wi and
xi, respectively, as the
element-wise diagonal condition.
Under this condition, the elements along the diagonal of the Jacobian are
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C196BF268027E86D3D2420C2A205AF28-depth005.92.svg)
:
(The large “0”s are a shorthand indicating all of the off-diagonal are 0.)
More succinctly, we can write:
and
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4935E4C6B875FD6C7C181871B566AB1A-depth003.25.svg)
constructs a matrix whose diagonal elements are taken from vector
x:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-358C0E8F99CB66017CCD32DBC6C23C50-depth003.25.svg)
.
I
represents the square identity matrix of appropriate dimensions that is
zero everywhere but the diagonal, which contains all ones. The
T exponent of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FAAEE783424BC0E27E9AA2F56A7B50B8-depth000.00.svg)
represents the transpose of the indicated vector. In this case, it flips a vertical vector to a horizontal vector.
Because we do lots of simple vector arithmetic, the general function
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5B39FE68B4FDD0AF04290BA579A993CB-depth003.25.svg)
in the binary element-wise operation is often just the vector
w. Any time the general function is a vector, we know that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD5A63074F44F11CB2ED06325816582A-depth003.25.svg)
reduces to
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F5BDA09DDE6E00806B01094F5BED3026-depth003.25.svg)
. For example, vector addition
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DF3F5FDED9142A243031D03CF82121AE-depth001.06.svg)
fits our element-wise diagonal condition because
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1ED79D2E2BBE9EE796433D13773157A7-depth003.25.svg)
has scalar equations
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F7C644AAA7A70D588A0E003C7C9E439E-depth003.25.svg)
that reduce to just
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-85550E18F87E4AF75645A38273B97A80-depth003.25.svg)
with partial derivatives:
That gives us
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-2935E504F134B53B2C03072175BCCD1F-depth004.67.svg)
, the identity matrix, because every element along the diagonal is 1.
Given the simplicity of this special case,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD5A63074F44F11CB2ED06325816582A-depth003.25.svg)
reducing to
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DB590A5BF8B0ACC05B1FEEDC07929CD7-depth003.25.svg)
, you should be able to derive the Jacobians for the common element-wise binary operations on vectors:
The
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-790C76CEB13E928D08EDC53D7AC4BB5C-depth001.08.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-98FF0549EBE322C195C2B36FD5EEAD33-depth001.08.svg)
operators are element-wise multiplication and division;
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-790C76CEB13E928D08EDC53D7AC4BB5C-depth001.08.svg)
is sometimes called the
Hadamard product.
There isn't a standard notation for element-wise multiplication and
division so we're using an approach consistent with our general binary
operation notation.
Derivatives involving scalar expansion
When we multiply or add scalars to vectors, we're implicitly
expanding the scalar to a vector and then performing an element-wise
binary operation. For example, adding scalar
z to vector
x,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6307CEA088D2D4E98E5B163B9CE8F510-depth002.33.svg)
, is really
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8D9C0B9B15490F45C353D9DE64565A4F-depth003.25.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3ABE4A0471143ABFC180C9FA485E5F0A-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD2335FC4BBF16BE9590D2501CE8C030-depth003.25.svg)
. (The notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C2C0146718E407005D0C74774C5C5FFC-depth000.00.svg)
represents a vector of ones of appropriate length.)
z is any scalar that doesn't depend on
x, which is useful because then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EBCC35EEAE3B420D59689973D8B6BD2E-depth005.92.svg)
for any
xi and that will simplify our partial derivative computations. (It's okay to think of variable
z as a constant for our discussion here.) Similarly, multiplying by a scalar,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C9A7B878F49F3D964AFEC9C1F78061CF-depth002.33.svg)
, is really
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-93A461AB49FD151E602D9344358732CD-depth003.25.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-790C76CEB13E928D08EDC53D7AC4BB5C-depth001.08.svg)
is the element-wise multiplication (Hadamard product) of the two vectors.
The partial derivatives of vector-scalar addition and multiplication with respect to vector
x use our element-wise rule:
This follows because functions
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3ABE4A0471143ABFC180C9FA485E5F0A-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD2335FC4BBF16BE9590D2501CE8C030-depth003.25.svg)
clearly satisfy our element-wise diagonal condition for the Jacobian (that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DAE17AB5EE9C0A7FFA3E9B1774E80201-depth003.25.svg)
refer at most to
xi and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9F8AE8514327A98189F8F05E2ECD6496-depth003.25.svg)
refers to the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-97361F12A3555FC4FC4E2FFCE1799AC3-depth000.14.svg)
value of the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-49508A178E36D0863E041575526BEA1A-depth001.05.svg)
vector).
Using the usual rules for scalar partial derivatives, we arrive at
the following diagonal elements of the Jacobian for vector-scalar
addition:
So,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5107D6819CDC50A8988D3EA0FB9B94CE-depth004.67.svg)
.
Computing the partial derivative with respect to the scalar parameter
z, however, results in a vertical vector, not a diagonal matrix. The elements of the vector are:
Therefore,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F05F04525E4A1B8959DE54DC7C692060-depth005.22.svg)
.
The diagonal elements of the Jacobian for vector-scalar multiplication involve the product rule for scalar derivatives:
So,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3F231096152DFB321FAC62F57A808C35-depth004.67.svg)
.
The partial derivative with respect to scalar parameter
z is a vertical vector whose elements are:
This gives us
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-82B897A0A23D9C6BB66EDF17D1D3CB02-depth005.22.svg)
.
Vector sum reduction
Summing up the elements of a vector is an important operation in deep
learning, such as the network loss function, but we can also use it as a
way to simplify computing the derivative of vector dot product and
other operations that reduce vectors to scalars.
Let
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DC5FB5DC7AEB54D8C206744EED4AD748-depth003.31.svg)
. Notice we were careful here to leave the parameter as a vector
x because each function
fi could use all values in the vector, not just
xi. The sum is over the
results of the function and not the parameter. The gradient (
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4C8971A9939B0BB2D8AF44195C5BD833-depth001.08.svg)
Jacobian) of vector summation is:
(The summation inside the gradient elements can be tricky so make sure to keep your notation consistent.)
Let's look at the gradient of the simple
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5FAFFB6A723E437AC6433DCA0B269846-depth003.25.svg)
. The function inside the summation is just
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E314355F5E2135483279531C62D7E8EC-depth003.25.svg)
and the gradient is then:
Because
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9B32CEB09BA5F0B84935A24BF81D3C9C-depth007.21.svg)
for
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B064F8555EC660F2F8BDC927D9636A06-depth002.72.svg)
, we can simplify to:
Notice that the result is a horizontal vector full of 1s, not a vertical vector, and so the gradient is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DFB70C45B6CCB149DFDA0E3690715F92-depth000.00.svg)
.
It's very important to keep the shape of all of your vectors and
matrices in order otherwise it's impossible to compute the derivatives
of complex functions.
As another example, let's sum the result of multiplying a vector by a constant scalar. If
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-729968724651D87C8269B6FFEAD6EA90-depth003.25.svg)
then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A5D03DE59DCDBA948F463FAABD04791D-depth003.25.svg)
. The gradient is:
The derivative with respect to scalar variable
z is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5ED2D4C114D036610B8E20271C5026EF-depth001.08.svg)
:
The Chain Rules
We can't compute partial derivatives of very complicated functions
using just the basic matrix calculus rules we've seen so far. For
example, we can't take the derivative of nested expressions like
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DCA4F9F0CE7F7CA365E8B26987ED972A-depth003.25.svg)
directly without reducing it to its scalar equivalent. We need to be
able to combine our basic vector rules using what we can call the
vector chain rule.
Unfortunately, there are a number of rules for differentiation that
fall under the name “chain rule” so we have to be careful which chain
rule we're talking about. Part of our goal here is to clearly define and
name three different chain rules and indicate in which situation they
are appropriate. To get warmed up, we'll start with what we'll call the
single-variable chain rule,
where we want the derivative of a scalar function with respect to a
scalar. Then we'll move on to an important concept called the
total derivative and use it to define what we'll pedantically call the
single-variable total-derivative chain rule. Then, we'll be ready for the vector chain rule in its full glory as needed for neural networks.
The chain rule is conceptually a divide and conquer strategy (like
Quicksort) that breaks complicated expressions into subexpressions whose
derivatives are easier to compute. Its power derives from the fact
that we can process each simple subexpression in isolation yet still
combine the intermediate results to get the correct overall result.
The chain rule comes into play when we need the derivative of an
expression composed of nested subexpressions. For example, we need the
chain rule when confronted with expressions like
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-06D67DC7FE74C1895AEF564F8295E918-depth004.58.svg)
. The outermost expression takes the
sin of an intermediate result, a nested subexpression that squares
x. Specifically, we need the single-variable chain rule, so let's start by digging into that in more detail.
Single-variable chain rule
Let's start with the solution to the derivative of our nested expression:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-AB5ECA885C5685990CD778580665B3A4-depth004.58.svg)
. It doesn't take a mathematical genius to recognize components of the solution that smack of scalar differentiation rules,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1A32AA532898DEBB80C0C7A818C5C70B-depth004.58.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DC7C2E5FB11394451EA6A2010904F0B2-depth004.58.svg)
.
It looks like the solution is to multiply the derivative of the outer
expression by the derivative of the inner expression or “chain the
pieces together,” which is exactly right. In this section, we'll explore
the general principle at work and provide a process that works for
highly-nested expressions of a single variable.
Chain rules are typically defined in terms of nested functions, such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-281717E562B6C04AA861AC9F2801D016-depth003.25.svg)
for single-variable chain rules. (You will also see the chain rule defined using function composition
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-88806005B64072FE5A06E4E609A9E251-depth003.25.svg)
, which is the same thing.) Some sources write the derivative using shorthand notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3D59985B51738C2BF54BA3D955AB8588-depth003.25.svg)
, but that hides the fact that we are introducing an intermediate variable:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8C36385E70550C1C0EA86E14320174DF-depth003.25.svg)
, which we'll see shortly. It's better to define the
single-variable chain rule of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8BC3A7E80988236E8F017205F413461C-depth003.25.svg)
explicitly so we never take the derivative with respect to the wrong
variable. Here is the formulation of the single-variable chain rule we
recommend:
To deploy the single-variable chain rule, follow these steps:
- Introduce intermediate variables for nested subexpressions and subexpressions for both binary and unary operators; e.g.,
is binary,
and other trigonometric functions are usually unary because there is a
single operand. This step normalizes all equations to single operators
or function applications.
- Compute derivatives of the intermediate variables with respect to their parameters.
- Combine all derivatives of intermediate variables by multiplying them together to get the overall result.
- Substitute intermediate variables back in if any are referenced in the derivative equation.
The third step puts the “chain” in “chain rule” because it chains
together intermediate results. Multiplying the intermediate derivatives
together is the common theme among all variations of the chain rule.
Let's try this process on
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-872F24FC57CA661E3704C0A10869C6B5-depth003.25.svg)
:
- Introduce intermediate variables. Let
represent subexpression
(shorthand for
). This gives us:
The order of these subexpressions does not affect the answer, but we
recommend working in the reverse order of operations dictated by the
nesting (innermost to outermost). That way, expressions and derivatives
are always functions of previously-computed elements.
- Compute derivatives.
- Combine.
- Substitute.
Notice how easy it is to compute the derivatives of the intermediate
variables in isolation! The chain rule says it's legal to do that and
tells us how to combine the intermediate results to get
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E4D3AF86AC3E4315148DA23A886A72EA-depth003.25.svg)
.
You can think of the combining step of the chain rule in terms of units canceling. If we let
y be gallons of gas,
x be the gallons in a gas tank, and
u as miles we can interpret
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-70047C98E4163674E78BB42D0CF4AEA8-depth004.58.svg)
as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F98E0EBA888E860C7E51565C7225EBFA-depth006.34.svg)
. The
gallon denominator and numerator cancel.
Another way to to think about the single-variable chain rule is to
visualize the overall expression as a dataflow diagram or chain of
operations (or
abstract syntax tree for compiler people):
Changes to function parameter
x bubble up through a squaring operation then through a
sin operation to change result
y. You can think of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B13DB657CBF4B8318DBF2799E687D1A1-depth004.58.svg)
as “getting changes from
x to
u” and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6E939E305B65D08F88FF95E7E028796B-depth004.58.svg)
as “getting changes from
u to
y.” Getting from
x to
y
requires an intermediate hop. The chain rule is, by convention, usually
written from the output variable down to the parameter(s),
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-70047C98E4163674E78BB42D0CF4AEA8-depth004.58.svg)
. But, the
x-to-
y perspective would be more clear if we reversed the flow and used the equivalent
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-256F5475DE15D99D8D55FE6F3A15CEA4-depth004.58.svg)
.
Conditions under which the single-variable chain rule applies. Notice that there is a single dataflow path from
x to the root
y. Changes in
x can influence output
y
in only one way. That is the condition under which we can apply the
single-variable chain rule. An easier condition to remember, though one
that's a bit looser, is that none of the intermediate subexpression
functions,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3BCB9E96DA63C9CDC1E56647C2071688-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C6E9D6FA0C33AA632E25E953C6E5C35D-depth003.25.svg)
, have more than one parameter. Consider
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-026E2EF5E8E6906B6CE75FF6CDB0F14E-depth003.25.svg)
, which would become
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A1829C9E1CDCFF4FE0BA9C0E7A70E635-depth003.25.svg)
after introducing intermediate variable
u. As we'll see in the next section,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-2C284C56D6E912E6D71990A11005902E-depth003.25.svg)
has multiple paths from
x to
y. To handle that situation, we'll deploy the single-variable total-derivative chain rule.
As an aside for those interested in automatic differentiation, papers and library documentation use terminology
forward differentiation and
backward differentiation
(for use in the back-propagation algorithm). From a dataflow
perspective, we are computing a forward differentiation because it
follows the normal data flow direction. Backward differentiation,
naturally, goes the other direction and we're asking how a change in the
output would affect function parameter
x.
Because backward differentiation can determine changes in all function
parameters at once, it turns out to be much more efficient for computing
the derivative of functions with lots of parameters. Forward
differentiation, on the other hand, must consider how a change in each
parameter, in turn, affects the function output
y. The following table emphasizes the order in which partial derivatives are computed for the two techniques.
Forward differentiation from x to y | Backward differentiation from y to x |
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-256F5475DE15D99D8D55FE6F3A15CEA4-depth004.58.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-70047C98E4163674E78BB42D0CF4AEA8-depth004.58.svg) |
Automatic differentiation is beyond the scope of this article, but we're setting the stage for a future article.
Many readers can solve
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-59B8E919C92D6DBE1DC50C5BE2CD8C1C-depth004.58.svg)
in their heads, but our goal is a process that will work even for very complicated expressions. This process is also how
automatic differentiation
works in libraries like PyTorch. So, by solving derivatives manually
in this way, you're also learning how to define functions for custom
neural networks in PyTorch.
With deeply nested expressions, it helps to think about deploying the
chain rule the way a compiler unravels nested function calls like
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-7559A5FC5EC5CA0B3E50011742D0A87B-depth003.25.svg)
into a sequence (chain) of calls. The result of calling function
fi is saved to a temporary variable called a register, which is then passed as a parameter to
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-36930D135F4763C21D1191803AC41B85-depth002.72.svg)
. Let's see how that looks in practice by using our process on a highly-nested equation like
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-642FD3E590A2B2D21AFE5254BE8E832F-depth003.25.svg)
:
- Introduce intermediate variables.
- Compute derivatives.
- Combine four intermediate values.
- Substitute.
Here is a visualization of the data flow through the chain of operations from
x to
y:
At this point, we can handle derivatives of nested expressions of a single variable,
x, using the chain rule but only if
x can affect
y through a single data flow path. To handle more complicated expressions, we need to extend our technique, which we'll do next.
Single-variable total-derivative chain rule
Our single-variable chain rule has limited applicability because all
intermediate variables must be functions of single variables. But, it
demonstrates the core mechanism of the chain rule, that of multiplying
out all derivatives of intermediate subexpressions. To handle more
general expressions such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4E2EFF28C6823738FA61BFC9A3DD6D0F-depth003.25.svg)
, however, we need to augment that basic chain rule.
Of course, we immediately see
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-18EF15EC4DDA421BE5BC86F0295D36CD-depth004.58.svg)
,
but that is using the scalar addition derivative rule, not the chain
rule. If we tried to apply the single-variable chain rule, we'd get the
wrong answer. In fact, the previous chain rule is meaningless in this
case because derivative operator
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-624FEBB9A49A3FC96353C861D175C806-depth004.58.svg)
does not apply to multivariate functions, such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-89AE78BE880A004AA5404AC874A01BFF-depth001.95.svg)
among our intermediate variables:
Let's try it anyway to see what happens. If we pretend that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-400C939294CFCAC13949F5A92DD9537A-depth005.85.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8B1AAFE58A962E6F06775EBD2808D5FE-depth004.58.svg)
, then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-69A0BF0F0F217A5C8CDB490B4C60ABEE-depth005.85.svg)
instead of the right answer
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-632334BF7A82AE1CEB6BF98756648B4E-depth001.06.svg)
.
Because
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3DD76E5D04C44A4E7170558B8BFE3219-depth003.25.svg)
has multiple parameters, partial derivatives come into play. Let's
blindly apply the partial derivative operator to all of our equations
and see what we get:
Ooops! The partial
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9A001AB7445AD6C36176920C0E5D253F-depth004.67.svg)
is wrong because it violates a key assumption for partial derivatives. When taking the partial derivative with respect to
x, the other variables must not vary as
x varies. Otherwise, we could not act as if the other variables were constants. Clearly, though,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-0F0BF7F8711E6437357749F43EF529D8-depth003.25.svg)
is a function of
x and therefore varies with
x.
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-78D6E87F2E36D9F4C482E9236F997C4C-depth004.67.svg)
because
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C9E3A40CD8239D58296F30B000154F1A-depth004.67.svg)
. A quick look at the data flow diagram for
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-AC2672C7A79B0E068FB3AC3D7FBF94C9-depth003.25.svg)
shows multiple paths from
x to
y, thus, making it clear we need to consider direct and indirect (through
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D8F3471EF0522DC14012F0DC5D01D570-depth003.25.svg)
) dependencies on
x:
A change in
x affects
y both as an operand of the addition and as the operand of the square operator. Here's an equation that describes how tweaks to
x affect the output:
Then,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-461DF6ABE0C0386D728B786B3116A5B1-depth002.65.svg)
, which we can read as “the change in
y is the difference between the original
y and
y at a tweaked
x.”
If we let
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A255512F9D61A6777BD5A304235BD26D-depth000.14.svg)
, then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-433AD860B59D47738D7AECAB6367A8AD-depth002.65.svg)
. If we bump
x by 1,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-60BD3A416346416B27D420F8EFEE9C9E-depth000.14.svg)
, then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4DCF3FC9093036B469E832AEDCFDA608-depth003.25.svg)
. The change in
y is not
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C4CA4238A0B923820DCC509A6F75849B-depth000.00.svg)
, as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BE27DFA2B74F7608759BD413AF458EB2-depth003.25.svg)
would lead us to believe, but
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F0B6923B35563AE91BDFC8B06222E495-depth001.08.svg)
!
Enter the “law” of
total derivatives, which basically says that to compute
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3BAFFD623D24688B6229E8808F4DD24A-depth004.58.svg)
, we need to sum up all possible contributions from changes in
x to the change in
y. The total derivative with respect to
x assumes all variables, such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D33DEF0EB4933F91B88EB4E784ADAF05-depth001.95.svg)
in this case, are functions of
x and potentially vary as
x varies. The total derivative of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-20596CA2E264644A086ABF3ABCA89367-depth003.25.svg)
that depends on
x directly and indirectly via intermediate variable
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D8F3471EF0522DC14012F0DC5D01D570-depth003.25.svg)
is given by:
Using this formula, we get the proper answer:
That is an application of what we can call the
single-variable total-derivative chain rule:
The total derivative assumes all variables are potentially codependent whereas the partial derivative assumes all variables but
x are constants.
There is something subtle going on here with the notation. All of the derivatives are shown as partial derivatives because
f and
ui are functions of multiple variables. This notation mirrors that of
MathWorld's notation but differs from
Wikipedia, which uses
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8A4956B34F845398E8CB25E9789E1477-depth003.25.svg)
instead (possibly to emphasize the total derivative nature of the
equation). We'll stick with the partial derivative notation so that it's
consistent with our discussion of the vector chain rule in the next
section.
In practice, just keep in mind that when you take the total derivative with respect to
x, other variables might also be functions of
x
so add in their contributions as well. The left side of the equation
looks like a typical partial derivative but the right-hand side is
actually the total derivative. It's common, however, that many
temporary variables are functions of a single parameter, which means
that the single-variable total-derivative chain rule degenerates to the
single-variable chain rule.
Let's look at a nested subexpression, such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FD3AC7E953C3B294864C747188F6F370-depth003.25.svg)
. We introduce three intermediate variables:
and partials:
where both
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A5D73F881FB427D3CD136DA4815CACA4-depth004.67.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-110F22244D49C5AD607D2BEFE91944A5-depth004.67.svg)
have
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4E8A57E5001AA58AEAA927CF98746B9C-depth004.67.svg)
terms that take into account the total derivative.
Also notice that the total derivative formula always
sums versus, say, multiplies terms
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-935DA2BFCBE67A179A8DAAB35E19A1DA-depth005.92.svg)
. It's tempting to think that summing up terms in the derivative makes sense because, for example,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FCCD52402617C355E7BB85B3336D0142-depth002.65.svg)
adds two terms. Nope. The total derivative is adding terms because it represents a weighted sum of all
x contributions to the change in
y. For example, given
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5C5A09876AA823C381141FDAC1D28BA6-depth002.65.svg)
instead of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C26DA96086B50304FCE872846D4BE19F-depth002.65.svg)
, the total-derivative chain rule formula still adds partial derivative terms. (
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-94E561C972A5FE1DDE82761012FB6DB1-depth001.08.svg)
simplifies to
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8C0FB3B076D9AEA142467B34F0F794EB-depth000.14.svg)
but for this demonstration, let's not combine the terms.) Here are the intermediate variables and partial derivatives:
The form of the total derivative remains the same, however:
It's the partials (weights) that change, not the formula, when the intermediate variable operators change.
Those readers with a strong calculus background might wonder why we
aggressively introduce intermediate variables even for the non-nested
subexpressions such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-32F5240D0DBF2CCBE75EF7F8EF2015E0-depth000.14.svg)
in
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-06671AD4FF442E623460886DA749C797-depth001.06.svg)
.
We use this process for three reasons: (i) computing the derivatives
for the simplified subexpressions is usually trivial, (ii) we can
simplify the chain rule, and (iii) the process mirrors how automatic
differentiation works in neural network libraries.
Using the intermediate variables even more aggressively, let's see
how we can simplify our single-variable total-derivative chain rule to
its final form. The goal is to get rid of the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F6E0346D1D3410B0FBE32B41B85999AA-depth004.67.svg)
sticking out on the front like a sore thumb:
We can achieve that by simply introducing a new temporary variable as an alias for
x:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-80A59ADB26E30571A3056E2EB3E8DDCB-depth002.69.svg)
. Then, the formula reduces to our final form:
This chain rule that takes into consideration the total derivative
degenerates to the single-variable chain rule when all intermediate
variables are functions of a single variable. Consequently, you can
remember this more general formula to cover both cases. As a bit of
dramatic foreshadowing, notice that the summation sure looks like a
vector dot product,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3F95AED50A8E6A5C514952D767B771CE-depth004.67.svg)
, or a vector multiply
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-51D7B0B0455FB229AB4920E9CA4AB032-depth004.67.svg)
.
Before we move on, a word of caution about terminology on the web.
Unfortunately, the chain rule given in this section, based upon the
total derivative, is universally called “multivariable chain rule” in
calculus discussions, which is highly misleading! Only the intermediate
variables are multivariate functions. The overall function, say,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A5A7963264669BCFA0CCFA897853A1E0-depth003.25.svg)
, is a scalar function that accepts a single parameter
x.
The derivative and parameter are scalars, not vectors, as one would
expect with a so-called multivariate chain rule. (Within the context of
a non-matrix calculus class, “multivariate chain rule” is likely
unambiguous.) To reduce confusion, we use “single-variable
total-derivative chain rule” to spell out the distinguishing feature
between the simple single-variable chain rule,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9D3919C42833D1FF1456DEA11D8CC927-depth004.58.svg)
, and this one.
Vector chain rule
Now that we've got a good handle on the total-derivative chain rule,
we're ready to tackle the chain rule for vectors of functions and vector
variables. Surprisingly, this more general chain rule is just as simple
looking as the single-variable chain rule for scalars. Rather than just
presenting the vector chain rule, let's rediscover it ourselves so we
get a firm grip on it. We can start by computing the derivative of a
sample vector function with respect to a scalar,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-82C28F66A012D28717CA0CFC8ED7F09B-depth003.25.svg)
, to see if we can abstract a general formula.
Let's introduce two intermediate variables,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E11BA37D5D784AF689E175BEC8A2F284-depth002.65.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D7F116997176D81A3BBD4E6DFC6FE6B0-depth002.65.svg)
, one for each
fi so that
y looks more like
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C5AF10997DA85D7540CDF87F1F10016C-depth003.25.svg)
:
The derivative of vector
y with respect to scalar
x is a vertical vector with elements computed using the single-variable total-derivative chain rule:
Ok, so now we have the answer using just the scalar rules, albeit
with the derivatives grouped into a vector. Let's try to abstract from
that result what it looks like in vector form. The goal is to convert
the following vector of scalar operations to a vector operation.
If we split the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-97D1ACA90CB316ED8AE1EDFFAED02C77-depth007.21.svg)
terms, isolating the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D513134B704605A9687A47F8841D7D29-depth004.67.svg)
terms into a vector, we get a matrix by vector multiplication:
That means that the Jacobian is the multiplication of two other Jacobians, which is kinda cool. Let's check our results:
Whew! We get the same answer as the scalar approach. This vector
chain rule for vectors of functions and a single parameter appears to be
correct and, indeed, mirrors the single-variable chain rule. Compare
the vector rule:
with the single-variable chain rule:
To make this formula work for multiple parameters or vector
x, we just have to change
x to vector
x in the equation. The effect is that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A24633DDE0B5346B4AE6B49395AC8B6D-depth004.67.svg)
and the resulting Jacobian,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A9A13E2AE753365278B4F2CD198BBF92-depth004.67.svg)
, are now matrices instead of vertical vectors. Our complete
vector chain rule is:
The beauty of the vector formula over the single-variable chain rule
is that it automatically takes into consideration the total derivative
while maintaining the same notational simplicity. The Jacobian contains
all possible combinations of
fi with respect to
gj and
gi with respect to
xj. For completeness, here are the two Jacobian components in their full glory:
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-99ADCB62C02EE5D10CC7B5F211BFA15B-depth003.25.svg)
,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6E37471488C004C0ABACDF4148B8F3D6-depth003.25.svg)
, and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-26312045D51B3E69C8357FF7FAF3BB3F-depth003.25.svg)
. The resulting Jacobian is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FBFEB9C8459FEE5A2BD529C07B881153-depth001.08.svg)
(an
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-2DC13003CC069B1027F12896B1A00631-depth001.08.svg)
matrix multiplied by a
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6D999BBC8A6ECBB820C04121088529A1-depth001.08.svg)
matrix).
Even within this
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A7896E2D919F33CF607DC9E972C70458-depth006.23.svg)
formula, we can simplify further because, for many applications, the Jacobians are square (
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A193EBC083B4745370F6F1343383D9CC-depth000.14.svg)
)
and the off-diagonal entries are zero. It is the nature of neural
networks that the associated mathematics deals with functions of vectors
not vectors of functions. For example, the neuron affine function has
term
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DE94554BD6F158BD8A829624C65169F4-depth003.25.svg)
and the activation function is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EA2DFCC1C759A6D2EE675EBD05B2C593-depth003.25.svg)
; we'll consider derivatives of these functions in the next section.
As we saw in a previous section, element-wise operations on vectors
w and
x yield diagonal matrices with elements
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D840B92230E9F1C5F2545BCA90B34038-depth005.92.svg)
because
wi is a function purely of
xi but not
xj for
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B064F8555EC660F2F8BDC927D9636A06-depth002.72.svg)
. The same thing happens here when
fi is purely a function of
gi and
gi is purely a function of
xi:
In this situation, the vector chain rule simplifies to:
Therefore, the Jacobian reduces to a diagonal matrix whose elements are the single-variable chain rule values.
After slogging through all of that mathematics, here's the payoff.
All you need is the vector chain rule because the single-variable
formulas are special cases of the vector chain rule. The following table
summarizes the appropriate components to multiply in order to get the
Jacobian.
The gradient of neuron activation
We now have all of the pieces needed to compute the derivative of a
typical neuron activation for a single neural network computation unit
with respect to the model parameters,
w and
b:
(This represents a neuron with fully connected weights and rectified
linear unit activation. There are, however, other affine functions such
as convolution and other activation functions, such as exponential
linear units, that follow similar logic.)
Let's worry about
max later and focus on computing
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-37826EBEE16CD487A60FA876F5038265-depth004.67.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-45DA2FE78D565E2361F35FF898D806A8-depth004.67.svg)
.
(Recall that neural networks learn through optimization of their
weights and biases.) We haven't discussed the derivative of the dot
product yet,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-31E193E45068EDA5F2E229B246720968-depth003.25.svg)
, but we can use the chain rule to avoid having to memorize yet another rule. (Note notation
y not
y as the result is a scalar not a vector.)
The dot product
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9CA8C3CD16894AF7620468A20C53D6FA-depth000.00.svg)
is just the summation of the element-wise multiplication of the elements:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4B722CF0EFB8F8ABCBB968086BB587E8-depth003.31.svg)
. (You might also find it useful to remember the linear algebra notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DE393E92B808CB595831DB7AF0D46F39-depth000.00.svg)
.) We know how to compute the partial derivatives of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1B8AC0FD13AAD81B7EFBE58CDD162D02-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-68EE6C0B6C39602AC2A620854B5785B2-depth001.08.svg)
but haven't looked at partial derivatives for
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-307751654E2CF1B689CE1D06776B3934-depth003.25.svg)
. We need the chain rule for that and so we can introduce an intermediate vector variable
u just as we did using the single-variable chain rule:
Once we've rephrased
y, we recognize two subexpressions for which we already know the partial derivatives:
The vector chain rule says to multiply the partials:
To check our results, we can grind the dot product down into a pure scalar function:
Then:
Hooray! Our scalar results match the vector chain rule results.
Now, let
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EABE2A2F7035C793F48B3885A2EA0009-depth002.65.svg)
, the full expression within the
max activation function call. We have two different partials to compute, but we don't need the chain rule:
Let's tackle the partials of the neuron activation,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-602D78206C69DAFE86EDC0775CF04CDB-depth003.25.svg)
. The use of the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-2E8B663420C79458FF788F7C8F198AA2-depth003.25.svg)
function call on scalar
z just says to treat all negative
z values as 0. The derivative of the max function is a piecewise function. When
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B960DEAE3DC80FA85FEA3304A9474DB5-depth001.72.svg)
, the derivative is 0 because
z is a constant. When
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E929E61EB1C05B5B7AC234027C595BE2-depth001.05.svg)
, the derivative of the max function is just the derivative of
z, which is
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-C4CA4238A0B923820DCC509A6F75849B-depth000.00.svg)
:
An aside on broadcasting functions across scalars. When one or both of the
max arguments are vectors, such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EA2DFCC1C759A6D2EE675EBD05B2C593-depth003.25.svg)
, we broadcast the single-variable function
max across the elements. This is an example of an element-wise unary operator. Just to be clear:
For the derivative of the broadcast version then, we get a vector of zeros and ones where:
To get the derivative of the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FD043EAC3999F8DEAF1FF1E131B3346C-depth003.25.svg)
function, we need the chain rule because of the nested subexpression,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6C156670CEC09096976A6722592523F3-depth001.06.svg)
. Following our process, let's introduce intermediate scalar variable
z to represent the affine function giving:
The vector chain rule tells us:
which we can rewrite as follows:
and then substitute
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-F394C21EEE331911707D5EDBD9BCAE20-depth001.06.svg)
back in:
That equation matches our intuition. When the activation function clips affine function output
z to 0, the derivative is zero with respect to any weight
wi. When
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E929E61EB1C05B5B7AC234027C595BE2-depth001.05.svg)
, it's as if the
max function disappears and we get just the derivative of
z with respect to the weights.
Turning now to the derivative of the neuron activation with respect to
b, we get:
Let's use these partial derivatives now to handle the entire loss function.
The gradient of the neural network loss function
Training a neuron requires that we take the derivative of our loss
or “cost” function with respect to the parameters of our model,
w and
b.
Because we train with multiple vector inputs (e.g., multiple images)
and scalar targets (e.g., one classification per image), we need some
more notation. Let
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BEDD04B41054D234623B0BB3759ABF73-depth003.25.svg)
, and then let
where
yi is a scalar. Then the cost equation becomes:
Following our chain rule process introduces these intermediate variables:
Let's compute the gradient with respect to
w first.
The gradient with respect to the weights
From before, we know:
and
Then, for the overall gradient, we get:
To interpret that equation, we can substitute an error term
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-A90A948C72E3B6E10EF49E9CA3323248-depth002.65.svg)
yielding:
From there, notice that this computation is a weighted average across all
xi in
X. The weights are the error terms, the difference between the target output and the actual neuron output for each
xi input. The resulting gradient will, on average, point in the direction of higher cost or loss because large
ei emphasize their associated
xi. Imagine we only had one input vector,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-75147DC2E59AB7AF04E48C0E3C2D71EA-depth003.25.svg)
, then the gradient is just
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-26DDD7C22F3D8F5B07FC7EE421EEE6ED-depth003.45.svg)
. If the error is 0, then the gradient is zero and we have arrived at the minimum loss. If
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3104BCC503B3B8C16FFA2940B56AAF1C-depth001.95.svg)
is some small positive difference, the gradient is a small step in the direction of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-262A3BD0318D5034272A8F904D6FAD24-depth001.95.svg)
. If
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3104BCC503B3B8C16FFA2940B56AAF1C-depth001.95.svg)
is large, the gradient is a large step in that direction. If
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3104BCC503B3B8C16FFA2940B56AAF1C-depth001.95.svg)
is negative, the gradient is reversed, meaning the highest cost is in the negative direction.
Of course, we want to reduce, not increase, the loss, which is why the
gradient descent recurrence relation takes the negative of the gradient to update the current position (for scalar learning rate
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FFE9F913124F345732E9F00FA258552E-depth002.65.svg)
):
Because the gradient indicates the direction of higher cost, we want to update
x in the opposite direction.
The derivative with respect to the bias
To optimize the bias,
b, we also need the partial with respect to
b. Here are the intermediate variables again:
We computed the partial with respect to the bias for equation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FBD5708C00E92A391A69A42591580BE0-depth003.25.svg)
previously:
For
v, the partial is:
And for the partial of the cost function itself we get:
As before, we can substitute an error term:
The partial derivative is then just the average error or zero,
according to the activation level. To update the neuron bias, we nudge
it in the opposite direction of increased cost:
In practice, it is convenient to combine
w and
b into a single vector parameter rather than having to deal with two different partials:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1C5C3BA710F818B84DF992E699DB50C3-depth003.25.svg)
. This requires a tweak to the input vector
x as well but simplifies the activation function. By tacking a 1 onto the end of
x,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-CDA92F9769DA156F5D82B4BF0D40A8B4-depth003.25.svg)
,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6C156670CEC09096976A6722592523F3-depth001.06.svg)
becomes
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-03741D422AA7DF7FF34288D8E4395143-depth000.00.svg)
.
This finishes off the optimization of the neural network loss
function because we have the two partials necessary to perform a
gradient descent.
Summary
Hopefully you've made it all the way through to this point. You're
well on your way to understanding matrix calculus! We've included a
reference that summarizes all of the rules from this article in the next
section. Also check out the annotated resource link below.
Your next step would be to learn about the partial derivatives of
matrices not just vectors. For example, you can take a look at the
matrix differentiation section of
Matrix calculus.
Acknowledgements. We thank
Yannet Interian (Faculty in MS data science program at University of San Francisco) and
David Uminsky (Faculty/director of MS data science) for their help with the notation presented here.
Matrix Calculus Reference
Gradients and Jacobians
The
gradient of a function of two variables is a horizontal 2-vector:
The
Jacobian of a vector-valued function that is a function of a vector is an
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FBFEB9C8459FEE5A2BD529C07B881153-depth001.08.svg)
(
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4B7BE5D4BAEFA7643CD9638A527AC10F-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-E3114C625CDDDC18ED29BA629242BD65-depth003.25.svg)
) matrix containing all possible scalar partial derivatives:
The Jacobian of the identity function
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3ABE4A0471143ABFC180C9FA485E5F0A-depth003.25.svg)
is
I.
Element-wise operations on vectors
Define generic
element-wise operations on vectors
w and
x using operator
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-1A74909B6CBAA4532A76D83B72C12DE0-depth002.52.svg)
such as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-26B17225B626FB9238849FD60EABDF60-depth001.06.svg)
:
The Jacobian with respect to
w (similar for
x) is:
Given the constraint (
element-wise diagonal condition) that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD5A63074F44F11CB2ED06325816582A-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5E597DAA4E5D9263DCBFB6AB02BDB67F-depth003.25.svg)
access at most
wi and
xi, respectively, the Jacobian simplifies to a diagonal matrix:
Here are some sample element-wise operators:
Scalar expansion
Adding scalar
z to vector
x,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6307CEA088D2D4E98E5B163B9CE8F510-depth002.33.svg)
, is really
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-8D9C0B9B15490F45C353D9DE64565A4F-depth003.25.svg)
where
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-3ABE4A0471143ABFC180C9FA485E5F0A-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-BD2335FC4BBF16BE9590D2501CE8C030-depth003.25.svg)
.
Scalar multiplication yields:
Vector reductions
The partial derivative of a vector sum with respect to one of the vectors is:
For
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-5FAFFB6A723E437AC6433DCA0B269846-depth003.25.svg)
:
For
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9D785998BBDB763E1D5EE5546D47E47E-depth003.25.svg)
and
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B00911788A7536593130B4C89B6653A2-depth003.25.svg)
, we get:
Vector dot product
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-CF556A1D1CDF0863FDD547594536501A-depth003.31.svg)
. Substituting
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-02A67275B737296ACC7D3FACA124192C-depth001.08.svg)
and using the vector chain rule, we get:
Similarly,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-D0E108FD2A63FFC84018DF9BCBE4C91B-depth004.58.svg)
.
Chain rules
The
vector chain rule is the general form as it degenerates to the others. When
f is a function of a single variable
x and all intermediate variables
u
are functions of a single variable, the single-variable chain rule
applies. When some or all of the intermediate variables are functions of
multiple variables, the single-variable total-derivative chain rule
applies. In all other cases, the vector chain rule applies.
Single-variable rule | Single-variable total-derivative rule | Vector rule |
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-32235B531450ABE9E39C9C91D083A8E2-depth004.58.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-ED4CD5FBA6B6EC8A51FA203E2AAFF531-depth004.67.svg) | ![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-6396586BD585A1CFB33959EB6FA8BFA0-depth006.23.svg) |
Notation
Lowercase letters in bold font such as
x are vectors and those in italics font like
x are scalars.
xi is the
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-97361F12A3555FC4FC4E2FFCE1799AC3-depth000.14.svg)
element of vector
x and is in italics because a single vector element is a scalar.
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-DEA8E196A572D082201CD5ABF2FA82DE-depth003.25.svg)
means “length of vector
x.”
The
T exponent of
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FAAEE783424BC0E27E9AA2F56A7B50B8-depth000.00.svg)
represents the transpose of the indicated vector.
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-60D8CDDDBF54CF43BC22AF322D2BB8E3-depth003.31.svg)
is just a for-loop that iterates
i from
a to
b, summing all the
xi.
Notation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-50BBD36E1FD2333108437A2CA378BE62-depth003.25.svg)
refers to a function called
f with an argument of
x.
I represents the square “identity matrix” of
appropriate dimensions that is zero everywhere but the diagonal, which
contains all ones.
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4935E4C6B875FD6C7C181871B566AB1A-depth003.25.svg)
constructs a matrix whose diagonal elements are taken from vector
x:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-358C0E8F99CB66017CCD32DBC6C23C50-depth003.25.svg)
.
The dot product
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-9CA8C3CD16894AF7620468A20C53D6FA-depth000.00.svg)
is the summation of the element-wise multiplication of the elements:
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-4B722CF0EFB8F8ABCBB968086BB587E8-depth003.31.svg)
. Or, you can look at it as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-CF0A6D64FC3321DB0EC98B7683024367-depth000.22.svg)
.
Differentiation
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-624FEBB9A49A3FC96353C861D175C806-depth004.58.svg)
is an operator that maps a function of one parameter to another function. That means that
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-EC0CEC5F9488EC510F8D688E7003222D-depth004.58.svg)
maps
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-50BBD36E1FD2333108437A2CA378BE62-depth003.25.svg)
to its derivative with respect to
x, which is the same thing as
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-07A5EA519C4CEA1A3539E3A7FC289163-depth004.58.svg)
. Also, if
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-FD91C508F91C2C84498680BD337C1D7A-depth003.25.svg)
, then
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-B1ED3CF9BA4D6F25A5A4F481C45EC658-depth004.58.svg)
.
The partial derivative of the function with respect to
x,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-2E335C72CE43928A54BED52C5A6FCC87-depth004.67.svg)
, performs the usual scalar derivative holding all other variables constant.
The gradient of
f with respect to vector
x,
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-69AA2CF3DFE9D41CB1DB567D1B0AD275-depth003.25.svg)
, organizes all of the partial derivatives for a specific scalar function.
The Jacobian organizes the gradients of multiple functions into a matrix by stacking them:
The following notation means that
y has the value
a upon
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-585B40029A25F6E19FF42DBC26AE5702-depth001.95.svg)
and value
b upon
![](https://parrt.cs.usfca.edu/doc/matrix-calculus/images/eqn-CBECF4275AFD44DAD4B312042088DA7E-depth001.95.svg)
.
Resources
When looking for resources on the web, search for “matrix calculus”
not “vector calculus.” Here are some comments on the top links that
come up from a
Google search:
To learn more about neural networks and the mathematics behind optimization and back propagation, we highly recommend
Michael Nielsen's book.
We reference the law of
total derivative, which is an important concept that just means derivatives with respect to
x must take into consideration the derivative with respect
x of all variables that are a function of
x.
No comments:
Post a Comment