Bielefeld, May 19, 1997

1. Basic Matrix Operations.

A matrix is simply a rectangular array of numbers, in itself nothing mysterious. However, the algebraic interactions of such arrays, and their geometric manifestations, open up a world of fascinating mathematical phenomena. This is matrix theory, also known as linear algebra.

To keep things simple, the present sketch will limit itself by and large to just two kinds of matrices:

A = [ \begin{matrix}a & b \\ c & d \end{matrix} ] and X = [ \begin{matrix}x \\ y \end{matrix} ] .

When we say “matrix”, we shall usually mean the first of these types, preferring the term “vector” for the second. To save space, we shall often write (x, y) instead of the vertical column displayed above when dealing with vectors alone.

Matrices can be added together and scaled in the most obvious manner, namely entry-by-entry. For these operations, a matrix is just like any string of numbers. The rectangular organization into rows and columns does not come in. Multiplication of matrices is another story, however, the product of two matrices being defined by:

[ \begin{matrix}a & b \\ c & d \end{matrix} ] [ \begin{matrix}x & u \\ y & v \end{matrix} ] = [ \begin{matrix}ax + by & au + bv \\ cx + dy & cu + dv \end{matrix} ] .

Note that the entry in row i and column j of the product AB is obtained by amalgamating the i-th row of the first factor A with the j-th column of the second factor B. We could also say that the k-th column of AB is obtained by letting A act on the k-th column of B.

This operation obeys most of the usual laws of multiplication, such as A(B + C) = AB + AC and (A + B)C = AC + BC (“distributivity”) as well as A(BC) = (AB)C (“associativity”), but it does not allow you to interchange the order of factors: AB ≠ BA in general. Another fact to be wary of is that AB = 0 is possible even if A ≠ 0 and B ≠ 0.

A hard look at the definition of the product should convince you that the two distributive laws do in fact hold. Associativity is not quite as obvious. One way to see it is to imagine every matrix expressed in terms of the so-called matrix units, which are matrices having a single entry = 1 and the others = 0. Any matrix can written as a linear combination (i.e. a sum of scalar multiples) of these, e.g.

[ \begin{matrix}a & b \\ c & d \end{matrix} ] = a [ \begin{matrix}1 & 0 \\ 0 & 0 \end{matrix} ] + b [ \begin{matrix}0 & 1 \\ 0 & 0 \end{matrix} ] + c [ \begin{matrix}0 & 0 \\ 1 & 0 \end{matrix} ] + d [ \begin{matrix}0 & 0 \\ 0 & 1 \end{matrix} ] .

The matrix unit having the 1 in row i and column j (and 0 elsewhere) is denoted by E_{ij}. It is a good exercise in matrix multiplication to check that E_{ij}E_{kl} equals E_{il} if j = k, and yields the zero-matrix (all of whose entries are 0) otherwise. Multiplying once more, we see that both E_{ij}(E_{kl}E_{mn}) and (E_{ij}E_{kl})E_{mn} give the same result (what is it?). To verify the general associative law A(BC) = (AB)C, we imagine these three matrices expressed as linear combinations of matrix units. Repeated use of the distributive law now reduces both sides of this equation to the special cases already checked.

Playing with matrix units also reveals some of the quirks of matrix multiplication. For instance, a look at the products E_{11}E_{12} and E_{12}E_{11} shows that factors cannot normally be interchanged and that a product of non-zero matrices can be zero (see?).

Exercises.

1. Compute E_{11}E_{12} and E_{12}E_{11}; then E_{ij}E_{kl}; finally E_{ij}(E_{kl}E_{mn}) as well as (E_{ij}E_{kl})E_{mn}.

2. From (AB)C = A(BC) deduce the equality of all 5 bracketings of A_{1}A_{2}A_{3}A_{4}.

3. In how many ways can you bracket A_{1}A_{2}A_{3}A_{4}A_{5}? Using the preceding exercise, prove that all these yield the same product.

2. The Inverse and Singularity

The matrix product is especially simple if one of the factors is a diagonal matrix D(α, β) = αE_{11} +βE_{22}:

[ \begin{matrix}α & 0 \\ 0 & β \end{matrix} ] [ \begin{matrix}a & b \\ c & d \end{matrix} ] = [ \begin{matrix}αa & αb \\ βc & βd \end{matrix} ] or [ \begin{matrix}a & b \\ c & d \end{matrix} ] [ \begin{matrix}α & 0 \\ 0 & β \end{matrix} ] = [ \begin{matrix}αa & βb \\ αc & βd \end{matrix} ] .

Note: left multiplication by a diagonal scales the rows, while right multiplication by a diagonal scales the columns, of the other matrix. If α = β = 1, we have the identity matrix I = D(1, 1) which, as a multiplier, has no effect at all: IA = AI = A for any A.

The obvious equation

[ \begin{matrix}a & b \\ c & d \end{matrix} ] [ \begin{matrix}d & −b \\ −c & a \end{matrix} ] = [ \begin{matrix}ad − bc & 0 \\ 0 & ad − bc \end{matrix} ]  (†)

turns out to be a real gold mine of information. The second factor appearing in it is visibly concocted from the first factor A, and is known as its adjoint, denoted A^{⋆}. The scalar (ad − bc) on the right hand side is called the determinant, det A. Our equation (†) can thus be abbreviated as AA^{⋆} = (det A)I.

You should check that A^{⋆}A = (det A)I, as well. Here is what we can deduce:

(1) If det A ≠ 0, the matrix A has an inverse

A^{−1} = \frac{1}{det A}A^{⋆},  (‡)

which is a matrix with the property that AA^{−1} = A^{−1}A = I.

(2) If det A = 0, then A is singular, which means that there is a B ≠ 0 with AB = 0 (e.g., B = A^{⋆}). By the way, to check singularity it suffices to find a single column X ≠ 0 such that AX = 0 (see?).

(3) Conversely, a singular A cannot have an inverse, because multiplying AB = 0 by A^{−1} would force B = 0.

Conclusion: every A is either invertible or singular, and the determinant says which.

(4) Noting that A^{⋆} = (a + d)I − A, and substituting the right hand side of this equation in (†) yields the famous Cayley-Hamilton relation

A^{2} − (a + d)A + (ad − bc)I = 0.

The scalar (a + d) is called the trace of A.

The following items not directly related to (†) should also be noted

(5) Inversion reverses the order of factors, i.e. (AB)^{−1} = B^{−1}A^{−1}, because ABB^{−1}A^{−1} = AIA^{−1} = I.

(6) Looking back to the product AB, as given in lesson 1, and computing its determinant, we obtain (ax + by)(cu + dv) − (cx + dy)(au + bv) = adxv + bcyu − bcxv − adyu = (ad − bc)(xv − yu), i.e. the miraculous formula:

det(AB) = (det A)(det B).

Exercises.

1. Prove that A is singular if and only if there is a matrix L ≠ 0 such that LA = 0.

2. Given that A has trace −1 and determinant 1, use the Cayley-Hamiltion relation to compute A^{3}. Show that there are infinitely many such matrices.

3. Matrices as Transformations

Recall that, by means of a cartesian coordinate system, any pair (x, y) of real numbers can be regarded as a point in the plane. A more dynamic interpretation of the same pair (x, y) is that of a vector, i.e. an imaginary shift “x units east and y units north” applicable to any point. This has the advantage of giving geometric meaning to addition (perform one shift after the other) and to scaling (lengthen or shorten the shift by a factor). A vector is often depicted by an arrow showing the shift applied to a suitable point (by default, the origin). We will not be fussy about interpreting X = (x, y) as either a point or a vector depending on circumstances.

Given a 2 × 2 matrix A, we now want to describe how multiplication by A transforms the points of the plane. We think of A as changing every point X to the “image point” AX. How is AX related to X?

As a vector, X can be written as X = xI^{(1)} +yI^{(2)}, where I^{(k)} stands for the k-th column of the identity matrix. AX is directly given by the definition

AX = xA^{(1)} + yA^{(2)},  (1)

where A^{(k)} denotes the k-th column of A. Thus AX is made from the columns of A in the same way that X is made from the columns of I: if you picture X as a diagonal arrow in a vector rectangle with sides xI^{(1)} and yI^{(2)}, the vector AX appears as the diagonal in the parallelogram with sides xA^{(1)} and yA^{(2)}.

In fact, if you imagine a rectangular coordinate grid overlaid on the plane as on graph paper, you will see it transformed into a generally skew grid made from copies of the parallelogram given by A^{(1)} and A^{(2)}. Every X is transformed into the AX having the same “coordinates” x and y with respect to the new grid.

Many useful transformations of the plane are unrelated to matrices (for instance, the one which gives rise to logarithmic graph paper). How, then, can we tell whether a transformation T of the plane comes from a matrix A? Answer: T must be linear, i.e. it must satisfy

T (V + W) = T (V) + T (W)  and  T (αW) = αT (W).  (2)

For X = (x, y), this obviously implies that T (X) = xT (I^{(1)}) + yT (I^{(2)}) (see?), which is the same as multiplying X by the matrix whose columns are T (I^{(1)}) and T (I^{(2)}).

For illustration we describe the action of three special kinds of matrices:

D(α, β) = [ \begin{matrix}α & 0 \\ 0 & β \end{matrix} ]  G_{12}(γ) = [ \begin{matrix}1 & γ \\ 0 & 1 \end{matrix} ]  R(θ) = [ \begin{matrix}cos θ & − sin θ \\ sin θ & cos θ \end{matrix} ] .

D(α, β) affects the coordinates of a point separately, multiplying each by the appropriate factor α or β. The coordinate axes stay put (though points on them may move), the grid remains rectangular (though it may get stretched or squeezed). If α ≠ β, any vector not parallel to one of the axes is tilted toward the axis with the stronger factor.

G_{12}(γ) produces a “shear”, which simply moves every point (x, y) horizontally by the amount γy. Every coordinate square is deformed into a parallelogram with the same base and height. Of course G_{12}(γ) has a sister G_{21}(γ) with the γ in the lower left corner, which moves points vertically in a similar mannner.

R(θ) acts as a rotation of the plane through the angle θ. Since its columns are the images of I^{(1)} and I^{(2)}, the definition of sine and cosine shows that these vectors are both rotated through θ. But then, so are xI^{(1)} and yI^{(2)}, and therefore the whole rectangle spanned by these.

Exercises.

1. Consider the matrix A which has a = 1.3, b = 0.5, c = 0.1, and d = 1.0. On graph paper, using 5 cm as your unit of length, draw the unit coordinate grid as transfomed by A. Do the same for A^{−1}.

2. On each of these diagrams, also draw the image of the circle (x − 1/2)^{2} + (y − 1/2)^{2} = 1/4.

4. The Dot Product

The basic building block of matrix multiplication is the product

[a b] [ \begin{matrix}x \\ y \end{matrix} ] = ax + by  (1)

of a row and a column. When dealing with vectors only, we usually do not wish to commit ourselves to a definite row or column lay-out of the coordinates, so we define the dot-product V •X of two vectors V = (a, b) and X = (x, y) by the same formula (1). We shall presently see that this algebraic operation has a surprising geometric interpretation.

To start with, we define the norm |V | of a vector V by the simple but crucial equation

|V |^{2} = V • V,  (2)

and observe that the general dot-product can be computed from norms by the obvious formula

2V • X = |V |^{2} + |X|^{2} − |V − X|^{2}.  (3)

We say that V and X are orthogonal to each other if V • X = 0, or equivalently |V − X|^{2} = |V |^{2} + |X|^{2}. In particular,

V • X = 0 ⟹ |V − X| ≥ |V |.  (4)

Obviously we can always find a scalar t such that (X − tV) • V = 0, and clearly the vector tV is unique. It is called the projection of X onto V and given by

proj_{V} (X) = tV = \frac{X • V}{V • V} V,  (5)

if V ≠ 0. For any scalar α, we can write (X − αV) = (X − tV) − (α − t)V as a difference of orthogonal vectors, and therefore get |X − αV | ≥ |X − tV |, by the inequality (4). This means that, among all vectors of the form αV, the projection tV is the closest to X, in the sense that it minimizes |X − αV |.

Now for the geometry. If our coordinate system is rectangular, a comparison of (1) and (2) shows that norm = length by the Theorem of Pythagoras. Moreover, if we represent the vectors V and X by arrows \overrightarrow{OP} and \overrightarrow{OQ}, the vector V − X = \overrightarrow{P Q} will lie along the third side of the triangle so formed (see?), and Pythagoras says that |V − X|^{2} = |V |^{2} + |X|^{2} if and only if the angle P OQ is a right angle. Hence formula (3) implies that orthogonality = perpendicularity. In the same setting, proj_{V} (X) = tV would show up as an arrow \overrightarrow{OP′} with P′ chosen so as to make OP′Q a right angle. Now, if ϕ denotes the angle between V and X, we have cos ϕ = t|V |/|X|, which turns into

cos ϕ = \frac{X • V}{|V | |X|},  (6)

by the formula (5) for tV. This shows that the dot-product governs angles as well as lengths.

Remark: Equations (2) through (6) as well as the arguments connecting them are valid verbatim for three or more dimensions, since they hinge only on Pythagoras and (2).

Exercises.

1. Let V and W be the columns of a matrix A. Show that det A = JV • W, where J = R(90^{∘}), and thereby interpret | det A| as the area of the parallelogram P(V, W) spanned by the columns of A.

2. If V ≠ 0 has the coordinates (u, v), find the matrix P_{V} such that P_{V} X = proj_{V} (X) for all X. What kind of transformation is given by F_{V} = 2P_{V} − I?