🔒 Private Site

This site is password-protected.

Calculusphile Logo
calculusphile

Engineering Mathematics-II — Unit II Notes

Course: Engineering Mathematics-II (Linear Algebra and Differential Equations)
Institute: Indian Institute of Information Technology Pune — F.Y. B.Tech Semester II
Syllabus Coverage: Linear Transformations; Kernel, Image & Rank-Nullity Theorem; Inner Product Spaces & Gram-Schmidt Orthogonalization; Eigenvalues, Eigenvectors & Diagonalization; Jordan Canonical Form; Quadratic Forms & Positive Definiteness (08 Hrs)
Text Books: Serge Lang — Introduction to Linear Algebra; Erwin Kreyszig — Advanced Engineering Mathematics

Table of Contents

1 — Linear Transformations, Kernel, Image & Rank-Nullity Theorem

2 — Inner Product Spaces & Gram-Schmidt Orthogonalization

3 — Eigenvalues, Eigenvectors & Diagonalization

4 — Jordan Canonical Form

5 — Quadratic Forms & Positive Definiteness

6 — Engineering & Real-Life Applications

Comprehensive Unit II Practice Set (with Answers)

Glossary — Key Terms at a Glance

Term Meaning
Linear Transformation A function $T: V \to W$ satisfying $T(u+v) = T(u)+T(v)$ and $T(\alpha v) = \alpha T(v)$.
Standard Matrix The matrix $A$ such that $T(x) = Ax$ for a linear transformation $T: \mathbb{R}^n \to \mathbb{R}^m$.
Kernel (Null Space) $\ker(T) = \lbrace v \in V : T(v) = 0 \rbrace$ — all vectors mapped to zero.
Image (Range) $\text{Im}(T) = \lbrace T(v) : v \in V \rbrace$ — all output vectors.
Rank $\text{rank}(T) = \dim(\text{Im}(T))$ — dimension of the image.
Nullity $\text{nullity}(T) = \dim(\ker(T))$ — dimension of the kernel.
Rank-Nullity Theorem $\dim(V) = \text{rank}(T) + \text{nullity}(T)$.
Inner Product A function $\langle u, v \rangle$ satisfying symmetry, linearity in the first argument, and positive-definiteness.
Induced Norm $\lVert v \rVert = \sqrt{\langle v, v \rangle}$ — length of a vector from the inner product.
Cauchy-Schwarz Inequality $\lvert\langle u, v \rangle\rvert \leq \lVert u \rVert \lVert v \rVert$.
Orthogonal Vectors $u, v$ with $\langle u, v \rangle = 0$.
Orthonormal An orthogonal set where every vector has unit norm.
Gram-Schmidt Process Algorithm to convert any basis into an orthonormal basis.
QR Decomposition $A = QR$ where $Q$ has orthonormal columns, $R$ is upper triangular.
Eigenvalue A scalar $\lambda$ such that $Av = \lambda v$ for some nonzero $v$.
Eigenvector A nonzero vector $v$ satisfying $Av = \lambda v$.
Characteristic Polynomial $p(\lambda) = \det(A - \lambda I)$ — roots are the eigenvalues.
Algebraic Multiplicity Multiplicity of $\lambda$ as a root of the characteristic polynomial.
Geometric Multiplicity $\dim(E_\lambda) = \dim(\text{null}(A - \lambda I))$.
Diagonalizable $A = PDP^{-1}$ where $D$ is diagonal — iff $A$ has $n$ linearly independent eigenvectors.
Spectral Theorem Symmetric matrices have real eigenvalues and orthogonal eigenvectors: $A = QDQ^T$.
Jordan Block A $k \times k$ matrix with $\lambda$ on diagonal and 1’s on superdiagonal.
Jordan Canonical Form Block diagonal matrix of Jordan blocks — exists for every square matrix.
Generalized Eigenvector Vector $v$ with $(A - \lambda I)^k v = 0$ for some $k > 1$.
Quadratic Form A homogeneous degree-2 polynomial $Q(x) = x^T A x$ with symmetric $A$.
Positive Definite $Q(x) > 0$ for all $x \neq 0$ — equivalently, all eigenvalues positive.
Sylvester’s Criterion $A$ is positive definite iff all leading principal minors are positive.
Principal Axes Theorem Orthogonal change of variables diagonalizes any quadratic form.
Signature The triple $(p, q, z)$ counting positive, negative, and zero eigenvalues.

1 — Linear Transformations, Kernel, Image & Rank-Nullity Theorem

1.1 Definition of a Linear Transformation

Definition: A function $T: V \to W$ between vector spaces is a linear transformation if for all vectors $u, v \in V$ and all scalars $\alpha$:

  1. Additivity: $T(u + v) = T(u) + T(v)$
  2. Homogeneity: $T(\alpha v) = \alpha\, T(v)$

Equivalently (combined): $T(\alpha u + \beta v) = \alpha\, T(u) + \beta\, T(v)$ for all scalars $\alpha, \beta$.

Intuition — What Does “Linear” Actually Mean?

Think of a linear transformation as a machine that reshapes space while obeying two simple rules:

  1. Lines stay lines — straight lines through the origin remain straight (they never bend or curve).
  2. Spacing stays even — evenly spaced points stay evenly spaced (they never bunch up or spread unevenly).

Imagine a rubber sheet pinned at the origin. You can stretch, compress, rotate, reflect, or shear it — all of these are linear transformations. But you cannot bend, tear, or crumple the sheet — those would break linearity.

Stretch — pull the circle wider along one axis:

Compress — squash it:

Rotate — spin the whole shape around the origin:

Reflect — mirror across an axis:

Shear — tilt the shape while keeping one axis fixed:

A practical way to check: Apply $T$ to two vectors individually, then apply $T$ to their sum. If $T(\text{sum}) = T(v_1) + T(v_2)$ every time, you have a linear transformation. That’s the additivity rule. Similarly, scaling a vector before or after applying $T$ gives the same result — that’s homogeneity.

What’s NOT linear? $T(x) = x + 3$ (shifts the origin), $T(x) = x^2$ (bends lines into curves), $T(x) = \lvert x \rvert$ (creates a kink at zero). All of these violate the rules above.

Why Study Linear Transformations?

Linear transformations are the “structure-preserving maps” of linear algebra. They model:

  • Rotations, reflections, projections in computer graphics
  • Differentiation and integration operators in calculus
  • Signal filters in electrical engineering
  • Coordinate changes in physics and robotics

Immediate Consequences

From the definition, every linear transformation satisfies:

  • $T(\vec{0}_V) = \vec{0}_W$ (maps zero to zero)
  • $T(-v) = -T(v)$
  • $T!\left(\sum_{i=1}^k \alpha_i v_i\right) = \sum_{i=1}^k \alpha_i\, T(v_i)$ (preserves arbitrary linear combinations)

1.2 Matrix Representation of Linear Transformations

Theorem: Every linear transformation $T: \mathbb{R}^n \to \mathbb{R}^m$ can be represented as matrix multiplication:

\[T(x) = Ax\]

where the standard matrix $A$ has columns $T(e_1), T(e_2), \ldots, T(e_n)$ (images of the standard basis vectors).

This means: to find $A$, simply compute what $T$ does to each standard basis vector $e_1, e_2, \ldots, e_n$ and place the results as columns.

Example: If $T: \mathbb{R}^2 \to \mathbb{R}^2$ is defined by $T(x, y) = (x + 2y,\; 3x - y)$, then:

\[T(e_1) = T(1, 0) = (1, 3), \quad T(e_2) = T(0, 1) = (2, -1)\] \[A = \begin{bmatrix} 1 & 2 \\ 3 & -1 \end{bmatrix}\]

1.3 Kernel (Null Space) of a Linear Transformation

Definition: The kernel (or null space) of a linear transformation $T: V \to W$ is the set of all vectors in $V$ that map to the zero vector in $W$:

\[\ker(T) = \lbrace v \in V : T(v) = \vec{0}_W \rbrace\]

Theorem (Kernel is a Subspace): For any linear transformation $T: V \to W$, the kernel $\ker(T)$ is a subspace of the domain $V$.

Proof:

  1. Contains zero: $T(\vec{0}_V) = \vec{0}_W$, so $\vec{0}_V \in \ker(T)$.
  2. Closed under addition: If $u, v \in \ker(T)$, then $T(u+v) = T(u) + T(v) = \vec{0} + \vec{0} = \vec{0}$, so $u + v \in \ker(T)$.
  3. Closed under scalar multiplication: If $v \in \ker(T)$ and $\alpha$ is a scalar, then $T(\alpha v) = \alpha T(v) = \alpha \cdot \vec{0} = \vec{0}$, so $\alpha v \in \ker(T)$. $\blacksquare$

Finding the Kernel

For $T: \mathbb{R}^n \to \mathbb{R}^m$ with standard matrix $A$:

\[\ker(T) = \text{null}(A) = \lbrace x \in \mathbb{R}^n : Ax = 0 \rbrace\]

Procedure: Row reduce $A$ and solve the homogeneous system $Ax = 0$. The free variables give a parametric description of the kernel.

Nullity: The nullity of $T$ is the dimension of the kernel:

\[\text{nullity}(T) = \dim(\ker(T))\]

1.4 Image (Range) of a Linear Transformation

Definition: The image (or range) of $T: V \to W$ is the set of all vectors in $W$ that are outputs of $T$:

\[\text{Im}(T) = \lbrace T(v) : v \in V \rbrace = \lbrace w \in W : w = T(v) \text{ for some } v \in V \rbrace\]

Theorem (Image is a Subspace): For any linear transformation $T: V \to W$, the image $\text{Im}(T)$ is a subspace of the codomain $W$.

Proof:

  1. Contains zero: $\vec{0}_W = T(\vec{0}_V) \in \text{Im}(T)$.
  2. Closed under addition: If $w_1, w_2 \in \text{Im}(T)$, then $w_1 = T(u_1)$ and $w_2 = T(u_2)$ for some $u_1, u_2 \in V$. So $w_1 + w_2 = T(u_1) + T(u_2) = T(u_1 + u_2) \in \text{Im}(T)$.
  3. Closed under scalar multiplication: If $w = T(u) \in \text{Im}(T)$, then $\alpha w = \alpha T(u) = T(\alpha u) \in \text{Im}(T)$. $\blacksquare$

Finding the Image

For $T: \mathbb{R}^n \to \mathbb{R}^m$ with standard matrix $A$:

\[\text{Im}(T) = \text{Col}(A) = \text{span}\lbrace\text{columns of } A\rbrace\]

Procedure:

  1. Write down the columns of $A$.
  2. Row reduce $A$ to identify pivot columns.
  3. The corresponding original columns of $A$ form a basis for $\text{Im}(T)$.

Rank: The rank of $T$ is the dimension of the image:

\[\text{rank}(T) = \dim(\text{Im}(T))\]

1.5 The Rank-Nullity Theorem

Theorem (Fundamental Theorem of Linear Maps): Let $T: V \to W$ be a linear transformation where $V$ is finite-dimensional. Then:

\[\dim(V) = \text{rank}(T) + \text{nullity}(T)\]

Equivalently:

\[\dim(V) = \dim(\text{Im}(T)) + \dim(\ker(T))\]

Proof Sketch: Let $\dim(V) = n$ and $\dim(\ker(T)) = k$. Choose a basis $\lbrace u_1, \ldots, u_k \rbrace$ for $\ker(T)$ and extend it to a basis $\lbrace u_1, \ldots, u_k, u_{k+1}, \ldots, u_n \rbrace$ for $V$.

Claim: $\lbrace T(u_{k+1}), \ldots, T(u_n) \rbrace$ is a basis for $\text{Im}(T)$.

Spanning: Any $w \in \text{Im}(T)$ has the form $w = T(v)$ for some $v \in V$. Writing $v = \alpha_1 u_1 + \cdots + \alpha_n u_n$ and applying $T$:

\[w = T(v) = \alpha_1 T(u_1) + \cdots + \alpha_k T(u_k) + \alpha_{k+1} T(u_{k+1}) + \cdots + \alpha_n T(u_n)\]

Since $u_1, \ldots, u_k \in \ker(T)$, we have $T(u_i) = 0$ for $i = 1, \ldots, k$. Thus $w = \alpha_{k+1} T(u_{k+1}) + \cdots + \alpha_n T(u_n)$.

Linear independence: If $\alpha_{k+1} T(u_{k+1}) + \cdots + \alpha_n T(u_n) = 0$, then $T(\alpha_{k+1} u_{k+1} + \cdots + \alpha_n u_n) = 0$, so $\alpha_{k+1} u_{k+1} + \cdots + \alpha_n u_n \in \ker(T)$. Writing this as a linear combination of $u_1, \ldots, u_k$ and using linear independence of $\lbrace u_1, \ldots, u_n \rbrace$ gives all $\alpha_i = 0$.

Therefore $\dim(\text{Im}(T)) = n - k$, giving $\dim(V) = \dim(\ker(T)) + \dim(\text{Im}(T))$. $\blacksquare$

Key Insight: The Rank-Nullity Theorem tells us there is always a tradeoff — as the kernel grows (more vectors map to zero), the image shrinks, and vice versa. The domain dimension is the total “budget.”

1.6 Injective, Surjective & Bijective Transformations

Property Definition Equivalent Condition
Injective (One-to-one) $T(u) = T(v) \Rightarrow u = v$ $\ker(T) = \lbrace\vec{0}\rbrace$, i.e., $\text{nullity}(T) = 0$
Surjective (Onto) $\text{Im}(T) = W$ $\text{rank}(T) = \dim(W)$
Bijective (Isomorphism) Both injective and surjective $\ker(T) = \lbrace\vec{0}\rbrace$ and $\text{Im}(T) = W$

Theorem: For $T: V \to W$ with $\dim(V) = \dim(W) = n$:

\[T \text{ is injective} \iff T \text{ is surjective} \iff T \text{ is bijective}\]

All three properties become equivalent for transformations between equal-dimensional spaces.

Proof (Injective ⇒ Surjective): If $T$ is injective, then $\ker(T) = \lbrace 0 \rbrace$, so $\text{nullity}(T) = 0$. By Rank-Nullity: $\text{rank}(T) = n - 0 = n = \dim(W)$, so $T$ is surjective.

Proof (Surjective ⇒ Injective): If $T$ is surjective, then $\text{rank}(T) = n$. By Rank-Nullity: $\text{nullity}(T) = n - n = 0$, so $\ker(T) = \lbrace 0 \rbrace$ and $T$ is injective. $\blacksquare$

1.7 Composition of Linear Transformations

If $T: U \to V$ and $S: V \to W$ are both linear, then the composition $S \circ T: U \to W$ defined by $(S \circ T)(x) = S(T(x))$ is also linear.

Properties of composition:

  • $\ker(T) \subseteq \ker(S \circ T)$
  • $\text{Im}(S \circ T) \subseteq \text{Im}(S)$
  • If $A$ and $B$ are the matrices of $T$ and $S$ respectively, then $BA$ is the matrix of $S \circ T$.

1.8 Worked Examples — Linear Transformations

Example 1: Verifying a Linear Transformation

Problem: Is $T: \mathbb{R}^2 \to \mathbb{R}^2$ defined by $T(x, y) = (x + 2y,\; 3x - y)$ linear?

Solution:

Let $u = (x_1, y_1)$ and $v = (x_2, y_2)$.

Additivity:

\[T(u + v) = T(x_1 + x_2,\; y_1 + y_2) = ((x_1+x_2) + 2(y_1+y_2),\; 3(x_1+x_2) - (y_1+y_2))\] \[= (x_1 + 2y_1,\; 3x_1 - y_1) + (x_2 + 2y_2,\; 3x_2 - y_2) = T(u) + T(v) \;\checkmark\]

Homogeneity:

\[T(\alpha u) = T(\alpha x_1,\; \alpha y_1) = (\alpha x_1 + 2\alpha y_1,\; 3\alpha x_1 - \alpha y_1) = \alpha(x_1 + 2y_1,\; 3x_1 - y_1) = \alpha T(u) \;\checkmark\]

Yes, $T$ is linear. Standard matrix: $A = \begin{bmatrix} 1 & 2 \ 3 & -1 \end{bmatrix}$.

Example 2: Finding Kernel and Image

Problem: For $T: \mathbb{R}^3 \to \mathbb{R}^2$ with standard matrix $A = \begin{bmatrix} 1 & 2 & 3 \ 2 & 4 & 6 \end{bmatrix}$, find the kernel, nullity, image, and rank.

Solution:

Kernel: Solve $Ax = 0$. Row reduce:

\[\begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \end{bmatrix} \xrightarrow{R_2 - 2R_1} \begin{bmatrix} 1 & 2 & 3 \\ 0 & 0 & 0 \end{bmatrix}\]

From RREF: $x_1 = -2x_2 - 3x_3$ with $x_2, x_3$ free. So:

\[\ker(T) = \text{span}\left\lbrace \begin{bmatrix} -2 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} -3 \\ 0 \\ 1 \end{bmatrix} \right\rbrace, \quad \text{nullity}(T) = 2\]

Image: The pivot column is column 1, so:

\[\text{Im}(T) = \text{span}\left\lbrace \begin{bmatrix} 1 \\ 2 \end{bmatrix} \right\rbrace, \quad \text{rank}(T) = 1\]

Verification: $\text{rank}(T) + \text{nullity}(T) = 1 + 2 = 3 = \dim(\mathbb{R}^3)$ $\checkmark$

Example 3: Rotation Transformation

Problem: $T: \mathbb{R}^2 \to \mathbb{R}^2$ rotates vectors counterclockwise by $\theta = 90°$.
(a) Find the standard matrix. (b) Find kernel and image. (c) Is $T$ bijective?

Solution:

(a) The rotation matrix for angle $\theta$:

\[R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}\]

For $\theta = 90°$: $A = \begin{bmatrix} 0 & -1 \ 1 & 0 \end{bmatrix}$.

(b) Solve $Ax = 0$: gives $-x_2 = 0$ and $x_1 = 0$, so $\ker(T) = \lbrace\vec{0}\rbrace$, $\text{nullity}(T) = 0$.

The columns are linearly independent and span $\mathbb{R}^2$, so $\text{Im}(T) = \mathbb{R}^2$, $\text{rank}(T) = 2$.

(c) $T$ is bijective (injective since kernel is trivial, surjective since image is all of $\mathbb{R}^2$).

Example 4: Projection onto the $xy$-plane

Problem: $T: \mathbb{R}^3 \to \mathbb{R}^3$ defined by $T(x, y, z) = (x, y, 0)$. Find kernel, image, and determine injectivity/surjectivity.

Solution:

Standard matrix: $A = \begin{bmatrix} 1 & 0 & 0 \ 0 & 1 & 0 \ 0 & 0 & 0 \end{bmatrix}$

\[\ker(T) = \text{span}\left\lbrace \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}\right\rbrace = z\text{-axis}, \quad \text{nullity}(T) = 1\] \[\text{Im}(T) = \text{span}\left\lbrace \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}\right\rbrace = xy\text{-plane}, \quad \text{rank}(T) = 2\]

Not injective (kernel is nontrivial). Not surjective (image $\neq \mathbb{R}^3$).

Example 5: Differentiation as a Linear Transformation

Problem: $D: \mathcal{P}_3 \to \mathcal{P}_2$ defined by $D(p) = p’$ (derivative). Find kernel, nullity, image, rank, and verify Rank-Nullity.

Solution:

Kernel: $D(p) = 0$ means $p’ = 0$, so $p$ is constant.

\[\ker(D) = \text{span}\lbrace 1 \rbrace, \quad \text{nullity}(D) = 1\]

Image: Any polynomial of degree $\leq 2$ is the derivative of some polynomial of degree $\leq 3$: e.g., $ax^2 + bx + c = D!\left(\tfrac{a}{3}x^3 + \tfrac{b}{2}x^2 + cx\right)$.

\[\text{Im}(D) = \mathcal{P}_2, \quad \text{rank}(D) = 3\]

Rank-Nullity: $\dim(\mathcal{P}_3) = 4 = 3 + 1 = \text{rank}(D) + \text{nullity}(D)$ $\checkmark$

Practice Questions — Linear Transformations

Problem Set 1: Verification of Linear Transformations

1.1 Determine whether $T: \mathbb{R}^2 \to \mathbb{R}^2$ defined by $T(x, y) = (x^2, y)$ is a linear transformation.

Click for solution No. $T(2, 0) = (4, 0)$ but $2 \cdot T(1, 0) = 2(1, 0) = (2, 0) \neq (4, 0)$. Homogeneity fails.

1.2 Determine whether $T: \mathbb{R}^3 \to \mathbb{R}^2$ defined by $T(x, y, z) = (x - 2y + z,\; 3x + y)$ is a linear transformation. If yes, find its standard matrix.

Click for solution Yes, $T$ is linear (each component is a linear combination of variables). Standard matrix: $A = \begin{bmatrix} 1 & -2 & 1 \\ 3 & 1 & 0 \end{bmatrix}$.

1.3 Show that $T: M_{2 \times 2} \to M_{2 \times 2}$ defined by $T(A) = A^T$ (transpose) is a linear transformation.

Click for solution $(A + B)^T = A^T + B^T$ and $(cA)^T = cA^T$. Both additivity and homogeneity hold, so $T$ is linear.

1.4 Determine whether $T: \mathbb{R}^2 \to \mathbb{R}$ defined by $T(x, y) = xy$ is a linear transformation.

Click for solution No. $T(1, 1) = 1$ but $T(1, 0) + T(0, 1) = 0 + 0 = 0 \neq 1$. Additivity fails.

1.5 Prove that $T: \mathbb{R}^3 \to \mathbb{R}^3$ defined by $T(x, y, z) = (x + y,\; y + z,\; x + z)$ is a linear transformation and find its standard matrix.

Click for solution Each component is a linear combination of $x, y, z$, so $T$ is linear. $A = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \end{bmatrix}$.

Problem Set 2: Finding Kernel and Image

2.1 For the linear transformation $T: \mathbb{R}^3 \to \mathbb{R}^2$ with matrix $A = \begin{bmatrix} 1 & 2 & 1 \ 2 & 4 & 2 \end{bmatrix}$, find $\ker(T)$, $\text{nullity}(T)$, $\text{Im}(T)$, and $\text{rank}(T)$.

Click for solution Row reduce: $\begin{bmatrix} 1 & 2 & 1 \\ 0 & 0 & 0 \end{bmatrix}$. Two free variables. $\ker(T) = \text{span}\lbrace(-2, 1, 0), (-1, 0, 1)\rbrace$, $\text{nullity}(T) = 2$. $\text{Im}(T) = \text{span}\lbrace(1, 2)\rbrace$, $\text{rank}(T) = 1$. Check: $1 + 2 = 3$ ✓.

2.2 Find the kernel and image of the linear transformation $T: \mathbb{R}^4 \to \mathbb{R}^3$ with matrix $A = \begin{bmatrix} 1 & 0 & 2 & 1 \ 0 & 1 & -1 & 3 \ 2 & 1 & 3 & 5 \end{bmatrix}$.

Click for solution After row reduction: $\text{rank}(T) = 2$ (third row becomes zero), $\text{nullity}(T) = 4 - 2 = 2$. $\ker(T) = \text{span}\lbrace(-2, 1, 1, 0), (-1, -3, 0, 1)\rbrace$. $\text{Im}(T) = \text{span}\lbrace(1, 0, 2), (0, 1, 1)\rbrace$.

2.3 Let $T: \mathbb{R}^3 \to \mathbb{R}^3$ be defined by $T(x, y, z) = (x + y,\; y + z,\; x + z)$. Find $\ker(T)$ and $\text{Im}(T)$.

Click for solution $A = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \end{bmatrix}$, $\det(A) = 2 \neq 0$. So $\ker(T) = \lbrace\vec{0}\rbrace$ and $\text{Im}(T) = \mathbb{R}^3$.

2.4 For the projection $T: \mathbb{R}^3 \to \mathbb{R}^3$ onto the $xz$-plane (i.e., $T(x, y, z) = (x, 0, z)$), find the kernel, nullity, image, and rank.

Click for solution $\ker(T) = \text{span}\lbrace(0, 1, 0)\rbrace$ ($y$-axis), $\text{nullity}(T) = 1$. $\text{Im}(T) = xz\text{-plane}$, $\text{rank}(T) = 2$. Check: $1 + 2 = 3$ ✓.

2.5 Let $T: P_2 \to P_2$ be defined by $T(p(x)) = p(x) + p’(x)$. Find the kernel of $T$.

Click for solution If $p(x) + p'(x) = 0$, then $p(x) = ce^{-x}$. But $e^{-x} \notin P_2$, so the only solution is $p = 0$. Thus $\ker(T) = \lbrace 0 \rbrace$.

Problem Set 3: Rank-Nullity Theorem Applications

3.1 If $T: \mathbb{R}^5 \to \mathbb{R}^3$ has $\text{nullity}(T) = 2$, what is $\text{rank}(T)$?

Click for solution By Rank-Nullity: $\text{rank}(T) = 5 - 2 = 3$.

3.2 A linear transformation $T: \mathbb{R}^6 \to \mathbb{R}^4$ has $\text{rank}(T) = 3$. What is $\dim(\ker(T))$?

Click for solution $\dim(\ker(T)) = 6 - 3 = 3$.

3.3 Let $T: \mathbb{R}^4 \to \mathbb{R}^3$ be a linear transformation with matrix $A = \begin{bmatrix} 1 & 2 & 0 & -1 \ 0 & 1 & 1 & 2 \ 1 & 3 & 1 & 1 \end{bmatrix}$. Find the rank and nullity of $T$, and verify the Rank-Nullity Theorem.

Click for solution Row reduce to find $\text{rank}(A) = 2$ (R3 = R1 + R2 after reduction), $\text{nullity}(T) = 4 - 2 = 2$. Verified: $2 + 2 = 4$ ✓.

3.4 Suppose $T: V \to W$ is a linear transformation where $\dim(V) = 7$, $\dim(W) = 5$, and $\ker(T) = \lbrace\vec{0}\rbrace$. What is $\dim(\text{Im}(T))$? Is $T$ surjective?

Click for solution $\dim(\text{Im}(T)) = 7 - 0 = 7$. But $\dim(W) = 5 < 7$ — contradiction. This situation is impossible: $\text{Im}(T) \subseteq W$ forces $\text{rank}(T) \leq 5$, so $\text{nullity}(T) \geq 2$.

3.5 Let $T: \mathbb{R}^n \to \mathbb{R}^m$ be a linear transformation with $m < n$. Show that $T$ cannot be injective.

Click for solution By Rank-Nullity: $n = \text{rank}(T) + \text{nullity}(T) \leq m + \text{nullity}(T)$. Since $m < n$, we get $\text{nullity}(T) \geq n - m > 0$, so $\ker(T) \neq \lbrace\vec{0}\rbrace$ and $T$ is not injective.

Problem Set 4: Injectivity and Surjectivity

4.1 Determine whether the linear transformation $T: \mathbb{R}^3 \to \mathbb{R}^3$ with matrix $A = \begin{bmatrix} 1 & 0 & 2 \ 0 & 1 & -1 \ 1 & 1 & 1 \end{bmatrix}$ is injective, surjective, or bijective.

Click for solution $\det(A) = 1(1+1) - 0 + 2(0-1) = 0$. So $\text{rank}(A) = 2$, $\text{nullity}(A) = 1$. Not injective, not surjective.

4.2 Let $T: \mathbb{R}^3 \to \mathbb{R}^2$ be defined by $T(x, y, z) = (x + y,\; y + z)$. Is $T$ injective? Is $T$ surjective?

Click for solution Not injective ($\dim(\mathbb{R}^3) > \dim(\mathbb{R}^2)$ implies $\text{nullity} \geq 1$). Surjective: $A = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}$ has $\text{rank} = 2 = \dim(\mathbb{R}^2)$ ✓.

4.3 Find conditions on $a$ and $b$ such that the linear transformation $T: \mathbb{R}^2 \to \mathbb{R}^2$ with matrix $A = \begin{bmatrix} a & 2 \ 3 & b \end{bmatrix}$ is bijective.

Click for solution Bijective iff $\det(A) = ab - 6 \neq 0$, i.e., $ab \neq 6$.

4.4 Show that the differentiation map $D: P_n \to P_{n-1}$ defined by $D(p) = p’$ is always surjective but never injective for $n \geq 1$.

Click for solution Surjective: every polynomial in $P_{n-1}$ can be obtained by differentiating its antiderivative in $P_n$. Not injective: $\ker(D)$ contains all constant polynomials (dimension 1).

4.5 Let $T: \mathbb{R}^4 \to \mathbb{R}^4$ be a linear transformation. If $\text{rank}(T) = 4$, prove that $T$ is bijective.

Click for solution $\text{nullity}(T) = 4 - 4 = 0 \Rightarrow T$ is injective. Since $\dim(\mathbb{R}^4) = \dim(\mathbb{R}^4)$, injectivity implies surjectivity, hence $T$ is bijective.

Problem Set 5: Geometric Transformations

5.1 Find the standard matrix of the linear transformation $T: \mathbb{R}^2 \to \mathbb{R}^2$ that reflects vectors across the line $y = x$.

Click for solution $T(e_1) = T(1,0) = (0,1)$ and $T(e_2) = T(0,1) = (1,0)$. So $A = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}$.

5.2 Find the matrix of the linear transformation $T: \mathbb{R}^2 \to \mathbb{R}^2$ that rotates vectors counterclockwise by $45°$ and then reflects across the $x$-axis.

Click for solution Rotation by $45°$: $R = \begin{bmatrix} \frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2} \\ \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2} \end{bmatrix}$. Reflection across $x$-axis: $S = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}$. Combined: $A = SR = \begin{bmatrix} \frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2} \\ -\frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2} \end{bmatrix}$.

5.3 Let $T: \mathbb{R}^3 \to \mathbb{R}^3$ project vectors onto the line $L = \text{span}\lbrace(1, 1, 1)\rbrace$. Find the kernel and image of $T$.

Click for solution $\ker(T) = \lbrace(x,y,z) : x+y+z = 0\rbrace$ (plane perpendicular to $(1,1,1)$). $\text{Im}(T) = \text{span}\lbrace(1,1,1)\rbrace$.

5.4 Find the standard matrix for the linear transformation $T: \mathbb{R}^3 \to \mathbb{R}^3$ that reflects vectors across the plane $x + y + z = 0$.

Click for solution Normal $n = (1,1,1)$. Reflection formula: $A = I - 2\frac{nn^T}{n^Tn} = I - \frac{2}{3}\begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix} = \frac{1}{3}\begin{bmatrix} 1 & -2 & -2 \\ -2 & 1 & -2 \\ -2 & -2 & 1 \end{bmatrix}$.

5.5 Describe geometrically the linear transformation $T: \mathbb{R}^3 \to \mathbb{R}^3$ with matrix $A = \begin{bmatrix} 2 & 0 & 0 \ 0 & 2 & 0 \ 0 & 0 & 2 \end{bmatrix}$.

Click for solution $T$ is uniform scaling by factor 2 (dilation from the origin). Every vector is stretched to twice its length in the same direction.

2 — Inner Product Spaces & Gram-Schmidt Orthogonalization

2.1 Definition of Inner Product

Definition: An inner product on a real vector space $V$ is a function $\langle \cdot, \cdot \rangle : V \times V \to \mathbb{R}$ satisfying, for all $u, v, w \in V$ and $\alpha \in \mathbb{R}$:

  1. Symmetry: $\langle u, v \rangle = \langle v, u \rangle$
  2. Linearity in first argument: $\langle \alpha u + v, w \rangle = \alpha\langle u, w \rangle + \langle v, w \rangle$
  3. Positive-definiteness: $\langle v, v \rangle \geq 0$, with equality iff $v = \vec{0}$

A vector space equipped with an inner product is called an inner product space.

2.2 Standard Inner Products

Space Inner Product Formula
$\mathbb{R}^n$ Standard dot product $\langle u, v \rangle = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n = u^T v$
$\mathbb{C}^n$ Standard Hermitian $\langle u, v \rangle = \sum_{i=1}^n \overline{u_i} v_i = u^* v$
$C[a,b]$ Function inner product $\langle f, g \rangle = \int_a^b f(x)\, g(x)\, dx$

2.3 Norm, Cauchy-Schwarz & Triangle Inequality

Induced Norm: $\lVert v \rVert = \sqrt{\langle v, v \rangle}$

Cauchy-Schwarz Inequality: For any $u, v$ in an inner product space:

\[\lvert \langle u, v \rangle \rvert \leq \lVert u \rVert \cdot \lVert v \rVert\]

Equality holds iff $u$ and $v$ are linearly dependent.

Triangle Inequality: For any $u, v$ in an inner product space:

\[\lVert u + v \rVert \leq \lVert u \rVert + \lVert v \rVert\]

2.4 Orthogonality & Orthonormal Sets

  • Orthogonal: Vectors $u, v$ are orthogonal if $\langle u, v \rangle = 0$.
  • Orthogonal set: A set $\lbrace v_1, \ldots, v_k \rbrace$ where $\langle v_i, v_j \rangle = 0$ for all $i \neq j$.
  • Orthonormal set: An orthogonal set where $\lVert v_i \rVert = 1$ for every $i$.

Theorem: An orthogonal set of nonzero vectors is linearly independent.

Pythagorean Theorem: If $u \perp v$ (orthogonal), then $\lVert u + v \rVert^2 = \lVert u \rVert^2 + \lVert v \rVert^2$.

2.5 Orthogonal Projection

Projection onto a single vector: The orthogonal projection of $u$ onto nonzero vector $v$:

\[\text{proj}_v(u) = \frac{\langle u, v \rangle}{\langle v, v \rangle}\, v\]

Projection onto a subspace: If $\lbrace v_1, \ldots, v_k \rbrace$ is an orthogonal basis for subspace $W$:

\[\text{proj}_W(u) = \sum_{i=1}^k \frac{\langle u, v_i \rangle}{\langle v_i, v_i \rangle}\, v_i\]

2.6 The Gram-Schmidt Process

Gram-Schmidt Algorithm: Given a basis $\lbrace v_1, v_2, \ldots, v_n \rbrace$ for a subspace, produce an orthogonal basis $\lbrace w_1, w_2, \ldots, w_n \rbrace$:

\[w_1 = v_1\] \[w_2 = v_2 - \frac{\langle v_2, w_1 \rangle}{\langle w_1, w_1 \rangle}\, w_1\] \[w_3 = v_3 - \frac{\langle v_3, w_1 \rangle}{\langle w_1, w_1 \rangle}\, w_1 - \frac{\langle v_3, w_2 \rangle}{\langle w_2, w_2 \rangle}\, w_2\]

In general:

\[w_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k, w_j \rangle}{\langle w_j, w_j \rangle}\, w_j\]

To get an orthonormal basis, normalize: $e_k = \frac{w_k}{\lVert w_k \rVert}$.

Geometric Intuition: At each step, we subtract off the components of $v_k$ in the directions of all previous orthogonal vectors, leaving only the part perpendicular to the span of $w_1, \ldots, w_{k-1}$.

2.7 Modified Gram-Schmidt (Numerical Stability)

In exact arithmetic, classical and modified Gram-Schmidt produce the same result. However, in floating-point computation, the Modified Gram-Schmidt (MGS) is numerically superior:

Instead of computing all projections using the original $v_k$, MGS updates $v_k$ sequentially:

\[v_k^{(1)} = v_k - \frac{\langle v_k, w_1 \rangle}{\langle w_1, w_1 \rangle} w_1, \quad v_k^{(2)} = v_k^{(1)} - \frac{\langle v_k^{(1)}, w_2 \rangle}{\langle w_2, w_2 \rangle} w_2, \quad \ldots, \quad w_k = v_k^{(k-1)}\]

2.8 QR Decomposition

Theorem: Any $m \times n$ matrix $A$ with linearly independent columns can be factored as $A = QR$ where:

  • $Q$ is $m \times n$ with orthonormal columns
  • $R$ is $n \times n$ upper triangular with positive diagonal entries

Construction via Gram-Schmidt: Apply Gram-Schmidt to the columns of $A$ to get $Q$. Then $R = Q^T A$ (the upper triangular matrix of inner products).

2.9 Best Approximation Theorem

Best Approximation Theorem: If $W$ is a finite-dimensional subspace of an inner product space $V$ and $u \in V$, then $\text{proj}_W(u)$ is the unique vector in $W$ closest to $u$:

\[\lVert u - \text{proj}_W(u) \rVert \leq \lVert u - w \rVert \quad \text{for all } w \in W\]

2.10 Worked Examples — Inner Products & Gram-Schmidt

Example 1: Inner Product in $\mathbb{R}^4$

Problem: Compute $\langle u, v \rangle$ for $u = (1, -2, 3, 1)$ and $v = (2, 1, 0, -3)$ in $\mathbb{R}^4$.

Solution:

\[\langle u, v \rangle = 1(2) + (-2)(1) + 3(0) + 1(-3) = 2 - 2 + 0 - 3 = -3\]

Since $\langle u, v \rangle \neq 0$, $u$ and $v$ are not orthogonal.

Example 2: Gram-Schmidt in $\mathbb{R}^3$

Problem: Apply Gram-Schmidt to the columns of $A = \begin{bmatrix} 1 & 1 & 0 \ 1 & 0 & 1 \ 0 & 1 & 1 \end{bmatrix}$ to obtain an orthonormal basis.

Solution:

Step 1: $w_1 = v_1 = (1, 1, 0)^T$

Step 2: $w_2 = v_2 - \frac{\langle v_2, w_1 \rangle}{\langle w_1, w_1 \rangle} w_1$

\[\langle v_2, w_1 \rangle = 1(1) + 0(1) + 1(0) = 1, \quad \langle w_1, w_1 \rangle = 1 + 1 + 0 = 2\] \[w_2 = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} - \frac{1}{2}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 1/2 \\ -1/2 \\ 1 \end{pmatrix}\]

Step 3: $w_3 = v_3 - \frac{\langle v_3, w_1 \rangle}{\langle w_1, w_1 \rangle} w_1 - \frac{\langle v_3, w_2 \rangle}{\langle w_2, w_2 \rangle} w_2$

\[\langle v_3, w_1 \rangle = 0(1) + 1(1) + 1(0) = 1, \quad \langle v_3, w_2 \rangle = 0(1/2) + 1(-1/2) + 1(1) = 1/2\] \[\langle w_2, w_2 \rangle = 1/4 + 1/4 + 1 = 3/2\] \[w_3 = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \frac{1}{2}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} - \frac{1/2}{3/2}\begin{pmatrix} 1/2 \\ -1/2 \\ 1 \end{pmatrix} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \begin{pmatrix} 1/2 \\ 1/2 \\ 0 \end{pmatrix} - \frac{1}{3}\begin{pmatrix} 1/2 \\ -1/2 \\ 1 \end{pmatrix}\] \[= \begin{pmatrix} -1/2 - 1/6 \\ 1/2 + 1/6 \\ 1 - 1/3 \end{pmatrix} = \begin{pmatrix} -2/3 \\ 2/3 \\ 2/3 \end{pmatrix}\]

Normalizing:

\[e_1 = \frac{1}{\sqrt{2}}(1, 1, 0), \quad e_2 = \frac{1}{\sqrt{6}}(1, -1, 2), \quad e_3 = \frac{1}{\sqrt{3}}(-1, 1, 1)\]

Example 3: Projection onto a Subspace

Problem: Project $u = (6, 3, 2)$ onto $v = (1, 2, 2)$ in $\mathbb{R}^3$.

Solution:

\[\text{proj}_v(u) = \frac{\langle u, v \rangle}{\langle v, v \rangle} v = \frac{6 + 6 + 4}{1 + 4 + 4}(1, 2, 2) = \frac{16}{9}(1, 2, 2) = \left(\frac{16}{9}, \frac{32}{9}, \frac{32}{9}\right)\]

Example 4: Inner Product on Function Space

Problem: In $C[0,1]$ with $\langle f, g \rangle = \int_0^1 f(x)g(x)\,dx$, compute $\langle f, g \rangle$ where $f(x) = x$ and $g(x) = x^2$.

Solution:

\[\langle f, g \rangle = \int_0^1 x \cdot x^2\,dx = \int_0^1 x^3\,dx = \left[\frac{x^4}{4}\right]_0^1 = \frac{1}{4}\]

Norms: $\lVert f \rVert = \sqrt{\int_0^1 x^2\,dx} = \sqrt{1/3} = \frac{1}{\sqrt{3}}$, $\lVert g \rVert = \sqrt{\int_0^1 x^4\,dx} = \sqrt{1/5} = \frac{1}{\sqrt{5}}$.

Example 5: QR Decomposition

Problem: Find the QR decomposition of $A = \begin{bmatrix} 1 & 1 & 0 \ 1 & 0 & 1 \ 0 & 1 & 1 \end{bmatrix}$.

Solution:

From Example 2, the orthonormal vectors are:

\[e_1 = \frac{1}{\sqrt{2}}(1, 1, 0), \quad e_2 = \frac{1}{\sqrt{6}}(1, -1, 2), \quad e_3 = \frac{1}{\sqrt{3}}(-1, 1, 1)\] \[Q = \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} & \frac{-1}{\sqrt{3}} \\[4pt] \frac{1}{\sqrt{2}} & \frac{-1}{\sqrt{6}} & \frac{1}{\sqrt{3}} \\[4pt] 0 & \frac{2}{\sqrt{6}} & \frac{1}{\sqrt{3}} \end{bmatrix}\]

$R = Q^T A$:

\[R = \begin{bmatrix} \sqrt{2} & \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\[4pt] 0 & \sqrt{\frac{3}{2}} & \frac{1}{\sqrt{6}} \\[4pt] 0 & 0 & \frac{2}{\sqrt{3}} \end{bmatrix}\]

Practice Questions — Inner Product Spaces

Problem Set 1: Inner Products and Orthogonality

1.1 In $\mathbb{R}^4$, compute $\langle u, v \rangle$ for $u = (1, -2, 3, 1)$ and $v = (2, 2, 1, -3)$. Are they orthogonal?

Click for solution $\langle u, v \rangle = 2 - 4 + 3 - 3 = -2 \neq 0$. Not orthogonal.

1.2 Verify the Cauchy-Schwarz inequality for $u = (1, 2, 3)$ and $v = (4, -1, 2)$ in $\mathbb{R}^3$.

Click for solution $\lvert\langle u, v \rangle\rvert = \lvert 4 - 2 + 6 \rvert = 8$. $\lVert u \rVert \cdot \lVert v \rVert = \sqrt{14}\sqrt{21} = \sqrt{294} \approx 17.15$. Since $8 < 17.15$, inequality holds $\checkmark$.

1.3 Find a vector in $\mathbb{R}^3$ that is orthogonal to both $u = (1, 2, 3)$ and $v = (4, -1, 2)$.

Click for solution Take the cross product: $u \times v = (2 \cdot 2 - 3 \cdot (-1),\; 3 \cdot 4 - 1 \cdot 2,\; 1 \cdot (-1) - 2 \cdot 4) = (7, 10, -9)$. Verify: $\langle(7,10,-9), u\rangle = 7 + 20 - 27 = 0$ ✓.

1.4 In $C[0,1]$ with $\langle f, g \rangle = \int_0^1 f(x)g(x)\,dx$, show that $f(x) = \sin(\pi x)$ and $g(x) = \cos(\pi x)$ are orthogonal.

Click for solution $\langle f, g \rangle = \int_0^1 \sin(\pi x)\cos(\pi x)\,dx = \frac{1}{2}\int_0^1 \sin(2\pi x)\,dx = \frac{1}{2}\left[-\frac{\cos(2\pi x)}{2\pi}\right]_0^1 = \frac{1}{2}\left(-\frac{1}{2\pi} + \frac{1}{2\pi}\right) = 0$ $\checkmark$.

1.5 If $u, v$ are orthonormal vectors, prove that $\lVert u - v \rVert = \sqrt{2}$.

Click for solution $\lVert u - v \rVert^2 = \langle u - v, u - v \rangle = \lVert u \rVert^2 - 2\langle u, v \rangle + \lVert v \rVert^2 = 1 - 0 + 1 = 2$, so $\lVert u - v \rVert = \sqrt{2}$.

Problem Set 2: Gram-Schmidt Process

2.1 Apply Gram-Schmidt to $\lbrace(1, 1, 1), (1, 1, 0), (1, 0, 0)\rbrace$ in $\mathbb{R}^3$ to find an orthonormal basis.

Click for solution $w_1 = (1, 1, 1)$, $w_2 = (1, 1, 0) - \frac{2}{3}(1, 1, 1) = (\frac{1}{3}, \frac{1}{3}, -\frac{2}{3})$, $w_3 = (1, 0, 0) - \frac{1}{3}(1, 1, 1) - \frac{1/3}{2/3}(\frac{1}{3}, \frac{1}{3}, -\frac{2}{3}) = (\frac{1}{2}, -\frac{1}{2}, 0)$. Orthonormal: $\lbrace\frac{1}{\sqrt{3}}(1,1,1),\; \frac{1}{\sqrt{6}}(1,1,-2),\; \frac{1}{\sqrt{2}}(1,-1,0)\rbrace$.

2.2 Apply Gram-Schmidt to $\lbrace(1, 0, 1, 0),\; (1, 1, 0, 0),\; (0, 1, 1, 1)\rbrace$ in $\mathbb{R}^4$.

Click for solution $w_1 = (1,0,1,0)$. $w_2 = (1,1,0,0) - \frac{1}{2}(1,0,1,0) = (\frac{1}{2}, 1, -\frac{1}{2}, 0)$. $w_3 = (0,1,1,1) - \frac{1}{2}(1,0,1,0) - \frac{1/2}{3/2}(\frac{1}{2}, 1, -\frac{1}{2}, 0) = (-\frac{2}{3}, \frac{2}{3}, \frac{2}{3}, 1)$. After normalizing, three orthonormal vectors in $\mathbb{R}^4$.

2.3 Apply Gram-Schmidt to $\lbrace 1, x, x^2 \rbrace$ in $C[-1, 1]$ with $\langle f, g \rangle = \int_{-1}^1 f(x)g(x)\,dx$ to obtain the first three Legendre polynomials.

Click for solution $p_0 = 1$. $p_1 = x - \frac{\langle x, 1\rangle}{\langle 1, 1\rangle} \cdot 1 = x$ (since $\int_{-1}^1 x\,dx = 0$). $p_2 = x^2 - \frac{\langle x^2, 1\rangle}{\langle 1, 1\rangle} - \frac{\langle x^2, x\rangle}{\langle x, x\rangle}x = x^2 - \frac{1}{3}$ (since $\int_{-1}^1 x^2\,dx = \frac{2}{3}$, $\int_{-1}^1 1\,dx = 2$, and $\int_{-1}^1 x^3\,dx = 0$). Legendre polynomials (scaled): $\lbrace 1,\; x,\; x^2 - \frac{1}{3}\rbrace$.

2.4 Find a vector $w$ in $\mathbb{R}^3$ such that $\lbrace(1, 2, 1),\; (1, 0, -1),\; w\rbrace$ is an orthogonal set.

Click for solution $w$ must satisfy $\langle w, (1,2,1)\rangle = 0$ and $\langle w, (1,0,-1)\rangle = 0$. From the cross product: $w = (1,2,1) \times (1,0,-1) = (-2, 2, -2) = -2(1, -1, 1)$. So $w = (1, -1, 1)$.

2.5 Apply Gram-Schmidt to the columns of $A = \begin{bmatrix} 1 & 1 \ 1 & 0 \ 0 & 1 \end{bmatrix}$ to find a matrix $Q$ with orthonormal columns.

Click for solution $v_1 = (1,1,0)$, $e_1 = \frac{1}{\sqrt{2}}(1,1,0)$. $v_2 = (1,0,1) - \frac{1}{2}(1,1,0) = (\frac{1}{2}, -\frac{1}{2}, 1)$, $e_2 = \frac{1}{\sqrt{3/2}}(\frac{1}{2}, -\frac{1}{2}, 1) = \frac{1}{\sqrt{6}}(1,-1,2)$. $Q = \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{6}} \\ 0 & \frac{2}{\sqrt{6}} \end{bmatrix}$.

Problem Set 3: Projections and Best Approximation

3.1 Find the projection of $u = (2, 3, 4)$ onto $v = (1, 2, 2)$.

Click for solution $\text{proj}_v(u) = \frac{\langle u, v\rangle}{\langle v, v\rangle}v = \frac{2+6+8}{1+4+4}(1,2,2) = \frac{16}{9}(1,2,2) = (\frac{16}{9}, \frac{32}{9}, \frac{32}{9})$.

3.2 Find the projection of $u = (1, 2, 3, 4)$ onto $W = \text{span}\lbrace(1, 0, 1, 0), (0, 1, 0, 1)\rbrace$.

Click for solution Basis vectors are already orthogonal. $\text{proj}_W(u) = \frac{\langle u, v_1 \rangle}{\langle v_1, v_1 \rangle}v_1 + \frac{\langle u, v_2 \rangle}{\langle v_2, v_2 \rangle}v_2 = \frac{4}{2}(1,0,1,0) + \frac{6}{2}(0,1,0,1) = (2, 3, 2, 3)$.

3.3 In $C[0, 1]$, find the best linear approximation $f(x) \approx a + bx$ to $g(x) = e^x$ by projecting onto $\text{span}\lbrace 1, x \rbrace$.

Click for solution Apply Gram-Schmidt to $\lbrace 1, x \rbrace$ first, then project. Using $\langle f, g\rangle = \int_0^1 f\,g\,dx$: $\langle e^x, 1\rangle = e - 1$, $\langle 1, 1\rangle = 1$. After orthogonal projection computations: $f(x) \approx (4e-10) + (18-6e)x \approx 0.873 + 1.690x$.

3.4 Find the distance from $u = (1, 2, 3)$ to $W = \text{span}\lbrace(1, 0, 1), (0, 1, 0)\rbrace$.

Click for solution $\text{proj}_W(u) = \frac{4}{2}(1,0,1) + \frac{2}{1}(0,1,0) = (2, 2, 2)$. Distance $= \lVert u - \text{proj}_W(u)\rVert = \lVert(-1, 0, 1)\rVert = \sqrt{2}$.

3.5 Prove that for any subspace $W$ and any vector $u$, $\lVert\text{proj}_W(u)\rVert \leq \lVert u \rVert$.

Click for solution Write $u = \text{proj}_W(u) + (u - \text{proj}_W(u))$ where the two components are orthogonal. By the Pythagorean theorem: $\lVert u \rVert^2 = \lVert\text{proj}_W(u)\rVert^2 + \lVert u - \text{proj}_W(u)\rVert^2 \geq \lVert\text{proj}_W(u)\rVert^2$.

Problem Set 4: QR Decomposition

4.1 Find the QR decomposition of $A = \begin{bmatrix} 1 & 1 \ 1 & 0 \ 0 & 1 \end{bmatrix}$.

Click for solution Apply Gram-Schmidt to columns of $A$: $Q = \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{6}} \\ 0 & \frac{2}{\sqrt{6}} \end{bmatrix}$, $R = \begin{bmatrix} \sqrt{2} & \frac{1}{\sqrt{2}} \\ 0 & \frac{3}{\sqrt{6}} \end{bmatrix}$.

4.2 Find the QR decomposition of the $4 \times 3$ matrix $A = \begin{bmatrix} 1 & -1 & 4 \ 1 & 4 & -2 \ 1 & 4 & 2 \ 1 & -1 & 0 \end{bmatrix}$.

Click for solution Apply Gram-Schmidt to columns. $Q$ is $4 \times 3$ with orthonormal columns, $R$ is $3 \times 3$ upper triangular. The entries are computed by: $r_{ij} = \langle a_j, q_i \rangle$ and $Q$ columns are the normalized orthogonal vectors.

4.3 Use QR decomposition to solve the least squares problem: find $x$ that minimizes $\lVert Ax - b \rVert$ where $A = \begin{bmatrix} 1 & 2 \ 1 & 3 \ 1 & 4 \end{bmatrix}$, $b = \begin{bmatrix} 2 \ 3 \ 5 \end{bmatrix}$.

Click for solution Compute $A = QR$, then solve $Rx = Q^Tb$. This avoids forming $A^TA$ and gives a numerically stable least squares solution.

4.4 Show that if $A = QR$ is the QR decomposition, then $A^TA = R^TR$.

Click for solution $A^TA = (QR)^T(QR) = R^TQ^TQR = R^TIR = R^TR$, since $Q^TQ = I$ (orthonormal columns).

4.5 Find the QR decomposition of the Vandermonde matrix $A = \begin{bmatrix} 1 & 0 & 0 \ 1 & 1 & 1 \ 1 & 2 & 4 \end{bmatrix}$.

Click for solution Apply Gram-Schmidt to columns of $A$. The result gives orthogonal polynomials evaluated at $\lbrace 0, 1, 2\rbrace$. $Q$ has orthonormal columns and $R$ is upper triangular with $R = Q^TA$.

Problem Set 5: Advanced Applications

5.1 Show that the set $\lbrace e^{inx} : n \in \mathbb{Z} \rbrace$ is orthogonal in $C[-\pi, \pi]$ with inner product $\langle f, g \rangle = \int_{-\pi}^{\pi} f(x)\overline{g(x)}\,dx$.

Click for solution $\langle e^{imx}, e^{inx} \rangle = \int_{-\pi}^{\pi} e^{i(m-n)x}\,dx$. For $m \neq n$: $= \frac{e^{i(m-n)x}}{i(m-n)}\Big|_{-\pi}^{\pi} = 0$ (since $e^{i(m-n)\pi} = e^{-i(m-n)\pi}$ for integer differences). For $m = n$: $= 2\pi$.

5.2 Prove that if $\lbrace e_1, e_2, \ldots, e_n \rbrace$ is an orthonormal basis for $V$, then for any $u \in V$: $u = \sum_{k=1}^{n} \langle u, e_k \rangle\, e_k$ (Parseval’s expansion).

Click for solution Write $u = \sum c_k e_k$. Taking inner product with $e_j$: $\langle u, e_j \rangle = \sum c_k \langle e_k, e_j \rangle = c_j$ (by orthonormality). Hence $u = \sum \langle u, e_k \rangle e_k$.

5.3 Let $W$ be a subspace of a finite-dimensional inner product space $V$. Prove that $V = W \oplus W^{\perp}$.

Click for solution Show $W \cap W^{\perp} = \lbrace 0 \rbrace$: if $v \in W \cap W^{\perp}$, then $\langle v, v \rangle = 0$, so $v = 0$. Show $V = W + W^{\perp}$: for any $v \in V$, write $v = \text{proj}_W(v) + (v - \text{proj}_W(v))$. The first part is in $W$, the second is in $W^{\perp}$. Since $\dim(W) + \dim(W^{\perp}) = \dim(V)$ and $W \cap W^{\perp} = \lbrace 0\rbrace$, we get $V = W \oplus W^{\perp}$.

5.4 Show that the Modified Gram-Schmidt and Classical Gram-Schmidt algorithms produce the same orthogonal vectors in exact arithmetic.

Click for solution Both produce the same span at each step. MGS updates projections immediately (subtracting from the already-orthogonalized vector), but in exact arithmetic this is mathematically equivalent to CGS. The difference appears only in finite-precision arithmetic where MGS has better numerical stability.

5.5 Prove Bessel’s inequality: If $\lbrace e_1, e_2, \ldots, e_n \rbrace$ is an orthonormal set in an inner product space and $u$ is any vector, then $\sum_{k=1}^{n} \lvert\langle u, e_k \rangle\rvert^2 \leq \lVert u \rVert^2$.

Click for solution Let $w = u - \sum_{k=1}^n \langle u, e_k \rangle e_k$. Then $w \perp e_k$ for all $k$, and by the Pythagorean theorem: $\lVert u \rVert^2 = \lVert w \rVert^2 + \sum_{k=1}^n \lvert\langle u, e_k\rangle\rvert^2 \geq \sum_{k=1}^n \lvert\langle u, e_k\rangle\rvert^2$ (since $\lVert w \rVert^2 \geq 0$).

3 — Eigenvalues, Eigenvectors & Diagonalization

3.1 Definition of Eigenvalues & Eigenvectors

Definition: Let $A$ be an $n \times n$ matrix. A nonzero vector $v \in \mathbb{R}^n$ is called an eigenvector of $A$ if there exists a scalar $\lambda$ such that:

\[Av = \lambda v\]

The scalar $\lambda$ is called an eigenvalue of $A$ corresponding to eigenvector $v$.

Geometric Interpretation

When $A$ acts on eigenvector $v$, the result is simply a scaled version of $v$ — the direction is preserved, only the magnitude changes by factor $\lambda$.

Key Observations:

  • $v \neq 0$ by definition (zero vector excluded)
  • If $v$ is an eigenvector, so is $cv$ for any $c \neq 0$
  • $\lambda > 0$: stretches in same direction; $\lambda < 0$: reverses direction; $\lambda = 0$: maps to zero ($A$ is singular)

Why Study Eigenvalues?

  1. Simplify Matrix Powers: If $A = PDP^{-1}$, then $A^k = PD^k P^{-1}$ — trivial to compute
  2. Differential Equations: Solutions of $\frac{dx}{dt} = Ax$ involve $e^{\lambda t}$ from eigenvalues
  3. Vibration Analysis: Natural frequencies are eigenvalues of stiffness/mass matrices
  4. Data Science (PCA): Directions of maximum variance are eigenvectors of covariance matrices
  5. Google PageRank: Page importance is the dominant eigenvector of a transition matrix
  6. Stability Analysis: System stability depends on eigenvalue location in the complex plane

3.2 The Characteristic Equation

Starting from $Av = \lambda v$:

\[Av - \lambda v = 0 \implies (A - \lambda I)v = 0\]

For a nontrivial solution $v \neq 0$, the matrix $(A - \lambda I)$ must be singular:

Characteristic Equation: $\det(A - \lambda I) = 0$

Characteristic Polynomial: $p(\lambda) = \det(A - \lambda I)$ — a degree $n$ polynomial whose roots are the eigenvalues.

Properties of the characteristic polynomial for $n \times n$ matrix $A$:

  1. Degree is exactly $n$; leading coefficient is $(-1)^n$
  2. $p(0) = \det(A)$
  3. Sum of eigenvalues = $\text{tr}(A) = a_{11} + a_{22} + \cdots + a_{nn}$
  4. Product of eigenvalues = $\det(A)$
  5. By the Fundamental Theorem of Algebra, $p(\lambda)$ has exactly $n$ roots (counting multiplicities) in $\mathbb{C}$

3.3 Eigenspaces & Multiplicities

Eigenspace: For eigenvalue $\lambda$:

\[E_\lambda = \text{null}(A - \lambda I) = \lbrace v \in \mathbb{R}^n : (A - \lambda I)v = 0 \rbrace\]
  • Geometric multiplicity of $\lambda$: $\dim(E_\lambda)$
  • Algebraic multiplicity of $\lambda$: its multiplicity as a root of $p(\lambda)$

Important Inequality: For any eigenvalue $\lambda$:

\[1 \leq \text{geometric multiplicity} \leq \text{algebraic multiplicity}\]

3.4 Properties of Eigenvectors

Theorem: Eigenvectors corresponding to distinct eigenvalues are linearly independent.

Proof: Suppose $v_1, \ldots, v_k$ are eigenvectors with distinct eigenvalues $\lambda_1, \ldots, \lambda_k$, and assume for contradiction they are linearly dependent. Let $m$ be the smallest index such that $v_m = c_1 v_1 + \cdots + c_{m-1} v_{m-1}$.

Applying $A$: $\lambda_m v_m = c_1 \lambda_1 v_1 + \cdots + c_{m-1} \lambda_{m-1} v_{m-1}$.

Multiplying the original by $\lambda_m$: $\lambda_m v_m = c_1 \lambda_m v_1 + \cdots + c_{m-1} \lambda_m v_{m-1}$.

Subtracting: $0 = c_1(\lambda_1 - \lambda_m)v_1 + \cdots + c_{m-1}(\lambda_{m-1} - \lambda_m)v_{m-1}$.

By minimality of $m$, $v_1, \ldots, v_{m-1}$ are linearly independent, so $c_i(\lambda_i - \lambda_m) = 0$ for all $i$. Since $\lambda_i \neq \lambda_m$, we get $c_i = 0$ — contradiction. $\blacksquare$

Corollary: An $n \times n$ matrix with $n$ distinct eigenvalues has $n$ linearly independent eigenvectors (and is therefore diagonalizable).

3.5 Diagonalization of Matrices

Definition: An $n \times n$ matrix $A$ is diagonalizable if there exists an invertible matrix $P$ and a diagonal matrix $D$ such that:

\[A = PDP^{-1} \quad \text{(equivalently, } P^{-1}AP = D\text{)}\]

Diagonalization Theorem: $A$ is diagonalizable if and only if $A$ has $n$ linearly independent eigenvectors.

Construction: If $\lambda_1, \ldots, \lambda_n$ are eigenvalues with corresponding linearly independent eigenvectors $v_1, \ldots, v_n$:

\[P = \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \quad D = \begin{bmatrix} \lambda_1 & 0 & \cdots & 0 \\ 0 & \lambda_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_n \end{bmatrix}\]

Proof: Consider $AP = A[v_1 \;\cdots\; v_n] = [Av_1 \;\cdots\; Av_n] = [\lambda_1 v_1 \;\cdots\; \lambda_n v_n] = PD$.

Since eigenvectors are linearly independent, $P$ is invertible, so $A = PDP^{-1}$. $\blacksquare$

3.6 When is a Matrix Diagonalizable?

Sufficient conditions for diagonalizability:

  1. $n$ distinct eigenvalues → always diagonalizable
  2. Symmetric matrix ($A = A^T$) → always diagonalizable with orthogonal eigenvectors
  3. Every eigenvalue’s geometric multiplicity equals its algebraic multiplicity

Example of a non-diagonalizable matrix:

\[A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}\]

Characteristic equation: $(1 - \lambda)^2 = 0$, so $\lambda = 1$ with algebraic multiplicity $2$.

Eigenspace: $(A - I)v = \begin{bmatrix} 0 & 1 \ 0 & 0 \end{bmatrix}v = 0$ gives $v = \begin{bmatrix} 1 \ 0 \end{bmatrix}$ only — geometric multiplicity $= 1 < 2$. Not diagonalizable.

3.7 Powers of Matrices via Diagonalization

Theorem: If $A = PDP^{-1}$, then for any positive integer $k$:

\[A^k = PD^k P^{-1}\]

where $D^k = \text{diag}(\lambda_1^k, \lambda_2^k, \ldots, \lambda_n^k)$.

Proof (by induction): Base: $A^1 = PDP^{-1}$. Inductive step: $A^{k+1} = A \cdot A^k = (PDP^{-1})(PD^k P^{-1}) = PD(P^{-1}P)D^k P^{-1} = PD^{k+1}P^{-1}$. $\blacksquare$

Application to difference equations: The system $x_{k+1} = Ax_k$ with initial condition $x_0$ has solution $x_k = A^k x_0 = PD^k P^{-1} x_0$.

3.8 Similarity & Invariants

Definition: Matrices $A$ and $B$ are similar if $B = P^{-1}AP$ for some invertible $P$.

Similar matrices share:

Invariant Why
Same eigenvalues $\det(B - \lambda I) = \det(P^{-1}(A-\lambda I)P) = \det(A - \lambda I)$
Same determinant $\det(B) = \det(P^{-1}AP) = \det(A)$
Same trace $\text{tr}(B) = \text{tr}(P^{-1}AP) = \text{tr}(A)$
Same rank Invertible $P$ preserves rank
Same characteristic polynomial Same eigenvalues with same algebraic multiplicities

3.9 Symmetric Matrices & the Spectral Theorem

Spectral Theorem: If $A$ is a real symmetric matrix ($A = A^T$), then:

  1. All eigenvalues of $A$ are real
  2. Eigenvectors corresponding to distinct eigenvalues are orthogonal
  3. $A$ is diagonalizable with orthonormal eigenvectors
  4. $A = QDQ^T$ where $Q$ is orthogonal ($Q^T Q = I$) and $D$ is diagonal

Special case — Triangular matrices: The eigenvalues of a triangular matrix (upper or lower) are its diagonal entries.

Complex eigenvalues: For real matrices, complex eigenvalues always occur in conjugate pairs: if $\lambda = a + bi$ is an eigenvalue, so is $\overline{\lambda} = a - bi$.

3.10 Worked Examples — Eigenvalues & Diagonalization

Example 1: Eigenvalues & Eigenvectors of a 2×2 Matrix

Problem: Find all eigenvalues and eigenvectors of $A = \begin{bmatrix} 4 & 2 \ 1 & 3 \end{bmatrix}$.

Solution:

Characteristic polynomial:

\[\det(A - \lambda I) = (4-\lambda)(3-\lambda) - 2 = \lambda^2 - 7\lambda + 10 = (\lambda - 5)(\lambda - 2) = 0\]

$\lambda_1 = 5, \quad \lambda_2 = 2$

Eigenvector for $\lambda_1 = 5$: Solve $(A - 5I)v = 0$:

\[\begin{bmatrix} -1 & 2 \\ 1 & -2 \end{bmatrix}v = 0 \implies -v_1 + 2v_2 = 0 \implies v_1 = 2v_2\] \[v_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}\]

Verify: $A\begin{bmatrix} 2 \ 1 \end{bmatrix} = \begin{bmatrix} 10 \ 5 \end{bmatrix} = 5\begin{bmatrix} 2 \ 1 \end{bmatrix}$ $\checkmark$

Eigenvector for $\lambda_2 = 2$: Solve $(A - 2I)v = 0$:

\[\begin{bmatrix} 2 & 2 \\ 1 & 1 \end{bmatrix}v = 0 \implies v_1 = -v_2\] \[v_2 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}\]

Verify: $A\begin{bmatrix} -1 \ 1 \end{bmatrix} = \begin{bmatrix} -2 \ 2 \end{bmatrix} = 2\begin{bmatrix} -1 \ 1 \end{bmatrix}$ $\checkmark$

Example 2: Diagonalization of a 3×3 Matrix

Problem: Diagonalize $A = \begin{bmatrix} 1 & 1 & 0 \ 0 & 2 & 0 \ 0 & 0 & 3 \end{bmatrix}$.

Solution:

Since $A$ is upper triangular, eigenvalues are the diagonal entries: $\lambda_1 = 1, \lambda_2 = 2, \lambda_3 = 3$ (all distinct → diagonalizable).

Eigenvectors:

$\lambda_1 = 1$: $(A - I)v = 0 \Rightarrow v_1 = (1, 0, 0)^T$

$\lambda_2 = 2$: $(A - 2I)v = 0 \Rightarrow v_2 = (1, 1, 0)^T$

$\lambda_3 = 3$: $(A - 3I)v = 0 \Rightarrow v_3 = (0, 0, 1)^T$

\[P = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \quad D = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \end{bmatrix}\]

$P^{-1} = \begin{bmatrix} 1 & -1 & 0 \ 0 & 1 & 0 \ 0 & 0 & 1 \end{bmatrix}$

Verify: $PDP^{-1} = A$ $\checkmark$

Example 3: Non-Diagonalizable Matrix

Problem: Show that $A = \begin{bmatrix} 2 & 1 \ 0 & 2 \end{bmatrix}$ is not diagonalizable.

Solution:

Characteristic equation: $(2 - \lambda)^2 = 0$, so $\lambda = 2$ with algebraic multiplicity $2$.

Eigenspace: $(A - 2I)v = \begin{bmatrix} 0 & 1 \ 0 & 0 \end{bmatrix}v = 0$ gives $v_2 = 0$, $v_1$ is free.

$E_2 = \text{span}\lbrace(1, 0)\rbrace$ — geometric multiplicity $= 1 < 2$. Not diagonalizable.

Example 4: Matrix Power via Diagonalization

Problem: Using Example 1, compute $A^{10}$ for $A = \begin{bmatrix} 4 & 2 \ 1 & 3 \end{bmatrix}$.

Solution:

$P = \begin{bmatrix} 2 & -1 \ 1 & 1 \end{bmatrix}$, $D = \begin{bmatrix} 5 & 0 \ 0 & 2 \end{bmatrix}$, $P^{-1} = \frac{1}{3}\begin{bmatrix} 1 & 1 \ -1 & 2 \end{bmatrix}$

\[A^{10} = PD^{10}P^{-1} = \begin{bmatrix} 2 & -1 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 5^{10} & 0 \\ 0 & 2^{10} \end{bmatrix}\frac{1}{3}\begin{bmatrix} 1 & 1 \\ -1 & 2 \end{bmatrix}\] \[= \frac{1}{3}\begin{bmatrix} 2(5^{10}) + 2^{10} & 2(5^{10}) - 2^{11} \\ 5^{10} - 2^{10} & 5^{10} + 2^{11} \end{bmatrix}\]

Practice Questions — Eigenvalues & Diagonalization

Problem Set 1: Finding Eigenvalues

1.1 Find the eigenvalues of $A = \begin{bmatrix} 3 & 1 \ 0 & 2 \end{bmatrix}$.

Click for solution $A$ is upper triangular, so eigenvalues are the diagonal entries: $\lambda_1 = 3$, $\lambda_2 = 2$.

1.2 Find the eigenvalues of $A = \begin{bmatrix} 1 & 5 \ 1 & 1 \end{bmatrix}$.

Click for solution Characteristic equation: $\lambda^2 - 2\lambda - 4 = 0$. Discriminant $= 4 + 16 = 20$. $\lambda = 1 \pm \sqrt{5}$. So $\lambda_1 = 1 + \sqrt{5} \approx 3.24$, $\lambda_2 = 1 - \sqrt{5} \approx -1.24$.

1.3 Find the characteristic polynomial and eigenvalues of $A = \begin{bmatrix} 1 & 2 & 1 \ 0 & 2 & 1 \ 0 & 0 & 1 \end{bmatrix}$.

Click for solution Upper triangular, so $p(\lambda) = (1 - \lambda)^2(2 - \lambda)$. Eigenvalues: $\lambda = 1$ (multiplicity 2), $\lambda = 2$ (multiplicity 1).

1.4 Show that if $A$ is an $n \times n$ matrix with eigenvalue $\lambda$, then $A^2$ has eigenvalue $\lambda^2$.

Click for solution If $Av = \lambda v$, then $A^2 v = A(Av) = A(\lambda v) = \lambda(Av) = \lambda(\lambda v) = \lambda^2 v$. So $v$ is an eigenvector of $A^2$ with eigenvalue $\lambda^2$.

1.5 If $A$ has eigenvalues $2, -1$, and $3$, what are the eigenvalues of $A - 4I$?

Click for solution If $Av = \lambda v$, then $(A - 4I)v = Av - 4v = \lambda v - 4v = (\lambda - 4)v$. Eigenvalues of $A - 4I$: $2 - 4 = -2$, $-1 - 4 = -5$, $3 - 4 = -1$.

Problem Set 2: Finding Eigenvectors

2.1 For $A = \begin{bmatrix} 3 & 1 \ 0 & 2 \end{bmatrix}$, find all eigenvalues and corresponding eigenvectors.

Click for solution $\lambda_1 = 3$: $(A - 3I)v = 0 \Rightarrow \begin{bmatrix} 0 & 1 \\ 0 & -1 \end{bmatrix}v = 0 \Rightarrow v_1 = (1, 0)^T$. $\lambda_2 = 2$: $(A - 2I)v = 0 \Rightarrow \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}v = 0 \Rightarrow v_2 = (-1, 1)^T$.

2.2 Find a basis for each eigenspace of $A = \begin{bmatrix} 1 & 0 & 0 \ 0 & 2 & 0 \ 0 & 0 & 3 \end{bmatrix}$.

Click for solution $A$ is diagonal, so eigenvalues are $\lambda_1 = 1, \lambda_2 = 2, \lambda_3 = 3$. Eigenspaces: $E_1 = \text{span}\lbrace(1,0,0)^T\rbrace$, $E_2 = \text{span}\lbrace(0,1,0)^T\rbrace$, $E_3 = \text{span}\lbrace(0,0,1)^T\rbrace$.

2.3 For $A = \begin{bmatrix} 2 & 1 & 0 \ 0 & 2 & 0 \ 6 & 0 & 2 \end{bmatrix}$, eigenvalue $\lambda = 2$ is known. Find the corresponding eigenvector(s).

Click for solution Solve $(A - 2I)v = 0$: $\begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 6 & 0 & 0 \end{bmatrix}v = 0$. From rows 1 and 3: $y = 0, x = 0$, $z$ free. Eigenvector: $v = (0, 0, 1)^T$.

2.4 Show that the eigenvectors corresponding to $\lambda_1 = 5$ and $\lambda_2 = 1$ of $A = \begin{bmatrix} 3 & 2 \ 2 & 3 \end{bmatrix}$ are orthogonal.

Click for solution For $\lambda_1 = 5$: $v_1 = (1, 1)^T$. For $\lambda_2 = 1$: $v_2 = (1, -1)^T$. Check: $v_1 \cdot v_2 = 1(1) + 1(-1) = 0$ ✓. (Expected since $A$ is symmetric.)

2.5 If $v$ is an eigenvector of $A$ with eigenvalue $\lambda$, show that $v$ is also an eigenvector of $A^n$ for any positive integer $n$.

Click for solution By induction. Base: $A^1 v = \lambda v$ ✓. Assume $A^k v = \lambda^k v$. Then $A^{k+1}v = A(A^k v) = A(\lambda^k v) = \lambda^k(Av) = \lambda^k \cdot \lambda v = \lambda^{k+1}v$.

Problem Set 3: Diagonalization

3.1 Diagonalize $A = \begin{bmatrix} 5 & -2 \ -2 & 2 \end{bmatrix}$ by finding matrices $P$ and $D$ such that $A = PDP^{-1}$.

Click for solution $p(\lambda) = \lambda^2 - 7\lambda + 6 = (\lambda - 6)(\lambda - 1)$. $\lambda_1 = 6$: $v_1 = (-2, 1)^T$. $\lambda_2 = 1$: $v_2 = (1, 2)^T$. $P = \begin{bmatrix} -2 & 1 \\ 1 & 2 \end{bmatrix}$, $D = \begin{bmatrix} 6 & 0 \\ 0 & 1 \end{bmatrix}$.

3.2 Determine whether $A = \begin{bmatrix} 2 & 1 \ 0 & 2 \end{bmatrix}$ is diagonalizable. Explain your answer.

Click for solution $\lambda = 2$ (multiplicity 2). $(A - 2I) = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$, nullity $= 1$. Geometric multiplicity ($1$) $<$ algebraic multiplicity ($2$), so $A$ is **NOT diagonalizable**.

3.3 Determine whether $A = \begin{bmatrix} 1 & 0 & 2 \ 0 & 1 & -1 \ 1 & 1 & 1 \end{bmatrix}$ is diagonalizable.

Click for solution $\det(A - \lambda I) = -\lambda^3 + 3\lambda^2 - 2\lambda = -\lambda(\lambda - 1)(\lambda - 2)$. Three distinct eigenvalues $\lambda = 0, 1, 2$ → **diagonalizable** (distinct eigenvalues guarantee independent eigenvectors).

3.4 If $A$ is $4 \times 4$ with eigenvalues $\lambda = 1, 2, 3, 4$, find $\det(A)$ and $\text{tr}(A)$.

Click for solution $\det(A) = 1 \cdot 2 \cdot 3 \cdot 4 = 24$. $\text{tr}(A) = 1 + 2 + 3 + 4 = 10$.

3.5 Show that if $A$ and $B$ are similar matrices (i.e., $B = P^{-1}AP$), then they have the same eigenvalues.

Click for solution $\det(B - \lambda I) = \det(P^{-1}AP - \lambda I) = \det(P^{-1}(A - \lambda I)P) = \det(P^{-1})\det(A - \lambda I)\det(P) = \det(A - \lambda I)$. Same characteristic polynomial → same eigenvalues.

Problem Set 4: Matrix Powers and Applications

4.1 Use diagonalization to compute $A^5$ where $A = \begin{bmatrix} 2 & 1 \ 0 & 1 \end{bmatrix}$.

Click for solution $\lambda_1 = 2, \lambda_2 = 1$. $v_1 = (1, 0)^T, v_2 = (-1, 1)^T$. $P = \begin{bmatrix} 1 & -1 \\ 0 & 1 \end{bmatrix}$, $D = \begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix}$. $A^5 = PD^5P^{-1} = \begin{bmatrix} 1 & -1 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 32 & 0 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 32 & 31 \\ 0 & 1 \end{bmatrix}$.

4.2 Given $A = \begin{bmatrix} 3 & 5 \ 1 & 3 \end{bmatrix}$, compute $A^{100}$ using diagonalization.

Click for solution Eigenvalues: $\lambda = 3 \pm \sqrt{5}$. Diagonalize as $A = PDP^{-1}$, then $A^{100} = PD^{100}P^{-1}$ where $D^{100}$ has $(3+\sqrt{5})^{100}$ and $(3-\sqrt{5})^{100}$ on the diagonal.

4.3 Solve the recurrence relation $x_{n+1} = Ax_n$ with $A = \begin{bmatrix} 3 & 1 \ 1 & 3 \end{bmatrix}$ and initial condition $x_0 = \begin{bmatrix} 1 \ 0 \end{bmatrix}$.

Click for solution Eigenvalues: $\lambda_1 = 4, \lambda_2 = 2$. Eigenvectors: $v_1 = (1,1)^T, v_2 = (1,-1)^T$. $x_0 = \frac{1}{2}v_1 + \frac{1}{2}v_2$. So $x_n = \frac{1}{2} \cdot 4^n v_1 + \frac{1}{2} \cdot 2^n v_2 = \begin{bmatrix} \frac{4^n + 2^n}{2} \\ \frac{4^n - 2^n}{2} \end{bmatrix}$.

4.4 Find $\lim_{n \to \infty} A^n$ where $A = \begin{bmatrix} 0.6 & 0.4 \ 0.3 & 0.7 \end{bmatrix}$ (a Markov/stochastic matrix).

Click for solution $\lambda_1 = 1, \lambda_2 = 0.3$. Since $|\lambda_2| < 1$, as $n \to \infty$, $D^n \to \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}$. The limit $A^n$ converges to the matrix with columns equal to the steady-state vector $\pi = \frac{1}{7}(3, 4)^T$: $\lim A^n = \begin{bmatrix} 3/7 & 3/7 \\ 4/7 & 4/7 \end{bmatrix}$.

4.5 Show that if $A = PDP^{-1}$, then $A^n = PD^nP^{-1}$ for any positive integer $n$.

Click for solution By induction. Base: $A^1 = PDP^{-1}$ ✓. Assume $A^k = PD^kP^{-1}$. Then $A^{k+1} = A \cdot A^k = (PDP^{-1})(PD^kP^{-1}) = PD(P^{-1}P)D^kP^{-1} = PD^{k+1}P^{-1}$.

Problem Set 5: Symmetric Matrices and Special Cases

5.1 Orthogonally diagonalize $A = \begin{bmatrix} 3 & 1 \ 1 & 3 \end{bmatrix}$ (find $Q$ and $D$ with $A = QDQ^T$).

Click for solution $\lambda_1 = 4, \lambda_2 = 2$. Normalized eigenvectors: $q_1 = \frac{1}{\sqrt{2}}(1, 1)^T$, $q_2 = \frac{1}{\sqrt{2}}(1, -1)^T$. $Q = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$, $D = \begin{bmatrix} 4 & 0 \\ 0 & 2 \end{bmatrix}$.

5.2 For the symmetric matrix $A = \begin{bmatrix} 2 & 1 & 0 \ 1 & 2 & 0 \ 0 & 0 & 3 \end{bmatrix}$, find the orthonormal eigenvectors.

Click for solution $\lambda_1 = 3$ (mult. 2), $\lambda_2 = 1$. For $\lambda = 3$: eigenspace includes $(0,0,1)^T$ and $(1,1,0)^T$. For $\lambda = 1$: eigenvector $(1,-1,0)^T$. Apply Gram-Schmidt if needed. Orthonormal set: $\lbrace\frac{1}{\sqrt{2}}(1,1,0)^T, (0,0,1)^T, \frac{1}{\sqrt{2}}(1,-1,0)^T\rbrace$.

5.3 Show that if $A$ is symmetric with distinct eigenvalues $\lambda_1 \neq \lambda_2$, then the corresponding eigenvectors are orthogonal.

Click for solution Let $Av_1 = \lambda_1 v_1$ and $Av_2 = \lambda_2 v_2$. Then: $\lambda_1 \langle v_1, v_2 \rangle = \langle Av_1, v_2 \rangle = \langle v_1, A^Tv_2 \rangle = \langle v_1, Av_2 \rangle = \lambda_2 \langle v_1, v_2 \rangle$. So $(\lambda_1 - \lambda_2)\langle v_1, v_2 \rangle = 0$. Since $\lambda_1 \neq \lambda_2$, we get $\langle v_1, v_2 \rangle = 0$.

5.4 Determine the eigenvalues and eigenvectors of the rotation matrix $R = \begin{bmatrix} \cos\theta & -\sin\theta \ \sin\theta & \cos\theta \end{bmatrix}$ for $\theta = \pi/4$.

Click for solution $p(\lambda) = \lambda^2 - 2\cos(\pi/4)\lambda + 1 = \lambda^2 - \sqrt{2}\lambda + 1$. Discriminant $= 2 - 4 = -2 < 0$. Complex eigenvalues: $\lambda = \frac{\sqrt{2}}{2} \pm \frac{\sqrt{2}}{2}i = e^{\pm i\pi/4}$. No real eigenvectors — rotation has no invariant directions (except when $\theta = 0$ or $\pi$).

5.5 Prove that a real symmetric matrix has only real eigenvalues.

Click for solution Let $Av = \lambda v$ where $v \neq 0$ (possibly complex). Then $\bar{v}^T A v = \lambda \bar{v}^T v$ and $\overline{\bar{v}^T A v} = \bar{v}^T A^T v = \bar{v}^T A v$ (since $A = A^T$ and $A$ is real). So $\bar{v}^T A v$ is real, meaning $\lambda \bar{v}^T v = \lambda \lVert v \rVert^2$ is real. Since $\lVert v \rVert^2 > 0$, $\lambda$ must be real.

4 — Jordan Canonical Form

4.1 Motivation — When Diagonalization Fails

Not every matrix is diagonalizable. When geometric multiplicity $<$ algebraic multiplicity for some eigenvalue, we cannot find enough linearly independent eigenvectors. The Jordan Canonical Form (JCF) is the “next best thing” to diagonalization — every square matrix has a JCF.

4.2 Jordan Blocks & Jordan Canonical Form

Jordan Block: A $k \times k$ matrix of the form:

\[J_k(\lambda) = \begin{bmatrix} \lambda & 1 & 0 & \cdots & 0 \\ 0 & \lambda & 1 & \cdots & 0 \\ \vdots & & \ddots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda & 1 \\ 0 & 0 & \cdots & 0 & \lambda \end{bmatrix}\]

with eigenvalue $\lambda$ on the diagonal and $1$’s on the superdiagonal.

Jordan Canonical Form (JCF): Every $n \times n$ matrix $A$ is similar to a block-diagonal matrix:

\[J = \begin{bmatrix} J_{k_1}(\lambda_1) & & \\ & J_{k_2}(\lambda_2) & \\ & & \ddots \\ & & & J_{k_r}(\lambda_r) \end{bmatrix}\]

That is, $A = PJP^{-1}$ for some invertible matrix $P$.

Key properties of JCF:

  1. Uniqueness: The JCF is unique up to reordering of blocks
  2. Eigenvalues on diagonal: The diagonal entries are the eigenvalues
  3. Block sizes: The sum of block sizes for eigenvalue $\lambda$ equals its algebraic multiplicity
  4. Number of blocks: The number of Jordan blocks for $\lambda$ equals its geometric multiplicity
  5. Diagonalizable case: All Jordan blocks are $1 \times 1$ iff $A$ is diagonalizable
  6. Minimal polynomial: The largest Jordan block for each $\lambda$ determines the minimal polynomial

4.3 Generalized Eigenvectors & Jordan Chains

Generalized Eigenvector: A vector $v$ is a generalized eigenvector of rank $k$ for eigenvalue $\lambda$ if:

\[(A - \lambda I)^k v = 0 \quad \text{but} \quad (A - \lambda I)^{k-1} v \neq 0\]

Jordan Chain: A sequence of vectors $v_1, v_2, \ldots, v_k$ (corresponding to a $k \times k$ Jordan block) satisfying:

\((A - \lambda I)v_1 = 0\) \((A - \lambda I)v_2 = v_1\) \((A - \lambda I)v_3 = v_2\) \(\vdots\) \((A - \lambda I)v_k = v_{k-1}\)

Here $v_1$ is an ordinary eigenvector and $v_2, \ldots, v_k$ are generalized eigenvectors.

4.4 Algorithm for Computing JCF

Step-by-step procedure:

  1. Find eigenvalues by solving $\det(A - \lambda I) = 0$
  2. Find algebraic multiplicities from the characteristic polynomial
  3. Find geometric multiplicities as $\dim(\text{null}(A - \lambda I))$ for each $\lambda$
  4. Determine block structure: For eigenvalue $\lambda$ with algebraic mult. $a$ and geometric mult. $g$:
    • Number of Jordan blocks = $g$
    • Total block sizes sum to $a$
    • Determine individual block sizes using $\dim(\text{null}(A - \lambda I)^k)$ for $k = 1, 2, \ldots$
  5. Find generalized eigenvectors by solving Jordan chain equations
  6. Form $P$ from the (generalized) eigenvectors as columns

4.5 Worked Examples — Jordan Form

Example 1: 2×2 Non-Diagonalizable Matrix

Problem: Find the Jordan form of $A = \begin{bmatrix} 3 & 1 \ 0 & 3 \end{bmatrix}$.

Solution:

Eigenvalue: $\lambda = 3$ with algebraic multiplicity $2$.

Eigenspace: $(A - 3I) = \begin{bmatrix} 0 & 1 \ 0 & 0 \end{bmatrix}$, nullity $= 1$. Geometric multiplicity $= 1$.

Since $1 < 2$, not diagonalizable. One Jordan block of size $2$:

\[J = \begin{bmatrix} 3 & 1 \\ 0 & 3 \end{bmatrix} = A\]

(In this case, $A$ is already in Jordan form with $P = I$.)

Example 2: 3×3 Matrix with Mixed Block Sizes

Problem: Find the JCF of $A = \begin{bmatrix} 2 & 1 & 0 \ 0 & 2 & 0 \ 0 & 0 & 3 \end{bmatrix}$.

Solution:

Eigenvalues: $\lambda_1 = 2$ (algebraic mult. $2$), $\lambda_2 = 3$ (algebraic mult. $1$).

For $\lambda_1 = 2$: $(A - 2I) = \begin{bmatrix} 0 & 1 & 0 \ 0 & 0 & 0 \ 0 & 0 & 1 \end{bmatrix}$, nullity $= 1$. Geometric mult. $= 1 < 2$.

For $\lambda_2 = 3$: $(A - 3I)$, nullity $= 1$. Geometric mult. $= 1$.

\[J = \begin{bmatrix} 2 & 1 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \end{bmatrix}\]

Again $A$ is already in Jordan form: one $2 \times 2$ block $J_2(2)$ and one $1 \times 1$ block $J_1(3)$.

Example 3: Computing $e^{At}$ via Jordan Form

Problem: For a $2 \times 2$ Jordan block $J_2(\lambda) = \begin{bmatrix} \lambda & 1 \ 0 & \lambda \end{bmatrix}$, compute $e^{Jt}$.

Solution:

\[e^{J_2(\lambda)t} = e^{\lambda t}\begin{bmatrix} 1 & t \\ 0 & 1 \end{bmatrix}\]

For a general $k \times k$ Jordan block:

\[e^{J_k(\lambda)t} = e^{\lambda t}\begin{bmatrix} 1 & t & \frac{t^2}{2!} & \cdots & \frac{t^{k-1}}{(k-1)!} \\ 0 & 1 & t & \cdots & \frac{t^{k-2}}{(k-2)!} \\ \vdots & & \ddots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 & t \\ 0 & 0 & \cdots & 0 & 1 \end{bmatrix}\]

Practice Questions — Jordan Form

Problem Set 1: Finding Jordan Forms

1.1 Find the Jordan form of $A = \begin{bmatrix} 5 & 1 \ 0 & 5 \end{bmatrix}$.

Click for solution $\lambda = 5$, algebraic mult. $= 2$, geometric mult. $= 1$. $J = \begin{bmatrix} 5 & 1 \\ 0 & 5 \end{bmatrix}$ (one $2 \times 2$ block). $A$ is already in Jordan form.

1.2 Find the Jordan canonical form of $A = \begin{bmatrix} 2 & 1 & 0 \ 0 & 2 & 0 \ 0 & 0 & 3 \end{bmatrix}$.

Click for solution $\lambda_1 = 2$ (alg. mult. 2, geom. mult. 1) and $\lambda_2 = 3$ (alg. mult. 1, geom. mult. 1). JCF: $J = \begin{bmatrix} 2 & 1 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \end{bmatrix}$. $A$ is already in Jordan form — one $J_2(2)$ block and one $J_1(3)$ block.

1.3 Determine the Jordan form of $A = \begin{bmatrix} 4 & 0 & 0 \ 0 & 4 & 0 \ 0 & 0 & 4 \end{bmatrix}$.

Click for solution $\lambda = 4$ with alg. mult. $= 3$ and geom. mult. $= 3$. Since geom. = alg. multiplicity, all blocks are $1 \times 1$. $J = 4I$ — three $J_1(4)$ blocks (diagonalizable).

1.4 Find the Jordan form of $A = \begin{bmatrix} 3 & 1 & 0 & 0 \ 0 & 3 & 0 & 0 \ 0 & 0 & 3 & 1 \ 0 & 0 & 0 & 3 \end{bmatrix}$.

Click for solution $\lambda = 3$ (alg. mult. 4). $(A - 3I)$ has rank 2, so nullity $= 2$ (geom. mult. $= 2$). Two $2 \times 2$ Jordan blocks: $J = \text{diag}(J_2(3), J_2(3))$. $A$ is already in Jordan form.

1.5 Find the Jordan form of $A = \begin{bmatrix} 1 & 1 & 0 \ 0 & 1 & 1 \ 0 & 0 & 1 \end{bmatrix}$.

Click for solution $\lambda = 1$ (alg. mult. 3, geom. mult. 1). One Jordan block of size 3: $J = J_3(1) = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 1 \end{bmatrix}$. $A$ is already in Jordan form.

Problem Set 2: Transformation Matrices

2.1 For $A = \begin{bmatrix} 5 & 4 \ -1 & 1 \end{bmatrix}$, find the Jordan form and the transformation matrix $P$ such that $P^{-1}AP = J$.

Click for solution $p(\lambda) = \lambda^2 - 6\lambda + 9 = (\lambda - 3)^2$. $\lambda = 3$ (alg. mult. 2). $(A - 3I) = \begin{bmatrix} 2 & 4 \\ -1 & -2 \end{bmatrix}$, rank 1, nullity 1. Geom. mult. $= 1 <$ alg. mult., so $J = J_2(3)$. Eigenvector: $v_1 = (2, -1)^T$. Generalized eigenvector: solve $(A - 3I)v_2 = v_1 \Rightarrow v_2 = (1, 0)^T$. $P = \begin{bmatrix} 2 & 1 \\ -1 & 0 \end{bmatrix}$.

2.2 Find $P$ such that $P^{-1}AP = J$ for $A = \begin{bmatrix} 0 & 1 & 0 \ 0 & 0 & 1 \ 0 & 0 & 0 \end{bmatrix}$.

Click for solution $\lambda = 0$ (alg. mult. 3, geom. mult. 1 since rank $(A) = 2$). $J = J_3(0)$. The chain: $v_1 = e_1 = (1,0,0)^T$, then $Av_2 = v_1 \Rightarrow v_2 = (0,1,0)^T$, $Av_3 = v_2 \Rightarrow v_3 = (0,0,1)^T$. $P = I$ — $A$ is already in Jordan form.

2.3 Given $J = \begin{bmatrix} 2 & 1 \ 0 & 2 \end{bmatrix}$ is the Jordan form of some $A$, and $P = \begin{bmatrix} 1 & 0 \ 1 & 1 \end{bmatrix}$, find $A$.

Click for solution $A = PJP^{-1}$. $P^{-1} = \begin{bmatrix} 1 & 0 \\ -1 & 1 \end{bmatrix}$. $A = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 2 & 1 \\ 0 & 2 \end{bmatrix}\begin{bmatrix} 1 & 0 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 3 & 3 \end{bmatrix}$.

2.4 For $A = \begin{bmatrix} 2 & 0 & 0 \ 1 & 2 & 0 \ 0 & 1 & 2 \end{bmatrix}$, find $P$ such that $P^{-1}AP$ is in Jordan form.

Click for solution $\lambda = 2$ (alg. mult. 3). $A - 2I = \begin{bmatrix} 0 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}$, rank 2, nullity 1. $J = J_3(2)$. Eigenvector: $v_1 = (0, 0, 1)^T$. Solve $(A-2I)v_2 = v_1 \Rightarrow v_2 = (0, 1, 0)^T$. Solve $(A-2I)v_3 = v_2 \Rightarrow v_3 = (1, 0, 0)^T$. $P = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0 \end{bmatrix}$.

2.5 Find the Jordan form and a basis for the Jordan chains of $A = \begin{bmatrix} 4 & 1 & 0 & 0 \ 0 & 4 & 1 & 0 \ 0 & 0 & 4 & 0 \ 0 & 0 & 0 & 4 \end{bmatrix}$.

Click for solution $\lambda = 4$ (alg. mult. 4). $A - 4I = \begin{bmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix}$, rank 2, nullity 2. Two Jordan chains: one of length 3 (block $J_3(4)$) and one of length 1 (block $J_1(4)$). $J = \text{diag}(J_3(4), J_1(4))$. $A$ is already in Jordan form.

Problem Set 3: Matrix Powers via Jordan Form

3.1 Compute $A^{10}$ where $A = \begin{bmatrix} 1 & 1 \ 0 & 1 \end{bmatrix}$ using Jordan form.

Click for solution $A = J_2(1)$. For a Jordan block: $J_2(\lambda)^n = \begin{bmatrix} \lambda^n & n\lambda^{n-1} \\ 0 & \lambda^n \end{bmatrix}$. So $A^{10} = \begin{bmatrix} 1 & 10 \\ 0 & 1 \end{bmatrix}$.

3.2 For $A = \begin{bmatrix} 2 & 1 & 0 \ 0 & 2 & 1 \ 0 & 0 & 2 \end{bmatrix}$, compute $A^n$ for general $n$.

Click for solution $A = J_3(2)$. Using the formula for powers of a Jordan block: $$A^n = \begin{bmatrix} 2^n & n \cdot 2^{n-1} & \binom{n}{2} \cdot 2^{n-2} \\ 0 & 2^n & n \cdot 2^{n-1} \\ 0 & 0 & 2^n \end{bmatrix}$$

3.3 If $A = PJP^{-1}$ with $J = \begin{bmatrix} 3 & 1 \ 0 & 3 \end{bmatrix}$, express $A^n$ in terms of $P$, $J$, and $n$.

Click for solution $A^n = PJ^nP^{-1} = P\begin{bmatrix} 3^n & n \cdot 3^{n-1} \\ 0 & 3^n \end{bmatrix}P^{-1}$.

3.4 Find the minimal polynomial of $A$ with Jordan form $J = \text{diag}(J_2(1), J_3(1))$.

Click for solution The minimal polynomial is $(\lambda - 1)^k$ where $k$ is the size of the largest Jordan block with eigenvalue 1. The largest block is $J_3(1)$, so $m(\lambda) = (\lambda - 1)^3$.

3.5 Determine the minimal polynomial of $A$ with Jordan form $J = \text{diag}(J_2(1), J_1(2), J_3(2))$.

Click for solution For $\lambda = 1$: largest block is $J_2(1)$, contributing $(\lambda - 1)^2$. For $\lambda = 2$: largest block is $J_3(2)$, contributing $(\lambda - 2)^3$. Minimal polynomial: $m(\lambda) = (\lambda - 1)^2(\lambda - 2)^3$.

Problem Set 4: Matrix Exponential and Differential Equations

4.1 Solve $\frac{dx}{dt} = Ax$ where $A = \begin{bmatrix} 1 & 1 \ 0 & 1 \end{bmatrix}$ and $x(0) = \begin{bmatrix} 1 \ 0 \end{bmatrix}$.

Click for solution $A$ is a Jordan block $J_2(1)$. $e^{At} = e^t\begin{bmatrix} 1 & t \\ 0 & 1 \end{bmatrix}$. So $x(t) = e^{At}x(0) = e^t\begin{bmatrix} 1 \\ 0 \end{bmatrix}$.

4.2 Compute $e^{At}$ for $A = \begin{bmatrix} 2 & 1 \ 0 & 2 \end{bmatrix}$.

Click for solution $A = 2I + N$ where $N = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$. Since $[2I, N] = 0$, $e^{At} = e^{2t}e^{Nt} = e^{2t}\begin{bmatrix} 1 & t \\ 0 & 1 \end{bmatrix}$.

4.3 Solve $\frac{dx}{dt} = Ax$ where $A = \begin{bmatrix} 3 & 0 \ 0 & 3 \end{bmatrix}$ with $x(0) = \begin{bmatrix} 2 \ 5 \end{bmatrix}$.

Click for solution $A = 3I$, so $e^{At} = e^{3t}I$. $x(t) = e^{3t}\begin{bmatrix} 2 \\ 5 \end{bmatrix}$.

4.4 Given $A$ with Jordan form $J = \begin{bmatrix} 0 & 1 & 0 \ 0 & 0 & 1 \ 0 & 0 & 0 \end{bmatrix}$, compute $e^{Jt}$.

Click for solution $J = J_3(0) = N$ (nilpotent, $N^3 = 0$). $e^{Nt} = I + Nt + \frac{(Nt)^2}{2!} = \begin{bmatrix} 1 & t & t^2/2 \\ 0 & 1 & t \\ 0 & 0 & 1 \end{bmatrix}$.

4.5 For a $2 \times 2$ matrix with distinct eigenvalues $\lambda_1, \lambda_2$ and eigenvectors $v_1, v_2$, write the general solution to $x’ = Ax$.

Click for solution General solution: $x(t) = c_1 e^{\lambda_1 t}v_1 + c_2 e^{\lambda_2 t}v_2$ where $c_1, c_2$ are determined by initial conditions.

Problem Set 5: Theoretical Questions

5.1 Prove: a matrix is diagonalizable if and only if its Jordan form has all $1 \times 1$ blocks.

Click for solution All $1 \times 1$ blocks means $J = D$ (diagonal). Then $A = PDP^{-1}$, which is the definition of diagonalizability. Conversely, if $A = PDP^{-1}$, the JCF is $D$ (all blocks are $1 \times 1$).

5.2 Show that two matrices with the same Jordan form are similar.

Click for solution If $A = P_1 J P_1^{-1}$ and $B = P_2 J P_2^{-1}$, then $B = P_2 P_1^{-1} A P_1 P_2^{-1} = Q A Q^{-1}$ (where $Q = P_2 P_1^{-1}$). So $A$ and $B$ are similar.

5.3 State and prove the Cayley-Hamilton theorem for a $2 \times 2$ matrix using Jordan form.

Click for solution The characteristic polynomial of a $2 \times 2$ matrix is $p(\lambda) = \lambda^2 - \text{tr}(A)\lambda + \det(A)$. **Case 1:** $J = \text{diag}(\lambda_1, \lambda_2)$. Then $p(J) = J^2 - (\lambda_1+\lambda_2)J + \lambda_1\lambda_2 I = 0$ (direct computation). **Case 2:** $J = \begin{bmatrix} \lambda & 1 \\ 0 & \lambda \end{bmatrix}$. Then $p(\lambda) = (\lambda - \lambda_0)^2$, and $p(J) = (J - \lambda_0 I)^2 = N^2 = 0$ ✓. Since $p(A) = Pp(J)P^{-1} = 0$.

5.4 Explain why the number of Jordan blocks for eigenvalue $\lambda$ equals the geometric multiplicity of $\lambda$.

Click for solution Each Jordan block $J_k(\lambda)$ contributes exactly one eigenvector (the first basis vector in the block). The eigenspace $\ker(A - \lambda I)$ has dimension equal to the number of Jordan blocks, which is the geometric multiplicity.

5.5 Given $A$ is $5 \times 5$ with $\lambda = 2$ (alg. mult. 5) and $\dim \ker(A - 2I) = 2$, $\dim \ker(A - 2I)^2 = 4$, $\dim \ker(A - 2I)^3 = 5$. Determine the Jordan block structure.

Click for solution Number of blocks $= \dim \ker(A - 2I) = 2$. Blocks of size $\geq 2$: $\dim \ker(A-2I)^2 - \dim \ker(A-2I) = 4 - 2 = 2$. Blocks of size $\geq 3$: $5 - 4 = 1$. So one block of size 3 ($J_3(2)$) and one block of size 2 ($J_2(2)$). Jordan form: $J = \text{diag}(J_3(2), J_2(2))$.

5 — Quadratic Forms & Positive Definiteness

5.1 Definition of Quadratic Forms

Definition: A quadratic form in $n$ variables $x_1, x_2, \ldots, x_n$ is a homogeneous polynomial of degree 2:

\[Q(x_1, x_2, \ldots, x_n) = \sum_{i=1}^n \sum_{j=1}^n a_{ij}\, x_i\, x_j\]

Why Study Quadratic Forms?

  1. Optimization: Local extrema characterized by the Hessian matrix (a quadratic form)
  2. Energy functions: Kinetic and potential energy are often quadratic: $KE = \frac{1}{2}\dot{q}^T M\dot{q}$
  3. Statistics: Mahalanobis distance $(x - \mu)^T \Sigma^{-1}(x - \mu)$
  4. Stability: Lyapunov functions for control system stability
  5. Least Squares: Error minimization $\lVert Ax - b \rVert^2 = x^T A^T A x - 2b^T Ax + b^T b$

5.2 Matrix Representation & Symmetric Convention

Matrix Representation: Every quadratic form can be written as:

\[Q(x) = x^T A x\]

where $A = [a_{ij}]$ is an $n \times n$ matrix.

Symmetric Convention: We always choose $A$ to be symmetric ($A = A^T$) by distributing mixed-term coefficients:

\[a_{ij} = a_{ji} = \frac{b_{ij} + b_{ji}}{2}\]

Example: $Q(x_1, x_2) = 3x_1^2 + 5x_1 x_2 + 2x_2^2$

\[A = \begin{bmatrix} 3 & 5/2 \\ 5/2 & 2 \end{bmatrix}\]

(The coefficient $5$ of $x_1 x_2$ is split as $5/2 + 5/2$ for the off-diagonal entries.)

Geometric Interpretation

The level set $x^T Ax = c$ (constant) defines:

  • Ellipse (2D) or ellipsoid (3D+) if $A$ is positive definite
  • Hyperbola (2D) or hyperboloid (3D+) if $A$ is indefinite
  • Degenerate conic if $A$ is singular

5.3 Definiteness Classification

Let $Q(x) = x^T Ax$ be a quadratic form with symmetric matrix $A$:

Type Condition
Positive Definite $Q(x) > 0$ for all $x \neq 0$
Positive Semidefinite $Q(x) \geq 0$ for all $x$, with $Q(x) = 0$ for some $x \neq 0$
Negative Definite $Q(x) < 0$ for all $x \neq 0$
Negative Semidefinite $Q(x) \leq 0$ for all $x$, with $Q(x) = 0$ for some $x \neq 0$
Indefinite $Q(x)$ takes both positive and negative values

5.4 Eigenvalue Characterization of Definiteness

Theorem: Let $A$ be a symmetric $n \times n$ matrix with eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_n$. Then:

Classification Eigenvalue Condition
Positive definite All $\lambda_i > 0$
Positive semidefinite All $\lambda_i \geq 0$
Negative definite All $\lambda_i < 0$
Negative semidefinite All $\lambda_i \leq 0$
Indefinite $\lambda_i$ have mixed signs

Proof (positive definite case): Since $A$ is symmetric, by the Spectral Theorem, $A = QDQ^T$ where $Q$ is orthogonal and $D = \text{diag}(\lambda_1, \ldots, \lambda_n)$.

For any $x \neq 0$, let $y = Q^T x$ (nonzero since $Q$ is invertible):

\[x^T Ax = x^T QDQ^T x = y^T Dy = \sum_{i=1}^n \lambda_i y_i^2\]

If all $\lambda_i > 0$ and $y \neq 0$, then $\sum \lambda_i y_i^2 > 0$. Conversely, choosing $x = Qe_i$ gives $x^T Ax = \lambda_i > 0$. $\blacksquare$

5.5 Sylvester's Criterion (Principal Minors Test)

Leading Principal Minors: The determinants of the upper-left $k \times k$ submatrices of $A$:

\[D_k = \det\begin{bmatrix} a_{11} & \cdots & a_{1k} \\ \vdots & \ddots & \vdots \\ a_{k1} & \cdots & a_{kk} \end{bmatrix}, \quad k = 1, 2, \ldots, n\]

Sylvester’s Criterion:

Positive Definite: $A$ is positive definite $\iff D_1 > 0, D_2 > 0, D_3 > 0, \ldots, D_n > 0$

Negative Definite: $A$ is negative definite $\iff (-1)^k D_k > 0$ for all $k = 1, \ldots, n$

Equivalently for negative definite: $D_1 < 0, D_2 > 0, D_3 < 0, D_4 > 0, \ldots$ (alternating signs).

Quick Check: For $A = \begin{bmatrix} 2 & 1 \ 1 & 3 \end{bmatrix}$:

$D_1 = 2 > 0$ $\checkmark$, $D_2 = \det(A) = 6 - 1 = 5 > 0$ $\checkmark$ → Positive definite.

5.6 Diagonalization of Quadratic Forms (Principal Axes Theorem)

Principal Axes Theorem: Let $Q(x) = x^T Ax$ be a quadratic form with symmetric matrix $A$. Then there exists an orthogonal transformation $x = Qy$ (where $Q$ is the orthogonal eigenvector matrix) that transforms $Q$ into:

\[Q(y) = \lambda_1 y_1^2 + \lambda_2 y_2^2 + \cdots + \lambda_n y_n^2\]

where $\lambda_1, \ldots, \lambda_n$ are the eigenvalues of $A$.

This is the canonical form — no cross terms. The columns of $Q$ (eigenvectors) define the principal axes of the quadric surface $x^T Ax = 1$.

5.7 Signature & Sylvester's Law of Inertia

Signature: The signature of a quadratic form is the triple $(p, q, z)$ where:

  • $p$ = number of positive eigenvalues
  • $q$ = number of negative eigenvalues
  • $z$ = number of zero eigenvalues

with $p + q + z = n$.

Sylvester’s Law of Inertia: The signature is invariant under any invertible change of variables — it depends only on the quadratic form, not the coordinate system.

Definiteness Signature
Positive definite $(n, 0, 0)$
Positive semidefinite $(p, 0, z)$ with $z > 0$
Negative definite $(0, n, 0)$
Indefinite Both $p > 0$ and $q > 0$

5.8 Worked Examples — Quadratic Forms

Example 1: Classification Using Eigenvalues

Problem: Classify $Q(x_1, x_2) = 3x_1^2 + 4x_1 x_2 + 6x_2^2$.

Solution:

\[A = \begin{bmatrix} 3 & 2 \\ 2 & 6 \end{bmatrix}\]

$p(\lambda) = (3-\lambda)(6-\lambda) - 4 = \lambda^2 - 9\lambda + 14 = (\lambda - 7)(\lambda - 2)$

$\lambda_1 = 7 > 0$, $\lambda_2 = 2 > 0$ → Positive definite.

Verification via Sylvester: $D_1 = 3 > 0$, $D_2 = 18 - 4 = 14 > 0$ $\checkmark$.

Example 2: Sylvester’s Criterion (3 variables)

Problem: Classify $Q = 2x_1^2 + 5x_2^2 + 5x_3^2 + 4x_1 x_2 - 2x_1 x_3 + 2x_2 x_3$.

Solution:

\[A = \begin{bmatrix} 2 & 2 & -1 \\ 2 & 5 & 1 \\ -1 & 1 & 5 \end{bmatrix}\]

$D_1 = 2 > 0$ $\checkmark$

$D_2 = \det\begin{bmatrix} 2 & 2 \ 2 & 5 \end{bmatrix} = 10 - 4 = 6 > 0$ $\checkmark$

$D_3 = \det(A) = 2(25-1) - 2(10+1) + (-1)(2+5) = 48 - 22 - 7 = 19 > 0$ $\checkmark$

All positive → Positive definite.

Example 3: Negative Definite Form

Problem: Classify $Q = -x_1^2 - 2x_2^2 - 3x_3^2 + 2x_1 x_2$.

Solution:

\[A = \begin{bmatrix} -1 & 1 & 0 \\ 1 & -2 & 0 \\ 0 & 0 & -3 \end{bmatrix}\]

Check $(-1)^k D_k > 0$:

$(-1)^1 D_1 = (-1)(-1) = 1 > 0$ $\checkmark$

$(-1)^2 D_2 = 1 \cdot \det\begin{bmatrix} -1 & 1 \ 1 & -2 \end{bmatrix} = 1(2-1) = 1 > 0$ $\checkmark$

$(-1)^3 D_3 = (-1)\det(A) = (-1)(-1)(6) - 1(-3)) = (-1)(-6+3) = (-1)(-3) = 3 > 0$ $\checkmark$

Negative definite.

Example 4: Diagonalization to Canonical Form

Problem: Reduce $Q(x_1, x_2) = 5x_1^2 - 4x_1 x_2 + 2x_2^2$ to canonical form.

Solution:

\[A = \begin{bmatrix} 5 & -2 \\ -2 & 2 \end{bmatrix}\]

$p(\lambda) = \lambda^2 - 7\lambda + 6 = (\lambda - 6)(\lambda - 1)$, so $\lambda_1 = 6, \lambda_2 = 1$.

For $\lambda_1 = 6$: $v_1 = (-2, 1)^T$, normalized: $\hat{v}_1 = \frac{1}{\sqrt{5}}(-2, 1)^T$.

For $\lambda_2 = 1$: $v_2 = (1, 2)^T$, normalized: $\hat{v}_2 = \frac{1}{\sqrt{5}}(1, 2)^T$.

Orthogonal transformation $x = Qy$ with $Q = \frac{1}{\sqrt{5}}\begin{bmatrix} -2 & 1 \ 1 & 2 \end{bmatrix}$:

\[Q(y) = 6y_1^2 + y_2^2\]

Example 5: Indefinite Quadratic Form

Problem: Show $Q(x_1, x_2, x_3) = x_1^2 - x_2^2 + 2x_1 x_3$ is indefinite.

Solution:

Method 1 (Direct): At $x = (1, 0, 0)^T$: $Q = 1 > 0$. At $x = (0, 1, 0)^T$: $Q = -1 < 0$. Both signs → indefinite.

Method 2 (Sylvester): $A = \begin{bmatrix} 1 & 0 & 1 \ 0 & -1 & 0 \ 1 & 0 & 0 \end{bmatrix}$

$D_1 = 1 > 0$, $D_2 = -1 < 0$. Minors have mixed signs → neither positive definite nor negative definite. Since $Q$ takes both signs, indefinite.

Practice Questions — Quadratic Forms

Problem Set 1: Classification of Quadratic Forms

1.1 Classify $Q(x_1, x_2) = 4x_1^2 + 12x_1 x_2 + 9x_2^2$ (positive definite, semidefinite, or indefinite?).

Click for solution $A = \begin{bmatrix} 4 & 6 \\ 6 & 9 \end{bmatrix}$, $\det(A) = 36 - 36 = 0$. Eigenvalues: $\lambda_1 = 13, \lambda_2 = 0$. **Positive semidefinite** (not positive definite since one eigenvalue is zero).

1.2 Classify $Q(x, y) = x^2 - 4xy + 4y^2$.

Click for solution $A = \begin{bmatrix} 1 & -2 \\ -2 & 4 \end{bmatrix}$. $\det(A) = 4 - 4 = 0$. Eigenvalues: $\lambda_1 = 5, \lambda_2 = 0$. **Positive semidefinite**. Note: $Q = (x - 2y)^2 \geq 0$.

1.3 Classify $Q(x, y, z) = x^2 + 2y^2 + 3z^2 + 2xy$.

Click for solution $A = \begin{bmatrix} 1 & 1 & 0 \\ 1 & 2 & 0 \\ 0 & 0 & 3 \end{bmatrix}$. Leading minors: $D_1 = 1 > 0$, $D_2 = 2 - 1 = 1 > 0$, $D_3 = 3 \cdot 1 = 3 > 0$. **Positive definite** by Sylvester's criterion.

1.4 Classify $Q(x, y) = 2x^2 - 3y^2$.

Click for solution $A = \begin{bmatrix} 2 & 0 \\ 0 & -3 \end{bmatrix}$. Eigenvalues: $2$ and $-3$ — one positive, one negative. **Indefinite**.

1.5 Classify $Q(x, y, z) = -x^2 - 2y^2 - 5z^2 + 2xz$.

Click for solution $A = \begin{bmatrix} -1 & 0 & 1 \\ 0 & -2 & 0 \\ 1 & 0 & -5 \end{bmatrix}$. $D_1 = -1 < 0$, $D_2 = 2 > 0$, $D_3 = -1(10) - 0 + 1(0 + 2) = -8 < 0$. Alternating signs: $D_1 < 0, D_2 > 0, D_3 < 0$ → **Negative definite**.

Problem Set 2: Sylvester’s Criterion

2.1 Use Sylvester’s criterion: Is $A = \begin{bmatrix} 2 & -1 & 0 \ -1 & 2 & -1 \ 0 & -1 & 2 \end{bmatrix}$ positive definite?

Click for solution $D_1 = 2 > 0$, $D_2 = 4-1 = 3 > 0$, $D_3 = 2(4-1) + 1(-2) = 6 - 2 = 4 > 0$. All positive → **yes, positive definite**.

2.2 For what values of $k$ is $A = \begin{bmatrix} 1 & 2 \ 2 & k \end{bmatrix}$ positive definite?

Click for solution By Sylvester: $D_1 = 1 > 0$ (always). $D_2 = k - 4 > 0 \Rightarrow k > 4$.

2.3 For what values of $a$ is $A = \begin{bmatrix} a & 1 & 0 \ 1 & a & 1 \ 0 & 1 & a \end{bmatrix}$ positive definite?

Click for solution $D_1 = a > 0$. $D_2 = a^2 - 1 > 0 \Rightarrow |a| > 1$. $D_3 = a(a^2 - 1) - 1(a) = a^3 - 2a > 0 \Rightarrow a(a^2 - 2) > 0$. Combined with $a > 1$: need $a^2 > 2$, i.e., $a > \sqrt{2}$.

2.4 Show that Sylvester’s criterion fails for positive semidefiniteness. Give a counterexample.

Click for solution $A = \begin{bmatrix} 0 & 0 \\ 0 & 1 \end{bmatrix}$ is positive semidefinite (eigenvalues $0, 1$), but $D_1 = 0$, which doesn't satisfy $D_1 > 0$. The criterion only works for **strict** positive definiteness. For semidefiniteness, check all eigenvalues $\geq 0$ or use all principal minors.

2.5 Apply Sylvester’s criterion to $A = \begin{bmatrix} 3 & 2 & 1 \ 2 & 5 & 3 \ 1 & 3 & 6 \end{bmatrix}$. Is it positive definite?

Click for solution $D_1 = 3 > 0$. $D_2 = 15 - 4 = 11 > 0$. $D_3 = 3(30-9) - 2(12-3) + 1(6-5) = 63 - 18 + 1 = 46 > 0$. All leading minors positive → **positive definite**.

Problem Set 3: Canonical Forms and Principal Axes

3.1 Reduce $Q = 2x_1^2 + 2x_1 x_2 + 2x_2^2$ to canonical form. What are the principal axes?

Click for solution $A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$, eigenvalues $\lambda = 3, 1$. Eigenvectors: $(1,1)^T$ and $(1,-1)^T$. Canonical form: $Q = 3y_1^2 + y_2^2$. Principal axes are along $\frac{1}{\sqrt{2}}(1,1)^T$ and $\frac{1}{\sqrt{2}}(1,-1)^T$.

3.2 Find the canonical form of $Q = x^2 + 4xy + 4y^2$ and identify the nature of the conic $Q = 1$.

Click for solution $A = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}$. Eigenvalues: $\lambda_1 = 5, \lambda_2 = 0$. Canonical form: $Q = 5y_1^2$. The conic $Q = 1 \Rightarrow y_1 = \pm 1/\sqrt{5}$ — a pair of parallel lines (degenerate, since one eigenvalue is zero).

3.3 Reduce $Q = 5x^2 + 8xy + 5y^2$ to canonical form by orthogonal transformation.

Click for solution $A = \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}$. Eigenvalues: $\lambda_1 = 9, \lambda_2 = 1$. Orthogonal matrix: $Q_{\text{orth}} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$. Canonical form: $Q = 9y_1^2 + y_2^2$ (an ellipse when $Q = c$).

3.4 Reduce $Q(x, y, z) = 2x^2 + 2y^2 + 2z^2 + 2xy + 2xz + 2yz$ to canonical form.

Click for solution $A = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix}$. Eigenvalues: $\lambda_1 = 4, \lambda_2 = \lambda_3 = 1$. Canonical form: $Q = 4y_1^2 + y_2^2 + y_3^2$.

3.5 Given $Q = 3x^2 + 3y^2 + 3z^2 + 2xy + 2xz + 2yz$, determine the rank, index, and signature.

Click for solution $A = \begin{bmatrix} 3 & 1 & 1 \\ 1 & 3 & 1 \\ 1 & 1 & 3 \end{bmatrix}$. Eigenvalues: $\lambda_1 = 5, \lambda_2 = \lambda_3 = 2$. All positive. Rank $= 3$ (number of nonzero eigenvalues), index $= 3$ (number of positive eigenvalues), signature $= 3 - 0 = 3$.

Problem Set 4: Applications

4.1 The conic $3x^2 + 2xy + 3y^2 = 4$ needs to be identified. Find the principal axes and classify the curve.

Click for solution $A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$. Eigenvalues: $4, 2$. In principal axes: $4y_1^2 + 2y_2^2 = 4 \Rightarrow y_1^2 + y_2^2/2 = 1$. This is an **ellipse** with semi-axes $a = 1, b = \sqrt{2}$, rotated $45°$.

4.2 Determine whether the function $f(x,y) = x^2 + xy + y^2$ has a minimum, maximum, or saddle point at the origin using the Hessian.

Click for solution Hessian: $H = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$. Eigenvalues: $3, 1$ (both positive). $H$ is positive definite → $f$ has a **local minimum** at the origin.

4.3 A spring system has potential energy $V = 3x_1^2 + 2x_2^2 + 4x_1 x_2$. Is the equilibrium at the origin stable?

Click for solution $A = \begin{bmatrix} 3 & 2 \\ 2 & 2 \end{bmatrix}$. $D_1 = 3 > 0$, $\det(A) = 6 - 4 = 2 > 0$ → positive definite. The equilibrium is **stable** (potential energy has a minimum).

4.4 Show that the constraint optimization problem “minimize $f(x) = x^TAx$ subject to $\lVert x \rVert = 1$” has solution $\lambda_{\min}$ (the smallest eigenvalue of $A$).

Click for solution By the spectral theorem, $A = QDQ^T$ with $D = \text{diag}(\lambda_1, \ldots, \lambda_n)$. Let $y = Q^Tx$, then $\lVert y \rVert = 1$ and $f = \sum \lambda_i y_i^2 \geq \lambda_{\min} \sum y_i^2 = \lambda_{\min}$. Equality when $y = e_k$ (eigenvector for $\lambda_{\min}$).

4.5 In PCA (Principal Component Analysis), data covariance matrix $\Sigma = \begin{bmatrix} 4 & 2 \ 2 & 3 \end{bmatrix}$. Find the principal components and variance explained.

Click for solution $\lambda_1 = 5, \lambda_2 = 2$. PC1 direction: eigenvector of $\lambda_1 = 5$ is $(2,1)^T/\sqrt{5}$. PC2 direction: $(-1,2)^T/\sqrt{5}$. Total variance $= 4 + 3 = 7$. PC1 explains $5/7 \approx 71.4\%$, PC2 explains $2/7 \approx 28.6\%$.

Problem Set 5: Advanced Topics

5.1 Prove: if $A$ is positive definite, then $\det(A) > 0$.

Click for solution If $A$ is positive definite, all eigenvalues $\lambda_i > 0$. Since $\det(A) = \prod \lambda_i > 0$.

5.2 Show that the sum of two positive definite matrices is positive definite.

Click for solution If $A, B$ are positive definite, then for any $x \neq 0$: $x^T(A+B)x = x^TAx + x^TBx > 0 + 0 = 0$. So $A + B$ is positive definite.

5.3 Let $A$ be positive definite. Show that $A^{-1}$ is also positive definite.

Click for solution Eigenvalues of $A^{-1}$ are $1/\lambda_i$ where $\lambda_i$ are eigenvalues of $A$. Since all $\lambda_i > 0$, all $1/\lambda_i > 0$. Also $A^{-1}$ is symmetric ($(A^{-1})^T = (A^T)^{-1} = A^{-1}$). So $A^{-1}$ is positive definite.

5.4 Find the Cholesky decomposition $A = LL^T$ for $A = \begin{bmatrix} 4 & 2 \ 2 & 5 \end{bmatrix}$.

Click for solution $l_{11} = \sqrt{4} = 2$. $l_{21} = 2/2 = 1$. $l_{22} = \sqrt{5 - 1} = 2$. $$L = \begin{bmatrix} 2 & 0 \\ 1 & 2 \end{bmatrix}, \quad LL^T = \begin{bmatrix} 2 & 0 \\ 1 & 2 \end{bmatrix}\begin{bmatrix} 2 & 1 \\ 0 & 2 \end{bmatrix} = \begin{bmatrix} 4 & 2 \\ 2 & 5 \end{bmatrix} \checkmark$$

5.5 Show that the quadratic form $Q(x) = x^TAx$ can be written as $Q(x) = \text{tr}(Axx^T)$ and explain why this is useful.

Click for solution $x^TAx$ is a scalar, so $x^TAx = \text{tr}(x^TAx) = \text{tr}(Axx^T)$ (using the cyclic property of trace: $\text{tr}(ABC) = \text{tr}(CAB)$). This is useful because: (1) it extends to expected values: $E[x^TAx] = \text{tr}(A \cdot E[xx^T])$, and (2) it simplifies matrix calculus with quadratic forms.

6 — Engineering & Real-Life Applications

6.1 Computer Graphics & 3D Transformations

Problem: A 3D game engine needs to (1) rotate a triangle about the $y$-axis by $30°$ and (2) project the result onto the $xy$-plane for screen display.

Key Linear Transformations:

Rotation about $y$-axis by $\theta$:

\[R_y(\theta) = \begin{bmatrix} \cos\theta & 0 & -\sin\theta \\ 0 & 1 & 0 \\ \sin\theta & 0 & \cos\theta \end{bmatrix}\]

Projection onto $xy$-plane:

\[P_{xy} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix}\]

Kernel/Image Analysis:

Transform $\ker$ $\text{Im}$ nullity rank
$R_y$ $\lbrace 0 \rbrace$ $\mathbb{R}^3$ 0 3
$P_{xy}$ $z$-axis $xy$-plane 1 2

Insight: Nullity of the projection $= 1$ tells us one dimension of information (depth) is lost when rendering 3D to 2D.

6.2 Principal Component Analysis (PCA)

Problem: Reduce a dataset of $1000$ images (each $100 \times 100 = 10{,}000$ dimensions) to $50$ dimensions.

PCA as a linear transformation $T: \mathbb{R}^{10000} \to \mathbb{R}^{50}$:

\[\text{rank}(T) = 50, \quad \text{nullity}(T) = 10000 - 50 = 9950\]

Interpretation:

  • Rank $= 50$: essential features preserved ($\sim$95% variance)
  • Nullity $= 9950$: redundant/noisy dimensions discarded
  • Compression: $\frac{50}{10000} = 0.5\%$ of dimensions retain $95\%$ of information

The eigenvectors of the covariance matrix (principal components) are orthogonal by the Spectral Theorem, and Gram-Schmidt ensures numerical orthogonality.

6.3 Electrical Circuit Analysis

Problem Context: Kirchhoff’s laws create systems of linear equations $AI = b$.

For a circuit with $3$ loops, $5$ resistors, and $2$ voltage sources:

\[\begin{bmatrix} 6 & -2 & 0 \\ -2 & 11 & -3 \\ 0 & -3 & 8 \end{bmatrix}\begin{bmatrix} I_1 \\ I_2 \\ I_3 \end{bmatrix} = \begin{bmatrix} 12 \\ 0 \\ 8 \end{bmatrix}\]

Rank-Nullity Analysis: $\text{rank}(A) = 3$, $\text{nullity}(A) = 0$ → unique solution exists. The kernel $\lbrace 0 \rbrace$ means no nontrivial current combination produces zero voltage drop.

6.4 Signal Processing — OFDM

Modern wireless systems (Wi-Fi 6, 5G) use OFDM — subcarriers $\phi_k(t) = e^{2\pi i f_k t}$ form an orthogonal set in the inner product space of complex functions:

\[\langle \phi_k, \phi_l \rangle = \int_0^T \phi_k(t)\overline{\phi_l(t)}\,dt = \begin{cases} T & k = l \\ 0 & k \neq l \end{cases}\]

This orthogonality allows receivers to extract individual data symbols via inner products, achieving 50% better spectral efficiency than traditional FDM.

6.5 Quantum Mechanics — Orthogonal Energy States

Particle in a box: Energy eigenstates $\psi_n(x) = \sqrt{\frac{2}{L}}\sin!\left(\frac{n\pi x}{L}\right)$ are orthonormal:

\[\langle \psi_m, \psi_n \rangle = \int_0^L \psi_m(x)\psi_n(x)\,dx = \delta_{mn}\]
Physical meaning: Orthogonal states represent perfectly distinguishable energy levels. Any state can be expanded: $ \psi\rangle = \sum_n c_n \psi_n\rangle$ where $ c_n ^2$ is the measurement probability.

6.6 Google PageRank Algorithm

PageRank models web pages as nodes in a directed graph. The importance vector $\pi$ is the dominant eigenvector (eigenvalue $\lambda = 1$) of the transition matrix $M$:

\[M\pi = \pi\]

For a web with billions of pages, this is solved iteratively using the power method, which relies on repeated matrix-vector multiplication converging to the dominant eigenvector.

Connection to diagonalization: If $M$ is diagonalizable, $M^k x_0 = PD^k P^{-1}x_0 \to$ component along eigenvector for $\lambda_1 = 1$ dominates as $k \to \infty$.

6.7 Differential Equations & System Dynamics (Jordan Form)

Two-Tank Chemical Reactor: Tanks in series with equal volumes and flow rates.

\[\frac{d}{dt}\begin{bmatrix} C_1 \\ C_2 \end{bmatrix} = \begin{bmatrix} -0.1 & 0 \\ 0.1 & -0.1 \end{bmatrix}\begin{bmatrix} C_1 \\ C_2 \end{bmatrix}\]

Eigenvalue $\lambda = -0.1$ with algebraic mult. $2$, geometric mult. $1$ → not diagonalizable; requires Jordan form.

Solution via Jordan form:

\[C_1(t) = 10e^{-0.1t}, \quad C_2(t) = t\, e^{-0.1t}\]

Tank 1 decays exponentially. Tank 2 shows characteristic $te^{-0.1t}$ rise-then-decay from the Jordan block — peak at $t = 10$ minutes.

Engineering significance: Jordan form predicts transient behavior in cascaded processes — critical for reactor design and wastewater treatment.

6.8 Optimization & Machine Learning (Quadratic Forms)

Hessian Matrix & Optimization: For $f: \mathbb{R}^n \to \mathbb{R}$ at a critical point:

  • Hessian positive definite → local minimum
  • Hessian negative definite → local maximum
  • Hessian indefinite → saddle point

Machine Learning (PCA): The covariance matrix $\Sigma$ is positive semidefinite. Its eigenvalues represent variances along principal directions, and its positive definiteness guarantees $\Sigma^{-1}$ exists for computing Mahalanobis distance.

Structural Engineering: Stiffness matrix $K$ must be positive definite for stable structures — $\frac{1}{2}x^T Kx > 0$ ensures positive potential energy for any nonzero displacement.

Comprehensive Unit II Practice Set

P1. Let $T: \mathbb{R}^3 \to \mathbb{R}^3$ be defined by $T(x,y,z) = (x+y, y+z, x+z)$. Find the standard matrix, kernel, image, and verify the Rank-Nullity Theorem.

Click for solution $A = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \end{bmatrix}$. $\det(A) = 1(1) - 1(0-1) + 0 = 2 \neq 0$. $\ker(T) = \lbrace 0 \rbrace$ (nullity $= 0$), $\text{Im}(T) = \mathbb{R}^3$ (rank $= 3$). Rank-Nullity: $3 = 3 + 0$ $\checkmark$.

P2. Apply Gram-Schmidt to $\lbrace (1,0,1,0), (1,1,0,0), (0,1,1,1) \rbrace$ in $\mathbb{R}^4$.

Click for solution $w_1 = (1,0,1,0)$. $w_2 = (1,1,0,0) - \frac{1}{2}(1,0,1,0) = (1/2, 1, -1/2, 0)$. $w_3 = (0,1,1,1) - \frac{1}{2}(1,0,1,0) - \frac{1/2}{3/2}(1/2, 1, -1/2, 0) = (-2/3, 2/3, 2/3, 1)$. After normalizing, this gives three orthonormal vectors in $\mathbb{R}^4$.

P3. Find all eigenvalues and eigenvectors of $A = \begin{bmatrix} 2 & -1 \ -1 & 2 \end{bmatrix}$. Diagonalize $A$.

Click for solution $p(\lambda) = (2-\lambda)^2 - 1 = \lambda^2 - 4\lambda + 3 = (\lambda-3)(\lambda-1)$. $\lambda_1 = 3$: $v_1 = (-1, 1)^T$. $\lambda_2 = 1$: $v_2 = (1, 1)^T$. $P = \begin{bmatrix} -1 & 1 \\ 1 & 1 \end{bmatrix}$, $D = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}$. Since $A$ is symmetric, eigenvectors are orthogonal $\checkmark$.

P4. Find the Jordan form of $A = \begin{bmatrix} 2 & 1 & 0 \ 0 & 2 & 1 \ 0 & 0 & 2 \end{bmatrix}$.

Click for solution $\lambda = 2$ with algebraic mult. $3$. $(A-2I) = \begin{bmatrix} 0&1&0\\0&0&1\\0&0&0\end{bmatrix}$, nullity $= 1$. So geometric mult. $= 1$, one Jordan block of size $3$: $J = \begin{bmatrix} 2&1&0\\0&2&1\\0&0&2 \end{bmatrix} = A$ (already in Jordan form).

P5. Determine the definiteness of $Q = x_1^2 + 4x_2^2 + x_3^2 + 2x_1 x_2 - 2x_2 x_3$.

Click for solution $A = \begin{bmatrix} 1 & 1 & 0 \\ 1 & 4 & -1 \\ 0 & -1 & 1 \end{bmatrix}$ $D_1 = 1 > 0$, $D_2 = 4-1 = 3 > 0$, $D_3 = 1(4-1) - 1(1-0) + 0 = 3-1 = 2 > 0$. All positive → **Positive definite** by Sylvester's criterion.

P6. Find the canonical form of $Q(x_1, x_2) = 3x_1^2 + 8x_1 x_2 + 3x_2^2$ and identify the principal axes.

Click for solution $A = \begin{bmatrix} 3 & 4 \\ 4 & 3 \end{bmatrix}$. $p(\lambda) = (3-\lambda)^2 - 16 = \lambda^2-6\lambda-7 = (\lambda-7)(\lambda+1)$. $\lambda_1 = 7$: $v_1 = (1,1)^T$. $\lambda_2 = -1$: $v_2 = (1,-1)^T$. Canonical form: $Q = 7y_1^2 - y_2^2$ (indefinite). Principal axes: $y_1$ along $(1,1)$, $y_2$ along $(1,-1)$.

P7. If $A$ is a $6 \times 4$ matrix with $\text{rank}(A) = 3$, what is $\text{nullity}(A)$? Can $T(x) = Ax$ be injective or surjective?

Click for solution $\text{nullity}(A) = 4 - 3 = 1$. Not injective (nullity $\neq 0$). Not surjective ($\text{rank} = 3 \neq 6 = \dim(\mathbb{R}^6)$).

P8. Prove: the sum of two positive definite matrices is positive definite.

Click for solution If $A, B$ are positive definite, then for any $x \neq 0$: $x^T(A+B)x = x^TAx + x^TBx > 0 + 0 = 0$. Since both $x^TAx > 0$ and $x^TBx > 0$ for $x \neq 0$.

Key Takeaways — Unit II

  1. Linear transformations preserve vector space structure; fully determined by action on a basis
  2. Kernel and image are subspaces; their dimensions are linked by the Rank-Nullity Theorem
  3. Injectivity $\iff$ trivial kernel; surjectivity $\iff$ full-rank image
  4. Inner products generalize angles and lengths; Cauchy-Schwarz is the fundamental inequality
  5. Gram-Schmidt converts any basis to orthonormal; essential for QR decomposition
  6. Eigenvalues are roots of the characteristic polynomial; eigenvectors are the “natural directions” of a matrix
  7. Diagonalization $A = PDP^{-1}$ simplifies matrix powers and differential equations
  8. Symmetric matrices are always diagonalizable with real eigenvalues and orthogonal eigenvectors (Spectral Theorem)
  9. Jordan Canonical Form generalizes diagonalization to all matrices; essential when geometric $<$ algebraic multiplicity
  10. Quadratic forms $x^TAx$ are classified by eigenvalues; Sylvester’s criterion provides a practical definiteness test
  11. Principal Axes Theorem diagonalizes quadratic forms via orthogonal change of variables

References

  1. Wikipedia. “Vector Space.” https://en.wikipedia.org/wiki/Vector_space
  2. Wikipedia. “Linear Map.” https://en.wikipedia.org/wiki/Linear_map
  3. Lay, D. C. Linear Algebra and Its Applications, 5th ed. Pearson, 2015.
  4. Strang, G. Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press, 2016.
  5. Axler, S. Linear Algebra Done Right, 3rd ed. Springer, 2015.
  6. Horn, R. A. & Johnson, C. R. Matrix Analysis, 2nd ed. Cambridge Univ. Press, 2012.
  7. Wikipedia. “Rotation Matrix.” https://en.wikipedia.org/wiki/Rotation_matrix
  8. Wikipedia. “3D Projection.” https://en.wikipedia.org/wiki/3D_projection
  9. Wikipedia. “Principal Component Analysis.” https://en.wikipedia.org/wiki/Principal_component_analysis
  10. Wikipedia. “Orthogonal Frequency-Division Multiplexing.” https://en.wikipedia.org/wiki/Orthogonal_frequency-division_multiplexing
  11. Wikipedia. “Matrix Exponential.” https://en.wikipedia.org/wiki/Matrix_exponential
  12. Wikipedia. “Quadratic Form.” https://en.wikipedia.org/wiki/Quadratic_form
  13. Wikipedia. “Definiteness of a Matrix.” https://en.wikipedia.org/wiki/Definiteness_of_a_matrix
  14. Wikipedia. “Quadric.” https://en.wikipedia.org/wiki/Quadric
  15. Wikipedia. “Hessian Matrix.” https://en.wikipedia.org/wiki/Hessian_matrix
  16. Wikipedia. “Kinetic Energy.” https://en.wikipedia.org/wiki/Kinetic_energy
  17. Wikipedia. “Mahalanobis Distance.” https://en.wikipedia.org/wiki/Mahalanobis_distance
  18. Wikipedia. “Lyapunov Stability.” https://en.wikipedia.org/wiki/Lyapunov_stability
  19. Wikipedia. “Least Squares.” https://en.wikipedia.org/wiki/Least_squares
  20. Wikipedia. “Spectral Theorem.” https://en.wikipedia.org/wiki/Spectral_theorem
  21. Wikipedia. “Sylvester’s Criterion.” https://en.wikipedia.org/wiki/Sylvester%27s_criterion
  22. Math LibreTexts. “Quadratic Forms.” https://math.libretexts.org/
  23. Wikipedia. “Sylvester’s Law of Inertia.” https://en.wikipedia.org/wiki/Sylvester%27s_law_of_inertia

← Back to EM-II Index