Solved Exercises on Spectral Theorem for Symmetric Matrices in Linear Algebra

Master the spectral theorem: eigenvalues, eigenvectors, orthogonal diagonalization, and quadratic forms through these 5 detailed exercises.

Spectral Theorem Fundamentals
\(A = QDQ^T\)
Spectral Theorem Formula
Key Definitions:

Symmetric Matrix: A matrix A where A = A^T (equal to its transpose)

Spectral Theorem: Every symmetric matrix A can be orthogonally diagonalized as A = QDQ^T

Orthogonal Matrix: Q^T Q = QQ^T = I (columns are orthonormal vectors)

Real Eigenvalues: Symmetric matrices have only real eigenvalues

Spectral Theorem Process:
  1. Verify symmetry: Check that A = A^T
  2. Find eigenvalues: Solve det(A - λI) = 0
  3. Find eigenvectors: For each eigenvalue, solve (A - λI)v = 0
  4. Orthogonalize: Apply Gram-Schmidt within each eigenspace
  5. Normalize: Make eigenvectors unit vectors
  6. Form matrices: Q from orthonormal eigenvectors, D from eigenvalues
Tip 1: Eigenvectors of distinct eigenvalues of symmetric matrices are automatically orthogonal.
Tip 2: Always normalize eigenvectors to form orthonormal basis.
Tip 3: For repeated eigenvalues, apply Gram-Schmidt to get orthogonal eigenvectors.
Tip 4: Use spectral theorem for efficient quadratic form analysis.
Key Properties: Real eigenvalues, orthogonal eigenvectors, diagonalizable by orthogonal matrix
Applications: Principal component analysis, optimization, physics, engineering
Essential Formulas:

• Spectral decomposition: A = QDQ^T where Q^T = Q^(-1)

• Characteristic equation: det(A - λI) = 0

• Eigenvector equation: Av = λv

• Orthogonality: vᵢ · vⱼ = 0 for i ≠ j

Solution: Exercises 1 to 3
1 Basic Spectral Theorem
Exercise 1
Find the spectral decomposition of: A =
[
4
2
2
1
]
Definition:

Spectral Decomposition: Express A as A = QDQ^T where Q is orthogonal and D is diagonal

Step-by-step Method:
  1. Verify A is symmetric: A = A^T
  2. Find characteristic polynomial det(A - λI) = 0
  3. Find eigenvalues λ
  4. Find eigenvectors v for each λ
  5. Normalize eigenvectors to form orthonormal basis
  6. Form Q from orthonormal eigenvectors, D from eigenvalues
Step 1: Verify A is symmetric

A =

[
4
2
2
1
]
, A^T =
[
4
2
2
1
]
✓

Step 2: Find characteristic polynomial

det(A - λI) = det

[
4-λ
2
2
1-λ
]

= (4-λ)(1-λ) - (2)(2) = (4-λ)(1-λ) - 4

= 4 - 4λ - λ + λ² - 4 = λ² - 5λ

= λ(λ - 5)

Step 3: Find eigenvalues

λ(λ - 5) = 0 → λ₁ = 0, λ₂ = 5

Step 4: Find eigenvector for λ₁ = 0

(A - 0I)v = 0 →

[
4
2
2
1
]
[
x
y
]
=
[
0
0
]

This gives: 4x + 2y = 0 and 2x + y = 0, so y = -2x

Eigenvector: v₁ =

[
1
-2
]

Step 5: Find eigenvector for λ₂ = 5

(A - 5I)v = 0 →

[
-1
2
2
-4
]
[
x
y
]
=
[
0
0
]

This gives: -x + 2y = 0, so x = 2y

Eigenvector: v₂ =

[
2
1
]

Step 6: Normalize eigenvectors

||v₁|| = √(1² + (-2)²) = √5

u₁ = v₁/||v₁|| =

[
1/√5
-2/√5
]

||v₂|| = √(2² + 1²) = √5

u₂ = v₂/||v₂|| =

[
2/√5
1/√5
]

Step 7: Verify orthogonality

u₁ · u₂ = (1/√5)(2/√5) + (-2/√5)(1/√5) = 2/5 - 2/5 = 0 ✓

A = QDQ^T where Q =
[
1/√5
2/√5
-2/√5
1/√5
]
, D =
[
0
0
0
5
]
Final answer:

A = QDQ^T with Q orthogonal and D diagonal containing eigenvalues

Applied rules:

• Symmetry verification: Essential first step

• Orthogonal eigenvectors: Automatic for distinct eigenvalues

• Normalization: Ensures Q is orthogonal

2 3x3 Symmetric Matrix
Exercise 2
Find the spectral decomposition of: A =
[
2
1
0
1
2
0
0
0
3
]
Definition:

3x3 Symmetric Matrix: Same process extended to higher dimensions

Step 1: Verify A is symmetric

A^T =

[
2
1
0
1
2
0
0
0
3
]
= A ✓

Step 2: Find characteristic polynomial

det(A - λI) = det

[
2-λ
1
0
1
2-λ
0
0
0
3-λ
]

= (3-λ)[(2-λ)(2-λ) - 1] = (3-λ)[(2-λ)² - 1]

= (3-λ)[4 - 4λ + λ² - 1] = (3-λ)[λ² - 4λ + 3]

= (3-λ)(λ - 1)(λ - 3) = -(λ - 3)²(λ - 1)

Step 3: Find eigenvalues

-(λ - 3)²(λ - 1) = 0 → λ₁ = 1, λ₂ = 3 (multiplicity 2)

Step 4: Find eigenvector for λ₁ = 1

(A - I)v = 0 →

[
1
1
0
1
1
0
0
0
2
]
[
x
y
z
]
=
[
0
0
0
]

This gives: x + y = 0 and z = 0, so y = -x and z = 0

Eigenvector: v₁ =

[
1
-1
0
]

Step 5: Find eigenvectors for λ₂ = 3

(A - 3I)v = 0 →

[
-1
1
0
1
-1
0
0
0
0
]
[
x
y
z
]
=
[
0
0
0
]

This gives: -x + y = 0, so x = y

Eigenspace is span{

[
1
1
0
]
,
[
0
0
1
]
}

Step 6: Verify orthogonality between eigenspaces

v₁ ·

[
1
1
0
]
= (1)(1) + (-1)(1) + (0)(0) = 0 ✓

v₁ ·

[
0
0
1
]
= (1)(0) + (-1)(0) + (0)(1) = 0 ✓

Q =
[
1/√2
1/√2
0
-1/√2
1/√2
0
0
0
1
]
, D =
[
1
0
0
0
3
0
0
0
3
]
Final answer:

A = QDQ^T with repeated eigenvalue handled properly

Applied rules:

• Repeated eigenvalues: Use Gram-Schmidt within eigenspace

• Orthogonality: Eigenspaces of different eigenvalues are orthogonal

• Diagonalization: Symmetric matrices are always diagonalizable

3 Verification Exercise
Exercise 3
Verify that the matrices Q and D from Exercise 1 satisfy A = QDQ^T
Definition:

Verification: Confirm spectral decomposition by computing QDQ^T and comparing to A

Step 1: Recall matrices from Exercise 1

Q =

[
1/√5
2/√5
-2/√5
1/√5
]
, D =
[
0
0
0
5
]

Step 2: Compute Q^T

Q^T =

[
1/√5
-2/√5
2/√5
1/√5
]

Step 3: Compute QD

QD =

[
1/√5
2/√5
-2/√5
1/√5
]
[
0
0
0
5
]
=
[
0
10/√5
0
5/√5
]
=
[
0
2√5
0
√5
]

Step 4: Compute QDQ^T

QDQ^T =

[
0
2√5
0
√5
]
[
1/√5
-2/√5
2/√5
1/√5
]

=

[
0×(1/√5) + 2√5×(2/√5)
0×(-2/√5) + 2√5×(1/√5)
0×(1/√5) + √5×(2/√5)
0×(-2/√5) + √5×(1/√5)
]

=

[
4
2
2
1
]
= A ✓

QDQ^T =
[
4
2
2
1
]
= A ✓
Final answer:

QDQ^T = A, confirming our spectral decomposition is correct

Applied rules:

• Orthogonal matrix: Q^T = Q^(-1) for symmetric matrices

• Verification: Essential to confirm correctness

• Matrix multiplication: Row by column rule

Solution: Exercises 4 to 5
4 Quadratic Form Application
Exercise 4
Use the spectral theorem to analyze the quadratic form: f(x,y) = 4x² + 4xy + y²
Definition:

Quadratic Form: f(x) = x^T Ax where A is symmetric matrix of coefficients

Step 1: Write in matrix form

f(x,y) = [x y]

[
4
2
2
1
]
[
x
y
]

So A =

[
4
2
2
1
]
(same as Exercise 1)

Step 2: Use spectral decomposition from Exercise 1

A = QDQ^T where Q =

[
1/√5
2/√5
-2/√5
1/√5
]
, D =
[
0
0
0
5
]

Step 3: Change of variables

Let [x y]^T = Q[u v]^T, so [u v]^T = Q^T[x y]^T

Then f(x,y) = x^T Ax = x^T QDQ^T x = (Q^T x)^T D (Q^T x) = [u v] D [u v]^T

= [u v]

[
0
0
0
5
]
[
u
v
]
= 5v²

Step 4: Interpret the result

In the new coordinate system (u,v), the quadratic form becomes f(u,v) = 5v²

This is a parabolic cylinder opening along the v-axis

Since one eigenvalue is 0, the form is positive semidefinite

f(x,y) = 4x² + 4xy + y² transforms to f(u,v) = 5v² in principal axes coordinates
Final answer:

The quadratic form represents a parabolic cylinder aligned with the eigenvector directions

Applied rules:

• Principal axes: Change to coordinate system of eigenvectors

• Quadratic form simplification: Eliminates cross terms

• Classification: Based on eigenvalue signs

5 Matrix Powers via Spectral Theorem
Exercise 5
Using the spectral decomposition from Exercise 1, compute A⁶ where A =
[
4
2
2
1
]
Definition:

Matrix Powers via Spectral Theorem: A^n = QD^nQ^T is extremely efficient for symmetric matrices

Step 1: Recall spectral decomposition from Exercise 1

Q =

[
1/√5
2/√5
-2/√5
1/√5
]
, D =
[
0
0
0
5
]

Step 2: Compute D⁶

Since D is diagonal, D⁶ =

[
0⁶
0
0
5⁶
]
=
[
0
0
0
15625
]

Step 3: Compute A⁶ = QD⁶Q^T

First compute QD⁶:

[
1/√5
2/√5
-2/√5
1/√5
]
[
0
0
0
15625
]
=
[
0
31250/√5
0
15625/√5
]

Then compute (QD⁶)Q^T:

[
0
31250/√5
0
15625/√5
]
[
1/√5
-2/√5
2/√5
1/√5
]

=

[
0×(1/√5) + (31250/√5)×(2/√5)
0×(-2/√5) + (31250/√5)×(1/√5)
0×(1/√5) + (15625/√5)×(2/√5)
0×(-2/√5) + (15625/√5)×(1/√5)
]

=

[
62500/5
31250/5
31250/5
15625/5
]
=
[
12500
6250
6250
3125
]

A⁶ =
[
12500
6250
6250
3125
]
Final answer:

A⁶ computed efficiently using spectral decomposition

Applied rules:

• Power efficiency: A^n = QD^nQ^T avoids n-1 matrix multiplications

• Diagonal powers: Just raise diagonal elements to power

• Orthogonal property: Q^T = Q^(-1) simplifies computation

Questions & Answers

Question: Why are symmetric matrices guaranteed to have real eigenvalues and orthogonal eigenvectors? What's the mathematical reason behind this?

Answer: The mathematical foundation lies in the properties of the inner product and the symmetry condition A = A^T:

  • Real eigenvalues: For eigenvalue λ with eigenvector v, we have Av = λv. Taking the inner product with v: ⟨Av, v⟩ = ⟨λv, v⟩. Since A is symmetric, ⟨Av, v⟩ = ⟨v, Av⟩ = ⟨v, λv⟩ = λ̄⟨v, v⟩. So λ⟨v, v⟩ = λ̄⟨v, v⟩, which implies λ = λ̄, meaning λ is real.
  • Orthogonal eigenvectors: For distinct eigenvalues λ₁ ≠ λ₂ with eigenvectors v₁ and v₂: Av₁ = λ₁v₁ and Av₂ = λ₂v₂. Then ⟨Av₁, v₂⟩ = ⟨λ₁v₁, v₂⟩ = λ₁⟨v₁, v₂⟩ and ⟨v₁, Av₂⟩ = ⟨v₁, λ₂v₂⟩ = λ₂⟨v₁, v₂⟩. Since A is symmetric, ⟨Av₁, v₂⟩ = ⟨v₁, Av₂⟩, so λ₁⟨v₁, v₂⟩ = λ₂⟨v₁, v₂⟩. Since λ₁ ≠ λ₂, we must have ⟨v₁, v₂⟩ = 0.

This is the fundamental reason why symmetric matrices are so special in linear algebra and why the spectral theorem holds.

Question: In Exercise 4, you used spectral theorem to analyze the quadratic form. How does this relate to optimization problems?

Answer: The spectral theorem is fundamental to optimization, especially for quadratic forms:

  • Positive definiteness: A symmetric matrix A is positive definite if all eigenvalues are positive, which means x^T Ax > 0 for all non-zero x. This is crucial for determining if a critical point is a minimum.
  • Optimization direction: The eigenvectors give the principal directions of variation. The largest eigenvalue corresponds to the direction of maximum increase.
  • Convergence rates: In optimization algorithms like gradient descent, convergence depends on the ratio of largest to smallest eigenvalues (condition number).

In Exercise 4, since we had eigenvalues 0 and 5, the quadratic form has one flat direction and one direction of maximum growth. This tells us the function's behavior near critical points.

Question: How does the spectral theorem relate to principal component analysis (PCA) in statistics?

Answer: PCA is fundamentally based on the spectral theorem applied to covariance matrices:

  1. Covariance matrix: Given data matrix X, compute covariance matrix C = (1/n)X^T X (which is symmetric)
  2. Spectral decomposition: C = VΛV^T where V contains eigenvectors (principal components) and Λ contains eigenvalues
  3. Dimensionality reduction: Keep only the top k eigenvectors corresponding to largest eigenvalues
  4. New coordinates: Project data onto principal components: Y = XV_k

The spectral theorem guarantees that we can find an orthonormal basis (the principal components) that captures the maximum variance in the data. The eigenvalues represent the amount of variance explained by each principal component.

This is why PCA is so effective for data compression and visualization!