Solved Exercises on Applications of Change of Basis in Linear Algebra

Master change of basis applications: principal component analysis, Fourier analysis basics, and coordinate system transformations through these 5 detailed exercises.

Change of Basis Applications
\([v]_B = P_{B \leftarrow C}[v]_C\)
Change of Basis Formula
Key Definitions:

Change of Basis: Transforming vector coordinates from one basis to another

Transition Matrix: P_{B←C} converts coordinates from basis C to basis B

Principal Component Analysis (PCA): Change to basis of eigenvectors to capture maximum variance

Fourier Analysis: Change to frequency domain basis using sine/cosine functions

Change of Basis Process:
  1. Identify bases: Original basis B and new basis C
  2. Form transition matrix: P_{B←C} with columns being C-basis vectors in B-coordinates
  3. Apply transformation: [v]_B = P_{B←C}[v]_C
  4. Verify: Check that vectors represent the same geometric entity
Tip 1: Transition matrix P_{B←C} = [P_{C←B}]⁻¹ (inverse relationship)
Tip 2: Orthonormal basis changes preserve lengths and angles
Tip 3: PCA uses eigendecomposition of covariance matrix
Tip 4: Fourier transform changes time domain to frequency domain
Key Properties: Linear transformation, determinant of transition matrix relates volumes
Applications: Data compression, signal processing, image analysis, machine learning
Essential Formulas:

• Change of basis: [v]_B = P_{B←C}[v]_C

• Transition matrix: P_{B←C} = [ [c₁]_B [c₂]_B ... [cₙ]_B ]

• PCA: Covariance matrix C = (1/n)X^T X, eigendecomposition C = VΛV^T

• Fourier series: f(t) = a₀/2 + Σ(aₙcos(nt) + bₙsin(nt))

Solution: Exercises 1 to 3
1 Basic Change of Basis
Exercise 1
Let B = {e₁, e₂} be the standard basis and C = {c₁, c₂} where c₁ = [2, 1], c₂ = [1, 3]. Find the transition matrix P_{B←C} and convert the vector [v]_C = [3, 2] to standard coordinates.
Definition:

Transition Matrix: Converts coordinates from basis C to basis B: [v]_B = P_{B←C}[v]_C

Step-by-step Method:
  1. Express each C-basis vector in B-coordinates
  2. Form transition matrix with these as columns
  3. Apply the transformation to convert coordinates
Step 1: Express C-basis vectors in standard coordinates

c₁ = [2, 1] in standard basis → [c₁]_B =

[
2
1
]

c₂ = [1, 3] in standard basis → [c₂]_B =

[
1
3
]

Step 2: Form the transition matrix P_{B←C}

P_{B←C} = [ [c₁]_B [c₂]_B ] =

[
2
1
1
3
]

Step 3: Convert [v]_C to [v]_B

[v]_B = P_{B←C}[v]_C =

[
2
1
1
3
]
[
3
2
]

=

[
2×3 + 1×2
1×3 + 3×2
]
=
[
8
9
]

Step 4: Verification

Original vector in C-coordinates: v = 3c₁ + 2c₂ = 3[2, 1] + 2[1, 3] = [6, 3] + [2, 6] = [8, 9] ✓

P_{B←C} =
[
2
1
1
3
]
, [v]_B =
[
8
9
]
Final answer:

Vector [v]_C = [3, 2] in basis C equals [v]_B = [8, 9] in standard basis

Applied rules:

• Transition matrix: Columns are new basis vectors in old coordinates

• Matrix multiplication: Row by column rule

• Verification: Check that vector representation is consistent

2 Principal Component Analysis (PCA) Intro
Exercise 2
Apply basic PCA to the dataset: X =
[
1
2
2
3
3
4
4
5
]
(each row is a data point)
Definition:

Principal Component Analysis: Change to basis that maximizes variance along principal components

Step 1: Center the data

Mean of each feature: μ₁ = (1+2+3+4)/4 = 2.5, μ₂ = (2+3+4+5)/4 = 3.5

X_centered =

[
1-2.5
2-3.5
2-2.5
3-3.5
3-2.5
4-3.5
4-2.5
5-3.5
]
=
[
-1.5
-1.5
-0.5
-0.5
0.5
0.5
1.5
1.5
]

Step 2: Compute covariance matrix

C = (1/n)X_centered^T X_centered = (1/4)

[
-1.5
-0.5
0.5
1.5
-1.5
-0.5
0.5
1.5
]
[
-1.5
-1.5
-0.5
-0.5
0.5
0.5
1.5
1.5
]

= (1/4)

[
(-1.5)² + (-0.5)² + (0.5)² + (1.5)²
(-1.5)(-1.5) + (-0.5)(-0.5) + (0.5)(0.5) + (1.5)(1.5)
(-1.5)(-1.5) + (-0.5)(-0.5) + (0.5)(0.5) + (1.5)(1.5)
(-1.5)² + (-0.5)² + (0.5)² + (1.5)²
]

= (1/4)

[
2.25 + 0.25 + 0.25 + 2.25
2.25 + 0.25 + 0.25 + 2.25
2.25 + 0.25 + 0.25 + 2.25
2.25 + 0.25 + 0.25 + 2.25
]
= (1/4)
[
5
5
5
5
]
=
[
1.25
1.25
1.25
1.25
]

Step 3: Find eigenvalues and eigenvectors of covariance matrix

det(C - λI) = det

[
1.25-λ
1.25
1.25
1.25-λ
]
= (1.25-λ)² - (1.25)² = 0

(1.25-λ)² = (1.25)² → 1.25-λ = ±1.25 → λ₁ = 2.5, λ₂ = 0

For λ₁ = 2.5: (C - 2.5I)v = 0 →

[
-1.25
1.25
1.25
-1.25
]
[
x
y
]
=
[
0
0
]

This gives: -1.25x + 1.25y = 0 → y = x, so v₁ =

[
1
1
]
/√2 =
[
1/√2
1/√2
]

For λ₂ = 0: (C)v = 0 → Similar process gives v₂ =

[
1/√2
-1/√2
]

Principal components: PC₁ =
[
1/√2
1/√2
]
(variance = 2.5), PC₂ =
[
1/√2
-1/√2
]
(variance = 0)
Final answer:

First principal component captures 100% of the variance (data lies on a line)

Applied rules:

• Data centering: Subtract mean from each feature

• Covariance matrix: Measures relationships between features

• Eigendecomposition: Finds principal directions of maximum variance

3 Fourier Series Basics
Exercise 3
Find the Fourier coefficients for f(t) = t on the interval [-π, π] using the orthonormal basis {1/√(2π), cos(t)/√π, sin(t)/√π, cos(2t)/√π, sin(2t)/√π, ...}
Definition:

Fourier Series: Representation of periodic function using sines and cosines as basis functions

Step 1: Find the constant term a₀

a₀ = ⟨f(t), 1/√(2π)⟩ = (1/√(2π)) ∫₋π^π t dt

= (1/√(2π)) [t²/2]₋π^π = (1/√(2π)) [π²/2 - π²/2] = 0

Step 2: Find cosine coefficients aₙ

aₙ = ⟨f(t), cos(nt)/√π⟩ = (1/√π) ∫₋π^π t cos(nt) dt

Using integration by parts: u = t, dv = cos(nt)dt → du = dt, v = sin(nt)/n

= (1/√π) [t sin(nt)/n]₋π^π - (1/√π) ∫₋π^π sin(nt)/n dt

= (1/√π) [π sin(nπ)/n - (-π) sin(-nπ)/n] - (1/√π) [-cos(nt)/(n²)]₋π^π

= (1/√π) [0 - 0] - (1/√π) [-cos(nπ)/(n²) - (-cos(-nπ))/(n²)]

= (1/√π) [cos(nπ)/(n²) - cos(nπ)/(n²)] = 0

Wait, let's recalculate more carefully: ∫₋π^π t cos(nt) dt = [t sin(nt)/n]₋π^π - ∫₋π^π sin(nt)/n dt

= π sin(nπ)/n - (-π) sin(-nπ)/n - [−cos(nt)/(n²)]₋π^π

= 0 - 0 - [−cos(nπ)/(n²) + cos(−nπ)/(n²)] = 0

Step 3: Find sine coefficients bₙ

bₙ = ⟨f(t), sin(nt)/√π⟩ = (1/√π) ∫₋π^π t sin(nt) dt

Using integration by parts: u = t, dv = sin(nt)dt → du = dt, v = -cos(nt)/n

= (1/√π) [-t cos(nt)/n]₋π^π + (1/√π) ∫₋π^π cos(nt)/n dt

= (1/√π) [-π cos(nπ)/n - (-(-π) cos(-nπ)/n)] + (1/√π) [sin(nt)/(n²)]₋π^π

= (1/√π) [-π cos(nπ)/n - π cos(nπ)/n] + (1/√π) [0 - 0]

= (1/√π) [-2π cos(nπ)/n] = -2π cos(nπ)/(n√π) = -2√π cos(nπ)/n

Since cos(nπ) = (-1)ⁿ, we have: bₙ = -2√π (-1)ⁿ/n = 2√π (-1)ⁿ⁺¹/n

Step 4: Write the Fourier series

f(t) = Σ bₙ sin(nt)/√π = Σ [2√π (-1)ⁿ⁺¹/n] × sin(nt)/√π = Σ 2(-1)ⁿ⁺¹ sin(nt)/n

= 2[sin(t) - sin(2t)/2 + sin(3t)/3 - sin(4t)/4 + ...]

f(t) = 2∑n=1^∞ [(-1)ⁿ⁺¹ sin(nt)/n] = 2[sin(t) - sin(2t)/2 + sin(3t)/3 - sin(4t)/4 + ...]
Final answer:

Only sine terms appear (since f(t) = t is odd), with coefficients bₙ = 2(-1)ⁿ⁺¹/n

Applied rules:

• Inner product: ⟨f,g⟩ = ∫f(t)g(t)dt for function spaces

• Orthonormal basis: Functions with ⟨φᵢ,φⱼ⟩ = δᵢⱼ

• Integration by parts: Essential for computing Fourier coefficients

Solution: Exercises 4 to 5
4 Change of Basis in PCA Context
Exercise 4
Given the data matrix X from Exercise 2 and the principal component basis from Exercise 2, transform the data point [3, 4] to the PCA coordinate system.
Definition:

PCA Coordinate Transformation: Change to basis of principal components to reduce dimensionality

Step 1: Recall the principal components

PC₁ =

[
1/√2
1/√2
]
, PC₂ =
[
1/√2
-1/√2
]

Step 2: Center the data point

Original point: [3, 4]

Centered point: [3, 4] - [2.5, 3.5] = [0.5, 0.5]

Step 3: Project onto principal components

Coordinate in PC₁ direction: ⟨[0.5, 0.5], PC₁⟩ = [0.5, 0.5] ·

[
1/√2
1/√2
]
= 0.5×(1/√2) + 0.5×(1/√2) = 1/√2

Coordinate in PC₂ direction: ⟨[0.5, 0.5], PC₂⟩ = [0.5, 0.5] ·

[
1/√2
-1/√2
]
= 0.5×(1/√2) + 0.5×(-1/√2) = 0

Step 4: Form the transformation matrix

The transformation matrix from standard coordinates to PCA coordinates is:

P_{PCA←std} =

[
PC₁[0]
PC₂[0]
PC₁[1]
PC₂[1]
]
=
[
1/√2
1/√2
1/√2
-1/√2
]

Wait, this is actually P_{std←PCA}. The correct transformation is P_{PCA←std} = (P_{std←PCA})^T =

[
1/√2
1/√2
1/√2
-1/√2
]

Step 5: Apply the transformation

[v]_{PCA} = P_{PCA←std}[v]_{std} =

[
1/√2
1/√2
1/√2
-1/√2
]
[
0.5
0.5
]

=

[
(1/√2)×0.5 + (1/√2)×0.5
(1/√2)×0.5 + (-1/√2)×0.5
]
=
[
1/√2
0
]

The point [3, 4] in standard coordinates becomes [1/√2, 0] in PCA coordinates
Final answer:

Point [3, 4] projects to [1/√2, 0] in PCA coordinate system (along PC₁ direction)

Applied rules:

• Data centering: Always center data before PCA transformation

• Projection: Inner product measures component along basis vector

• Dimensionality reduction: Small components can be ignored

5 Signal Processing via Fourier Transform
Exercise 5
A digital signal is represented by the vector x = [1, 2, 1, 0]. Find its discrete Fourier transform (DFT) and interpret the results in terms of frequency components.
Definition:

Discrete Fourier Transform: Change of basis from time domain to frequency domain using complex exponentials

Step 1: Define the DFT formula

X[k] = Σn=0^{N-1} x[n] e^{-2πikn/N} for k = 0, 1, ..., N-1

where N = 4 (length of input signal), x = [1, 2, 1, 0]

Step 2: Calculate X[0] (DC component)

X[0] = Σn=0³ x[n] e^{-2πi×0×n/4} = Σn=0³ x[n] = 1 + 2 + 1 + 0 = 4

Step 3: Calculate X[1] (fundamental frequency)

X[1] = Σn=0³ x[n] e^{-2πi×1×n/4} = x[0]e^0 + x[1]e^{-πi/2} + x[2]e^{-πi} + x[3]e^{-3πi/2}

= 1×1 + 2×e^{-πi/2} + 1×e^{-πi} + 0×e^{-3πi/2}

= 1 + 2×(-i) + 1×(-1) + 0

= 1 - 2i - 1 = -2i

Step 4: Calculate X[2] (second harmonic)

X[2] = Σn=0³ x[n] e^{-2πi×2×n/4} = Σn=0³ x[n] e^{-πin}

= x[0]e^0 + x[1]e^{-πi} + x[2]e^{-2πi} + x[3]e^{-3πi}

= 1×1 + 2×(-1) + 1×1 + 0×(-1)

= 1 - 2 + 1 + 0 = 0

Step 5: Calculate X[3] (third harmonic)

X[3] = Σn=0³ x[n] e^{-2πi×3×n/4} = x[0]e^0 + x[1]e^{-3πi/2} + x[2]e^{-3πi} + x[3]e^{-9πi/2}

= 1×1 + 2×e^{-3πi/2} + 1×e^{-3πi} + 0×e^{-πi/2}

= 1 + 2×i + 1×(-1) + 0

= 1 + 2i - 1 = 2i

Step 6: Interpret the results

DFT: X = [4, -2i, 0, 2i]

Magnitude spectrum: |X| = [|4|, |-2i|, |0|, |2i|] = [4, 2, 0, 2]

Phase spectrum: arg(X) = [0, -π/2, 0, π/2]

Interpretation: - DC component (frequency 0): magnitude 4 - Fundamental frequency: magnitude 2, phase -π/2 - Second harmonic: no contribution - Third harmonic: magnitude 2, phase π/2

X = [4, -2i, 0, 2i] in frequency domain, showing dominant DC and alternating components
Final answer:

The signal has strong DC component and alternating pattern at fundamental frequency

Applied rules:

• Complex exponentials: e^{iθ} = cos(θ) + i sin(θ)

• Frequency interpretation: Higher indices correspond to higher frequencies

• Hermitian symmetry: For real signals, X[N-k] = X*[k]

Questions & Answers

Question: I'm confused about the difference between PCA and regular change of basis. How is PCA different?

Answer: Great question! The key difference is in the selection criteria for the new basis:

  • Regular change of basis: Any set of linearly independent vectors can form a new basis
  • PCA: The new basis vectors (principal components) are specifically chosen to maximize variance along each direction

In PCA, the basis vectors are the eigenvectors of the covariance matrix, ordered by their corresponding eigenvalues (which represent the variance explained). This ensures that the first few components capture most of the data's variability.

In Exercise 2, we saw that the first principal component captured 100% of the variance because our data lay perfectly on a line. This is what makes PCA so powerful for dimensionality reduction!

Question: In Exercise 3, why did we only get sine terms in the Fourier series for f(t) = t? Shouldn't we have both sine and cosine terms?

Answer: This is a beautiful example of how function symmetry determines Fourier coefficients:

  • Odd function: f(t) = t satisfies f(-t) = -f(t)
  • Even functions: cos(nt) satisfy cos(-nt) = cos(nt)
  • Integral property: ∫₋π^π (odd × even) dt = 0

Since f(t) = t is odd and cos(nt) is even, their product t cos(nt) is odd, so ∫₋π^π t cos(nt) dt = 0. This means all cosine coefficients aₙ = 0.

On the other hand, both f(t) = t and sin(nt) are odd, so their product t sin(nt) is even, giving non-zero sine coefficients bₙ. This is why only sine terms appear in the Fourier series of an odd function.

Question: How do we decide how many principal components to keep in PCA for dimensionality reduction?

Answer: There are several common approaches to select the number of principal components:

  1. Variance threshold: Keep enough components to explain, say, 95% of total variance
  2. Elbow method: Plot eigenvalues and look for the "elbow" where additional components add little
  3. Kaiser criterion: Keep components with eigenvalues > 1 (for standardized data)
  4. Cross-validation: Select components that optimize performance on held-out data

In Exercise 2, we had eigenvalues [2.5, 0], so the first component explains 100% of variance. In real applications, the eigenvalues typically decay gradually, requiring a trade-off between dimensionality and information loss.

The cumulative proportion of variance explained is often plotted to guide this decision!