SVD: Any m×n matrix A can be factored as A = UΣV^T where U and V are orthogonal matrices and Σ is diagonal
Singular Values σᵢ: Square roots of eigenvalues of A^T A (or AA^T), arranged in descending order
Left Singular Vectors uᵢ: Orthonormal eigenvectors of AA^T
Right Singular Vectors vᵢ: Orthonormal eigenvectors of A^T A
- Compute A^T A: Find eigenvalues and eigenvectors
- Find singular values: σᵢ = √λᵢ where λᵢ are eigenvalues of A^T A
- Find right singular vectors: vᵢ are normalized eigenvectors of A^T A
- Find left singular vectors: uᵢ = (1/σᵢ)Avᵢ
- Form matrices: U from left vectors, Σ from singular values, V from right vectors
• SVD: A = UΣV^T
• Singular values: σᵢ = √λᵢ(A^T A)
• Left singular vectors: uᵢ = (1/σᵢ)Avᵢ
• Matrix norm: ||A||₂ = σ₁
SVD Decomposition: Express A as A = UΣV^T where U and V are orthogonal and Σ is diagonal
- Compute A^T A
- Find eigenvalues and eigenvectors of A^T A
- Calculate singular values σᵢ = √λᵢ
- Find right singular vectors vᵢ
- Find left singular vectors uᵢ = (1/σᵢ)Avᵢ
- Form U, Σ, and V matrices
A^T A =
det(A^T A - λI) = det
Therefore: λ₁ = 25, λ₂ = 0
σ₁ = √λ₁ = √25 = 5, σ₂ = √λ₂ = √0 = 0
For λ₁ = 25: (A^T A - 25I)v = 0 →
This gives y = 0, so v₁ =
For λ₂ = 0: (A^T A)v = 0 →
This gives x = 0, so v₂ =
u₁ = (1/σ₁)Av₁ = (1/5)
For u₂, we need a unit vector orthogonal to u₁. We can choose u₂ =
A = UΣV^T with one non-zero singular value
• SVD computation: Systematic approach to factor any matrix
• Rank: Number of non-zero singular values equals rank
• Orthogonality: U and V are orthogonal matrices
Full Rank SVD: When A has full column rank, all singular values are non-zero
A^T A =
det(A^T A - λI) = det
= (1-λ)(2-λ) - 1 = 2 - 3λ + λ² - 1 = λ² - 3λ + 1
Using quadratic formula: λ = (3 ± √(9-4))/2 = (3 ± √5)/2
So λ₁ = (3 + √5)/2 ≈ 2.618, λ₂ = (3 - √5)/2 ≈ 0.382
σ₁ = √λ₁ = √((3 + √5)/2) ≈ 1.618, σ₂ = √λ₂ = √((3 - √5)/2) ≈ 0.618
For λ₁ = (3 + √5)/2: (A^T A - λ₁I)v = 0
Row 1: ((-1-√5)/2)x + y = 0, so y = ((1+√5)/2)x
Normalizing: v₁ = (1/√(1 + ((1+√5)/2)²))
For λ₂ = (3 - √5)/2: Similarly, v₂ ≈
u₁ = (1/σ₁)Av₁ ≈ (1/1.618)
u₂ = (1/σ₂)Av₂ ≈ (1/0.618)
A = UΣV^T with both singular values non-zero (full rank)
• Full rank: All singular values are positive
• Condition number: Ratio of largest to smallest singular value
• Accuracy: Exact symbolic values preferred over approximations
Verification: Confirm SVD by computing UΣV^T and comparing to A
U =
V^T =
ΣV^T =
UΣV^T =
=
UΣV^T = A, confirming our SVD decomposition is correct
• Matrix multiplication: Row by column rule
• Verification: Essential to confirm correctness
• Orthogonality: U^T U = I and V^T V = I
Low-Rank Approximation: A_k = U_k Σ_k V_k^T using only first k singular values/vectors
A = UΣV^T where U =
For rank-1 approximation, keep only the first singular value and corresponding vectors:
U₁ =
A₁ = u₁σ₁v₁^T =
=
By the Eckart-Young theorem, A₁ minimizes ||A - B||₂ among all rank-1 matrices B
In this case, A₁ = A since A already has rank 1
Best rank-1 approximation is the matrix itself since A has rank 1
• Eckart-Young theorem: SVD provides optimal low-rank approximations
• Rank determination: Number of non-zero singular values
• Compression: Store only k columns of U, k rows of V, and k singular values
Frobenius norm: ||A||_F = √(Σᵢ σᵢ²) (square root of sum of squared singular values)
Spectral norm: ||A||₂ = σ₁ (largest singular value)
From Exercise 2: σ₁ = √((3 + √5)/2) ≈ 1.618, σ₂ = √((3 - √5)/2) ≈ 0.618
||A||_F = √(σ₁² + σ₂²) = √((3 + √5)/2 + (3 - √5)/2) = √(3 + √5 + 3 - √5)/2 = √3
Verification: ||A||_F = √(1² + 1² + 0² + 1²) = √3 ✓
||A||₂ = σ₁ = √((3 + √5)/2) ≈ 1.618
The spectral norm ||A||₂ = σ₁ represents the maximum "stretching" factor of A
The Frobenius norm ||A||_F = √3 represents the total energy in the matrix
Relationship: ||A||₂ ≤ ||A||_F ≤ √rank(A) × ||A||₂
Frobenius norm: √3, Spectral norm: √((3 + √5)/2)
• Norm computation: SVD makes matrix norms trivial to calculate
• Spectral norm: Equals largest singular value
• Frobenius norm: Equals square root of sum of squared singular values