Solved Exercises on Orthogonal Projections onto Subspaces in Linear Algebra

Master orthogonal projections onto subspaces: vector projections, matrix representations, applications, and geometric interpretations through these 5 detailed exercises.

Solution: Exercises 1 to 3
1 Projection onto Line in R²
Exercise 1
Find the orthogonal projection of the vector
\(\vec{v} = \begin{pmatrix} 3 \\ 4 \end{pmatrix}\)
onto the line spanned by
\(\vec{u} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}\).
Definition:

Orthogonal projection: The closest point in the subspace to the given vector, obtained by dropping a perpendicular from the vector to the subspace

Projection formula:

\(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

  1. Calculate the dot product \(\vec{v} \cdot \vec{u}\)
  2. Calculate the dot product \(\vec{u} \cdot \vec{u}\)
  3. Divide the results and multiply by \(\vec{u}\)
Input
\(\vec{v} = \begin{pmatrix} 3 \\ 4 \end{pmatrix}, \vec{u} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}\)
Dot Products
\(\vec{v} \cdot \vec{u} = 11, \vec{u} \cdot \vec{u} = 5\)
Result
\(\text{proj}_{\vec{u}}\vec{v} = \frac{11}{5}\vec{u}\)
Step 1: Calculate \(\vec{v} \cdot \vec{u}\)

\(\vec{v} \cdot \vec{u} = (3)(1) + (4)(2) = 3 + 8 = 11\)

Step 2: Calculate \(\vec{u} \cdot \vec{u}\)

\(\vec{u} \cdot \vec{u} = (1)(1) + (2)(2) = 1 + 4 = 5\)

Step 3: Apply projection formula

\(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u} = \frac{11}{5}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\)

Step 4: Calculate final result

\(\text{proj}_{\vec{u}}\vec{v} = \begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix}\)

Step 5: Verify orthogonality

\(\vec{v} - \text{proj}_{\vec{u}}\vec{v} = \begin{pmatrix} 3 \\ 4 \end{pmatrix} - \begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix} = \begin{pmatrix} \frac{4}{5} \\ -\frac{2}{5} \end{pmatrix}\)

\(\left(\vec{v} - \text{proj}_{\vec{u}}\vec{v}\right) \cdot \vec{u} = \frac{4}{5}(1) + \left(-\frac{2}{5}\right)(2) = \frac{4}{5} - \frac{4}{5} = 0\)

\(\text{proj}_{\vec{u}}\vec{v} = \begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix}\)
Final answer:

The orthogonal projection of \(\vec{v}\) onto the line spanned by \(\vec{u}\) is \(\begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix}\)

Applied rules:

• Projection Formula: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

• Orthogonality Property: \(\vec{v} - \text{proj}_{\vec{u}}\vec{v} \perp \vec{u}\)

• Dot Product: \(\vec{a} \cdot \vec{b} = \sum_{i=1}^{n} a_i b_i\)

2 Projection onto Plane in R³
Exercise 2
Find the orthogonal projection of the vector
\(\vec{w} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\)
onto the plane spanned by
\(\vec{u_1} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\)
and
\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}\).
Definition:

Plane projection: When the basis vectors are orthogonal, the projection is the sum of individual projections onto each basis vector

Input
\(\vec{w}, \vec{u_1}, \vec{u_2}\)
Individual Projections
\(\text{proj}_{\vec{u_1}}\vec{w}, \text{proj}_{\vec{u_2}}\vec{w}\)
Total Projection
\(\text{proj}_{W}\vec{w} = \text{proj}_{\vec{u_1}}\vec{w} + \text{proj}_{\vec{u_2}}\vec{w}\)
Step 1: Calculate projection onto \(\vec{u_1}\)

\(\text{proj}_{\vec{u_1}}\vec{w} = \frac{\vec{w} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1} = \frac{(1)(1) + (2)(0) + (3)(0)}{1}\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\)

Step 2: Calculate projection onto \(\vec{u_2}\)

\(\text{proj}_{\vec{u_2}}\vec{w} = \frac{\vec{w} \cdot \vec{u_2}}{\vec{u_2} \cdot \vec{u_2}}\vec{u_2} = \frac{(1)(0) + (2)(1) + (3)(0)}{1}\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}\)

Step 3: Sum the projections

\(\text{proj}_{W}\vec{w} = \text{proj}_{\vec{u_1}}\vec{w} + \text{proj}_{\vec{u_2}}\vec{w} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} + \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\)

Step 4: Verify result

The projected vector \(\begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\) lies in the xy-plane (z=0), which is the subspace spanned by \(\vec{u_1}\) and \(\vec{u_2}\)

Step 5: Check orthogonality

\(\vec{w} - \text{proj}_{W}\vec{w} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} - \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 3 \end{pmatrix}\)

This residual vector is orthogonal to both \(\vec{u_1}\) and \(\vec{u_2}\) (has zero z-components)

\(\text{proj}_{W}\vec{w} = \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\)
Final answer:

The orthogonal projection of \(\vec{w}\) onto the plane spanned by \(\vec{u_1}\) and \(\vec{u_2}\) is \(\begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\)

Applied rules:

• Subspace Projection: \(\text{proj}_{W}\vec{v} = \sum_{i=1}^{k} \text{proj}_{\vec{u_i}}\vec{v}\) (for orthogonal basis)

• Orthogonal Basis: When basis vectors are orthogonal, projections add directly

• Residual Vector: \(\vec{v} - \text{proj}_{W}\vec{v}\) is orthogonal to the subspace

3 Projection Matrix Construction
Exercise 3
Find the projection matrix \(P\) that projects any vector in \(\mathbb{R}^3\) onto the subspace spanned by
\(\vec{u_1} = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}\)
and
\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}\).
Definition:

Projection matrix: A matrix \(P\) such that \(P\vec{v}\) gives the orthogonal projection of \(\vec{v}\) onto the subspace

Basis Matrix
\(A = [\vec{u_1}, \vec{u_2}]\)
Matrix Formula
\(P = A(A^TA)^{-1}A^T\)
Result
Projection matrix \(P\)
Step 1: Form matrix A

\(A = \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix}\)

Step 2: Calculate \(A^T\)

\(A^T = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}\)

Step 3: Calculate \(A^TA\)

\(A^TA = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\)

Step 4: Calculate \((A^TA)^{-1}\)

\(\det(A^TA) = (2)(2) - (1)(1) = 3\)

\((A^TA)^{-1} = \frac{1}{3}\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}\)

Step 5: Calculate final projection matrix

\(P = A(A^TA)^{-1}A^T = \frac{1}{3}\begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}\)

\(P = \frac{1}{3}\begin{bmatrix} 2 & -1 \\ 1 & 1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}\)

\(P = \frac{1}{3}\begin{bmatrix} 2 & 1 & -1 \\ 1 & 2 & 1 \\ -1 & 1 & 2 \end{bmatrix}\)

Step 6: Verify properties of projection matrix

Check idempotence: \(P^2 = P\)

Check symmetry: \(P^T = P\)

\(P = \frac{1}{3}\begin{bmatrix} 2 & 1 & -1 \\ 1 & 2 & 1 \\ -1 & 1 & 2 \end{bmatrix}\)
Final answer:

The projection matrix is \(P = \frac{1}{3}\begin{bmatrix} 2 & 1 & -1 \\ 1 & 2 & 1 \\ -1 & 1 & 2 \end{bmatrix}\)

Applied rules:

• Matrix Formula: \(P = A(A^TA)^{-1}A^T\) where columns of A form basis

• Idempotence: \(P^2 = P\) for any projection matrix

• Symmetry: \(P^T = P\) for orthogonal projection matrices

Complete Theory: Orthogonal Projections
\(P = A(A^TA)^{-1}A^T\)
Projection Matrix Formula
Line Proj.
\(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)
Single vector
Subspace Proj.
\(\text{proj}_{W}\vec{v} = \sum_{i=1}^{k} \text{proj}_{\vec{u_i}}\vec{v}\)
Orthogonal basis
Properties
\(P^2 = P, P^T = P\)
Idempotent & symmetric
Key definitions:

Orthogonal Projection: The closest point in a subspace to a given vector, minimizing the distance between them

Projection Matrix: A matrix \(P\) such that \(P\vec{v}\) is the orthogonal projection of \(\vec{v}\) onto a subspace

Subspace: A subset of a vector space that is closed under addition and scalar multiplication

Orthogonal Complement: The set of all vectors orthogonal to a given subspace

Complete methodology:
  1. Single vector projection: Use the standard projection formula
  2. Subspace projection: For orthogonal basis, sum individual projections
  3. General subspace: Use Gram-Schmidt to orthogonalize, then sum projections
  4. Matrix approach: Use the matrix formula \(P = A(A^TA)^{-1}A^T\)
Tip 1: Always verify that the residual vector is orthogonal to the subspace.
Tip 2: For orthogonal bases, projections add directly without Gram-Schmidt.
Tip 3: Projection matrices are always idempotent (\(P^2 = P\)).
Tip 4: The range of a projection matrix is the subspace being projected onto.
Applications: Least squares regression, signal processing, computer graphics, data compression.
Advanced Uses: Principal Component Analysis (PCA), eigenvalue problems, optimization.
Formulas to know by heart:

• Single vector: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

• Subspace (orthogonal basis): \(\text{proj}_{W}\vec{v} = \sum_{i=1}^{k} \frac{\vec{v} \cdot \vec{u_i}}{\vec{u_i} \cdot \vec{u_i}}\vec{u_i}\)

• Matrix form: \(P = A(A^TA)^{-1}A^T\)

• Orthogonality: \(\vec{v} - P\vec{v} \perp \text{range}(P)\)

• Properties: \(P^2 = P\) (idempotent), \(P^T = P\) (symmetric)

Solution: Exercises 4 to 5
4 Least Squares Application
Exercise 4
Find the best fit line through the points \((1,2)\), \((2,3)\), and \((3,5)\) using orthogonal projections.
Definition:

Least squares: The method of finding the line that minimizes the sum of squared residuals, equivalent to orthogonal projection onto the model subspace

Data Points
\((1,2), (2,3), (3,5)\)
Design Matrix
\(A = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}\)
Solution
\(y = 1.5 + x\)
Step 1: Set up the system

We want to find coefficients \(a\) and \(b\) for the line \(y = a + bx\) that best fits the data

Set up the system: \(\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \\ 5 \end{bmatrix}\)

Step 2: Form the normal equations

Since the system is overdetermined, solve \(A^TA\vec{x} = A^T\vec{b}\)

\(A^T = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix}\)

\(A^TA = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}\)

\(A^T\vec{b} = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \\ 5 \end{bmatrix} = \begin{bmatrix} 10 \\ 23 \end{bmatrix}\)

Step 3: Solve the normal equations

\(\begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 10 \\ 23 \end{bmatrix}\)

From the first equation: \(3a + 6b = 10\), so \(a = \frac{10 - 6b}{3}\)

Substituting into the second: \(6(\frac{10 - 6b}{3}) + 14b = 23\)

\(2(10 - 6b) + 14b = 23\)

\(20 - 12b + 14b = 23\)

\(2b = 3\), so \(b = 1.5\)

Then \(a = \frac{10 - 6(1.5)}{3} = \frac{10 - 9}{3} = \frac{1}{3}\)

Step 4: Verify the solution

The best fit line is \(y = \frac{1}{3} + 1.5x\)

Predicted values: \((1, 1.83), (2, 3.33), (3, 4.83)\)

Residuals: \((0.17, -0.33, 0.17)\) - small and balanced

Best fit line: \(y = \frac{1}{3} + 1.5x\)
Final answer:

The best fit line is \(y = \frac{1}{3} + 1.5x\) (approximately \(y = 0.33 + 1.5x\))

Applied rules:

• Normal Equations: \(A^TA\vec{x} = A^T\vec{b}\) for least squares solution

• Geometric Interpretation: Projection of \(\vec{b}\) onto the column space of \(A\)

• Minimization: Minimizes \(\|A\vec{x} - \vec{b}\|^2\)

5 Orthogonal Decomposition
Exercise 5
Let \(W\) be the subspace spanned by
\(\vec{u_1} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\)
and
\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}\).
Decompose
\(\vec{v} = \begin{pmatrix} 2 \\ 3 \\ 1 \end{pmatrix}\)
as
\(\vec{v} = \vec{w} + \vec{z}\)
where
\(\vec{w} \in W\)
and
\(\vec{z} \in W^\perp\).
Definition:

Orthogonal decomposition: Any vector can be uniquely written as the sum of a component in a subspace and a component in its orthogonal complement

Input Vector
\(\vec{v} = \begin{pmatrix} 2 \\ 3 \\ 1 \end{pmatrix}\)
Subspace Proj.
\(\vec{w} = \text{proj}_{W}\vec{v}\)
Complement
\(\vec{z} = \vec{v} - \vec{w}\)
Step 1: Apply Gram-Schmidt to get orthogonal basis

\(\vec{u_1} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\)

\(\vec{u_2}' = \vec{u_2} - \text{proj}_{\vec{u_1}}\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \frac{1}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix}\)

Step 2: Calculate projection onto \(\vec{u_1}\)

\(\text{proj}_{\vec{u_1}}\vec{v} = \frac{\vec{v} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1} = \frac{(2)(1) + (3)(0) + (1)(1)}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \frac{3}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} 3/2 \\ 0 \\ 3/2 \end{pmatrix}\)

Step 3: Calculate projection onto \(\vec{u_2}'\)

\(\text{proj}_{\vec{u_2}'}\vec{v} = \frac{\vec{v} \cdot \vec{u_2}'}{\vec{u_2}' \cdot \vec{u_2}'}\vec{u_2}'\)

\(\vec{v} \cdot \vec{u_2}' = (2)(-1/2) + (3)(1) + (1)(1/2) = -1 + 3 + 1/2 = 2.5\)

\(\vec{u_2}' \cdot \vec{u_2}' = (-1/2)^2 + (1)^2 + (1/2)^2 = 1/4 + 1 + 1/4 = 1.5\)

\(\text{proj}_{\vec{u_2}'}\vec{v} = \frac{2.5}{1.5}\begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix} = \frac{5}{3}\begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix} = \begin{pmatrix} -5/6 \\ 5/3 \\ 5/6 \end{pmatrix}\)

Step 4: Calculate total projection

\(\vec{w} = \text{proj}_{W}\vec{v} = \text{proj}_{\vec{u_1}}\vec{v} + \text{proj}_{\vec{u_2}'}\vec{v}\)

\(\vec{w} = \begin{pmatrix} 3/2 \\ 0 \\ 3/2 \end{pmatrix} + \begin{pmatrix} -5/6 \\ 5/3 \\ 5/6 \end{pmatrix} = \begin{pmatrix} 9/6 - 5/6 \\ 5/3 \\ 9/6 + 5/6 \end{pmatrix} = \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix}\)

Step 5: Calculate orthogonal component

\(\vec{z} = \vec{v} - \vec{w} = \begin{pmatrix} 2 \\ 3 \\ 1 \end{pmatrix} - \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix} = \begin{pmatrix} 4/3 \\ 4/3 \\ -4/3 \end{pmatrix}\)

Step 6: Verify orthogonality

\(\vec{z} \cdot \vec{u_1} = \frac{4}{3}(1) + \frac{4}{3}(0) + \left(-\frac{4}{3}\right)(1) = \frac{4}{3} - \frac{4}{3} = 0\)

\(\vec{z} \cdot \vec{u_2} = \frac{4}{3}(0) + \frac{4}{3}(1) + \left(-\frac{4}{3}\right)(1) = \frac{4}{3} - \frac{4}{3} = 0\)

\(\vec{v} = \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix} + \begin{pmatrix} 4/3 \\ 4/3 \\ -4/3 \end{pmatrix}\)
Final answer:

\(\vec{v} = \vec{w} + \vec{z}\) where \(\vec{w} = \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix} \in W\) and \(\vec{z} = \begin{pmatrix} 4/3 \\ 4/3 \\ -4/3 \end{pmatrix} \in W^\perp\)

Applied rules:

• Orthogonal Decomposition: \(\vec{v} = \text{proj}_{W}\vec{v} + (\vec{v} - \text{proj}_{W}\vec{v})\)

• Orthogonal Basis: Required for direct summation of projections

• Direct Sum: \(\mathbb{R}^n = W \oplus W^\perp\)

Detailed Summary: Orthogonal Projections onto Subspaces
\(\vec{v} = \text{proj}_{W}\vec{v} + \text{proj}_{W^\perp}\vec{v}\)
Orthogonal Decomposition
Key definitions:

Orthogonal Projection: The mapping of a vector to the closest point in a subspace, measured by Euclidean distance

Projection Matrix: A matrix \(P\) satisfying \(P^2 = P\) that implements the orthogonal projection operation

Orthogonal Complement: The set of all vectors orthogonal to every vector in a given subspace

Subspace: A vector space contained within another vector space, closed under vector addition and scalar multiplication

Complete methodology:
  1. Identify the subspace: Determine a basis for the subspace onto which you're projecting
  2. Orthogonalize if needed: Use Gram-Schmidt to create an orthogonal basis
  3. Apply projection formula: Sum individual projections onto each orthogonal basis vector
  4. Verify results: Check that the residual vector is orthogonal to the subspace
Tip 1: For orthogonal bases, you can sum projections directly without Gram-Schmidt.
Tip 2: Always verify that your projection matrix is idempotent (\(P^2 = P\)).
Tip 3: The rank of a projection matrix equals the dimension of the subspace.
Tip 4: The trace of a projection matrix equals its rank.
Applications: Least squares regression, signal filtering, image compression, computer vision, machine learning.
Advanced Uses: Principal Component Analysis (PCA), singular value decomposition, eigenvalue problems, optimization theory.
Core principles:

• Best Approximation: The projection is the closest point in the subspace to the original vector

• Orthogonality Principle: The error vector is orthogonal to the subspace being projected onto

• Decomposition: Every vector can be uniquely decomposed into subspace and orthogonal complement components

• Minimization: Orthogonal projection minimizes the Euclidean distance to the subspace

Visualization: Projection Geometry
Exercise 6: Geometric Interpretation
Understanding the geometric meaning of orthogonal projections:
- The shortest distance from a point to a subspace is along the perpendicular
- The residual vector is always orthogonal to the subspace
- Projections preserve linearity and structure

Analysis: The visualization shows how orthogonal projections minimize distance and maintain geometric relationships.

  • Original vector: The starting point
  • Projected vector: Closest point in the subspace
  • Residual vector: The error, always perpendicular to subspace

Questions & Answers

Question: What's the relationship between orthogonal projections and least squares problems?

Answer: There's a fundamental connection between orthogonal projections and least squares:

  • Geometric interpretation: Finding the least squares solution means projecting the observation vector onto the column space of the design matrix.
  • Minimization: The projection minimizes the Euclidean norm of the residual vector.
  • Normal equations: The least squares solution comes from the condition that the residual is orthogonal to the column space.

When solving \(A\vec{x} = \vec{b}\) where the system is inconsistent, the least squares solution \(\vec{x}^*\) satisfies \(A\vec{x}^* = \text{proj}_{\text{col}(A)}\vec{b}\), which is the closest point to \(\vec{b}\) in the column space of \(A\).

This is why the normal equations \(A^TA\vec{x} = A^T\vec{b}\) arise naturally from the orthogonality condition.

Question: Why are projection matrices idempotent (\(P^2 = P\))? What's the intuition?

Answer: The idempotence property has a beautiful geometric interpretation:

  • Once projected, stay projected: If you project a vector onto a subspace, and then project the result again, you get the same thing back.
  • No further reduction: The projection has already found the closest point in the subspace, so projecting again doesn't change anything.
  • Algebraic verification: \(P^2\vec{v} = P(P\vec{v}) = P\vec{v}\) since \(P\vec{v}\) is already in the subspace.

Think of it like flattening a 3D object onto a 2D surface - once it's flat, flattening it again doesn't change anything. The projection matrix "flattens" vectors onto the subspace, and once flattened, they stay flattened.

This property characterizes projection operators completely: any matrix satisfying \(P^2 = P\) is a projection matrix.

Question: How do orthogonal projections relate to statistics and regression analysis?

Answer: Orthogonal projections are fundamental to statistical analysis:

  • Linear regression: The fitted values are the orthogonal projection of the response vector onto the column space of predictors.
  • Residuals: The difference between observed and fitted values is the component orthogonal to the predictor space.
  • Variance decomposition: Total variance = Explained variance + Residual variance (Pythagorean theorem).

In multiple regression, if \(Y\) is the response vector and \(X\) is the design matrix, then:

  • Fitted values: \(\hat{Y} = PY\) where \(P = X(X^TX)^{-1}X^T\)
  • Residuals: \(e = Y - \hat{Y} = (I-P)Y\)
  • Total sum of squares: \(Y^TY = \hat{Y}^T\hat{Y} + e^Te\) (orthogonal decomposition)

This geometric perspective provides deep insights into the properties of regression estimators and hypothesis tests.