\(\vec{v} = \begin{pmatrix} 3 \\ 4 \end{pmatrix}\)
onto the line spanned by
\(\vec{u} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}\).
Orthogonal projection: The closest point in the subspace to the given vector, obtained by dropping a perpendicular from the vector to the subspace
\(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)
- Calculate the dot product \(\vec{v} \cdot \vec{u}\)
- Calculate the dot product \(\vec{u} \cdot \vec{u}\)
- Divide the results and multiply by \(\vec{u}\)
\(\vec{v} \cdot \vec{u} = (3)(1) + (4)(2) = 3 + 8 = 11\)
\(\vec{u} \cdot \vec{u} = (1)(1) + (2)(2) = 1 + 4 = 5\)
\(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u} = \frac{11}{5}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\)
\(\text{proj}_{\vec{u}}\vec{v} = \begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix}\)
\(\vec{v} - \text{proj}_{\vec{u}}\vec{v} = \begin{pmatrix} 3 \\ 4 \end{pmatrix} - \begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix} = \begin{pmatrix} \frac{4}{5} \\ -\frac{2}{5} \end{pmatrix}\)
\(\left(\vec{v} - \text{proj}_{\vec{u}}\vec{v}\right) \cdot \vec{u} = \frac{4}{5}(1) + \left(-\frac{2}{5}\right)(2) = \frac{4}{5} - \frac{4}{5} = 0\)
The orthogonal projection of \(\vec{v}\) onto the line spanned by \(\vec{u}\) is \(\begin{pmatrix} \frac{11}{5} \\ \frac{22}{5} \end{pmatrix}\)
• Projection Formula: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)
• Orthogonality Property: \(\vec{v} - \text{proj}_{\vec{u}}\vec{v} \perp \vec{u}\)
• Dot Product: \(\vec{a} \cdot \vec{b} = \sum_{i=1}^{n} a_i b_i\)
\(\vec{w} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\)
onto the plane spanned by
\(\vec{u_1} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\)
and
\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}\).
Plane projection: When the basis vectors are orthogonal, the projection is the sum of individual projections onto each basis vector
\(\text{proj}_{\vec{u_1}}\vec{w} = \frac{\vec{w} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1} = \frac{(1)(1) + (2)(0) + (3)(0)}{1}\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\)
\(\text{proj}_{\vec{u_2}}\vec{w} = \frac{\vec{w} \cdot \vec{u_2}}{\vec{u_2} \cdot \vec{u_2}}\vec{u_2} = \frac{(1)(0) + (2)(1) + (3)(0)}{1}\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}\)
\(\text{proj}_{W}\vec{w} = \text{proj}_{\vec{u_1}}\vec{w} + \text{proj}_{\vec{u_2}}\vec{w} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} + \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\)
The projected vector \(\begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\) lies in the xy-plane (z=0), which is the subspace spanned by \(\vec{u_1}\) and \(\vec{u_2}\)
\(\vec{w} - \text{proj}_{W}\vec{w} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} - \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 3 \end{pmatrix}\)
This residual vector is orthogonal to both \(\vec{u_1}\) and \(\vec{u_2}\) (has zero z-components)
The orthogonal projection of \(\vec{w}\) onto the plane spanned by \(\vec{u_1}\) and \(\vec{u_2}\) is \(\begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}\)
• Subspace Projection: \(\text{proj}_{W}\vec{v} = \sum_{i=1}^{k} \text{proj}_{\vec{u_i}}\vec{v}\) (for orthogonal basis)
• Orthogonal Basis: When basis vectors are orthogonal, projections add directly
• Residual Vector: \(\vec{v} - \text{proj}_{W}\vec{v}\) is orthogonal to the subspace
\(\vec{u_1} = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}\)
and
\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}\).
Projection matrix: A matrix \(P\) such that \(P\vec{v}\) gives the orthogonal projection of \(\vec{v}\) onto the subspace
\(A = \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix}\)
\(A^T = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}\)
\(A^TA = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\)
\(\det(A^TA) = (2)(2) - (1)(1) = 3\)
\((A^TA)^{-1} = \frac{1}{3}\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}\)
\(P = A(A^TA)^{-1}A^T = \frac{1}{3}\begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}\)
\(P = \frac{1}{3}\begin{bmatrix} 2 & -1 \\ 1 & 1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}\)
\(P = \frac{1}{3}\begin{bmatrix} 2 & 1 & -1 \\ 1 & 2 & 1 \\ -1 & 1 & 2 \end{bmatrix}\)
Check idempotence: \(P^2 = P\)
Check symmetry: \(P^T = P\)
The projection matrix is \(P = \frac{1}{3}\begin{bmatrix} 2 & 1 & -1 \\ 1 & 2 & 1 \\ -1 & 1 & 2 \end{bmatrix}\)
• Matrix Formula: \(P = A(A^TA)^{-1}A^T\) where columns of A form basis
• Idempotence: \(P^2 = P\) for any projection matrix
• Symmetry: \(P^T = P\) for orthogonal projection matrices
Orthogonal Projection: The closest point in a subspace to a given vector, minimizing the distance between them
Projection Matrix: A matrix \(P\) such that \(P\vec{v}\) is the orthogonal projection of \(\vec{v}\) onto a subspace
Subspace: A subset of a vector space that is closed under addition and scalar multiplication
Orthogonal Complement: The set of all vectors orthogonal to a given subspace
- Single vector projection: Use the standard projection formula
- Subspace projection: For orthogonal basis, sum individual projections
- General subspace: Use Gram-Schmidt to orthogonalize, then sum projections
- Matrix approach: Use the matrix formula \(P = A(A^TA)^{-1}A^T\)
• Single vector: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)
• Subspace (orthogonal basis): \(\text{proj}_{W}\vec{v} = \sum_{i=1}^{k} \frac{\vec{v} \cdot \vec{u_i}}{\vec{u_i} \cdot \vec{u_i}}\vec{u_i}\)
• Matrix form: \(P = A(A^TA)^{-1}A^T\)
• Orthogonality: \(\vec{v} - P\vec{v} \perp \text{range}(P)\)
• Properties: \(P^2 = P\) (idempotent), \(P^T = P\) (symmetric)
Least squares: The method of finding the line that minimizes the sum of squared residuals, equivalent to orthogonal projection onto the model subspace
We want to find coefficients \(a\) and \(b\) for the line \(y = a + bx\) that best fits the data
Set up the system: \(\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \\ 5 \end{bmatrix}\)
Since the system is overdetermined, solve \(A^TA\vec{x} = A^T\vec{b}\)
\(A^T = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix}\)
\(A^TA = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}\)
\(A^T\vec{b} = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \\ 5 \end{bmatrix} = \begin{bmatrix} 10 \\ 23 \end{bmatrix}\)
\(\begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 10 \\ 23 \end{bmatrix}\)
From the first equation: \(3a + 6b = 10\), so \(a = \frac{10 - 6b}{3}\)
Substituting into the second: \(6(\frac{10 - 6b}{3}) + 14b = 23\)
\(2(10 - 6b) + 14b = 23\)
\(20 - 12b + 14b = 23\)
\(2b = 3\), so \(b = 1.5\)
Then \(a = \frac{10 - 6(1.5)}{3} = \frac{10 - 9}{3} = \frac{1}{3}\)
The best fit line is \(y = \frac{1}{3} + 1.5x\)
Predicted values: \((1, 1.83), (2, 3.33), (3, 4.83)\)
Residuals: \((0.17, -0.33, 0.17)\) - small and balanced
The best fit line is \(y = \frac{1}{3} + 1.5x\) (approximately \(y = 0.33 + 1.5x\))
• Normal Equations: \(A^TA\vec{x} = A^T\vec{b}\) for least squares solution
• Geometric Interpretation: Projection of \(\vec{b}\) onto the column space of \(A\)
• Minimization: Minimizes \(\|A\vec{x} - \vec{b}\|^2\)
\(\vec{u_1} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\)
and
\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}\).
Decompose
\(\vec{v} = \begin{pmatrix} 2 \\ 3 \\ 1 \end{pmatrix}\)
as
\(\vec{v} = \vec{w} + \vec{z}\)
where
\(\vec{w} \in W\)
and
\(\vec{z} \in W^\perp\).
Orthogonal decomposition: Any vector can be uniquely written as the sum of a component in a subspace and a component in its orthogonal complement
\(\vec{u_1} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\)
\(\vec{u_2}' = \vec{u_2} - \text{proj}_{\vec{u_1}}\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \frac{1}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix}\)
\(\text{proj}_{\vec{u_1}}\vec{v} = \frac{\vec{v} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1} = \frac{(2)(1) + (3)(0) + (1)(1)}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \frac{3}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} 3/2 \\ 0 \\ 3/2 \end{pmatrix}\)
\(\text{proj}_{\vec{u_2}'}\vec{v} = \frac{\vec{v} \cdot \vec{u_2}'}{\vec{u_2}' \cdot \vec{u_2}'}\vec{u_2}'\)
\(\vec{v} \cdot \vec{u_2}' = (2)(-1/2) + (3)(1) + (1)(1/2) = -1 + 3 + 1/2 = 2.5\)
\(\vec{u_2}' \cdot \vec{u_2}' = (-1/2)^2 + (1)^2 + (1/2)^2 = 1/4 + 1 + 1/4 = 1.5\)
\(\text{proj}_{\vec{u_2}'}\vec{v} = \frac{2.5}{1.5}\begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix} = \frac{5}{3}\begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix} = \begin{pmatrix} -5/6 \\ 5/3 \\ 5/6 \end{pmatrix}\)
\(\vec{w} = \text{proj}_{W}\vec{v} = \text{proj}_{\vec{u_1}}\vec{v} + \text{proj}_{\vec{u_2}'}\vec{v}\)
\(\vec{w} = \begin{pmatrix} 3/2 \\ 0 \\ 3/2 \end{pmatrix} + \begin{pmatrix} -5/6 \\ 5/3 \\ 5/6 \end{pmatrix} = \begin{pmatrix} 9/6 - 5/6 \\ 5/3 \\ 9/6 + 5/6 \end{pmatrix} = \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix}\)
\(\vec{z} = \vec{v} - \vec{w} = \begin{pmatrix} 2 \\ 3 \\ 1 \end{pmatrix} - \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix} = \begin{pmatrix} 4/3 \\ 4/3 \\ -4/3 \end{pmatrix}\)
\(\vec{z} \cdot \vec{u_1} = \frac{4}{3}(1) + \frac{4}{3}(0) + \left(-\frac{4}{3}\right)(1) = \frac{4}{3} - \frac{4}{3} = 0\)
\(\vec{z} \cdot \vec{u_2} = \frac{4}{3}(0) + \frac{4}{3}(1) + \left(-\frac{4}{3}\right)(1) = \frac{4}{3} - \frac{4}{3} = 0\)
\(\vec{v} = \vec{w} + \vec{z}\) where \(\vec{w} = \begin{pmatrix} 2/3 \\ 5/3 \\ 7/3 \end{pmatrix} \in W\) and \(\vec{z} = \begin{pmatrix} 4/3 \\ 4/3 \\ -4/3 \end{pmatrix} \in W^\perp\)
• Orthogonal Decomposition: \(\vec{v} = \text{proj}_{W}\vec{v} + (\vec{v} - \text{proj}_{W}\vec{v})\)
• Orthogonal Basis: Required for direct summation of projections
• Direct Sum: \(\mathbb{R}^n = W \oplus W^\perp\)
Orthogonal Projection: The mapping of a vector to the closest point in a subspace, measured by Euclidean distance
Projection Matrix: A matrix \(P\) satisfying \(P^2 = P\) that implements the orthogonal projection operation
Orthogonal Complement: The set of all vectors orthogonal to every vector in a given subspace
Subspace: A vector space contained within another vector space, closed under vector addition and scalar multiplication
- Identify the subspace: Determine a basis for the subspace onto which you're projecting
- Orthogonalize if needed: Use Gram-Schmidt to create an orthogonal basis
- Apply projection formula: Sum individual projections onto each orthogonal basis vector
- Verify results: Check that the residual vector is orthogonal to the subspace
• Best Approximation: The projection is the closest point in the subspace to the original vector
• Orthogonality Principle: The error vector is orthogonal to the subspace being projected onto
• Decomposition: Every vector can be uniquely decomposed into subspace and orthogonal complement components
• Minimization: Orthogonal projection minimizes the Euclidean distance to the subspace
- The shortest distance from a point to a subspace is along the perpendicular
- The residual vector is always orthogonal to the subspace
- Projections preserve linearity and structure
Analysis: The visualization shows how orthogonal projections minimize distance and maintain geometric relationships.
- Original vector: The starting point
- Projected vector: Closest point in the subspace
- Residual vector: The error, always perpendicular to subspace