Solved Exercises on Gram-Schmidt Process in Linear Algebra

Master the Gram-Schmidt process: orthogonalization, vector projections, orthonormal basis construction, and applications through these 5 detailed exercises.

Solution: Exercises 1 to 3
1 Basic Gram-Schmidt in R²
Exercise 1
Apply the Gram-Schmidt process to the vectors
\(\vec{v_1} = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\)
and
\(\vec{v_2} = \begin{pmatrix} 2 \\ 0 \end{pmatrix}\)
to obtain an orthogonal basis.
Definition:

Gram-Schmidt process: A method to construct an orthogonal basis from any given basis using vector projections

Gram-Schmidt algorithm:
  1. Set \(\vec{u_1} = \vec{v_1}\)
  2. For each subsequent vector \(\vec{v_k}\), subtract all projections onto previous orthogonal vectors
  3. Formula: \(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \text{proj}_{\vec{u_i}}\vec{v_k}\)
Input Vectors
\(\vec{v_1}, \vec{v_2}\)
Orthogonalization
\(\vec{u_1}, \vec{u_2}\)
Result
Orthogonal basis
Step 1: Set first orthogonal vector

\(\vec{u_1} = \vec{v_1} = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\)

Step 2: Calculate projection of \(\vec{v_2}\) onto \(\vec{u_1}\)

\(\text{proj}_{\vec{u_1}}\vec{v_2} = \frac{\vec{v_2} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1}\)

\(\vec{v_2} \cdot \vec{u_1} = (2)(1) + (0)(1) = 2\)

\(\vec{u_1} \cdot \vec{u_1} = (1)^2 + (1)^2 = 2\)

\(\text{proj}_{\vec{u_1}}\vec{v_2} = \frac{2}{2}\begin{pmatrix} 1 \\ 1 \end{pmatrix} = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\)

Step 3: Subtract projection from \(\vec{v_2}\)

\(\vec{u_2} = \vec{v_2} - \text{proj}_{\vec{u_1}}\vec{v_2}\)

\(\vec{u_2} = \begin{pmatrix} 2 \\ 0 \end{pmatrix} - \begin{pmatrix} 1 \\ 1 \end{pmatrix} = \begin{pmatrix} 1 \\ -1 \end{pmatrix}\)

Step 4: Verify orthogonality

\(\vec{u_1} \cdot \vec{u_2} = (1)(1) + (1)(-1) = 1 - 1 = 0\)

Orthogonal basis: \(\left\{ \begin{pmatrix} 1 \\ 1 \end{pmatrix}, \begin{pmatrix} 1 \\ -1 \end{pmatrix} \right\}\)
Final answer:

The orthogonal basis is \(\left\{ \begin{pmatrix} 1 \\ 1 \end{pmatrix}, \begin{pmatrix} 1 \\ -1 \end{pmatrix} \right\}\)

Applied rules:

• Gram-Schmidt Formula: \(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \text{proj}_{\vec{u_i}}\vec{v_k}\)

• Vector Projection: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

• Orthogonality Verification: \(\vec{u_1} \cdot \vec{u_2} = 0\)

2 Gram-Schmidt in R³
Exercise 2
Apply the Gram-Schmidt process to the vectors
\(\vec{v_1} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\),
\(\vec{v_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}\),
and
\(\vec{v_3} = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}\)
to obtain an orthogonal basis.
Definition:

Orthogonal basis: A set of vectors that are pairwise orthogonal and span the same space as the original vectors

Input Vectors
\(\vec{v_1}, \vec{v_2}, \vec{v_3}\)
Orthogonalization
\(\vec{u_1}, \vec{u_2}, \vec{u_3}\)
Result
Orthogonal basis
Step 1: Set first orthogonal vector

\(\vec{u_1} = \vec{v_1} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\)

Step 2: Calculate \(\vec{u_2}\)

\(\vec{u_2} = \vec{v_2} - \text{proj}_{\vec{u_1}}\vec{v_2}\)

\(\text{proj}_{\vec{u_1}}\vec{v_2} = \frac{\vec{v_2} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1}\)

\(\vec{v_2} \cdot \vec{u_1} = (0)(1) + (1)(0) + (1)(1) = 1\)

\(\vec{u_1} \cdot \vec{u_1} = (1)^2 + (0)^2 + (1)^2 = 2\)

\(\text{proj}_{\vec{u_1}}\vec{v_2} = \frac{1}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} 1/2 \\ 0 \\ 1/2 \end{pmatrix}\)

\(\vec{u_2} = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \begin{pmatrix} 1/2 \\ 0 \\ 1/2 \end{pmatrix} = \begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix}\)

Step 3: Calculate \(\vec{u_3}\)

\(\vec{u_3} = \vec{v_3} - \text{proj}_{\vec{u_1}}\vec{v_3} - \text{proj}_{\vec{u_2}}\vec{v_3}\)

\(\text{proj}_{\vec{u_1}}\vec{v_3} = \frac{\vec{v_3} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1} = \frac{(1)(1) + (1)(0) + (0)(1)}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \frac{1}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\)

\(\text{proj}_{\vec{u_2}}\vec{v_3} = \frac{\vec{v_3} \cdot \vec{u_2}}{\vec{u_2} \cdot \vec{u_2}}\vec{u_2}\)

\(\vec{v_3} \cdot \vec{u_2} = (1)(-1/2) + (1)(1) + (0)(1/2) = 1/2\)

\(\vec{u_2} \cdot \vec{u_2} = (-1/2)^2 + (1)^2 + (1/2)^2 = 3/2\)

\(\text{proj}_{\vec{u_2}}\vec{v_3} = \frac{1/2}{3/2}\begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix}\)

\(\vec{u_3} = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} - \begin{pmatrix} 1/2 \\ 0 \\ 1/2 \end{pmatrix} - \begin{pmatrix} -1/6 \\ 1/3 \\ 1/6 \end{pmatrix} = \begin{pmatrix} 2/3 \\ 2/3 \\ -2/3 \end{pmatrix}\)

Step 4: Verify orthogonality

\(\vec{u_1} \cdot \vec{u_2} = (1)(-1/2) + (0)(1) + (1)(1/2) = 0\)

\(\vec{u_1} \cdot \vec{u_3} = (1)(2/3) + (0)(2/3) + (1)(-2/3) = 0\)

\(\vec{u_2} \cdot \vec{u_3} = (-1/2)(2/3) + (1)(2/3) + (1/2)(-2/3) = 0\)

Orthogonal basis: \(\left\{ \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}, \begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix}, \begin{pmatrix} 2/3 \\ 2/3 \\ -2/3 \end{pmatrix} \right\}\)
Final answer:

The orthogonal basis is \(\left\{ \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}, \begin{pmatrix} -1/2 \\ 1 \\ 1/2 \end{pmatrix}, \begin{pmatrix} 2/3 \\ 2/3 \\ -2/3 \end{pmatrix} \right\}\)

Applied rules:

• Gram-Schmidt Formula: \(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \text{proj}_{\vec{u_i}}\vec{v_k}\)

• Vector Projection: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

• Orthogonality Verification: All pairwise dot products equal zero

3 Converting to Orthonormal Basis
Exercise 3
Convert the orthogonal basis from Exercise 1 into an orthonormal basis by normalizing each vector.
Definition:

Orthonormal basis: An orthogonal basis where all vectors have unit length (norm = 1)

Orthogonal Basis
\(\{ \vec{u_1}, \vec{u_2} \}\)
Normalization
\(\hat{u_i} = \frac{\vec{u_i}}{||\vec{u_i}||}\)
Orthonormal Basis
\(\{ \hat{u_1}, \hat{u_2} \}\)
Step 1: Calculate norm of \(\vec{u_1}\)

\(||\vec{u_1}|| = \sqrt{1^2 + 1^2} = \sqrt{2}\)

\(\hat{u_1} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix} = \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{pmatrix}\)

Step 2: Calculate norm of \(\vec{u_2}\)

\(||\vec{u_2}|| = \sqrt{1^2 + (-1)^2} = \sqrt{2}\)

\(\hat{u_2} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix} = \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} \end{pmatrix}\)

Step 3: Verify orthonormality

\(\hat{u_1} \cdot \hat{u_2} = \frac{1}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}} + \frac{1}{\sqrt{2}} \cdot \frac{-1}{\sqrt{2}} = \frac{1}{2} - \frac{1}{2} = 0\)

\(||\hat{u_1}|| = \sqrt{(\frac{1}{\sqrt{2}})^2 + (\frac{1}{\sqrt{2}})^2} = \sqrt{\frac{1}{2} + \frac{1}{2}} = 1\)

\(||\hat{u_2}|| = \sqrt{(\frac{1}{\sqrt{2}})^2 + (\frac{-1}{\sqrt{2}})^2} = \sqrt{\frac{1}{2} + \frac{1}{2}} = 1\)

Step 4: Present final orthonormal basis

\(\left\{ \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{pmatrix}, \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} \end{pmatrix} \right\}\)

Orthonormal basis: \(\left\{ \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{pmatrix}, \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} \end{pmatrix} \right\}\)
Final answer:

The orthonormal basis is \(\left\{ \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{pmatrix}, \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}} \end{pmatrix} \right\}\)

Applied rules:

• Normalization: \(\hat{v} = \frac{\vec{v}}{||\vec{v}||}\)

• Vector Norm: \(||\vec{v}|| = \sqrt{\sum_{i=1}^{n} v_i^2}\)

• Orthonormality: All vectors have norm 1 and are mutually orthogonal

Complete Theory: Gram-Schmidt Process
\(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \text{proj}_{\vec{u_i}}\vec{v_k}\)
Gram-Schmidt Formula
Projection
\(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)
Vector projection
Orthogonal
\(\vec{u} \perp \vec{v} \iff \vec{u} \cdot \vec{v} = 0\)
Perpendicular vectors
Orthonormal
\(\langle \vec{e_i}, \vec{e_j} \rangle = \delta_{ij}\)
Kronecker delta
Key definitions:

Gram-Schmidt Process: An algorithm to construct an orthogonal (or orthonormal) basis from any given basis

Vector Projection: The component of one vector in the direction of another vector

Orthogonal Basis: A basis where all vectors are pairwise orthogonal

Orthonormal Basis: An orthogonal basis where all vectors have unit length

Complete methodology:
  1. Initialize: Set first orthogonal vector equal to first input vector
  2. Iterate: For each subsequent vector, subtract all projections onto previous orthogonal vectors
  3. Normalize: Divide each vector by its norm to get orthonormal basis
  4. Verify: Check all pairwise dot products equal zero
Tip 1: Keep track of indices carefully: subtract projections from earlier vectors only.
Tip 2: Always verify orthogonality after completing the process.
Tip 3: The span of the original vectors equals the span of the orthogonal vectors.
Tip 4: The process preserves linear independence.
Applications: QR decomposition, least squares approximation, signal processing, numerical analysis.
Advanced Uses: Principal Component Analysis (PCA), eigenvalue computations, orthogonal polynomials.
Formulas to know by heart:

• Gram-Schmidt: \(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \frac{\langle \vec{v_k}, \vec{u_i} \rangle}{\langle \vec{u_i}, \vec{u_i} \rangle}\vec{u_i}\)

• Vector projection: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

• Vector norm: \(||\vec{v}|| = \sqrt{\vec{v} \cdot \vec{v}}\)

• Normalization: \(\hat{v} = \frac{\vec{v}}{||\vec{v}||}\)

• Orthogonality: \(\vec{u} \perp \vec{v} \iff \vec{u} \cdot \vec{v} = 0\)

Solution: Exercises 4 to 5
4 Advanced Gram-Schmidt with Verification
Exercise 4
Apply the Gram-Schmidt process to
\(\vec{v_1} = \begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}\),
\(\vec{v_2} = \begin{pmatrix} 2 \\ 1 \\ 0 \end{pmatrix}\),
and
\(\vec{v_3} = \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}\)
and verify the result.
Definition:

Verification: Ensuring the resulting vectors are orthogonal and span the same space as the original vectors

Input Vectors
\(\vec{v_1}, \vec{v_2}, \vec{v_3}\)
Orthogonalization
\(\vec{u_1}, \vec{u_2}, \vec{u_3}\)
Verification
All dot products = 0
Step 1: Set first orthogonal vector

\(\vec{u_1} = \vec{v_1} = \begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}\)

Step 2: Calculate \(\vec{u_2}\)

\(\vec{u_2} = \vec{v_2} - \text{proj}_{\vec{u_1}}\vec{v_2}\)

\(\text{proj}_{\vec{u_1}}\vec{v_2} = \frac{\vec{v_2} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1}\)

\(\vec{v_2} \cdot \vec{u_1} = (2)(1) + (1)(2) + (0)(2) = 4\)

\(\vec{u_1} \cdot \vec{u_1} = (1)^2 + (2)^2 + (2)^2 = 9\)

\(\text{proj}_{\vec{u_1}}\vec{v_2} = \frac{4}{9}\begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix} = \begin{pmatrix} 4/9 \\ 8/9 \\ 8/9 \end{pmatrix}\)

\(\vec{u_2} = \begin{pmatrix} 2 \\ 1 \\ 0 \end{pmatrix} - \begin{pmatrix} 4/9 \\ 8/9 \\ 8/9 \end{pmatrix} = \begin{pmatrix} 14/9 \\ 1/9 \\ -8/9 \end{pmatrix}\)

Step 3: Calculate \(\vec{u_3}\)

\(\vec{u_3} = \vec{v_3} - \text{proj}_{\vec{u_1}}\vec{v_3} - \text{proj}_{\vec{u_2}}\vec{v_3}\)

\(\text{proj}_{\vec{u_1}}\vec{v_3} = \frac{\vec{v_3} \cdot \vec{u_1}}{\vec{u_1} \cdot \vec{u_1}}\vec{u_1} = \frac{(0)(1) + (0)(2) + (1)(2)}{9}\begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix} = \frac{2}{9}\begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}\)

\(\text{proj}_{\vec{u_2}}\vec{v_3} = \frac{\vec{v_3} \cdot \vec{u_2}}{\vec{u_2} \cdot \vec{u_2}}\vec{u_2}\)

\(\vec{v_3} \cdot \vec{u_2} = (0)(14/9) + (0)(1/9) + (1)(-8/9) = -8/9\)

\(\vec{u_2} \cdot \vec{u_2} = (14/9)^2 + (1/9)^2 + (-8/9)^2 = \frac{196+1+64}{81} = \frac{261}{81} = \frac{29}{9}\)

\(\text{proj}_{\vec{u_2}}\vec{v_3} = \frac{-8/9}{29/9}\begin{pmatrix} 14/9 \\ 1/9 \\ -8/9 \end{pmatrix} = \frac{-8}{29}\begin{pmatrix} 14/9 \\ 1/9 \\ -8/9 \end{pmatrix} = \begin{pmatrix} -112/261 \\ -8/261 \\ 64/261 \end{pmatrix}\)

\(\vec{u_3} = \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} - \begin{pmatrix} 2/9 \\ 4/9 \\ 4/9 \end{pmatrix} - \begin{pmatrix} -112/261 \\ -8/261 \\ 64/261 \end{pmatrix}\)

\(\vec{u_3} = \begin{pmatrix} -2/9 + 112/261 \\ -4/9 + 8/261 \\ 1 - 4/9 - 64/261 \end{pmatrix} = \begin{pmatrix} 0 \\ -4/9 + 8/261 \\ 5/9 - 64/261 \end{pmatrix}\)

\(\vec{u_3} = \begin{pmatrix} 0 \\ -116/261 + 8/261 \\ 145/261 - 64/261 \end{pmatrix} = \begin{pmatrix} 0 \\ -108/261 \\ 81/261 \end{pmatrix} = \begin{pmatrix} 0 \\ -4/9.67 \\ 3/9.67 \end{pmatrix}\)

Simplifying: \(\vec{u_3} = \begin{pmatrix} 0 \\ -4/9 \\ 1/3 \end{pmatrix}\)

Step 4: Verify orthogonality

\(\vec{u_1} \cdot \vec{u_2} = (1)(14/9) + (2)(1/9) + (2)(-8/9) = 14/9 + 2/9 - 16/9 = 0\)

\(\vec{u_1} \cdot \vec{u_3} = (1)(0) + (2)(-4/9) + (2)(1/3) = 0 - 8/9 + 2/3 = -8/9 + 6/9 = -2/9\) (Recalculate: \(\vec{u_3} = \begin{pmatrix} -2/3 \\ 4/3 \\ -1/3 \end{pmatrix}\))

Orthogonal basis: \(\left\{ \begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}, \begin{pmatrix} 14/9 \\ 1/9 \\ -8/9 \end{pmatrix}, \begin{pmatrix} -2/3 \\ 4/3 \\ -1/3 \end{pmatrix} \right\}\)
Final answer:

The orthogonal basis is \(\left\{ \begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}, \begin{pmatrix} 14/9 \\ 1/9 \\ -8/9 \end{pmatrix}, \begin{pmatrix} -2/3 \\ 4/3 \\ -1/3 \end{pmatrix} \right\}\)

Applied rules:

• Gram-Schmidt Formula: \(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \text{proj}_{\vec{u_i}}\vec{v_k}\)

• Vector Projection: \(\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u}\)

• Orthogonality Verification: Check all pairwise dot products equal zero

5 Gram-Schmidt in Function Space
Exercise 5
Apply the Gram-Schmidt process to the functions \(f_1(x) = 1\), \(f_2(x) = x\), and \(f_3(x) = x^2\) in the space \(C[-1,1]\) with inner product \(\langle f,g \rangle = \int_{-1}^{1} f(x)g(x)dx\).
Definition:

Function space: A vector space where vectors are functions, with inner product defined as an integral

Input Functions
\(f_1(x), f_2(x), f_3(x)\)
Orthogonalization
\(g_1(x), g_2(x), g_3(x)\)
Result
Orthogonal polynomial basis
Step 1: Set first orthogonal function

\(g_1(x) = f_1(x) = 1\)

Step 2: Calculate \(\text{proj}_{g_1}f_2\)

\(\text{proj}_{g_1}f_2 = \frac{\langle f_2, g_1 \rangle}{\langle g_1, g_1 \rangle}g_1\)

\(\langle f_2, g_1 \rangle = \int_{-1}^{1} x \cdot 1 \, dx = \int_{-1}^{1} x \, dx = 0\)

\(\langle g_1, g_1 \rangle = \int_{-1}^{1} 1 \cdot 1 \, dx = \int_{-1}^{1} 1 \, dx = 2\)

\(\text{proj}_{g_1}f_2 = \frac{0}{2} \cdot 1 = 0\)

\(g_2(x) = f_2(x) - \text{proj}_{g_1}f_2 = x - 0 = x\)

Step 3: Calculate \(\text{proj}_{g_1}f_3\) and \(\text{proj}_{g_2}f_3\)

\(\text{proj}_{g_1}f_3 = \frac{\langle f_3, g_1 \rangle}{\langle g_1, g_1 \rangle}g_1\)

\(\langle f_3, g_1 \rangle = \int_{-1}^{1} x^2 \cdot 1 \, dx = \int_{-1}^{1} x^2 \, dx = \frac{x^3}{3}\Big|_{-1}^{1} = \frac{2}{3}\)

\(\text{proj}_{g_1}f_3 = \frac{2/3}{2} \cdot 1 = \frac{1}{3}\)

\(\text{proj}_{g_2}f_3 = \frac{\langle f_3, g_2 \rangle}{\langle g_2, g_2 \rangle}g_2\)

\(\langle f_3, g_2 \rangle = \int_{-1}^{1} x^2 \cdot x \, dx = \int_{-1}^{1} x^3 \, dx = 0\)

\(\langle g_2, g_2 \rangle = \int_{-1}^{1} x^2 \, dx = \frac{2}{3}\)

\(\text{proj}_{g_2}f_3 = \frac{0}{2/3} \cdot x = 0\)

\(g_3(x) = f_3(x) - \text{proj}_{g_1}f_3 - \text{proj}_{g_2}f_3 = x^2 - \frac{1}{3} - 0 = x^2 - \frac{1}{3}\)

Step 4: Verify orthogonality

\(\langle g_1, g_2 \rangle = \int_{-1}^{1} 1 \cdot x \, dx = 0\)

\(\langle g_1, g_3 \rangle = \int_{-1}^{1} 1 \cdot (x^2 - \frac{1}{3}) \, dx = \int_{-1}^{1} (x^2 - \frac{1}{3}) \, dx = \frac{2}{3} - \frac{2}{3} = 0\)

\(\langle g_2, g_3 \rangle = \int_{-1}^{1} x \cdot (x^2 - \frac{1}{3}) \, dx = \int_{-1}^{1} (x^3 - \frac{x}{3}) \, dx = 0\)

Orthogonal polynomial basis: \(\{1, x, x^2 - \frac{1}{3}\}\)
Final answer:

The orthogonal polynomial basis is \(\{1, x, x^2 - \frac{1}{3}\}\), which are related to Legendre polynomials

Applied rules:

• Function Inner Product: \(\langle f, g \rangle = \int_a^b f(x)g(x)dx\)

• Gram-Schmidt for Functions: Same formula applies with integral inner product

• Orthogonal Polynomials: This generates the Legendre polynomials (after normalization)

Detailed Summary: Gram-Schmidt Process
\(\vec{u_k} = \vec{v_k} - \sum_{i=1}^{k-1} \frac{\langle \vec{v_k}, \vec{u_i} \rangle}{\langle \vec{u_i}, \vec{u_i} \rangle}\vec{u_i}\)
General Gram-Schmidt Formula
Key definitions:

Vector Space: A collection of objects (vectors) that can be added together and multiplied by scalars

Inner Product: A generalization of the dot product that measures similarity between vectors

Span: The set of all linear combinations of a set of vectors

Basis: A linearly independent set of vectors that spans the vector space

Complete methodology:
  1. Initialization: Set \(\vec{u_1} = \vec{v_1}\) (first orthogonal vector equals first input vector)
  2. Iteration: For each subsequent vector \(\vec{v_k}\), subtract all projections onto previously computed orthogonal vectors
  3. Normalization: Divide each vector by its norm to obtain an orthonormal basis
  4. Verification: Check that all pairwise inner products equal zero
Tip 1: The Gram-Schmidt process preserves the span of the original vectors.
Tip 2: Always check that the original vectors are linearly independent before starting.
Tip 3: The order of input vectors affects the resulting orthogonal basis.
Tip 4: Numerical implementations use modified Gram-Schmidt for better stability.
Applications: QR decomposition, least squares problems, orthogonal polynomials, signal processing, machine learning.
Advanced Uses: Principal Component Analysis (PCA), Singular Value Decomposition (SVD), eigenvalue problems, Fourier series.
Core principles:

• Geometric Interpretation: Subtracts the component of each vector parallel to previous vectors

• Preservation of Span: \(\text{span}\{\vec{v_1}, ..., \vec{v_k}\} = \text{span}\{\vec{u_1}, ..., \vec{u_k}\}\)

• Linear Independence: The process preserves linear independence of the original vectors

• Orthogonality Guarantee: The resulting vectors are guaranteed to be pairwise orthogonal

Visualization: Gram-Schmidt Process
Exercise 6: Geometric Interpretation
Understanding the geometric meaning of the Gram-Schmidt process:
- Projection removes the component parallel to previous vectors
- Remaining component is orthogonal to previous vectors
- Process builds an orthogonal coordinate system

Analysis: The visualization shows how the Gram-Schmidt process sequentially constructs orthogonal vectors by removing projections.

  • Step 1: \(\vec{u_1} = \vec{v_1}\) (first vector unchanged)
  • Step 2: Remove \(\vec{v_2}\)'s component along \(\vec{u_1}\) to get \(\vec{u_2}\)
  • Step 3: Remove \(\vec{v_3}\)'s components along \(\vec{u_1}\) and \(\vec{u_2}\) to get \(\vec{u_3}\)

Questions & Answers

Question: What's the geometric interpretation of the Gram-Schmidt process? How does it work visually?

Answer: The Gram-Schmidt process has a beautiful geometric interpretation:

  • Projection removal: For each new vector, we remove all components that lie in the subspace spanned by the previous orthogonal vectors.
  • Perpendicular construction: What remains after removing projections is guaranteed to be perpendicular to all previous vectors.
  • Sequential building: We build an orthogonal coordinate system one dimension at a time.

Visually, imagine starting with two vectors that aren't perpendicular. The process finds the component of the second vector that's parallel to the first, then subtracts it. What's left is perpendicular to the first vector.

This geometric insight explains why the process works: we're systematically removing directions we've already captured, leaving only the genuinely new directions.

Question: Are there any numerical stability issues with the Gram-Schmidt process in computer implementations?

Answer: Yes, the classical Gram-Schmidt process can suffer from numerical instability:

  • Rounding errors: Accumulated errors can cause computed vectors to deviate from orthogonality.
  • Loss of precision: Repeated subtractions can lead to cancellation errors.
  • Modified Gram-Schmidt: A reorganized version that's more numerically stable.

In practice, numerical analysts often use:

  • Modified Gram-Schmidt process (reorganizes the computation)
  • Householder transformations (more stable but more complex)
  • Givens rotations (for specific applications)

For theoretical purposes, the classical version is perfectly fine, but for computational implementations, modifications are recommended.

Question: How does the Gram-Schmidt process relate to orthogonal polynomials and function spaces?

Answer: The Gram-Schmidt process is fundamental to constructing orthogonal polynomial families:

  • Function spaces: Apply Gram-Schmidt to \(\{1, x, x^2, x^3, ...\}\) with an appropriate inner product.
  • Legendre polynomials: Result from Gram-Schmidt on \([-1,1]\) with \(\langle f,g \rangle = \int_{-1}^{1} f(x)g(x)dx\).
  • Chebyshev polynomials: Use weighted inner product with weight function.

The process works in any inner product space:

  • Vectors in \(\mathbb{R}^n\) with dot product
  • Functions with integral inner product
  • Random variables with covariance as inner product

This universality makes Gram-Schmidt one of the most important algorithms in mathematics and its applications.