Solved Exercises on Null Space and Column Space

Master null space, column space, kernel, range, and applications through these 5 detailed exercises.

Null Space and Column Space - Comprehensive Summary

Definitions

Null Space (Kernel): \(\text{null}(A) = \{x \in \mathbb{R}^n \mid Ax = 0\}\), all vectors that map to zero vector

Column Space (Range): \(\text{col}(A) = \{Ax \mid x \in \mathbb{R}^n\}\), all possible outputs of matrix multiplication

Rank: Dimension of column space

Nullity: Dimension of null space

Core Rules

• \(\text{rank}(A) + \text{nullity}(A) = n\) (number of columns)

• Null space is always a subspace of domain

• Column space is always a subspace of codomain

• Both spaces are subspaces of their respective vector spaces

Methods

1. Find null space: Solve homogeneous system \(Ax = 0\)

2. Find column space: Use pivot columns of original matrix

3. Determine dimensions: Count free variables and pivot columns

4. Verify: Use rank-nullity theorem

Key Tips

• Always row reduce for null space computation

• Use original columns for column space basis

• Check rank-nullity theorem as verification

• Pivot columns correspond to basic variables

Solution: Exercises 1 to 3
1 Finding Null Space
Exercise 1
Find the null space of \(A = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 2 & 5 \end{pmatrix}\).
Definition:

Null Space: The null space of matrix A, denoted \(\text{null}(A)\), is the set of all vectors \(\vec{x}\) such that \(A\vec{x} = \vec{0}\). It's also called the kernel of A.

Null Space Method:
  1. Set up the homogeneous system \(A\vec{x} = \vec{0}\)
  2. Row reduce the augmented matrix \([A | \vec{0}]\)
  3. Identify pivot and free variables
  4. Express basic variables in terms of free variables
  5. Write the general solution as a linear combination of vectors
Row Reduction
\(A \sim \begin{pmatrix} 1 & 2 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\)
Free Variable
\(x_2 = t\)
Null Space
\(\text{span}\left\{\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}\right\}\)
Step 1: Set up the homogeneous system

We need to solve: \(\begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 2 & 5 \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}\)

Step 2: Row reduce the augmented matrix

\(\begin{pmatrix} 1 & 2 & 3 & | & 0 \\ 2 & 4 & 6 & | & 0 \\ 1 & 2 & 5 & | & 0 \end{pmatrix}\)

\(R_2 \leftarrow R_2 - 2R_1\): \(\begin{pmatrix} 1 & 2 & 3 & | & 0 \\ 0 & 0 & 0 & | & 0 \\ 1 & 2 & 5 & | & 0 \end{pmatrix}\)

\(R_3 \leftarrow R_3 - R_1\): \(\begin{pmatrix} 1 & 2 & 3 & | & 0 \\ 0 & 0 & 0 & | & 0 \\ 0 & 0 & 2 & | & 0 \end{pmatrix}\)

\(R_3 \leftarrow \frac{1}{2}R_3\): \(\begin{pmatrix} 1 & 2 & 3 & | & 0 \\ 0 & 0 & 0 & | & 0 \\ 0 & 0 & 1 & | & 0 \end{pmatrix}\)

\(R_1 \leftarrow R_1 - 3R_3\): \(\begin{pmatrix} 1 & 2 & 0 & | & 0 \\ 0 & 0 & 0 & | & 0 \\ 0 & 0 & 1 & | & 0 \end{pmatrix}\)

Step 3: Identify pivot and free variables

From the reduced matrix: \(x_1 + 2x_2 = 0\) and \(x_3 = 0\)

Pivot variables: \(x_1, x_3\)

Free variable: \(x_2\)

Step 4: Express solution in terms of free variables

From \(x_1 + 2x_2 = 0\): \(x_1 = -2x_2\)

From \(x_3 = 0\): \(x_3 = 0\)

Let \(x_2 = t\) where \(t\) is any real number.

Then: \(\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} -2t \\ t \\ 0 \end{pmatrix} = t\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}\)

Step 5: Write the null space

\(\text{null}(A) = \left\{t\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix} \mid t \in \mathbb{R}\right\} = \text{span}\left\{\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}\right\}\)

\(\text{null}(A) = \text{span}\left\{\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}\right\}\)
Final answer:

The null space of A is \(\text{span}\left\{\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix}\right\}\).

Applied rules:

• Row reduction: Preserves solution space of homogeneous system

• Free variable identification: Non-pivot columns correspond to free variables

• Parametric form: Express solution in terms of free variables

2 Finding Column Space
Exercise 2
Find a basis for the column space of \(A = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 2 & 5 \end{pmatrix}\).
Definition:

Column Space: The column space of matrix A, denoted \(\text{col}(A)\), is the span of the column vectors of A. It's also called the range of A.

Row Reduction
\(A \sim \begin{pmatrix} 1 & 2 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\)
Pivot Columns
Columns 1, 3
Basis
\(\left\{\begin{pmatrix} 1 \\ 2 \\ 1 \end{pmatrix}, \begin{pmatrix} 3 \\ 6 \\ 5 \end{pmatrix}\right\}\)
Step 1: Row reduce matrix A

From Exercise 1: \(A \sim \begin{pmatrix} 1 & 2 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\)

Step 2: Identify pivot columns

In the row-reduced form, the pivot columns are columns 1 and 3 (columns with leading 1s).

Step 3: Select corresponding columns from original matrix

The pivot columns in the original matrix A are:

Column 1: \(\begin{pmatrix} 1 \\ 2 \\ 1 \end{pmatrix}\)

Column 3: \(\begin{pmatrix} 3 \\ 6 \\ 5 \end{pmatrix}\)

Step 4: Form the basis

These columns form a basis for the column space: \(\left\{\begin{pmatrix} 1 \\ 2 \\ 1 \end{pmatrix}, \begin{pmatrix} 3 \\ 6 \\ 5 \end{pmatrix}\right\}\)

Step 5: Verify linear independence

Check that these two vectors are linearly independent:

\(c_1\begin{pmatrix} 1 \\ 2 \\ 1 \end{pmatrix} + c_2\begin{pmatrix} 3 \\ 6 \\ 5 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}\)

This gives us: \(c_1 + 3c_2 = 0\), \(2c_1 + 6c_2 = 0\), \(c_1 + 5c_2 = 0\)

From first equation: \(c_1 = -3c_2\)

Substituting into third: \(-3c_2 + 5c_2 = 0\), so \(2c_2 = 0\), thus \(c_2 = 0\)

Therefore \(c_1 = 0\), confirming linear independence. ✓

Basis for \(\text{col}(A)\): \(\left\{\begin{pmatrix} 1 \\ 2 \\ 1 \end{pmatrix}, \begin{pmatrix} 3 \\ 6 \\ 5 \end{pmatrix}\right\}\)
Final answer:

A basis for the column space of A is \(\left\{\begin{pmatrix} 1 \\ 2 \\ 1 \end{pmatrix}, \begin{pmatrix} 3 \\ 6 \\ 5 \end{pmatrix}\right\}\).

Applied rules:

• Pivot column method: Use original columns corresponding to pivot positions

• Linear independence: Pivot columns are always linearly independent

• Span preservation: Pivot columns span the same space as all columns

3 Rank and Nullity
Exercise 3
For the matrix \(A = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 2 & 5 \end{pmatrix}\), find the rank and nullity of A and verify the Rank-Nullity Theorem.
Definition:

Rank: The dimension of the column space of A (equivalently, the row space of A).

Nullity: The dimension of the null space of A.

Rank-Nullity Theorem: For any \(m \times n\) matrix A: \(\text{rank}(A) + \text{nullity}(A) = n\).

Rank
\(\text{rank}(A) = 2\)
Nullity
\(\text{nullity}(A) = 1\)
Verification
\(2 + 1 = 3\)
Step 1: Find the rank

From Exercise 2, the column space has dimension 2 (since the basis has 2 vectors).

Therefore: \(\text{rank}(A) = 2\)

Step 2: Find the nullity

From Exercise 1, the null space has dimension 1 (since the basis has 1 vector).

Therefore: \(\text{nullity}(A) = 1\)

Step 3: Verify the Rank-Nullity Theorem

Matrix A is \(3 \times 3\), so \(n = 3\).

\(\text{rank}(A) + \text{nullity}(A) = 2 + 1 = 3 = n\). ✓

Step 4: Alternative calculation of rank

From row-reduced form in Exercise 1: \(\begin{pmatrix} 1 & 2 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\)

Number of pivot positions = 2, so \(\text{rank}(A) = 2\). ✓

Step 5: Alternative calculation of nullity

Number of free variables = 3 - 2 = 1, so \(\text{nullity}(A) = 1\). ✓

\(\text{rank}(A) = 2\), \(\text{nullity}(A) = 1\), and \(\text{rank}(A) + \text{nullity}(A) = 3\)
Final answer:

The rank of A is 2, the nullity of A is 1, and the Rank-Nullity Theorem is verified: \(2 + 1 = 3\).

Applied rules:

• Dimension counting: Count vectors in basis to find dimension

• Rank-Nullity theorem: Fundamental relationship between dimensions

• Verification: Use theorem to check calculations

Null Space and Column Space Properties
\(\text{rank}(A) + \text{nullity}(A) = n\)
Rank-Nullity Theorem
Null Space
\(\text{null}(A) = \{x \in \mathbb{R}^n \mid Ax = 0\}\)
Kernel of the linear transformation
Column Space
\(\text{col}(A) = \{Ax \mid x \in \mathbb{R}^n\}\)
Range of the linear transformation
Rank Equality
\(\text{rank}(A) = \text{rank}(A^T)\)
Row rank equals column rank
Subspace Properties: Both null space and column space are subspaces of their respective vector spaces
Dimension Relationship: The sum of dimensions equals the number of columns
Injectivity Connection: A is injective (one-to-one) ⟺ null(A) = {0}
Solution: Exercises 4 to 5
4 Invertibility and Null Space
Exercise 4
Prove that an \(n \times n\) matrix A is invertible if and only if \(\text{null}(A) = \{\vec{0}\}\).
Definition:

Invertibility Criterion: A square matrix A is invertible if and only if the equation \(A\vec{x} = \vec{0}\) has only the trivial solution \(\vec{x} = \vec{0}\).

Forward Direction
Invertible ⟹ null(A) = {0}
Reverse Direction
null(A) = {0} ⟹ Invertible
Equivalence
⟺
Step 1: Forward direction (⇒)

Assume A is invertible. We need to show that \(\text{null}(A) = \{\vec{0}\}\).

Suppose \(\vec{x} \in \text{null}(A)\), so \(A\vec{x} = \vec{0}\).

Multiply both sides by \(A^{-1}\): \(A^{-1}A\vec{x} = A^{-1}\vec{0}\).

This gives us: \(I\vec{x} = \vec{0}\), so \(\vec{x} = \vec{0}\).

Therefore, the only vector in \(\text{null}(A)\) is \(\vec{0}\), so \(\text{null}(A) = \{\vec{0}\}\). ✓

Step 2: Reverse direction (⇐)

Assume \(\text{null}(A) = \{\vec{0}\}\). We need to show that A is invertible.

Since \(\text{null}(A) = \{\vec{0}\}\), the homogeneous system \(A\vec{x} = \vec{0}\) has only the trivial solution.

This means the columns of A are linearly independent.

Since A is \(n \times n\) and its columns are linearly independent, A has full rank (rank = n).

For an \(n \times n\) matrix, full rank implies invertibility. ✓

Step 3: Equivalence established

We've shown both directions: A is invertible ⟺ \(\text{null}(A) = \{\vec{0}\}\).

Step 4: Alternative characterization

This is equivalent to saying: A is invertible ⟺ the columns of A form a basis for \(\mathbb{R}^n\).

A is invertible ⟺ \(\text{null}(A) = \{\vec{0}\}\)
Final answer:

An \(n \times n\) matrix A is invertible if and only if its null space contains only the zero vector.

Applied rules:

• Contrapositive reasoning: Use both directions of equivalence

• Linear independence: Trivial null space ⟹ independent columns

• Full rank criterion: Independent columns in square matrix ⟹ invertibility

5 Column Space and Linear Systems
Exercise 5
For the matrix \(A = \begin{pmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{pmatrix}\), determine for which vectors \(\vec{b}\) the system \(A\vec{x} = \vec{b}\) has a solution. Specifically, solve for \(\vec{b_1} = \begin{pmatrix} 3 \\ 6 \\ 9 \end{pmatrix}\) and \(\vec{b_2} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\).
Definition:

Consistency Condition: The system \(A\vec{x} = \vec{b}\) has a solution if and only if \(\vec{b} \in \text{col}(A)\). This is because the column space consists of all possible outputs of the matrix transformation.

Column Space
\(\text{span}\left\{\begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\right\}\)
Solution for b₁
\(\vec{x} = \begin{pmatrix} 3 \\ 0 \end{pmatrix}\)
Solution for b₂
\(\vec{x} = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\)
Step 1: Find the column space of A

\(A = \begin{pmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{pmatrix}\)

Notice that column 2 is twice column 1: \(\begin{pmatrix} 2 \\ 4 \\ 6 \end{pmatrix} = 2\begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\).

So \(\text{col}(A) = \text{span}\left\{\begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\right\}\).

Step 2: Check if \(\vec{b_1}\) is in column space

\(\vec{b_1} = \begin{pmatrix} 3 \\ 6 \\ 9 \end{pmatrix} = 3\begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\).

Since \(\vec{b_1}\) is a scalar multiple of the column space generator, \(\vec{b_1} \in \text{col}(A)\).

Therefore, \(A\vec{x} = \vec{b_1}\) has a solution. ✓

Step 3: Find solution for \(\vec{b_1}\)

We need \(\begin{pmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} 3 \\ 6 \\ 9 \end{pmatrix}\).

This gives us: \(x_1 + 2x_2 = 3\), \(2x_1 + 4x_2 = 6\), \(3x_1 + 6x_2 = 9\).

The second and third equations are multiples of the first, so we have one equation with two unknowns.

Let \(x_2 = 0\), then \(x_1 = 3\).

So one solution is \(\vec{x} = \begin{pmatrix} 3 \\ 0 \end{pmatrix}\).

Step 4: Check if \(\vec{b_2}\) is in column space

\(\vec{b_2} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} = 1\begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\).

Since \(\vec{b_2}\) equals the column space generator, \(\vec{b_2} \in \text{col}(A)\).

Therefore, \(A\vec{x} = \vec{b_2}\) has a solution. ✓

Step 5: Find solution for \(\vec{b_2}\)

We need \(\begin{pmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}\).

This gives us: \(x_1 + 2x_2 = 1\), \(2x_1 + 4x_2 = 2\), \(3x_1 + 6x_2 = 3\).

Again, the second and third equations are multiples of the first.

Let \(x_2 = 0\), then \(x_1 = 1\).

So one solution is \(\vec{x} = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\).

\(\vec{b_1}, \vec{b_2} \in \text{col}(A)\), so both systems have solutions.
Final answer:

Both systems have solutions: For \(\vec{b_1}\), \(\vec{x} = \begin{pmatrix} 3 \\ 0 \end{pmatrix}\) is a solution; for \(\vec{b_2}\), \(\vec{x} = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\) is a solution.

Applied rules:

• Consistency test: System has solution ⟺ \(\vec{b} \in \text{col}(A)\)

• Dependent columns: Reduces dimension of column space

• Free variables: Allow for multiple solutions

Null Space and Column Space: Advanced Concepts
\(\text{null}(A) \oplus \text{row}(A) = \mathbb{R}^n\)
Fundamental Subspaces
Key definitions:

Null Space: The set of all vectors that are mapped to the zero vector by the matrix transformation. It represents the "kernel" of the transformation.

Column Space: The set of all possible outputs of the matrix transformation. It represents the "range" of the transformation.

Row Space: The span of the row vectors of A, which has the same dimension as the column space.

Complete methodology:
  1. For null space: Row reduce [A | 0] and express solution in parametric form
  2. For column space: Row reduce A and select original columns corresponding to pivot positions
  3. For rank: Count number of pivots in row-reduced form
  4. For nullity: Count number of free variables or use rank-nullity theorem
Tip 1: Always use the original matrix columns for column space basis, not the reduced form.
Tip 2: The number of free variables equals the nullity.
Tip 3: Use the rank-nullity theorem as a verification check.
Tip 4: For square matrices, invertibility is equivalent to null space being trivial.
Common errors: Using reduced columns for column space, miscounting dimensions, forgetting to verify solutions.
Applications: Solving linear systems, differential equations, least squares, machine learning, computer graphics.
Properties to know by heart:

• \(\text{rank}(A) = \text{rank}(A^T)\) (row rank equals column rank)

• \(\text{null}(A) \perp \text{row}(A)\) (null space is orthogonal to row space)

• \(A\) is surjective ⟺ \(\text{col}(A)\) equals codomain

• \(A\) is injective ⟺ \(\text{null}(A) = \{\vec{0}\}\)

• \(A\) is bijective ⟺ \(A\) is square and invertible

Null Space and Column Space Visualization: Geometric Interpretation
Exercise 6: Geometric Relationships
Visualize the relationship between null space and column space for a \(2 \times 3\) matrix \(A = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \end{pmatrix}\). Show how the null space (in domain \(\mathbb{R}^3\)) relates to the column space (in codomain \(\mathbb{R}^2\)).

Geometric Interpretation: The null space represents all vectors in the domain that are "collapsed" to zero in the codomain, while the column space represents all possible outputs of the transformation.

  • Domain: \(\mathbb{R}^3\) (3D space)
  • Null space: Line through origin in \(\mathbb{R}^3\)
  • Codomain: \(\mathbb{R}^2\) (2D space)
  • Column space: Line through origin in \(\mathbb{R}^2\)

Questions & Answers

Question: I'm confused about why we call the null space the "kernel". What's the connection between these terms?

Answer: The terms "null space" and "kernel" refer to the same concept, but come from different historical contexts:

Null Space: This name emphasizes that these are the vectors that are mapped to the "null" or zero vector by the transformation.

Kernel: This term comes from German "Kern" (meaning core or kernel) introduced by mathematicians studying abstract algebra. The kernel represents the "core" of what gets "annihilated" by the transformation.

Both terms are used interchangeably in linear algebra. The kernel terminology is more common in abstract algebra and functional analysis, while null space is more common in matrix theory and applied mathematics.

Formally: \(\ker(A) = \text{null}(A) = \{x \mid Ax = 0\}\)

Question: Why do we use the original columns for the column space basis instead of the columns from the row-reduced matrix?

Answer: This is a crucial distinction:

Row operations preserve linear relationships: Row operations don't change which columns are linearly independent, but they do change the actual column vectors.

Example: If \(A = \begin{pmatrix} 1 & 2 \\ 3 & 6 \end{pmatrix}\), then after row reduction we get \(\begin{pmatrix} 1 & 2 \\ 0 & 0 \end{pmatrix}\).

The column space of A is spanned by \(\left\{\begin{pmatrix} 1 \\ 3 \end{pmatrix}\right\}\), not by \(\left\{\begin{pmatrix} 1 \\ 0 \end{pmatrix}\right\}\).

Row operations preserve the span of the columns, but they change the vectors themselves. The pivot positions tell us which original columns form a basis for the column space.

Question: What's the intuition behind the Rank-Nullity Theorem? Why should rank + nullity = number of columns?

Answer: Think of it in terms of degrees of freedom:

Input dimensions: An \(m \times n\) matrix takes vectors from \(\mathbb{R}^n\), which has \(n\) degrees of freedom.

Output dimensions: The rank tells us how many dimensions we "preserve" in the output (dimension of column space).

Collapsed dimensions: The nullity tells us how many dimensions are "lost" (dimension of null space).

Intuitively: Total input dimensions = Preserved dimensions + Lost dimensions

It's like a conservation law for vector space dimensions. The matrix "uses up" its input dimensions in two ways: some dimensions contribute to the output (rank), and some dimensions get mapped to zero (nullity).

Question: How are null spaces and column spaces used in machine learning?

Answer: Null spaces and column spaces are fundamental in machine learning:

Principal Component Analysis (PCA):

  • Uses eigenvectors which are related to null spaces of shifted matrices
  • Column spaces determine the principal components

Linear Regression:

  • The solution space is related to the null space of the feature matrix
  • The fitted values lie in the column space of the feature matrix

Regularization:

  • Penalizes solutions that have large components in certain directions
  • Related to the geometry of the null and column spaces

Understanding these spaces helps in feature selection, dimensionality reduction, and model interpretability.

Question: How do null spaces and column spaces relate to solving systems of linear equations in physics?

Answer: These concepts are essential in physics:

Consistency of physical systems:

  • \(A\vec{x} = \vec{b}\) has a solution if and only if \(\vec{b}\) is in the column space of A
  • This determines whether a physical system has a solution

Freedom in solutions:

  • The null space represents degrees of freedom in the solution
  • In mechanics, this might represent gauge freedom
  • In circuit theory, it represents currents that don't affect voltages

Stability analysis:

  • Small perturbations in \(\vec{b}\) lead to large changes in solution if A has small singular values
  • This relates to the geometry of the null and column spaces

These concepts help determine solvability, uniqueness, and stability of physical models.