Fight4354

The learning process over the past couple of days has gradually clarified my understanding of linear algebra. With this post, I summarize some key new content.

When facing matrices and singular values, one should establish the following understanding:
✅ A matrix is a spatial operator.
✅ Singular value decomposition helps you break down the essence of a matrix: rotation → stretching → rotation.
✅ The size order of singular values tells you: in which directions the matrix truly has strength and which directions are ineffective.

1. Orthogonal Matrix#

The core definition of an Orthogonal Matrix

An $n \times n$ real matrix $Q$ is called an orthogonal matrix if it satisfies

Copy$$Q^{\mathsf T}\,Q \;=\; Q\,Q^{\mathsf T}\;=\;I_n$$

Here, $Q^{\mathsf T}$ is the transpose, and $I_n$ is the $n$-dimensional identity matrix.

The inverse is the transpose

This simplifies calculations and ensures numerical stability.
An orthogonal matrix is a real matrix that "preserves the inner product"—it rotates or flips the coordinate system but never stretches or distorts it.

2. Angle Brackets#

Here

is the symbol for the "inner product." In the most common case—real vector space $\mathbb R^n$—it is equivalent to the dot product we are familiar with:

3. Matrix Position Exchange#

1. Eliminate the left $Q$

• The $Q$ attached to the left of $S$

• Multiply both sides by its inverse $Q^{-1}$ from the left:

Note:

• Must multiply from the left consistently;

• Do not attempt to multiply $Q^{-1}$ to the right (that would disrupt the order of multiplication).

2. Eliminate the right $Q^{\mathsf T}$

• The $Q^{\mathsf T}$ attached to the right of $S$

• Multiply both sides by its inverse $(Q^{\mathsf T})^{-1}$ from the right:

If $Q$ is an orthogonal matrix, $Q^{-1}=Q^{\mathsf T}$, thus

Why can't the order be reversed?

• Once you multiply on the wrong side, the symbols will "insert" into different positions:
  $AQ^{-1} \neq Q^{-1}A$.

• The same operation must be performed symmetrically on both sides of the equation for it to hold.

• This is essentially the same as the order of function composition or coordinate transformation: which transformation to perform first and which second must be written in the corresponding positions of the product, and cannot be arbitrarily exchanged.

4. Similar Diagonalizable Matrix#

A similar diagonalizable matrix (commonly referred to as a "diagonalizable matrix") means:

There exists an invertible matrix $P$ such that

Copy$$P^{-1}AP = D,$$

where $D$ is a diagonal matrix.
At this point, we say $A$ can be diagonalized through a similarity transformation, or simply $A$ is diagonalizable.

The "mechanical process" of diagonalization

1. Find the eigenvalues: Solve $\det(A-\lambda I)=0$.

2. Find the eigenvectors: For each $\lambda$, solve $(A-\lambda I)x=0$.

3. Assemble $P$: Form the matrix $P$ by stacking the $n$ linearly independent eigenvectors as columns.

4. Obtain $D$: Fill the corresponding eigenvalues into the diagonal: $D=\operatorname{diag}(\lambda_1,\dots,\lambda_n)$.
   Then we have $A = P D P^{-1}$.

5. Singular Value Decomposition#

The expression $A=Q S Q^{\mathsf T}$ is called orthogonal similarity diagonalization; geometrically, it means "rotate (or mirror) the coordinate system → A only retains independent stretching."

1. Why can "real symmetric matrices always be orthogonally diagonalized"?

Spectral Theorem:

For any real symmetric matrix $A$, there exists an orthogonal matrix $Q$ such that $Q^{\mathsf T} A Q$ is diagonal, and the diagonal elements are the eigenvalues of $A$.

• Real eigenvalues: Symmetry guarantees all eigenvalues are real.
• Orthogonal eigenvectors: If $\lambda_i \ne \lambda_j$, the corresponding eigenvectors must be orthogonal.
• Repeated roots can also take orthogonal bases: The same eigenvalue may correspond to multiple vectors; in this case, perform Gram–Schmidt in the subspace they span.

2. Step-by-step textual analysis

• If $\lambda$ has duplicates (geometric multiplicity >1), first take a set of linearly independent vectors, then perform Gram-Schmidt in that subspace to make them pairwise orthogonal and normalize (adjust length to 1). |
  | 4. Arrange the modified eigenvectors horizontally according to the order of eigenvalues on the diagonal to obtain the orthogonal matrix $Q$ | Arrange the modified eigenvectors as columns in $Q=[q_1\ \cdots\ q_n]$ according to the order of the diagonal eigenvalues. At this point, $Q^{\mathsf T}Q=I$, and we have $A = Q S Q^{\mathsf T}$. |

3. A specific small example of $2 \times 2$

Let

① Find the eigenvalues

② Find the eigenvectors

• $\lambda_1=5$: $(A-5I)v=0 \ \Rightarrow\ v_1=\begin{bmatrix}1\\1\end{bmatrix}$
• $\lambda_2=3$: $(A-3I)v=0 \ \Rightarrow\ v_2=\begin{bmatrix}1\\-1\end{bmatrix}$

③ Normalize

④ Assemble and verify

6. Determinant (det · )#

The determinant (determinant) is an operation that maps a square matrix $A$ to a scalar $\det A$.
This scalar encapsulates the most core geometric and algebraic information of the matrix: volume scaling factor, invertibility, product of eigenvalues, etc.

Formula

3×3 Manual Calculation Example

Let

Expand along the first row:

In summary

"Taking the determinant" means: collapsing an $n\times n$ square matrix into a single number through a set of alternating, linear rules, and this number encodes key information such as the matrix's volume scaling, direction, invertibility, and product of eigenvalues.

7. Rank of a Matrix#

What exactly is the "rank" of a matrix?

Linear Independence
Below are three comparative cases using a small $3 \times 3$ matrix to make the statement "rank = how many linearly independent column (or row) vectors can be selected" clear.

| Matrix $A$ | Column Vectors Written as $\bigl(v_1\,|\,v_2\,|\,v_3\bigr)$ | Linear Relationship | Rank |
|------------|-----------------------------------------------|-----------|--------|
| $\displaystyle\begin{bmatrix}1&2&3\\2&4&6\\3&6&9\end{bmatrix}$ | $v_1=\begin{bmatrix}1\\2\\3\end{bmatrix}$
$v_2=\begin{bmatrix}2\\4\\6\end{bmatrix}=2v_1$
$v_3=\begin{bmatrix}3\\6\\9\end{bmatrix}=3v_1$ | All three columns lie on the same line—only 1 independent vector | 1 |
| $\displaystyle\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}$ | $v_1=\begin{bmatrix}1\\0\\1\end{bmatrix}$
$v_2=\begin{bmatrix}0\\1\\1\end{bmatrix}$
$v_3=v_1+v_2$ | $v_1, v_2$ are not collinear ⇒ 2-dimensional plane; $v_3$ lies in this plane | 2 |
| $\displaystyle\begin{bmatrix}1&0&1\\0&1&1\\1&1&0\end{bmatrix}$ | $v_1=\begin{bmatrix}1\\0\\1\end{bmatrix}$
$v_2=\begin{bmatrix}0\\1\\1\end{bmatrix}$
$v_3=\begin{bmatrix}1\\1\\0\end{bmatrix}$ | Any two columns cannot linearly express the third column ⇒ All three columns span the entire $\mathbb R^{3}$ | 3 |

How to determine "independence"?

• Manual Calculation Form a matrix with the columns and perform elimination → The number of non-zero rows is the rank.

• Concept If there exist constants $c_1,c_2,c_3$ such that $c_1v_1+c_2v_2+c_3v_3=0$ and not all are 0, the vectors are dependent; otherwise, they are independent.

  • Case 1: $2v_1-v_2=0$ → dependent

  • Case 2: Only $v_3=v_1+v_2$ is dependent, $v_1,v_2$ are independent

  • Case 3: Any non-trivial combination ≠ 0 → All three vectors are independent

In summary: The rank = how much independent information (dimension) this matrix can truly "hold."

8. Low-Rank Approximation#

Why does truncating SVD (low-rank approximation) only require storing k (m+n)+k numbers?

When the original matrix

is truncated to rank $k$ and written as

Adding these three parts gives

• $U_k$ and $V_k$: Each has $k$ columns, and each column stores a vector of length equal to the number of rows
  $\Rightarrow mk + nk$ scalars.

• $\Sigma_k$: It is a diagonal matrix, only needing those $k$ diagonal elements—not $k^2$.

Therefore, using a rank-$k$ SVD approximation to replace the original $m\times n$ storage, the parameter count reduces from $mn$ to $k(m+n)+k$.
If $k \ll \min(m,n)$, the saved space is quite considerable.

Lower rank = reduced information dimension, low-rank storage = reduced parameter count/memory simultaneously

9. Norms#

The "double vertical bars" $|,\cdot,|$ in linear algebra represent norms.

• For a vector $v\in\mathbb R^m$, the most commonly used is the Euclidean norm: $\|v\|=\sqrt{v^{\mathsf T}v}=\sqrt{\sum_{i=1}^{m}v_i^{2}},\qquad \|v\|^{2}=v^{\mathsf T}v.$
  In the figure, $\|Xw-y\|^{2}$ is the sum of the squares of each component of the vector $Xw-y$.

• For a matrix $A$, if also written as $\|A\|$, it commonly refers to the Frobenius norm: $\|A\|_F=\sqrt{\sum_{i,j}A_{ij}^{2}}$. However, the figures here involve vectors.

In contrast, the single vertical bar $|\,\cdot\,|$ usually denotes absolute value (scalar) or determinant $|A|$. Thus, double vertical bars signify the "length" of vectors/matrices, while single vertical bars denote scalar size or determinant—different objects and meanings.

Common Euclidean Distance for Vectors -- 2-Norm (L2 Norm)#

import torch

b = torch.tensor([3.0, 4.0])
print(b.norm())  # Outputs 5.0

.norm() is a method of PyTorch tensors (torch.Tensor).

Common Frobenius Norm for Matrices#

Matrices also have "length"—the commonly used is the Frobenius norm

1. Why can it also be written as "matrix dot product"

The commonly used inner product in matrix space is

where $\operatorname{tr}(\cdot)$ is the trace operation (sum of diagonal elements).
Taking the inner product of $A$ with itself gives

Thus:

  

Copy$$\boxed{\;\|A\|_F^{2}= \operatorname{tr}(A^{\mathsf T}A)\;}$$

This is the matrix version of $\|v\|^{2}=v^{\mathsf T}v$—just replacing the vector dot product with the "trace dot product."
The Frobenius norm is indeed equal to the square root of the sum of the squares of all singular values, that is:

Here:

• $| A |_F$ is the Frobenius norm of matrix $A$

• $\sigma_i$ is the singular value of $A$

The Frobenius norm is indeed equal to the square root of the sum of the squares of all singular values
Expanded explanation:

The Frobenius norm is defined as:

But singular value decomposition (SVD) tells us:

where $\Sigma$ is a diagonal matrix, with singular values $\sigma_1, \sigma_2, \dots$ on the main diagonal.

Since the Frobenius norm does not change (unit orthogonal transformations do not change the norm), we can directly compute:

Thus, ultimately:

Beware of misconceptions

Note:
✅ Not the square root of a single singular value, nor the maximum singular value
✅ Is the sum of the squares of all singular values, then taking the square root

The spectral norm looks at "the direction that stretches the most," while the Frobenius norm accumulates all energy.

Spectral Norm of a Matrix#

✅ Definition of Spectral Norm

The spectral norm of matrix $A$ is defined as:

Simply put, it is the maximum value to which matrix $A$ stretches a unit vector.
Singular values inherently represent the stretching transformations of the matrix.

It equals the maximum singular value of $A$:

From another perspective: the spectral norm ≈ the maximum length to which a unit vector is stretched after being input into the matrix.
✅ Its relationship with the Frobenius norm

• Frobenius Norm → Looks at overall energy (sum of squares of matrix elements)

• Spectral Norm → Looks at the maximum stretching amount in a single direction

In other words:

• The Frobenius norm is like the total "volume" of the matrix

• The spectral norm is like the "most extreme" stretching rate in a single direction

✅ Example: Why is it important?

Imagine a linear layer $W$ in a neural network:

• If $\| W \|_2$ is very large, even small perturbations in the input will be amplified,
  making the network prone to overfitting and sensitive to noise.

• If $\| W \|_2$ is moderate, the output changes will be stable against input perturbations,
  leading to better generalization.

Thus, modern methods (like spectral normalization)
will directly constrain the spectral norm of $W$ within a certain range during training.

⚠ Straightforward drawbacks

The spectral norm is powerful, but:

• It only focuses on a single maximum direction, ignoring the stretching in other directions;

• It is more complex to compute than the Frobenius norm (requires singular value decomposition, rather than a simple sum of squares).

Summary Comparison

1. Same idea, different dimensions

• The Euclidean norm is the square root of a vector's dot product with itself.

• The Frobenius norm treats all matrix elements as a long vector and does the same; written in matrix language as

  Copy$$\|A\|_F=\sqrt{\operatorname{tr}(A^{\mathsf T}A)}.$$

  This is "transpose → multiply → take trace."

2. When to use which?

3. Intuitive Differences

• Euclidean: Measures the length in a single direction;

• Frobenius: Measures the total energy of each element, so for matrices, no particular row or column is special; all elements are treated equally.

One-sentence memory:

Euclidean Norm: The "ruler" for vectors.
Frobenius Norm: Measuring the overall size of a matrix after "flattening" it with the same ruler.

10. Transpose of Matrix Multiplication#

In matrix algebra, there is a fixed "flip order" rule for the transpose of the product of two (or more) matrices:

This means transpose each matrix first, then reverse the order of multiplication.
This property holds for any dimension-matching real (or complex) matrices and can be recursively extended:

xLog Editing Markdown Document Notes

• Ensure all mathematical expressions are enclosed in $$ … $$

• If there are single $n \times n$ types, change them to n × n or $$n\\times n$$

Reference video:

Click here to watch the Bilibili video