1.1.8 The Singular Value Decomposition

We now con­sid­er the sin­gu­lar value de­com­po­si­tion of a matrix to in­crease our con­cep­tu­al un­der­stand­ing of the geom­e­try in­volved in the linear trans­for­ma­tion process. All ma­tri­ces have a sin­gu­lar value de­com­po­si­tion, and the image of the unit sphere for any size, $m \times n$, takes on the form of a hy­per­el­lipse. The hy­per­el­lipse can be de­fined $\in \mathbb{C}^{m}\ or\ \mathbb{R}^{m}$, to be the ob­tained sur­face that re­sults from stretch­ing the unit sphere $\in \mathbb{C}^{m}\ or\ \mathbb{R}^{m}$, by the scal­ing fac­tors $\sigma_{1},\sigma_{2},\ldots\sigma_{m}$, in or­thog­o­nal di­rec­tions $u_{1},u_{2},\ldots u_{m}$. When taken to­geth­er, the set of vec­tors $\{\sigma_{j}u_{j}\}$, are the prin­ci­ple semi­ax­es of the hy­per­el­lipse, which are scaled by $\sigma_{1},\sigma_{2},\ldots\sigma_{m}$, equat­ing to the length of these semi­ax­es. The rank of the matrix being con­sid­ered for the sin­gu­lar value de­com­po­si­tion de­ter­mines the number of non-zero scal­ing el­e­ments $\sigma_{j}$ present in the de­com­po­si­tion. In what fol­lows, we use a sim­i­lar SVD de­f­i­n­i­tion to that de­scribed by Tre­fethen and Bau.¹

The SVD is one of the more im­por­tant means to un­der­stand how a matrix will act on a math­e­mat­i­cal object like a vector or an­oth­er matrix. It also yields one of the most im­por­tant in­sights into the com­plex con­ju­gate trans­pose of a matrix $\left( M^{\dagger} \right).$The SVD de­com­po­si­tion for a matrix $M$, is de­fined

for uni­tary

We note that $M^{\dagger}$, or in some texts, $M^{*},\ M^{\text{H}}$, means the (com­plex) con­ju­gate trans­pose of the matrix $M,$and that the com­plex con­ju­gate trans­pose of a uni­tary matrix equals the in­verse of that matrix. For the matrix $\Sigma$, we have di­ag­o­nal en­tries that are the real number scal­ing fac­tors $\sigma_{1},\sigma_{2},\ldots\sigma_{m}$, arranged in de­scend­ing order along the di­ag­o­nal. The map­ping $V^{\dagger}$, will rotate or re­flect the sphere, while pre­serv­ing its shape, the matrix $\Sigma$ stretch­es the sphere by sin­gu­lar values $\sigma_{1},\sigma_{2},\ldots\sigma_{m} \neq 1,$and the final matrix $U$, will then rotate or re­flect the hy­per­el­lipse. Figure 1.1.2 graph­i­cal­ly de­picts the SVD of a matrix $M$, for the ac­tions of $V^{\dagger}$, $\Sigma$, and $U$, which, when taken to­geth­er, ac­com­plish the linear trans­for­ma­tion $Mv$.

1. Tre­fethen & Bau, "Nu­mer­i­cal Linear Al­ge­bra", 1997