Machine Learning - Assignment 1

$\begin{array}{|l|} \hline \text { Machine Learning } \\\\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textbf { Assignment 1 (Linear Algebra) }\\\\ \text {Instructor: Beilun Wang }\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{Name:Daiyang Luan\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{ID:61518421}}\\\\ \hline \end{array}$
​

Problem 1

​
Let two vectors $a=(1,2,3)^{\mathrm{T}}$ and $b=(-8,1,2)^{\mathrm{T}}$.Answer the following equations:
​
(1) Compute the $\ell_{2}$ norm of $a$ and $b$
​
(2) Calculate the Euclidean distance between $a$ and $b$
​
(3) Are $a$ and $b$ orthogonal?
​
Solution:

(1)The $\ell_{2}$ norm of $a$ is $\sqrt{14}$ and the $\ell_{2}$ norm of $b$ is $\sqrt{69}$.

(2)The Euclidean distance between $a$ and $b$ is $\sqrt{83}$.

(3)As $a^{\mathrm{T}}b=1\times (-8)+2\times 1+3\times 2=0$, $a$ and $b$ is orthogonal.
​

Problem 2

​
Suppose $A=\left[\begin{array}{ccc}{1} & {-3} & {3} \\ {3} & {-5} & {3} \\ {6} & {-6} & {4}\end{array}\right]$, answer the following questions:
​
(1) Calculate $A^{-1}$ and $\operatorname{det}(A)$.
​
(2) The Rank of $A$ is?
​
(3) The trace of $A$ is?
​
(4) Calculate $A+A^{T}$
​
(5) Is $A$ an orthogonal matrix? State your reason.
​
(6) Calculate all the eigenvalue $\lambda$ and corresponding eigenvectors of $A$.
​
(7) Diagonalize the matrix $A$.
​
(8) Calculate the $\ell_{2,1}$ norm $\|A\|_{2,1}$ and the Frobenius norm (i.e. $\ell_{2}$ norm) $\|A\|_{F}$
​
(9) Calculate the nuclear norm $\|A\|_*$ and the spectral norm $\|A\|_{2}$

Solution:
(1) $\left[\begin{array}{ccc} A &I\end{array}\right]=\left[\begin{array}{ccc}1&-3&3&1&0&0\\3&-5&3&0&1&0 \\6&-6&4&0&0&1\end{array}\right]\stackrel{row }{\longrightarrow}\left[\begin{array}{ccc}1&0&0&-1/8&-3/8&3/8\\0&1&0&3/8&-7/8&3/8 \\0&0&1&3/4&-3/4&1/4\end{array}\right]=\left[\begin{array}{ccc} I &A^{-1}\end{array}\right]$
Hence, $A^{-1}=\left[\begin{array}{ccc}-1/8&-3/8&3/8\\3/8&-7/8&3/8 \\3/4&-3/4&1/4\end{array}\right]$
$det(A)= \left|\begin{array}{cccc} 1 & -3 & 3 \\ 3 & -5 & 3\\ 6 & -6 & 4 \end{array}\right| =\left|\begin{array}{cccc} 1 & -3 & 3 \\ 0 & 4 & -6\\ 0 & 0 & 4 \end{array}\right|=16$

(2)As $det(A)\not=0$, $A$ is a full-rank matrix. Thus, the rank of $A$ is $3$.

(3) $tr(A)=1+(-5)+4=0$. That is, the trace of $A$ is $0$.

(4) $A+A^{T}=\left[\begin{array}{ccc}1&-3&3\\3&-5&3\\6&-6&4\end{array}\right]+\left[\begin{array}{ccc}1&3&6\\-3&-5&-6\\3&3&4\end{array}\right]=\left[\begin{array}{ccc}2&0&9\\0&-10&-3\\9&-3&8\end{array}\right]$

(5) $A^{T}A=\left[\begin{array}{ccc}46&-54&36\\-54&70&-48\\36&-48&34\end{array}\right]\not=I$, so $A$ is not an orthogonal matrix.

(6)The characteristic determinant of $A$ is $\left|\begin{array}{cccc} \lambda-1 & 3 & -3 \\ -3 & \lambda+5 & -3\\ -6 & 6 & \lambda-4 \end{array}\right|=(\lambda+2)^{2}(\lambda-4).$ Thus, all the eigenvalues of $A$ are $\lambda_{1}=\lambda_{2}=-2,\lambda_{3}=4.$ Let $A\alpha_{i}=\lambda_{i}\alpha_{i},i=1,2,3$. Then we have $\alpha_{1}=\left[\begin{array}{ccc}1\\1\\0\end{array}\right],\alpha_{2}=\left[\begin{array}{ccc}0\\1\\1\end{array}\right],\alpha_{3}=\left[\begin{array}{ccc}1\\1\\2\end{array}\right]$. $\alpha_{i}(i=1,2,3)$ are the corresponding eigenvectors.

(7)The diagonal matrix corresponding to matrix $A$ is $\left[\begin{array}{cccc} -2 & 0 & 0 \\ 0 & -2 & 0\\ 0 &0 & 4 \end{array}\right]$

(8)In order to calculate the $\ell_{2,1}$ norm $\|A\|_{2,1}$, we first calculate the 2-norm of each row: $\sqrt{19},\sqrt{43},2\sqrt{22}$. Thus, $\|A\|_{2,1}=\sqrt{19}+\sqrt{43}+2\sqrt{22}$.
$\Vert A \Vert_F=\left({\sum\limits_{i=1}^{m}{\sum\limits_{j=1}^n{(a_{ij})^2}}}\right)^{{\frac{1}{2}}}=\sqrt{1+9+9+9+25+9+36+36+16}=\sqrt{150}.$

(9)The nuclear norm $\|A\|_*$ is defined as the sum of all the singular values of matrix $A$. As is calculated above, $A^{T}A=\left[\begin{array}{ccc}46&-54&36\\-54&70&-48\\36&-48&34\end{array}\right]$. Supposing the eigenvalues of $A^TA$ are $\lambda_i, i=1,2,3$, we have $|\lambda I-A|=0$.
That is,
$\left|{\begin{array}{l} \lambda-46&54&-36\\ 54&\lambda-70&48\\ -36&48&\lambda-34 \end{array}}\right|=0$
Hence, we have $\lambda^3-150\lambda^2+648\lambda-256=0$
The solution of the equation is:
$\lambda_1=4$ $\lambda_2=73+9\sqrt{65}$ $\lambda_3=73-9\sqrt{65}$
Thus, $\|A\|_*=2+\sqrt{73+9\sqrt{65}}+\sqrt{73-9\sqrt{65}}\approx14.727922061357859$.
$\|A\|_2=\sqrt{max(A^TA})=\sqrt{73+9\sqrt{65}}\approx 12.064838156174618$

Problem 3​

Please give some proper steps to show how you get the answer. Let $x=\left(x_{1}, x_{2}, x_{3}\right)^{T}$ and
$\left\{\begin{array}{l} 2 x_{1}+2 x_{2}+3 x_{3}=1 \\ x_{1}-x_{2}=-1 \\ -x_{1}+2 x_{2}+x_{3}=2 \end{array}\right.$
Answer the following questions:
​
(1) Solve the linear equations
​
(2) Write it into matrix form(i.e. $A x=b$ ) and we will use the same $A$ and $b$ in the following questions.
​
(3) The Rank of $A$ is?
​
(4) Calculate $A^{-1}$ and $\operatorname{det}(A)$
​
(5) Use (4) to solve the linear equations
​
(6) Calculate the inner product and outer product of $x$ and $b$.(i.e. $\langle x, b\rangle$ and $x \otimes b$ )
​
(7) Calculate the $\ell_{1}, \ell_{2}$ and $\ell_{\infty}$ norm of $b$
​
(8) Suppose $y=\left(y_{1}, y_{2}, y_{3}\right),$ calculate $y^{T} A y, \nabla_{y} y^{T} A y$
​
(9) We add one linear equation $-x_{1}+2 x_{2}+x_{3}=2$ into linear equations above. Write it into matrix form(i.e. $\left.A_{1} x=b\right)$
​
(10) The rank of $A_{1}$ is?
​
(11) Could these linear equations $A_{1} x=b$ be solved? State reasons.
​
​Solution:
(1)Solving the linear equations, we have: $x_1=-1, x_2=0, x_3=1$.

(2)The linear equation can be written into matrix form $Ax=b$ where
$A=\left[\begin{array}{l} 2&2&3 \\ 1&-1&0 \\ -1&2&1 \end{array}\right]$
and
$b=\left[\begin{array}{l} 1\\-1\\2 \end{array}\right]$

(3)The rank of $A$ is 3.

(4) $A^{-1}=\left[\begin{array}{l} 1&-4&-3 \\ 1&-5&-3 \\ -1&6&4 \end{array}\right]$
$det(A)=-1.$

(5) $x=A^{-1}b=\left[\begin{array}{l} 1&-4&-3 \\ 1&-5&-3 \\ -1&6&4 \end{array}\right]\left[\begin{array}{l} 1\\-1\\2 \end{array}\right]=\left[\begin{array}{l} -1\\0\\1 \end{array}\right]$
That is, $x_1=-1, x_2=0, x_3=1$, which is consistent with the result of question1.

(6) $<x,b>=1,x\bigotimes b=\left[\begin{array}{l} 1&3&1 \end{array}\right]^T$

(7)The $\ell_1$ norm of $b$ is $\|b\|_1=1+1+2=5$.
The $\ell_2$ norm of $b$ is $\|b\|_2=\sqrt{1+1+4}=\sqrt{6}$.
The $\ell_\infty$ norm of $b$ is $\|b\|_\infty=max(1,1,2)=2$.

(8) $y^TAy=\left[\begin{array}{l} y_1&y_2&y_3 \end{array}\right]\left[\begin{array}{l} 2&2&3 \\ 1&-1&0 \\ -1&2&1 \end{array}\right]\left[\begin{array}{l} y_1\\y_2\\y_3 \end{array}\right]=2y_1^2-y_2^2+y_3^2+3y_1y_2+2y_2y_3+2y_1y_3$
$\nabla_yy^TAy=\left[\begin{array}{l} 4y_1+3y_2+2y_3\\3y_1-2y_2+2y_3\\2y_1+2y_2+2y_3 \end{array}\right]$

(9)The new linear equation can be written into matrix form $A_1x=b_1$ where
$A_1=\left[\begin{array}{l} 2&2&3 \\ 1&-1&0 \\ -1&2&1\\-1&2&1 \end{array}\right]$
and
$b_1=\left[\begin{array}{l} 1\\-1\\2\\2 \end{array}\right]$

(10)The rank of $A_1$ is 3.

(11)Yes.
The number of variables is the same as the rank of the new matrix $A_1$ and thus there is no more than one solution to the non homogeneous linear equations. Moreover, after diagonalizing the matrix $A$, we can see that after deleting the row whose elements are all zero, determinant of the new matrix is not zero. This indicates that a solution exists for these linear equations.