4. Matrix calculation

4.1 Scalar derivatives

 4.2 Subderivatives

 4.3 Derivation

4.3.1 Gradient

① If y is a vector and x is a scalar, the derivative is a scalar.

② If y is a vector and x is a vector, the derivative is a matrix.

 4.3.2 Scalar versus Vector Derivation

③  is an ellipse, and the gradient points in the direction of the largest value change.

 4.3.3 Vector versus scalar differentiation

 4.3.4 Vector-to-Vector Derivation

 4.3.5 Extended matrix-to-matrix derivatives