Matrix differential is time to get up!

Preface and thanks

• Ago have matrix derivation psychological shadow, in the end not transpose, in front of which the matrix, which matrix on the back, how do the chain rule, the first big problem generated by such a range of people. Among them, the most intellectual is often put out some of the online algorithm formula is derived fundamentally wrong, because matrix dimensions do not match, so these unscrupulous publishers may simply not push off, every day, you do copy paste Dafa , most of the time a lot of duplicate content errors. Recently read some material, feeling came up with some of the numbers, and therefore want to summarize this article, a one-time sort out clearly. Matrix differential is time to get up!
• Here, we must thank the people to write articles for a detailed analysis of this issue, especially the authors [2,3,8] are, they are very hard to stand beginner point of view these issues. Indeed, as a blogger said, I also think matrix differential belong limbo area, whether it is a few points, high generation or optimization course, both former teachers feel that this issue does not belong to the main line of knowledge, do not teach, the latter category teachers feel that this course is essentially also belong to the contents of linear Algebra + calculus, should get in the basic courses, thus creating such a situation, hates few books to use when ah!

Derivative with a predetermined symbol arrangement

• Sign convention
• $ X $: scalar
• $ Y $: scalar
• $ \ Mathbf {x} $: $ m $ dimensional column vector
• $ \ Mathbf {y} $: $ n $ dimensional column vector
• $ \ Mathbf {X} $: size $ m × n $ matrix
• Derivative layout $ (Layout) $

 

References:

1. Zhang Xian of Matrix Analysis and Applications, 2004
2. Xia ghost footer length. Matrix derivation procedure (a), ( https://zhuanlan.zhihu.com/p/24709748 )
3. Jianping Matrix vector machine learning derivation, ( https://www.cnblogs.com/pinard/ )
4. Kaare Brandt Petersen, Michael Syskind Pedersen. "The Matrix Cookbook", 2008
5. Thomas P. Minka. "Old and New Matrix Algebra Useful for Statistics", 2000
6. Searle Shayle R. "Matrix Algebra Useful for Statistics", 1982
7. Jan R. Magnus, Heinz Neudecker. "Matrix Differential Calculus with Applications in Statistics and Econometrics", 2007
8. "Matrix Vector Derivatives for Machine Learning", (作者邮箱：[email protected])