矩阵求导笔记

1. ML中为什么需要矩阵求导

• 简洁
  用方程式表示如下：
  $$y_1=w_1{X}_{11}+w_2{X}_{12}\text{\hspace{1em}(1)}$$
  $$y_2=w_1{X}_{21}+w_2{X}_{22}\text{\hspace{1em}(2)}$$
  转换成矩阵表示如下：
  $$Y=XW\text{\hspace{1em}(3)}$$
  $$Y=\left[\begin{array}{c}{y}_1\\ \\ y_2\end{array}\right],X=\left[\begin{array}{ccc}{x}_{11}& & x_{12}\\ & & \\ x_{21}& & x_{22}\end{array}\right],W=\left[\begin{array}{c}{w}_1\\ \\ w_2\end{array}\right]\text{\hspace{1em}(4)}$$

• 快速
  当使用python 中的numpy 库时候，在相对于 for 循环，Numpy 本身的计算提速相当快

• 源代码

import time
import numpy as npif __name__ == "__main__":N = 1000000a = np.random.rand(N)b = np.random.rand(N)start = time.time()c = np.dot(a,b)stop = time.time()print(f"c={c}")print("vectorized version: " + str(1000*(stop-start))+"ms")c = 0start1 = time.time()for i in range(N):c += a[i]*b[i]stop1 = time.time()print(f"c={c}")print("for loop: " + str(1000*(stop1-start1))+"ms")times1 = (stop1-start1)/(stop-start)print(f"times1={times1}")

• 结果

c=250071.8870070607
vectorized version: 6.549358367919922ms
c=250071.88700706122
for loop: 265.43641090393066ms
times1=40.52861303239898# 向量化居然比单独的for循环快40倍

2. 向量函数与矩阵求导初印象

• 标量函数:输出为标量的函数
  $$f(x)=x^2\Rightarrow x\in R\to x^2\in R$$
  $$f(x)=x_1^2+x_2^2\Rightarrow \left[\begin{array}{c}{x}_1\\ \\ x_2\end{array}\right]\in R^2\to x_1^2+x_2^2\in R$$
• 向量函数:输出为向量或矩阵的函数
  <1> 输入标量，输出向量
  $$f(x)=\left[\begin{array}{c}{f}_1(x)=x\\ \\ f_2(x)=x^2\end{array}\right]\Rightarrow x\in R,\left[\begin{array}{c}x\\ \\ x^2\end{array}\right]\in R^2$$
  <2> 输入标量，输出矩阵
  $$f(x)=\left[\begin{array}{ccc}{f}_{11}(x)=x& & f_{12}(x)=x^2\\ & & \\ f_{21}(x)=x^3& & f_{22}(x)=x^4\end{array}\right]\Rightarrow x\in R,\left[\begin{array}{ccc}x& & x^2\\ & & \\ x^3& & x^4\end{array}\right]\in R^{2\times 2}$$
  <3> 输入向量，输出矩阵
  $$f(x)=\left[\begin{array}{ccc}{f}_{11}(x)=x_1+x_2& & f_{12}(x)=x_1^2+x_2^2\\ & & \\ f_{21}(x)=x_1^3+x_2^3& & f_{22}(x)=x_1^4+x_2^4\end{array}\right]\Rightarrow \left[\begin{array}{c}{x}_1\\ \\ x_2\end{array}\right]\in R^2,\left[\begin{array}{ccc}{x}_1+x_2& & x_1^2+x_2^2\\ & & \\ x_1^3+x_2^3& & x_1^4+x_2^4\end{array}\right]\in R^{2\times 2}$$
• 总结
  矩阵求导的本质
  $$\frac{\mathrm{d}A}{\mathrm{d}B}=矩阵A中的每个元素对矩阵B中的每个元素求导$$

3. YX 拉伸术

3.1 f(x)为标量，X为列向量

• 标量不变，向量拉伸
• YX中，Y前面横向拉，X后面纵向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x},Y=f(x)为标量，X=\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]为列向量$$
  $$f(x)=f(x_1,x_2,....,x_n)为标量$$
• 标量$f(x)$不变，向量X 因为在YX拉伸术中在Y后面，所以向量X纵向拉伸，实际上就是将多元函数的偏导写在一个列向量中
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$

3.2 f(x)为列向量，X 为标量

$$f(x)=\left[\begin{array}{c}{f}_1(x)\\ \\ f_2(x)\\ \\ ⋮\\ \\ f_n(x)\end{array}\right];X为标量$$

• 标量不变，向量拉伸
• YX中，Y前面横向拉，X后面纵向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{ccccccc}\frac{\partial f_1(x)}{\partial x}& & \frac{\partial f_2(x)}{\partial x}& & \dots & & \frac{\partial f_n(x)}{\partial x}\end{array}\right]$$

3.3 f(x)为列向量，X 为列向量

$$f(x)=\left[\begin{array}{c}{f}_1(x)\\ \\ f_2(x)\\ \\ ⋮\\ \\ f_n(x)\end{array}\right];X=\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]为列向量$$

• 第一步先固定Y ，将 X 纵向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$
• 第二步，看每一个项$\frac{\partial f(x)}{\partial x_1}$,其中f(x)为列向量，$x_1$为标量，那么可以看出要进行 Y 横向拉
  $$\frac{\partial f(x)}{\partial x_1}=\left[\begin{array}{ccccccc}\frac{\partial f_1(x)}{\partial x_1}& & \frac{\partial f_2(x)}{\partial x_1}& & \dots & & \frac{\partial f_n(x)}{\partial x_1}\end{array}\right]$$
• 第三步 ，将每项整合如下
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{ccccccc}\frac{\partial f_1(x)}{\partial x_1}& & \frac{\partial f_2(x)}{\partial x_1}& & \dots & & \frac{\partial f_n(x)}{\partial x_1}\\ & & & & & & \\ \frac{\partial f_1(x)}{\partial x_2}& & \frac{\partial f_2(x)}{\partial x_2}& & \dots & & \frac{\partial f_n(x)}{\partial x_2}\\ & & & & & & \\ ⋮& & ⋮& & \dots & & ⋮\\ & & & & & & \\ \frac{\partial f_1(x)}{\partial x_n}& & \frac{\partial f_2(x)}{\partial x_n}& & \dots & & \frac{\partial f_n(x)}{\partial x_n}\end{array}\right]$$

4. 常见矩阵求导公式

4.1 $Y=A^{T}X$

$$f(x)=A^{T}X;A=[a_1,a_2,\dots ,a_n{]}^T;X=[x_1,x_2,\dots ,x_n{]}^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X}$$

• 由于$A^T=1\times n,X=n\times 1,那么f(x)为标量，即表示数值$，
• 标量不变，向量拉伸
• YX中，Y前面横向拉，X后面纵向拉
  $$f(x)=\sum _{i=1}^N{a}_i{x}_i$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$
• 可以计算$\frac{\partial f(x)}{\partial x_i}$
  $$\frac{\partial f(x)}{\partial x_i}=a_i$$
• 可得如下：
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}{a}_1\\ \\ a_2\\ \\ ⋮\\ \\ a_n\end{array}\right]=A$$
• 结论：
  $$当f(x)=A^{T}X$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=A$$

4.2 $Y=X^{T}AX$

$$f(x)=X^{T}AX;A=\left[\begin{array}{ccccccc}{a}_{11}& & a_{12}& & \dots & & a_{1n}\\ & & & & & & \\ a_{21}& & a_{22}& & \dots & & a_{2n}\\ & & & & & & \\ ⋮& & ⋮& & \dots & & ⋮\\ & & & & & & \\ a_{n1}& & a_{n2}& & \dots & & a_{nn}\end{array}\right];X=[x_1,x_2,\dots ,x_n{]}^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X}$$
$$f(x)=\sum _{i=1}^N\sum _{j=1}^N{a}_{ij}{x}_i{x}_j$$

• 标量不变，YX拉伸术，X纵向拉伸
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$
  $$\frac{\partial f(x)}{\partial x_i}=\left[\begin{array}{cccc}{a}_{i1}& a_{i2}& \dots & a_{in}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{1i}& a_{2i}& \dots & a_{ni}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ & & & \\ a_{21}& a_{22}& \dots & a_{2n}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{n1}& a_{n2}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ & & & \\ a_{12}& a_{22}& \dots & a_{n2}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]$$
• 已知$A,A^T$表示如下：
  $$A=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ & & & \\ a_{21}& a_{22}& \dots & a_{2n}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{n1}& a_{n2}& \dots & a_{nn}\end{array}\right];A^T=\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ & & & \\ a_{12}& a_{22}& \dots & a_{n2}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]$$
• 综上所述如下：
  当 $f(x)=X^{T}AX$时
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=AX+A^{T}X=(A+A^T)X$$