矩阵的偏导数

设 $X=(x_{ij}{)}_{m\times n}$，函数 $f(X)=f(x_{11},x_{12},\dots ,x_{1n},x_{21},\dots ,x_{mn})$ 是一个 $m\times n$ 元的多元函数，且偏导数

$$\frac{\partial f}{\partial x_{ij}}(i=1,2,\dots ,m,j=1,2,\dots ,n)$$

都存在。定义 $f(X)$ 对矩阵 $X$ 的导数为：

$$\frac{df(X)}{dX}={\left(\frac{\partial f}{\partial x_{ij}}\right)}_{m\times n}=\left[\begin{array}{ccc}\frac{\partial f}{\partial x_{11}}& \cdots & \frac{\partial f}{\partial x_{1n}}\\ ⋮& \ddots & ⋮\\ \frac{\partial f}{\partial x_{m1}}& \cdots & \frac{\partial f}{\partial x_{mn}}\end{array}\right]$$

(1) 设 $\mathbf{x}=({\xi }_1,{\xi }_2,\cdots ,{\xi }_n{)}^{\top }$，$n$ 元函数 $f(\mathbf{x})$，求 $\frac{df}{d{\mathbf{x}}^{\top }}$、$\frac{df}{d\mathbf{x}}$ 和 $\frac{d^{2}f}{d{\mathbf{x}}^2}$。

$$\frac{df}{d{\mathbf{x}}^{\top }}=\left(\begin{array}{c}\frac{\partial f}{\partial {\xi }_1},\frac{\partial f}{\partial {\xi }_2},\cdots ,\frac{\partial f}{\partial {\xi }_n}\end{array}\right)$$

$$\nabla f(\mathbf{x})=\frac{df}{d\mathbf{x}}=\left(\begin{array}{c}\frac{\partial f}{\partial {\xi }_1}\\ \frac{\partial f}{\partial {\xi }_2}\\ ⋮\\ \frac{\partial f}{\partial {\xi }_n}\end{array}\right)\text{，这就是梯度。}$$

$$H(\mathbf{x})={\nabla }^{2}f(\mathbf{x})=\frac{{\partial }^{2}f}{\partial \mathbf{x}\partial {\mathbf{x}}^{\top }}=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial {\xi }_1^2}& \frac{{\partial }^{2}f}{\partial {\xi }_1\partial {\xi }_2}& \cdots & \frac{{\partial }^{2}f}{\partial {\xi }_1\partial {\xi }_n}\\ \frac{{\partial }^{2}f}{\partial {\xi }_2\partial {\xi }_1}& \frac{{\partial }^{2}f}{\partial {\xi }_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial {\xi }_2\partial {\xi }_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial {\xi }_n\partial {\xi }_1}& \frac{{\partial }^{2}f}{\partial {\xi }_n\partial {\xi }_2}& \cdots & \frac{{\partial }^{2}f}{\partial {\xi }_n^2}\end{array}\right],\text{这就是Hessian 矩阵，它是对称的。}$$

(2) 设 $\mathbf{a}={\left(\begin{array}{c}{a}_1,a_2,\cdots ,a_n\end{array}\right)}^{\top }$ 为向量变量，且 $f(\mathbf{x})=f(\mathbf{x},\mathbf{a})$，求 $\frac{\partial f}{\partial \mathbf{x}}$。

解：由于 $f(\mathbf{x})={\sum }_{i=1}^n{a}_i{\xi }_j$，$\frac{\partial f}{\partial {\xi }_j}=a_j$，$(j=1,2,\cdots ,n)$，所以

$$\frac{\partial f}{\partial \mathbf{x}}=\left(\begin{array}{c}\frac{\partial f}{\partial {\xi }_1}\\ \frac{\partial f}{\partial {\xi }_2}\\ ⋮\\ \frac{\partial f}{\partial {\xi }_n}\end{array}\right)=\left(\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right)=\mathbf{a}$$

(3) 设 $A={\left(a_{ij}\right)}_{m\times n}$ 为常矩阵，$X={\left(x_{ij}\right)}_{n\times m}$ 为矩阵变量，且 $f(\mathbf{X})=\mathrm{tr}(\mathbf{AX})$，求 $\frac{\partial f}{\partial X}$。

分析：
$$\left(\begin{array}{ccc}{c}_{11}& \cdots & c_{1m}\\ ⋮& \ddots & ⋮\\ c_{m1}& \cdots & c_{mm}\end{array}\right)=\left(\begin{array}{ccc}{a}_{11}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ a_{m1}& \cdots & a_{mn}\end{array}\right)\left(\begin{array}{ccc}{x}_{11}& \cdots & x_{1n}\\ ⋮& \ddots & ⋮\\ x_{n1}& \cdots & x_{nm}\end{array}\right)$$

展开后得：
$$\begin{array}{c}\begin{array}{rl}{c}_{11}& =a_{11}{x}_{11}+a_{12}{x}_{21}+\cdots +a_{1n}{x}_{n1},\\ c_{22}& =a_{21}{x}_{12}+a_{22}{x}_{22}+\cdots +a_{2n}{x}_{n2},\\ & ⋮\\ c_{mn}& =a_{m1}{x}_{1m}+a_{m2}{x}_{2m}+\cdots +a_{mn}{x}_{nm}\end{array}\end{array}$$

规律：每个 $x$ 只会被用到一次，$x$ 的下标和 $a$ 的下标是相反的。

解：由于 $AX={\left({\sum }_{i=1}^n{a}_{ik}{x}_{ki}\right)}_{m\times m}$，

所以：$f(\mathbf{X})=\mathrm{tr}(\mathbf{AX})={\sum }_{s=1}^m{\sum }_{k=1}^n{a}_{sk}{x}_{ks}$。

而：
$${\left(\frac{\partial f}{\partial x_{ij}}\right)}_{n\times m}=(a_{ji}{)}_{n\times m}(i=1,2,\cdots ,n,j=1,2,\cdots ,m)$$

故：
$$\frac{\partial f}{\partial X}=\left(\frac{\partial f}{\partial x_{ij}}\right)=(a_{ji}{)}_{n\times m}=A^{\top }$$

(4) 设 $\mathbf{x}={\left({\xi }_1,{\xi }_2,\cdots ,{\xi }_n\right)}^{\top }$，矩阵 $A={\left(a_{ij}\right)}_{n\times n}$，$n$ 元函数 $f(\mathbf{x})={\mathbf{x}}^{\top }A\mathbf{x}$，求导数 $\frac{df}{d\mathbf{x}}$。

解：因
$$\begin{array}{rl}f\left(\mathbf{x}\right)& ={\mathbf{x}}^{\top }A\mathbf{x}\\ & =\left({\xi }_1,{\xi }_2,\cdots ,{\xi }_n\right)\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right)\left(\begin{array}{c}{\xi }_1\\ {\xi }_2\\ {\xi }_3\\ ⋮\\ {\xi }_n\end{array}\right)\\ & =\left(\begin{array}{cccccc}{\xi }_1& {\xi }_2& \cdots & {\xi }_k& \cdots & {\xi }_n\end{array}\right)\left(\begin{array}{c}\sum _{i=1}^n{a}_{1i}{\xi }_i\\ \sum _{i=1}^n{a}_{2i}{\xi }_i\\ ⋮\\ \sum _{i=1}^n{a}_{ki}{\xi }_i\\ ⋮\\ \sum _{i=1}^n{a}_{ni}{\xi }_i\end{array}\right)\\ & ={\xi }_1\sum _{j=1}^n{a}_{1j}{\xi }_j+\cdots +{\xi }_k\sum _{j=1}^n{a}_{kj}{\xi }_j+\cdots +{\xi }_n\sum _{j=1}^n{a}_{nj}{\xi }_j\end{array}$$

所以：
$$\begin{array}{rl}\frac{\partial f(\mathbf{x})}{\partial {\xi }_k}& ={\xi }_1{a}_{1k}+\cdots +{\xi }_{k-1}{a}_{k-1,k}+\left(\sum _{j=1}^n{a}_{kj}{\xi }_j+{\xi }_k{a}_{kk}\right)+{\xi }_{k+1}{a}_{k+1,k}+\cdots +{\xi }_n{a}_{nk}\\ & =\sum _{i=1}^n{a}_{ik}{\xi }_i+\sum _{j=1}^n{a}_{kj}{\xi }_j,k=1,2,\cdots ,n\end{array}$$
所以：
$$\begin{array}{rl}\frac{df}{d\mathbf{x}}& =\left(\begin{array}{c}\frac{\partial f}{\partial {\xi }_1}\\ \frac{\partial f}{\partial {\xi }_2}\\ ⋮\\ \frac{\partial f}{\partial {\xi }_n}\end{array}\right)=\left(\begin{array}{c}\sum _{j=1}^n{a}_{1j}{\xi }_j\\ \sum _{j=1}^n{a}_{2j}{\xi }_j\\ ⋮\\ \sum _{j=1}^n{a}_{nj}{\xi }_j\end{array}\right)+\left(\begin{array}{c}\sum _{i=1}^n{a}_{i1}{\xi }_i\\ \sum _{i=1}^n{a}_{i2}{\xi }_i\\ ⋮\\ \sum _{i=1}^n{a}_{in}{\xi }_i\end{array}\right)\\ & =Ax+A^{\top }x=(A+A^{\top })x\end{array}$$
特别地，当A为对称矩阵时，$\frac{df}{d\mathbf{x}}=2Ax$