【机器学习】【线性回归】梯度下降

个人主页：丷从心

系列专栏：机器学习

---

数据集

$$\left(x^{(i)},y^{(i)}\right),i=1,2,\cdots ,m$$

---

实际值

$$y^{(i)}$$

---

估计值

$$h_{\theta }\left(x^{(i)}\right)={\theta }_0+{\theta }_1{x}^{(i)}$$

---

估计误差

$$h_{\theta }\left(x^{(i)}\right)-y^{(i)}$$

---

代价函数

$$J(\theta )=J({\theta }_0,{\theta }_1)=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)}^2=\frac{1}{2m}\sum _{i=1}^m{\left({\theta }_0+{\theta }_1{x}^{(i)}-y^{(i)}\right)}^2$$

---

学习率

• $\alpha$是学习率，一个大于$0$的很小的经验值，决定代价函数下降的程度

---

参数更新

$$\Delta {\theta }_j=\frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1)$$

$${\theta }_j:={\theta }_j-\alpha \Delta {\theta }_j={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1)$$

$$\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]:=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\alpha \left[\begin{array}{c}\frac{\partial J({\theta }_0,{\theta }_1)}{\partial {\theta }_0}\\ \frac{\partial J({\theta }_0,{\theta }_1)}{\partial {\theta }_1}\end{array}\right]$$

$$\left[\begin{array}{c}\frac{\partial J({\theta }_0,{\theta }_1)}{\partial {\theta }_0}\\ \frac{\partial J({\theta }_0,{\theta }_1)}{\partial {\theta }_1}\end{array}\right]=\left[\begin{array}{c}\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)\\ \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)x^{(i)}\end{array}\right]=\left[\begin{array}{c}\frac{1}{m}\sum _{i=1}^m{e}^{(i)}\\ \frac{1}{m}\sum _{i=1}^m{e}^{(i)}{x}^{(i)}\end{array}\right]e^{(i)}=h_{\theta }\left(x^{(i)}\right)-y^{(i)}$$

$$\begin{array}{rl}\left[\begin{array}{c}\frac{\partial J({\theta }_0,{\theta }_1)}{\partial {\theta }_0}\\ \frac{\partial J({\theta }_0,{\theta }_1)}{\partial {\theta }_1}\end{array}\right]& =\left[\begin{array}{c}\frac{1}{m}\sum _{i=1}^m{e}^{(i)}\\ \frac{1}{m}\sum _{i=1}^m{e}^{(i)}{x}^{(i)}\end{array}\right]=\left[\begin{array}{c}\frac{1}{m}\left(e^{(1)}+e^{(2)}+\cdots +e^{(m)}\right)\\ \frac{1}{m}\left(e^{(1)}+e^{(2)}+\cdots +e^{(m)}\right)x^{(i)}\end{array}\right]\\ & =\frac{1}{m}\left[\begin{array}{cccc}1& 1& \cdots & 1\\ x^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]\left[\begin{array}{c}{e}^{(1)}\\ e^{(2)}\\ ⋮\\ e^{(m)}\end{array}\right]=\frac{1}{m}{X}^{T}e=\frac{1}{m}{X}^T(X\theta -y)\end{array}$$

• 由上述推导得

$$\Delta \theta =\frac{1}{m}{X}^{T}e$$

$$\theta :=\theta -\alpha \Delta \theta =\theta -\alpha \frac{1}{m}{X}^{T}e$$

---

Python实现

导包

import numpy as np
import matplotlib.pyplot as plt

数据预处理

x = np.array([4, 3, 3, 4, 2, 2, 0, 1, 2, 5, 1, 2, 5, 1, 3])
y = np.array([8, 6, 6, 7, 4, 4, 2, 4, 5, 9, 3, 4, 8, 3, 6])m = len(x)x = np.c_[np.ones([m, 1]), x]
y = y.reshape(m, 1)
theta = np.zeros([2, 1])

迭代过程

alpha = 0.01
iter_cnt = 1000  # 迭代次数
cost = np.zeros([iter_cnt])  # 代价数据for i in range(iter_cnt):h = x.dot(theta)  # 估计值error = h - y  # 误差值cost[i] = 1 / (2 * m) * error.T.dot(error)  # 代价值# 更新参数delta_theta = 1 / m * x.T.dot(error)theta -= alpha * delta_theta

数据可视化

# 回归结果
plt.scatter(x[:, 1], y, c='blue')
plt.plot(x[:, 1], h, 'r-')
plt.show()# 代价结果
plt.plot(cost)
plt.show()

完整代码

import numpy as np
import matplotlib.pyplot as pltx = np.array([4, 3, 3, 4, 2, 2, 0, 1, 2, 5, 1, 2, 5, 1, 3])
y = np.array([8, 6, 6, 7, 4, 4, 2, 4, 5, 9, 3, 4, 8, 3, 6])m = len(x)x = np.c_[np.ones([m, 1]), x]
y = y.reshape(m, 1)
theta = np.zeros([2, 1])alpha = 0.01
iter_cnt = 1000  # 迭代次数
cost = np.zeros([iter_cnt])  # 代价数据for i in range(iter_cnt):h = x.dot(theta)  # 估计值error = h - y  # 误差值cost[i] = 1 / (2 * m) * error.T.dot(error)  # 代价值# 更新参数delta_theta = 1 / m * x.T.dot(error)theta -= alpha * delta_theta# 线性拟合结果
plt.scatter(x[:, 1], y, c='blue')
plt.plot(x[:, 1], h, 'r-')
plt.show()# 代价结果
plt.plot(cost)
plt.show()

---

线性拟合结果

---

代价结果

---