【机器学习】【线性回归】梯度下降

文章目录

• • @[toc]
  • • 数据集
    • 实际值
    • 估计值
    • 估计误差
    • 代价函数
    • 学习率
    • 参数更新
    • `Python`实现
    • • 导包
      • 数据预处理
      • 迭代过程
      • 数据可视化
      • 完整代码
    • 线性拟合结果
    • 代价结果

个人主页：丷从心

系列专栏：机器学习

---

数据集

$$\left(x^{(i)},y^{(i)}\right),i=1,2,?,m$$

---

实际值

$$y^{(i)}$$

---

估计值

$$h_{\theta }\left(x^{(i)}\right)={\theta }_0+{\theta }_1{x}^{(i)}$$

---

估计误差

$$h_{\theta }\left(x^{(i)}\right)?y^{(i)}$$

---

代价函数

$$J(\theta )=J({\theta }_0,{\theta }_1)=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }\left(x^{(i)}\right)?y^{(i)}\right)}^2=\frac{1}{2m}\sum _{i=1}^m{\left({\theta }_0+{\theta }_1{x}^{(i)}?y^{(i)}\right)}^2$$

---

学习率

• $\alpha$是学习率，一个大于$0$的很小的经验值，决定代价函数下降的程度

---

参数更新

$$\Delta {\theta }_j=\frac{?}{?{\theta }_j}J({\theta }_0,{\theta }_1)$$

$${\theta }_j:={\theta }_j?\alpha \Delta {\theta }_j={\theta }_j?\alpha \frac{?}{?{\theta }_j}J({\theta }_0,{\theta }_1)$$

$$\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]:=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]?\alpha \left[\begin{array}{c}\frac{?J({\theta }_0,{\theta }_1)}{?{\theta }_0}\\ \frac{?J({\theta }_0,{\theta }_1)}{?{\theta }_1}\end{array}\right]$$

$$\left[\begin{array}{c}\frac{?J({\theta }_0,{\theta }_1)}{?{\theta }_0}\\ \frac{?J({\theta }_0,{\theta }_1)}{?{\theta }_1}\end{array}\right]=\left[\begin{array}{c}\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)?y^{(i)}\right)\\ \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)?y^{(i)}\right)x^{(i)}\end{array}\right]=\left[\begin{array}{c}\frac{1}{m}\sum _{i=1}^m{e}^{(i)}\\ \frac{1}{m}\sum _{i=1}^m{e}^{(i)}{x}^{(i)}\end{array}\right]e^{(i)}=h_{\theta }\left(x^{(i)}\right)?y^{(i)}$$

$$\begin{array}{rl}\left[\begin{array}{c}\frac{?J({\theta }_0,{\theta }_1)}{?{\theta }_0}\\ \frac{?J({\theta }_0,{\theta }_1)}{?{\theta }_1}\end{array}\right]& =\left[\begin{array}{c}\frac{1}{m}\sum _{i=1}^m{e}^{(i)}\\ \frac{1}{m}\sum _{i=1}^m{e}^{(i)}{x}^{(i)}\end{array}\right]=\left[\begin{array}{c}\frac{1}{m}\left(e^{(1)}+e^{(2)}+?+e^{(m)}\right)\\ \frac{1}{m}\left(e^{(1)}+e^{(2)}+?+e^{(m)}\right)x^{(i)}\end{array}\right]\\ & =\frac{1}{m}\left[\begin{array}{cccc}1& 1& ?& 1\\ x^{(1)}& x^{(2)}& ?& x^{(m)}\end{array}\right]\left[\begin{array}{c}{e}^{(1)}\\ e^{(2)}\\ ?\\ e^{(m)}\end{array}\right]=\frac{1}{m}{X}^{T}e=\frac{1}{m}{X}^T(X\theta ?y)\end{array}$$

• 由上述推导得

$$\Delta \theta =\frac{1}{m}{X}^{T}e$$

$$\theta :=\theta ?\alpha \Delta \theta =\theta ?\alpha \frac{1}{m}{X}^{T}e$$

---

Python实现

导包

import numpy as np
import matplotlib.pyplot as plt

数据预处理

x = np.array([4, 3, 3, 4, 2, 2, 0, 1, 2, 5, 1, 2, 5, 1, 3])
y = np.array([8, 6, 6, 7, 4, 4, 2, 4, 5, 9, 3, 4, 8, 3, 6])

m = len(x)

x = np.c_[np.ones([m, 1]), x]
y = y.reshape(m, 1)
theta = np.zeros([2, 1])

迭代过程

alpha = 0.01
iter_cnt = 1000  # 迭代次数
cost = np.zeros([iter_cnt])  # 代价数据

for i in range(iter_cnt):
    h = x.dot(theta)  # 估计值
    error = h - y  # 误差值
    cost[i] = 1 / (2 * m) * error.T.dot(error)  # 代价值

    # 更新参数
    delta_theta = 1 / m * x.T.dot(error)
    theta -= alpha * delta_theta

数据可视化

# 回归结果
plt.scatter(x[:, 1], y, c='blue')
plt.plot(x[:, 1], h, 'r-')
plt.show()

# 代价结果
plt.plot(cost)
plt.show()

完整代码

import numpy as np
import matplotlib.pyplot as plt

x = np.array([4, 3, 3, 4, 2, 2, 0, 1, 2, 5, 1, 2, 5, 1, 3])
y = np.array([8, 6, 6, 7, 4, 4, 2, 4, 5, 9, 3, 4, 8, 3, 6])

m = len(x)

x = np.c_[np.ones([m, 1]), x]
y = y.reshape(m, 1)
theta = np.zeros([2, 1])

alpha = 0.01
iter_cnt = 1000  # 迭代次数
cost = np.zeros([iter_cnt])  # 代价数据

for i in range(iter_cnt):
    h = x.dot(theta)  # 估计值
    error = h - y  # 误差值
    cost[i] = 1 / (2 * m) * error.T.dot(error)  # 代价值

    # 更新参数
    delta_theta = 1 / m * x.T.dot(error)
    theta -= alpha * delta_theta

# 线性拟合结果
plt.scatter(x[:, 1], y, c='blue')
plt.plot(x[:, 1], h, 'r-')
plt.show()

# 代价结果
plt.plot(cost)
plt.show()

---

线性拟合结果

---

代价结果

---