Double DQN缓解动作价值的高估问题

1、算法：

Selection using DQN：
$$a^{\star }=\underset{a}{\mathrm{argmax}}Q(s_{t+1},a;\mathbf{w}).$$
Evaluation using target network:
$$y_t=r_t+\gamma \cdot Q(s_{t+1},a^{\star };{\mathbf{w}}^{-}).$$

2、算法实现：

class DoubleDQN:def __init__(self, dim_obs=None, num_act=None, discount=0.9):self.discount = discountself.model = QNet(dim_obs, num_act)self.target_model = QNet(dim_obs, num_act)self.target_model.load_state_dict(self.model.state_dict())def get_action(self, obs):qvals = self.model(obs)return qvals.argmax()def compute_loss(self, s_batch, a_batch, r_batch, d_batch, next_s_batch):# Compute current Q value based on current states and actions.qvals = self.model(s_batch).gather(1, a_batch.unsqueeze(1)).squeeze()# next state的value不参与导数计算，避免不收敛。next_qvals, _ = self.target_model(next_s_batch).detach().max(dim=1)loss = F.mse_loss(r_batch + self.discount * next_qvals * (1 - d_batch), qvals)return loss