注意力机制 attention Transformer 笔记

动手学深度学习

注意力

注意力汇聚的输出为值的加权和

查询的长度为q，键的长度为k，值的长度为v。
$\mathbf{q}\in {}^{1\times q},\mathbf{k}\in {}^{1\times k},\mathbf{v}\in {\mathbb{R}}^{1\times v}$
n个查询和m个键-值对
$\mathbf{Q}\in {}^{n\times q},\mathbf{K}\in {}^{m\times k},\mathbf{V}\in {\mathbb{R}}^{m\times v}$
$\mathbf{a}\left(\mathbf{Q},\mathbf{K}\right)\in {\mathbb{R}}^{n\times m}$是注意力评分函数
$\mathbf{\alpha }\left(\mathbf{Q},\mathbf{K}\right)=\mathrm{softmax}\left(\mathbf{a}\left(\mathbf{Q},\mathbf{K}\right)\right)=\frac{\exp \left(\mathbf{a}\left(\mathbf{Q},\mathbf{K}\right)\right)}{\sum _{j=1}^m\exp \left(\mathbf{a}\left(\mathbf{Q},\mathbf{K}\right)\right)}\in {\mathbb{R}}^{n\times m}$是注意力权重
$f(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{\alpha }{\left(\mathbf{Q},\mathbf{K}\right)}^{\top }\mathbf{V}\in {\mathbb{R}}^{n\times v}$是注意力汇聚函数

加性注意力

$\mathbf{q}\in {\mathbb{R}}^{1\times q},\mathbf{k}\in {\mathbb{R}}^{1\times k}$
${\mathbf{W}}_q\in {\mathbb{R}}^{h\times q},{\mathbf{W}}_k\in {\mathbb{R}}^{h\times k},{\mathbf{w}}_v\in {\mathbb{R}}^{h\times 1}$
$a(\mathbf{q},\mathbf{k})={\mathbf{w}}_v^{\top }\tanh ({\mathbf{W}}_q{\mathbf{q}}^{\top }+{\mathbf{W}}_k{\mathbf{k}}^{\top })\in \mathbb{R}$是注意力评分函数

缩放点积注意力

$\mathbf{q}\in \mathbb{R}{}^{1\times d},\mathbf{k}\in \mathbb{R}{}^{1\times d},\mathbf{v}\in {\mathbb{R}}^{1\times v}$
$a\left(\mathbf{q},\mathbf{k}\right)=\frac{1}{\sqrt{d}}\mathbf{q}{\mathbf{k}}^{\top }\in \mathbb{R}$是注意力评分函数
$f(\mathbf{q},\mathbf{k},\mathbf{v})=\alpha {\left(\mathbf{q},\mathbf{k}\right)}^{\top }\mathbf{v}=\mathrm{softmax}\left(\frac{1}{\sqrt{d}}\mathbf{q}{\mathbf{k}}^{\top }\right)\mathbf{v}\in {\mathbb{R}}^{1\times v}$是注意力汇聚函数

n个查询和m个键-值对
$\mathbf{Q}\in {\mathbb{R}}^{n\times d},\mathbf{K}\in {\mathbb{R}}^{m\times d},\mathbf{V}\in {\mathbb{R}}^{m\times v}$
$\mathbf{a}\left(\mathbf{Q},\mathbf{K}\right)=\frac{1}{\sqrt{d}}\mathbf{Q}{\mathbf{K}}^{\top }\in {\mathbb{R}}^{n\times m}$是注意力评分函数
$f(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{\alpha }{\left(\mathbf{Q},\mathbf{K}\right)}^{\top }\mathbf{V}=\mathrm{softmax}\left(\frac{1}{\sqrt{d}}\mathbf{Q}{\mathbf{K}}^{\top }\right)\mathbf{V}\in {\mathbb{R}}^{n\times v}$是注意力汇聚函数

多头注意力

$\mathbf{q}\in {\mathbb{R}}^{1\times d_q},\mathbf{k}\in {\mathbb{R}}^{1\times d_k},\mathbf{v}\in {\mathbb{R}}^{1\times d_v}$
${\mathbf{W}}_i^{(q)}\in {\mathbb{R}}^{p_q\times d_q},{\mathbf{W}}_i^{(k)}\in {\mathbb{R}}^{p_k\times d_k},{\mathbf{W}}_i^{(v)}\in {\mathbb{R}}^{p_v\times d_v}$
${\mathbf{h}}_i=f\left({\mathbf{W}}_i^{(q)}{\mathbf{q}}^{\top },{\mathbf{W}}_i^{(k)}{\mathbf{k}}^{\top },{\mathbf{W}}_i^{(v)}{\mathbf{v}}^{\top }\right)\in {\mathbb{R}}^{1\times p_v}$是注意力头

${\mathbf{W}}_o\in {\mathbb{R}}^{p_o\times h{p}_v}$
${\mathbf{W}}_o\left[\begin{array}{c}{\mathbf{h}}_1^{\top }\\ ⋮\\ {\mathbf{h}}_h^{\top }\end{array}\right]\in {\mathbb{R}}^{p_o}$

$p_{q}h=p_{k}h=p_{v}h=p_o$

多头注意力：多个注意力头连结然后线性变换

自注意力

${\mathbf{x}}_i\in {\mathbb{R}}^{1\times d},\mathbf{X}=\left[\begin{array}{c}{\mathbf{x}}_1\\ \cdots \\ {\mathbf{x}}_n\end{array}\right]\in {\mathbb{R}}^{n\times d}$
$\mathbf{Q}=\mathbf{X},\mathbf{K}=\mathbf{X},\mathbf{V}=\mathbf{X}$
$f(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{\alpha }{\left(\mathbf{Q},\mathbf{K}\right)}^{\top }\mathbf{V}=\mathrm{softmax}\left(\frac{1}{\sqrt{d}}\mathbf{Q}{\mathbf{K}}^{\top }\right)\mathbf{V}\in {\mathbb{R}}^{n\times d}$
${\mathbf{y}}_i=f\left({\mathbf{x}}_i,\left({\mathbf{x}}_1,{\mathbf{x}}_1\right),\dots ,\left({\mathbf{x}}_n,{\mathbf{x}}_n\right)\right)\in {\mathbb{R}}^d$

n个查询和m个键-值对
$\mathbf{Q}=\tanh \left({\mathbf{W}}_q\mathbf{X}\right)\in {\mathbb{R}}^{n\times d}$
$\mathbf{K}=\tanh \left({\mathbf{W}}_k\mathbf{X}\right)\in {\mathbb{R}}^{m\times d}$
$\mathbf{V}=\tanh \left({\mathbf{W}}_v\mathbf{X}\right)\in {\mathbb{R}}^{m\times v}$

J. Xu, F. Zhong, and Y. Wang, “Learning multi-agent coordination for enhancing target coverage in directional sensor networks,” in Proc. Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada, Dec. 2020, pp. 1–16.
https://github.com/XuJing1022/HiT-MAC/blob/main/perception.py

${\mathbf{x}}_i\in {\mathbb{R}}^{1\times d_{in}},\mathbf{X}=\left[\begin{array}{c}{\mathbf{x}}_1\\ \cdots \\ {\mathbf{x}}_{nm}\end{array}\right]\in {\mathbb{R}}^{nm\times d_{in}}$
$\mathbf{W}\in {\mathbb{R}}^{d_{att}\times d_{in}}$
$\mathbf{Q}=\tanh {\left({\mathbf{W}}_q{\mathbf{X}}^{\top }\right)}^{\top }\in {\mathbb{R}}^{nm\times d_{att}}$
$\mathbf{K}=\tanh {\left({\mathbf{W}}_k{\mathbf{X}}^{\top }\right)}^{\top }\in {\mathbb{R}}^{nm\times d_{att}}$
$\mathbf{V}=\tanh {\left({\mathbf{W}}_v{\mathbf{X}}^{\top }\right)}^{\top }\in {\mathbb{R}}^{nm\times d_{att}}$
$f(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{\alpha }{\left(\mathbf{Q},\mathbf{K}\right)}^{\top }\mathbf{V}=\mathrm{softmax}\left(\frac{1}{\sqrt{d}}\mathbf{Q}{\mathbf{K}}^{\top }\right)\mathbf{V}\in {\mathbb{R}}^{nm\times d_{att}}$

class AttentionLayer(torch.nn.Module):def __init__(self, feature_dim, weight_dim, device):super(AttentionLayer, self).__init__()self.in_dim = feature_dimself.device = deviceself.Q = xavier_init(nn.Linear(self.in_dim, weight_dim))self.K = xavier_init(nn.Linear(self.in_dim, weight_dim))self.V = xavier_init(nn.Linear(self.in_dim, weight_dim))self.feature_dim = weight_dimdef forward(self, x):# param x: [num_agent, num_target, in_dim]# return z: [num_agent, num_target, weight_dim]# z = softmax(Q,K)*Vq = torch.tanh(self.Q(x))  # [batch_size, sequence_len, weight_dim]k = torch.tanh(self.K(x))  # [batch_size, sequence_len, weight_dim]v = torch.tanh(self.V(x))  # [batch_size, sequence_len, weight_dim]z = torch.bmm(F.softmax(torch.bmm(q, k.permute(0, 2, 1)), dim=2), v)  # [batch_size, sequence_len, weight_dim]global_feature = z.sum(dim=1)return z, global_feature

Transformer