先啰嗦直接看代码

Q3 Image Captioning with Transformers

MultiHeadAttention.forward

题面

解析

让我们实现多头注意力的前向计算

可以先看看这篇李沐老师对transformer的讲解

如果你看了上面的这个视频，那么你应该能大概明白多头注意力是个什么东西
然后我们来看init部分的代码

首先多头注意力让我们对单纯的注意力的Q,K,V,计算的时候进行了投影，那么具体的投影方法，就是上面定义的

1. self.key 投影key
2. self.query 投影query
3. self.value 投影 value

我们可以看到，这几个线性层的输入输出维度是相同的，接下来只需要计算单纯的权重就好了，如果看不懂我的解析,可以看我的代码注释，同时建议看看上面那个链接的视频

代码

    def forward(self, query, key, value, attn_mask=None):"""Calculate the masked attention output for the provided data, computingall attention heads in parallel.In the shape definitions below, N is the batch size, S is the sourcesequence length, T is the target sequence length, and E is the embeddingdimension.Inputs:- query: Input data to be used as the query, of shape (N, S, E)- key: Input data to be used as the key, of shape (N, T, E)- value: Input data to be used as the value, of shape (N, T, E)- attn_mask: Array of shape (S, T) where mask[i,j] == 0 indicates tokeni in the source should not influence token j in the target.Returns:- output: Tensor of shape (N, S, E) giving the weighted combination ofdata in value according to the attention weights calculated using keyand query."""N, S, E = query.shapeN, T, E = value.shape# Create a placeholder, to be overwritten by your code below.output = torch.empty((N, S, E))############################################################################# TODO: Implement multiheaded attention using the equations given in       ## Transformer_Captioning.ipynb.                                            ## A few hints:                                                             ##  1) You'll want to split your shape from (N, T, E) into (N, T, H, E/H),  ##     where H is the number of heads.                                      ##  2) The function torch.matmul allows you to do a batched matrix multiply.##     For example, you can do (N, H, T, E/H) by (N, H, E/H, T) to yield a  ##     shape (N, H, T, T). For more examples, see                           ##     https://pytorch.org/docs/stable/generated/torch.matmul.html          ##  3) For applying attn_mask, think how the scores should be modified to   ##     prevent a value from influencing output. Specifically, the PyTorch   ##     function masked_fill may come in handy.                              ############################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****# 投影QKVQ = self.query(query)K = self.key(key)V = self.value(value)# 获取投影个数H = self.n_head# 获取投影维度D = self.head_dim# 矩阵分割 与 转置 这里需要结合QKV的形状来理解Q = Q.reshape(N, S, H, D).transpose(1, 2)  # (N H S D)K = K.reshape(N, T, H, D).transpose(1, 2)  # (N H T D)V = V.reshape(N, T, H, D).transpose(1, 2)  # (N H T D)# 矩阵乘法算权重  (N, H, S, K) * (N, H, K, T) -> N, H, S, Tenergy = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(D)  # (N H S T)# 判断是否需要maskif attn_mask is not None:energy = energy.masked_fill(attn_mask == 0, float('-inf'))# softmax计算A = torch.softmax(energy, dim=3)  # 对第四维度进行softmax# 使用dropoutA = self.attn_drop(A)# 计算加权和  (N, H, S, T) * (N, H, T, K) -> (N, H, S, K)Y = A.matmul(V)# 再投影回去Y = Y.transpose(1, 2).reshape(N, S, E)  # (N, S, E)output = self.proj(Y)  # (N, S, E)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****#############################################################################                             END OF YOUR CODE                             #############################################################################return output

输出

Positional Encoding

题面

因为transformer能够很容易的主要的输入的任何部分，但是注意力机制没有顺序的概念，但是，对于许多人物，尤其是自然语言处理任务，顺序非常重要，因此作者添加了位置标记对每个单词的token

定于矩阵p为

我们并不直接把x输入进网络，而是将 x + p输入进来

解析

看懂上面的题面，就好了，还不懂就看代码

代码

class PositionalEncoding(nn.Module):"""Encodes information about the positions of the tokens in the sequence. Inthis case, the layer has no learnable parameters, since it is a simplefunction of sines and cosines."""def __init__(self, embed_dim, dropout=0.1, max_len=5000):"""Construct the PositionalEncoding layer.Inputs:- embed_dim: the size of the embed dimension- dropout: the dropout value- max_len: the maximum possible length of the incoming sequence"""super().__init__()self.dropout = nn.Dropout(p=dropout)assert embed_dim % 2 == 0# Create an array with a "batch dimension" of 1 (which will broadcast# across all examples in the batch).pe = torch.zeros(1, max_len, embed_dim)############################################################################# TODO: Construct the positional encoding array as described in            ## Transformer_Captioning.ipynb.  The goal is for each row to alternate     ## sine and cosine, and have exponents of 0, 0, 2, 2, 4, 4, etc. up to      ## embed_dim. Of course this exact specification is somewhat arbitrary, but ## this is what the autograder is expecting. For reference, our solution is ## less than 5 lines of code.                                               ############################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****# 创建一个 0 到 maxlen 的行向量并转为列向量 就相当于公式里的icol = torch.arange(max_len)[:, None]# 获取公式中的矩阵的行向量row = torch.pow(10000, -torch.arange(0, embed_dim, 2) / embed_dim)# 计算p 矩阵pe[0, :, 0::2] = torch.sin(col * row)pe[0, :, 1::2] = torch.cos(col * row)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****#############################################################################                             END OF YOUR CODE                             ############################################################################## Make sure the positional encodings will be saved with the model# parameters (mostly for completeness).self.register_buffer('pe', pe)def forward(self, x):"""Element-wise add positional embeddings to the input sequence.Inputs:- x: the sequence fed to the positional encoder model, of shape(N, S, D), where N is the batch size, S is the sequence length andD is embed dimReturns:- output: the input sequence + positional encodings, of shape (N, S, D)"""N, S, D = x.shape# Create a placeholder, to be overwritten by your code below.output = torch.empty((N, S, D))############################################################################# TODO: Index into your array of positional encodings, and add the         ## appropriate ones to the input sequence. Don't forget to apply dropout    ## afterward. This should only take a few lines of code.                    ############################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****output = x + self.pe[:, :S]output = self.dropout(output)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****#############################################################################                             END OF YOUR CODE                             #############################################################################return output

输出

transformer.forward

题面

解析

如果是正确理解了transformer模型的话，应该不会在这里卡壳，建议多看看相应的课程和论文讲解，我在代码中也写上了详尽的注释

代码

    def forward(self, features, captions):"""Given image features and caption tokens, return a distribution over thepossible tokens for each timestep. Note that since the entire sequenceof captions is provided all at once, we mask out future timesteps.Inputs:- features: image features, of shape (N, D)- captions: ground truth captions, of shape (N, T)Returns:- scores: score for each token at each timestep, of shape (N, T, V)"""N, T = captions.shape# Create a placeholder, to be overwritten by your code below.scores = torch.empty((N, T, self.vocab_size))############################################################################# TODO: Implement the forward function for CaptionTransformer.             ## A few hints:                                                             ##  1) You first have to embed your caption and add positional              ##     encoding. You then have to project the image features into the same  ##     dimensions.                                                          ##  2) You have to prepare a mask (tgt_mask) for masking out the future     ##     timesteps in captions. torch.tril() function might help in preparing ##     this mask.                                                           ##  3) Finally, apply the decoder features on the text & image embeddings   ##     along with the tgt_mask. Project the output to scores per token      ############################################################################## *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****# 第一步，嵌入说明和添加位置编码# captions: (N, T) -> (N, T, W)captions = self.embedding(captions)captions = self.positional_encoding(captions)# 第二步，将图像特征投影到相同的维度# features: (N, D) -> (N,W) -> (N, T, W)features_projected = self.visual_projection(features).unsqueeze(1)# 第三步，准备一个掩码（tgt_mask）来屏蔽说明中的未来时间步骤tgt_mask = torch.tril(torch.ones((T, T), dtype=torch.bool))# 第四部，将解码器特征应用于文本和图像嵌入以及tgt_mask# scores: (N, T, W) -> (N, T, V)scores = self.transformer(captions, features_projected, tgt_mask=tgt_mask)scores = self.output(scores)# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****#############################################################################                             END OF YOUR CODE                             #############################################################################return scores

输出