前言：

• 利用torchtext类来处理一个著名的数据集，包含了一些英文和德文句子。利用该数据处理sequence-to-sequence模型，通过注意力机制，可以将德语翻译成英语。
• Torchtext：它是 PyTorch 生态系统中的一个库，主要用于自然语言处理（NLP）相关的任务。它提供了一系列便捷的工具和功能来处理文本数据，使得在进行 NLP 项目开发时更加高效。
• 创建可轻松迭代的数据集：在训练语言模型（这里特指语言翻译模型）时，需要大量的文本数据作为训练素材。这些数据要以一种合适的格式组织起来，以便模型能够逐批次地读取和处理。Torchtext 提供的工具可以帮助我们把原始的文本数据（比如来自不同语言的句子对，用于翻译任务）整理成这样一种可以方便地按批次依次访问的数据集形式，就像在一个循环中能够轻松地逐个获取数据样本进行后续处理一样。
• Spacy 被选用是因为它在对除英语之外的其他众多语言进行分词处理时，能够给予非常强有力的支持。虽然 torchtext 这个工具也提供了像 basic_english 标记器这样专门针对英语的标记器，而且还支持其他一些英语标记器（比如 Moses 标记器等），但是当涉及到语言翻译任务的时候，往往需要处理多种不同的语言，而在这种多种语言并存的情况下，Spacy 就成了最佳的选择，它能更好地应对不同语言的分词需求，以便为后续的语言翻译相关工作打好基础

       本教程介绍了如何使用torchtext预处理包含英语和德语句子的著名数据集的数据，并使用它来训练序列到序列模型，并能将德语句子翻译成英语

Field and TranslationDataset

        torchtext可用来实现语言翻译模型。一个主要的类是 Field, 其可以指定每个句子如何被处理。另一个是TranslationDataset 。torchtext有许多这样的数据集。在本文，我们利用 Multi30k dataset,其中包含了30000句子（平均长约13个单词）同时有英语和德语。

注意： 本教程中的标记化（tokenization）依赖Spacy使用Spacy是因为其提供了除了英语外的强有力的标记化支持。torchtext提供了basic_english标记化及其他英语标记化支持（如，Moses）,但是对于语言翻译——需要多种语言，Spacy是最佳选择。

为了运行代码需要安装利用pip或者conda安装sapcy

pip install spacy

然后下载英语和德语 Spacy 分词器的原始数据：

python -m spacy download en
python -m spacy download de

安装了Spacy之后，随后的代码将基于Field中定义的标记器（tokenizer）将TranslationDataset中的每个句子进行标记化。

from torchtext.datasets import Multi30k
from torchtext.data import Field, BucketIteratorSRC = Field(tokenize = "spacy",tokenizer_language="de",init_token = '<sos>',eos_token = '<eos>',lower = True)TRG = Field(tokenize = "spacy",tokenizer_language="en",init_token = '<sos>',eos_token = '<eos>',lower = True)train_data, valid_data, test_data = Multi30k.splits(exts = ('.de', '.en'),fields = (SRC, TRG))

        定义train_data后，可以看到torchtext的Field的作用：build_vocab方法允许我们创建和每种语言相关的字典

SRC.build_vocab(train_data, min_freq = 2)
TRG.build_vocab(train_data, min_freq = 2)

运行这些后，SRC.vocab.stoi将会是一个字典：tokens为键，对应的indices为值。SRC.vocab.itos是同样的字典。

BucketIterator

最后利用BucketIterator，利用TranslationDataset 作为第一个参数。定义一个迭代器，用于将相似长度的示例批处理在一起。在为每个新epoch生产新的洗批量时，最小化所需的填充量。

import torchdevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')BATCH_SIZE = 128train_iterator, valid_iterator, test_iterator = BucketIterator.splits((train_data, valid_data, test_data),batch_size = BATCH_SIZE,device = device)

这些迭代器类似于DataLoader：

for i, batch in enumerate(iterator):

每个批量有两无属性：

src = batch.src
trg = batch.trg

Defining our nn.Module and Optimizer

import random
from typing import Tupleimport torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch import Tensorclass Encoder(nn.Module):def __init__(self,input_dim: int,emb_dim: int,enc_hid_dim: int,dec_hid_dim: int,dropout: float):super().__init__()self.input_dim = input_dimself.emb_dim = emb_dimself.enc_hid_dim = enc_hid_dimself.dec_hid_dim = dec_hid_dimself.dropout = dropoutself.embedding = nn.Embedding(input_dim, emb_dim)self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional = True)self.fc = nn.Linear(enc_hid_dim * 2, dec_hid_dim)self.dropout = nn.Dropout(dropout)def forward(self,src: Tensor) -> Tuple[Tensor]:embedded = self.dropout(self.embedding(src))outputs, hidden = self.rnn(embedded)hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)))return outputs, hiddenclass Attention(nn.Module):def __init__(self,enc_hid_dim: int,dec_hid_dim: int,attn_dim: int):super().__init__()self.enc_hid_dim = enc_hid_dimself.dec_hid_dim = dec_hid_dimself.attn_in = (enc_hid_dim * 2) + dec_hid_dimself.attn = nn.Linear(self.attn_in, attn_dim)def forward(self,decoder_hidden: Tensor,encoder_outputs: Tensor) -> Tensor:src_len = encoder_outputs.shape[0]repeated_decoder_hidden = decoder_hidden.unsqueeze(1).repeat(1, src_len, 1)encoder_outputs = encoder_outputs.permute(1, 0, 2)energy = torch.tanh(self.attn(torch.cat((repeated_decoder_hidden,encoder_outputs),dim = 2)))attention = torch.sum(energy, dim=2)return F.softmax(attention, dim=1)class Decoder(nn.Module):def __init__(self,output_dim: int,emb_dim: int,enc_hid_dim: int,dec_hid_dim: int,dropout: int,attention: nn.Module):super().__init__()self.emb_dim = emb_dimself.enc_hid_dim = enc_hid_dimself.dec_hid_dim = dec_hid_dimself.output_dim = output_dimself.dropout = dropoutself.attention = attentionself.embedding = nn.Embedding(output_dim, emb_dim)self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)self.out = nn.Linear(self.attention.attn_in + emb_dim, output_dim)self.dropout = nn.Dropout(dropout)def _weighted_encoder_rep(self,decoder_hidden: Tensor,encoder_outputs: Tensor) -> Tensor:a = self.attention(decoder_hidden, encoder_outputs)a = a.unsqueeze(1)encoder_outputs = encoder_outputs.permute(1, 0, 2)weighted_encoder_rep = torch.bmm(a, encoder_outputs)weighted_encoder_rep = weighted_encoder_rep.permute(1, 0, 2)return weighted_encoder_repdef forward(self,input: Tensor,decoder_hidden: Tensor,encoder_outputs: Tensor) -> Tuple[Tensor]:input = input.unsqueeze(0)embedded = self.dropout(self.embedding(input))weighted_encoder_rep = self._weighted_encoder_rep(decoder_hidden,encoder_outputs)rnn_input = torch.cat((embedded, weighted_encoder_rep), dim = 2)output, decoder_hidden = self.rnn(rnn_input, decoder_hidden.unsqueeze(0))embedded = embedded.squeeze(0)output = output.squeeze(0)weighted_encoder_rep = weighted_encoder_rep.squeeze(0)output = self.out(torch.cat((output,weighted_encoder_rep,embedded), dim = 1))return output, decoder_hidden.squeeze(0)class Seq2Seq(nn.Module):def __init__(self,encoder: nn.Module,decoder: nn.Module,device: torch.device):super().__init__()self.encoder = encoderself.decoder = decoderself.device = devicedef forward(self,src: Tensor,trg: Tensor,teacher_forcing_ratio: float = 0.5) -> Tensor:batch_size = src.shape[1]max_len = trg.shape[0]trg_vocab_size = self.decoder.output_dimoutputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)encoder_outputs, hidden = self.encoder(src)# first input to the decoder is the <sos> tokenoutput = trg[0,:]for t in range(1, max_len):output, hidden = self.decoder(output, hidden, encoder_outputs)outputs[t] = outputteacher_force = random.random() < teacher_forcing_ratiotop1 = output.max(1)[1]output = (trg[t] if teacher_force else top1)return outputsINPUT_DIM = len(SRC.vocab)
OUTPUT_DIM = len(TRG.vocab)
# ENC_EMB_DIM = 256
# DEC_EMB_DIM = 256
# ENC_HID_DIM = 512
# DEC_HID_DIM = 512
# ATTN_DIM = 64
# ENC_DROPOUT = 0.5
# DEC_DROPOUT = 0.5ENC_EMB_DIM = 32
DEC_EMB_DIM = 32
ENC_HID_DIM = 64
DEC_HID_DIM = 64
ATTN_DIM = 8
ENC_DROPOUT = 0.5
DEC_DROPOUT = 0.5enc = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, ENC_DROPOUT)attn = Attention(ENC_HID_DIM, DEC_HID_DIM, ATTN_DIM)dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, DEC_DROPOUT, attn)model = Seq2Seq(enc, dec, device).to(device)def init_weights(m: nn.Module):for name, param in m.named_parameters():if 'weight' in name:nn.init.normal_(param.data, mean=0, std=0.01)else:nn.init.constant_(param.data, 0)model.apply(init_weights)optimizer = optim.Adam(model.parameters())def count_parameters(model: nn.Module):return sum(p.numel() for p in model.parameters() if p.requires_grad)print(f'The model has {count_parameters(model):,} trainable parameters')

注意：在对语言翻译模型的性能进行评分时，必须告诉nn.CrossEntropyLoss函数忽略目标只是填充的索引。

PAD_IDX = TRG.vocab.stoi['<pad>']criterion = nn.CrossEntropyLoss(ignore_index=PAD_IDX)

最后训练和评估：

import math
import timedef train(model: nn.Module,iterator: BucketIterator,optimizer: optim.Optimizer,criterion: nn.Module,clip: float):model.train()epoch_loss = 0for _, batch in enumerate(iterator):src = batch.srctrg = batch.trgoptimizer.zero_grad()output = model(src, trg)output = output[1:].view(-1, output.shape[-1])trg = trg[1:].view(-1)loss = criterion(output, trg)loss.backward()torch.nn.utils.clip_grad_norm_(model.parameters(), clip)optimizer.step()epoch_loss += loss.item()return epoch_loss / len(iterator)def evaluate(model: nn.Module,iterator: BucketIterator,criterion: nn.Module):model.eval()epoch_loss = 0with torch.no_grad():for _, batch in enumerate(iterator):src = batch.srctrg = batch.trgoutput = model(src, trg, 0) #turn off teacher forcingoutput = output[1:].view(-1, output.shape[-1])trg = trg[1:].view(-1)loss = criterion(output, trg)epoch_loss += loss.item()return epoch_loss / len(iterator)def epoch_time(start_time: int,end_time: int):elapsed_time = end_time - start_timeelapsed_mins = int(elapsed_time / 60)elapsed_secs = int(elapsed_time - (elapsed_mins * 60))return elapsed_mins, elapsed_secsN_EPOCHS = 10
CLIP = 1best_valid_loss = float('inf')for epoch in range(N_EPOCHS):start_time = time.time()train_loss = train(model, train_iterator, optimizer, criterion, CLIP)valid_loss = evaluate(model, valid_iterator, criterion)end_time = time.time()epoch_mins, epoch_secs = epoch_time(start_time, end_time)print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')print(f'\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')print(f'\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')test_loss = evaluate(model, test_iterator, criterion)print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')

全部代码：

import torch# 打印PyTorch版本
print(torch.__version__)
# 根据是否有可用的CUDA设备来选择设备（GPU或CPU）
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(str(device) + ':' + str(torch.cuda.is_available()))from torchtext.utils import download_from_url, extract_archive# 数据集的基础URL
url_base = 'https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/'
# 训练集、验证集、测试集的URL文件名元组
train_urls = ('train.de.gz', 'train.en.gz')
val_urls = ('val.de.gz', 'val.en.gz')
test_urls = ('test_2016_flickr.de.gz', 'test_2016_flickr.en.gz')# 以下三行原先是通过下载和解压获取文件路径，这里注释掉了，替换为自定义路径示例
# train_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in train_urls]
train_filepaths = ['... your path\\.data\\train.de', '... your path\\.data\\train.en']
val_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in val_urls]
test_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in test_urls]from torchtext.data.utils import get_tokenizer# 获取德语和英语的分词器，使用Spacy，指定对应的语言模型
de_tokenizer = get_tokenizer('spacy', language='de_core_news_sm')
en_tokenizer = get_tokenizer('spacy', language='en_core_web_sm')from collections import Counter
from torchtext.vocab import Vocab
import io# 构建词汇表的函数，根据给定文件路径和分词器统计词频并创建词汇表
def build_vocab(filepath, tokenizer):counter = Counter()with io.open(filepath, encoding="utf8") as f:for string_ in f:counter.update(tokenizer(string_))return Vocab(counter, specials=['<unk>', '<pad>', '<bos>', '<eos>'])# 构建德语和英语的词汇表
de_vocab = build_vocab(train_filepaths[0], de_tokenizer)
en_vocab = build_vocab(train_filepaths[1], en_tokenizer)# 数据处理函数，将文本数据转换为张量形式，配对德语和英语句子
def data_process(filepaths):raw_de_iter = iter(io.open(filepaths[0], encoding="utf8"))raw_en_iter = iter(io.open(filepaths[1], encoding="utf8"))data = []for (raw_de, raw_en) in zip(raw_de_iter, raw_en_iter):de_tensor_ = torch.tensor([de_vocab[token] for token in de_tokenizer(raw_de)], dtype=torch.long)en_tensor_ = torch.tensor([en_vocab[token] for token in en_tokenizer(raw_en)], dtype=torch.long)data.append((de_tensor_, en_tensor_))return data# 处理训练集、验证集、测试集数据
train_data = data_process(train_filepaths)
val_data = data_process(val_filepaths)
test_data = data_process(test_filepaths)BATCH_SIZE = 128
PAD_IDX = de_vocab['<pad>']
BOS_IDX = de_vocab['<bos>']
EOS_IDX = de_vocab['<eos>']from torch.nn.utils.rnn import pad_sequence
from torch.utils.data import DataLoader# 生成批次数据的函数，添加起始、结束标记并填充序列
def generate_batch(data_batch):de_batch, en_batch = [], []for (de_item, en_item) in data_batch:de_batch.append(torch.cat([torch.tensor([BOS_IDX]), de_item, torch.tensor([EOS_IDX])], dim=0))en_batch.append(torch.cat([torch.tensor([BOS_IDX]), en_item, torch.tensor([EOS_IDX])], dim=0))de_batch = pad_sequence(de_batch, padding_value=PAD_IDX)en_batch = pad_sequence(en_batch, padding_value=PAD_IDX)return de_batch, en_batch# 创建训练集、验证集、测试集的数据加载器
train_iter = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch)
valid_iter = DataLoader(val_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch)
test_iter = DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch)import torch.nn as nn
from typing import Tuple# 编码器类，对输入进行编码
class Encoder(nn.Module):def __init__(self, input_dim: int, emb_dim: int, enc_hid_dim: int, dec_hid_dim: int, dropout: float):super().__init__()self.input_dim = input_dimself.emb_dim = emb_dimself.enc_hid_dim = enc_hid_dimself.dec_hid_dim = dec_hid_dimself.dropout = dropoutself.embedding = nn.Embedding(input_dim, emb_dim)self.dropout = nn.Dropout(dropout)self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional=True)self.fc = nn.Linear(enc_hid_dim * 2, dec_hid_dim)def forward(self, src: torch.Tensor) -> Tuple[torch.Tensor]:embedded = self.dropout(self.embedding(src))outputs, hidden = self.rnn(embedded)hidden = torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1)hidden = self.fc(hidden)hidden = torch.tanh(hidden)return outputs, hiddenimport torch.nn.functional as F# 注意力机制类，计算注意力权重
class Attention(nn.Module):def __init__(self, enc_hid_dim: int, dec_hid_dim: int, attn_dim: int):super().__init__()self.enc_hid_dim = enc_hid_dimself.dec_hid_dim = dec_hid_dimself.attn_in = (enc_hid_dim * 2) + dec_hid_dimself.attn = nn.Linear(self.attn_in, attn_dim)def forward(self, decoder_hidden, encoder_outputs) -> torch.Tensor:src_len = encoder_outputs.shape[0]repeated_decoder_hidden = decoder_hidden.unsqueeze(1).repeat(1, src_len, 1)encoder_outputs = encoder_outputs.permute(1, 0, 2)energy = torch.tanh(self.attn(torch.cat((repeated_decoder_hidden, encoder_outputs), dim=2)))attention = torch.sum(energy, dirm=2)return F.softmax(attention, dim=1)# 解码器类，根据编码信息和注意力机制生成输出
class Decoder(nn.Module):def __init__(self, output_dim: int, emb_dim: int, enc_hid_dim: int, dec_hid_dim: int, dropout: int, attention: nn.Module):super().__init__()self.emb_dim = emb_dimself.enc_hid_dim = enc_hid_dimself.dec_hid_dim = dec_hid_dimself.output_dim = output_dimself.dropout = dropoutself.attention = attentionself.embedding = nn.Embedding(output_dim, emb_dim)self.dropout = nn.Dropout(dropout)self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)self.out = nn.Linear(self.attention.attn_in + emb_dim, output_dim)def _weighted_encoder_rep(self, decoder_hidden, encoder_outputs) -> torch.Tensor:a = self.attention(decoder_hidden, encoder_outputs)a = a.unsqueeze(1)encoder_outputs = encoder_outputs.permute(1, 0, 2)weighted_encoder_rep = torch.bmm(a, encoder_outputs)weighted_encoder_rep = weighted_encoder_rep.permute(1, 0, 2)return weighted_encoder_repdef forward(self, input: torch.Tensor, decoder_hidden: torch.Tensor, encoder_outputs: torch.Tensor) -> Tuple[torch.Tensor]:input = input.unsqueeze(0)embedded = self.dropout(self.embedding(input))weighted_encoder_rep = self._weighted_encoder_rep(decoder_hidden, encoder_outputs)rnn_input = torch.cat((embedded, weighted_encoder_rep), dim=2)output, decoder_hidden = self.rnn(rnn_input, decoder_hidden.unsqueeze(0))embedded = embedded.squeeze(0)output = output.squeeze(0)weighted_encoder_rep = weighted_encoder_rep.squeeze(0)output = self.out(torch.cat((output, weighted_encoder_rep, embedded), dim=1))return output, decoder_hidden.squeeze(0)import random# 序列到序列模型类，结合编码器和解码器进行翻译
class Seq2Seq(nn.Module):def __init__(self, encoder: nn.Module, decoder: nn.Module, device: torch.device):super().__init__()self.encoder = encoderself.decoder = decoderself.device = devicedef forward(self, src: torch.Tensor, trg: torch.Tensor, teacher_forcing_ratio: float = 0.5) -> torch.Tensor:batch_size = src.shape[1]max_len = trg.shape[0]trg_vocab_size = self.decoder.output_dimoutputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)encoder_outputs, hidden = self.encoder(src)output = trg[0,:]for t in range(1, max_len):output, hidden = self.decoder(output, hidden, encoder_outputs)outputs[t] = outputteacher_force = random.random() < teacher_forcing_ratiotop1 = output.max(1)[1]output = (trg[t] if teacher_force else top1)return outputs# 定义输入、输出维度及各种模型参数
INPUT_DIM = len(de_vocab)
OUTPUT_DIM = len(en_vocab)
ENC_EMB_DIM = 32
DEC_EMB_DIM = 32
ENC_HID_DIM = 64
DEC_HID_DIM = 64
ATTN_DIM = 8
ENC_DROPOUT = 0.5
DEC_DROPOUT = 0.5# 创建编码器、注意力机制、解码器和序列到序列模型实例，并将模型移到指定设备
enc = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, ENC_DROPOUT)
attn = Attention(ENC_HID_DIM, DEC_HID_DIM, ATTN_DIM)
dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, DEC_DROPOUT, attn)
model = Seq2Seq(enc, dec, device).to(device)# 初始化模型权重的函数
def init_weights(m: nn.Module):for name, param in m.named_parameters():if 'weight' in name:nn.init.normal_(param.data, mean=0, std=0.01)else:nn.init.constant_(param.data, 0)model.apply(init_weights)import torch.optim as optim# 创建优化器和损失函数
optimizer = optim.Adam(model.parameters())
criterion = nn.CrossEntropyLoss(ignore_index=en_vocab.stoi['<pad>'])import math
import time# 训练函数，执行一个训练 epoch，更新模型参数
def train(model: nn.Module, iterator: torch.utils.data.DataLoader, optimizer: optim.Optimizer, criterion: nn.Module, clip: float):model.train()epoch_loss = 0for _, (src, trg) in enumerate(iterator):src, trg = src.to(device), trg.to(device)optimizer.zero_grad()output = model(src, trg)output = output[1:].view(-1, output.shape[-1])trg = trg[1:].view(-1)loss = criterion(output, trg)loss.backward()torch.nn.utils.clip_grad_norm_(model.parameters(), clip)optimizer.step()epoch_loss += loss.item()return epoch_loss / len(iterator)# 评估函数，在给定数据集上评估模型性能
def evaluate(model: nn.Module, iterator: torch.utils.data.DataLoader, criterion: nn.Module):model.eval()epoch_loss = 0with torch.no_grad():for _, (src, trg) in enumerate(iterator):src, trg = src.to(device), trg.to(device)output = model(src, trg, 0)output = output[1:].view(-1, output.shape[-1])trg = trg[1:].view(-1)loss = criterion(output, trg)epoch_loss += loss.item()return epoch_loss / len(iterator)# 计算一个 epoch 训练或评估所花费时间的函数
def epoch_time(start_time, end_time):elapsed_time = end_time - start_timeelapsed_mins = int(elapsed_time / 60)elapsed_secs = int(elapsed_time - (elapsed_mins * 60))return elapsed_mins, elapsed_secsif __name__ == '__main__':# 以下部分原先是训练模型并保存，这里注释掉了# N_EPOCHS = 10# CLIP = 1# best_valid_loss = float('inf')# for epoch in range(N_EPOCHS):#     start_time = time.time()#     train_loss = train(model, train_iter, optimizer, criterion, CLIP)#     valid_loss = evaluate(model, valid_iter, criterion)#     end_time = time.time()#     epoch_mins, epoch_secs = epoch_time(start_time, end_time)##     print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')#     print(f'\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')#     print(f'\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')## test_loss = evaluate(model, test_iter, criterion)## print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')## torch.save(model.state_dict(), '... your path\\model_Translate.pth')# 以下为测试部分，加载预训练模型并进行翻译预测model.load_state_dict(torch.load('... your path\\model_Translate.pth'))str = 'Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.'de_tensor = torch.tensor([de_vocab[token] for token in de_tokenizer(str)], dtype=torch.long)de_tensor = torch.cat([torch.tensor([BOS_IDX]), de_tensor, torch.tensor([EOS_IDX])], dim=0)de_tensor = de_tensor.unsqueeze(1).to(device)output = model(de_tensor, torch.zeros(50,1).long().to(device),0)output = output[1:].view(-1, output.shape[-1])result = []for i in range(output.size()[0]):index = output[i].data.topk(1)[1].item()if en_vocab.itos[index] == '<eos>':breakresult.append(en_vocab.itos[index])print(' '.join(result))

        这段代码主要实现了一个基于序列到序列（Seq2Seq）架构的机器翻译模型，包括数据预处理、模型构建（编码器、解码器、注意力机制等）、训练和评估函数以及最后的测试部分，用于将德语句子翻译成英语句子。

Next steps

• Check out the rest of Ben Trevett’s tutorials using torchtext here
• Stay tuned for a tutorial using other torchtext features along with nn.Transformer for language modeling via next word prediction!