文本分类与TorchText

译者：PengboLiu

校验：PengboLiu

本教程将说明如何在torchtext中使用文本分类数据集，其中包括：

  1. - AG_NEWS,
  2. - SogouNews,
  3. - DBpedia,
  4. - YelpReviewPolarity,
  5. - YelpReviewFull,
  6. - YahooAnswers,
  7. - AmazonReviewPolarity,
  8. - AmazonReviewFull

下面的例子显示如何使用TextClassification中的数据集训练有监督的文本分类模型。

以ngrams的方式加载数据

使用ngrams可以捕捉句子中有关单词顺序的一些信息。实际上，应用二元语法或三元语法作为单词组可以比仅一个单词组成句子提供更多的好处。一个例子：

  1. "load data with ngrams"
  2. Bi-grams results: "load data", "data with", "with ngrams"
  3. Tri-grams results: "load data with", "data with ngrams"

TextClassification数据集支持n元语法方法。如果NGRAMS 设置为2，数据集中的示例文本将是单个单词加上二元语法字符串的列表。

  1. import torch
  2. import torchtext
  3. from torchtext.datasets import text_classification
  4. NGRAMS = 2
  5. import os
  6. if not os.path.isdir('./.data'):
  7.     os.mkdir('./.data')
  8. train_dataset, test_dataset = text_classification.DATASETS['AG_NEWS'](
  9.     root='./.data', ngrams=NGRAMS, vocab=None)
  10. BATCH_SIZE = 16
  11. device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

定义模型

该模型是由 EmbeddingBag层和线性层(参见下图）组成的。

nn.EmbeddingBag计算“袋”中embeddings的平均值。此处的文本具有不同的长度。nn.EmbeddingBag不需要填充，因为文本长度会以偏移量保存。

此外，由于nn.EmbeddingBag会动态累积嵌入中的平均值，因此nn.EmbeddingBag可以提高性能和存储效率，以处理张量序列。

text_sentiment_ngrams_model

  1. import torch.nn as nn
  2. import torch.nn.functional as F
  3. class TextSentiment(nn.Module):
  4.     def __init__(self, vocab_size, embed_dim, num_class):
  5.         super().__init__()
  6.         self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)
  7.         self.fc = nn.Linear(embed_dim, num_class)
  8.         self.init_weights()
  10.     def init_weights(self):
  11.         initrange = 0.5
  12.         self.embedding.weight.data.uniform_(-initrange, initrange)
  13.         self.fc.weight.data.uniform_(-initrange, initrange)
  14.         self.fc.bias.data.zero_()
  16.     def forward(self, text, offsets):
  17.         embedded = self.embedding(text, offsets)
  18.         return self.fc(embedded)

初始化一个实例

AG_NEWS数据集有四个标签，也就是一共有四类。分别如下：

  1. 1 : World
  2. 2 : Sports
  3. 3 : Business
  4. 4 : Sci/Tec

VOCAB_SIZE等于词汇表的长度(包括单个单词和ngrams）。NUN_CLASS等于标签的数量，在AG_NEWS情况下为4。

  1. VOCAB_SIZE = len(train_dataset.get_vocab())
  2. EMBED_DIM = 32
  3. NUN_CLASS = len(train_dataset.get_labels())
  4. model = TextSentiment(VOCAB_SIZE, EMBED_DIM, NUN_CLASS).to(device)

生成批数据的方法

由于文本条目的长度不同，因此使用自定义函数generate_batch(）被用于生成批量数据和偏移量。该函数在torch.utils.data.DataLoader中传递给collate_fn在`。collate_fn的输入是batch_size大小的张量列表，collat​​e_fn函数把它们打包成一个小规模的批处理(mini-batch）。请注意将collate_fn`在程序顶层声明。 这样可以确保该功能在每个程序中均可用。

原始数据批处理输入中的文本条目打包到一个列表中，并作为单个张量串联在一起，作为nn.EmbeddingBag的输入。 偏移量是分解符的张量，表示文本张量中各个序列的起始索引。 Label是一个保存单个文本条目的标签张量。

  1. def generate_batch(batch):
  2.     label = torch.tensor([entry[0] for entry in batch])
  3.     text = [entry[1] for entry in batch]
  4.     offsets = [0] + [len(entry) for entry in text]
  5.     # torch.Tensor.cumsum returns the cumulative sum
  6.     # of elements in the dimension dim.
  7.     # torch.Tensor([1.0, 2.0, 3.0]).cumsum(dim=0)
  9.     offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)
  10.     text = torch.cat(text)
  11.     return text, offsets, label

定义功能训练模型和评估结果。

建议PyTorch用户使用torch.utils.data.DataLoader，它使数据并行加载变得容易(此处有一个教程）。 我们在这里使用DataLoader加载AG_NEWS数据集并将其发送到模型以进行训练/验证。

  1. from torch.utils.data import DataLoader
  3. def train_func(sub_train_):
  5.     # Train the model
  6.     train_loss = 0
  7.     train_acc = 0
  8.     data = DataLoader(sub_train_, batch_size=BATCH_SIZE, shuffle=True,
  9.                       collate_fn=generate_batch)
  10.     for i, (text, offsets, cls) in enumerate(data):
  11.         optimizer.zero_grad()
  12.         text, offsets, cls = text.to(device), offsets.to(device), cls.to(device)
  13.         output = model(text, offsets)
  14.         loss = criterion(output, cls)
  15.         train_loss += loss.item()
  16.         loss.backward()
  17.         optimizer.step()
  18.         train_acc += (output.argmax(1) == cls).sum().item()
  20.     # Adjust the learning rate
  21.     scheduler.step()
  23.     return train_loss / len(sub_train_), train_acc / len(sub_train_)
  25. def test(data_):
  26.     loss = 0
  27.     acc = 0
  28.     data = DataLoader(data_, batch_size=BATCH_SIZE, collate_fn=generate_batch)
  29.     for text, offsets, cls in data:
  30.         text, offsets, cls = text.to(device), offsets.to(device), cls.to(device)
  31.         with torch.no_grad():
  32.             output = model(text, offsets)
  33.             loss = criterion(output, cls)
  34.             loss += loss.item()
  35.             acc += (output.argmax(1) == cls).sum().item()
  37.     return loss / len(data_), acc / len(data_)

拆分数据集和运行模型

由于原始AG_NEWS数据没有验证集，我们用的0.95(训练集）和0.05(验证集）的比例分割训练数据集。在这里，我们使用PyTorch核心库中的torch.utils.data.dataset.random_split函数。

CrossEntropyLoss将nn.LogSoftmax(）和nn.NLLLoss(）合并到一个类中。在训练多分类模型时很有用。 SGD将随机梯度下降方法用作优化程序。 初始学习率设置为4.0。 StepLR用于每轮调整学习率。

  1. import time
  2. from torch.utils.data.dataset import random_split
  3. N_EPOCHS = 5
  4. min_valid_loss = float('inf')
  6. criterion = torch.nn.CrossEntropyLoss().to(device)
  7. optimizer = torch.optim.SGD(model.parameters(), lr=4.0)
  8. scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)
  10. train_len = int(len(train_dataset) * 0.95)
  11. sub_train_, sub_valid_ = \
  12.     random_split(train_dataset, [train_len, len(train_dataset) - train_len])
  14. for epoch in range(N_EPOCHS):
  16.     start_time = time.time()
  17.     train_loss, train_acc = train_func(sub_train_)
  18.     valid_loss, valid_acc = test(sub_valid_)
  20.     secs = int(time.time() - start_time)
  21.     mins = secs / 60
  22.     secs = secs % 60
  24.     print('Epoch: %d' %(epoch + 1), " | time in %d minutes, %d seconds" %(mins, secs))
  25.     print(f'\tLoss: {train_loss:.4f}(train)\t|\tAcc: {train_acc * 100:.1f}%(train)')
  26.     print(f'\tLoss: {valid_loss:.4f}(valid)\t|\tAcc: {valid_acc * 100:.1f}%(valid)')

输出：

  1. Epoch: 1  | time in 0 minutes, 8 seconds
  2.         Loss: 0.0261(train)     |       Acc: 84.8%(train)
  3.         Loss: 0.0000(valid)     |       Acc: 90.4%(valid)
  4. Epoch: 2  | time in 0 minutes, 8 seconds
  5.         Loss: 0.0120(train)     |       Acc: 93.5%(train)
  6.         Loss: 0.0000(valid)     |       Acc: 91.2%(valid)
  7. Epoch: 3  | time in 0 minutes, 8 seconds
  8.         Loss: 0.0070(train)     |       Acc: 96.4%(train)
  9.         Loss: 0.0000(valid)     |       Acc: 90.8%(valid)
  10. Epoch: 4  | time in 0 minutes, 8 seconds
  11.         Loss: 0.0039(train)     |       Acc: 98.1%(train)
  12.         Loss: 0.0001(valid)     |       Acc: 91.0%(valid)
  13. Epoch: 5  | time in 0 minutes, 8 seconds
  14.         Loss: 0.0023(train)     |       Acc: 99.0%(train)
  15.         Loss: 0.0001(valid)     |       Acc: 90.9%(valid)

运行在GPU上的相关信息：

Epoch：1 |时间为0分钟，11秒

​
​ Loss: 0.0263(train) | Acc: 84.5%(train) ​ Loss: 0.0001(valid) | Acc: 89.0%(valid)

Epoch：2 |时间0分钟，10秒

​
​ Loss: 0.0119(train) | Acc: 93.6%(train) ​ Loss: 0.0000(valid) | Acc: 89.6%(valid)

Epoch：3 |时间0分钟，9秒

​
​ Loss: 0.0069(train) | Acc: 96.4%(train) ​ Loss: 0.0000(valid) | Acc: 90.5%(valid)

Epoch：4 |时间为0分钟，11秒

​
​ Loss: 0.0038(train) | Acc: 98.2%(train) ​ Loss: 0.0000(valid) | Acc: 90.4%(valid)

Epoch：5 |时间为0分钟，11秒

​
​ Loss: 0.0022(train) | Acc: 99.0%(train) ​ Loss: 0.0000(valid) | Acc: 91.0%(valid)

使用测试数据集评估模型

  1. print('Checking the results of test dataset...')
  2. test_loss, test_acc = test(test_dataset)
  3. print(f'\tLoss: {test_loss:.4f}(test)\t|\tAcc: {test_acc * 100:.1f}%(test)')

Out:

  1. Checking the results of test dataset...
  2.         Loss: 0.0002(test)      |       Acc: 89.3%(test)

检查测试数据集的结果…

  1. Loss: 0.0237(test)      |       Acc: 90.5%(test)

在随机新闻上测试

使用到目前为止最好的模型并测试高尔夫新闻。标签信息在这里可用。 import re from torchtext.data.utils import ngrams_iterator from torchtext.data.utils import get_tokenizer

  1. ag_news_label = {1 : "World",
  2.                  2 : "Sports",
  3.                  3 : "Business",
  4.                  4 : "Sci/Tec"}
  6. def predict(text, model, vocab, ngrams):
  7.     tokenizer = get_tokenizer("basic_english")
  8.     with torch.no_grad():
  9.         text = torch.tensor([vocab[token]
  10.                             for token in ngrams_iterator(tokenizer(text), ngrams)])
  11.         output = model(text, torch.tensor([0]))
  12.         return output.argmax(1).item() + 1
  14. ex_text_str = "MEMPHIS, Tenn. – Four days ago, Jon Rahm was \
  15.     enduring the season's worst weather conditions on Sunday at The \
  16.     Open on his way to a closing 75 at Royal Portrush, which \
  17.     considering the wind and the rain was a respectable showing. \
  18.     Thursday's first round at the WGC-FedEx St. Jude Invitational \
  19.     was another story. With temperatures in the mid-80s and hardly any \
  20.     wind, the Spaniard was 13 strokes better in a flawless round. \
  21.     Thanks to his best putting performance on the PGA Tour, Rahm \
  22.     finished with an 8-under 62 for a three-stroke lead, which \
  23.     was even more impressive considering he'd never played the \
  24.     front nine at TPC Southwind."
  26. vocab = train_dataset.get_vocab()
  27. model = model.to("cpu")
  29. print("This is a %s news" %ag_news_label[predict(ex_text_str, model, vocab, 2)])

Out:

  1. This is a Sports news

这是一个体育新闻

你可以在此处找到对应的代码示例。

脚本的总运行时间： (1分钟26.424秒）