我的开源
OpenNMT-py初体验
背景
OpenNMT是一个开源的机器翻译系统。
OpenNMT有两个实现，OpenNMT-py和OpenNMT-tf，两者略有差异。
本文以OpenNMT-py为例，来体验。
初体验
安装
pip install OpenNMT-py
快速入门
数据准备
wget https://s3.amazonaws.com/opennmt-trainingdata/toy-ende.tar.gz tar xvf toy-ende.tar.gz cd toy-ende
数据预处理
$ onmt_preprocess -train_src toy-ende/src-train.txt -train_tgt toy-ende/tgt-train.txt -valid_src toy-ende/src-val.txt -valid_tgt toy-ende/tgt-val.txt -save_data toy-ende/demo [2020-10-08 14:06:44,423 INFO] Extracting features... [2020-10-08 14:06:44,425 INFO] * number of source features: 0. [2020-10-08 14:06:44,425 INFO] * number of target features: 0. [2020-10-08 14:06:44,425 INFO] Building `Fields` object... [2020-10-08 14:06:44,425 INFO] Building & saving training data... [2020-10-08 14:06:44,479 INFO] Building shard 0. [2020-10-08 14:06:44,998 INFO] * saving 0th train data shard to toy-ende/demo.train.0.pt. [2020-10-08 14:06:45,690 INFO] * tgt vocab size: 35820. [2020-10-08 14:06:45,735 INFO] * src vocab size: 24997. [2020-10-08 14:06:45,940 INFO] Building & saving validation data... [2020-10-08 14:06:46,187 INFO] Building shard 0. [2020-10-08 14:06:46,267 INFO] * saving 0th valid data shard to toy-ende/demo.valid.0.pt.
结果文件：
demo.train.pt: serialized PyTorch file containing training datademo.valid.pt: serialized PyTorch file containing validation datademo.vocab.pt: serialized PyTorch file containing vocabulary data
模型训练
$ onmt_train -data toy-ende/demo -save_model demo-model [2020-10-08 14:09:02,315 INFO] * src vocab size = 24997 [2020-10-08 14:09:02,315 INFO] * tgt vocab size = 35820 [2020-10-08 14:09:02,315 INFO] Building model... [2020-10-08 14:09:03,134 INFO] NMTModel( (encoder): RNNEncoder( (embeddings): Embeddings( (make_embedding): Sequential( (emb_luts): Elementwise( (0): Embedding(24997, 500, padding_idx=1) ) ) ) (rnn): LSTM(500, 500, num_layers=2, dropout=0.3) ) (decoder): InputFeedRNNDecoder( (embeddings): Embeddings( (make_embedding): Sequential( (emb_luts): Elementwise( (0): Embedding(35820, 500, padding_idx=1) ) ) ) (dropout): Dropout(p=0.3, inplace=False) (rnn): StackedLSTM( (dropout): Dropout(p=0.3, inplace=False) (layers): ModuleList( (0): LSTMCell(1000, 500) (1): LSTMCell(500, 500) ) ) (attn): GlobalAttention( (linear_in): Linear(in_features=500, out_features=500, bias=False) (linear_out): Linear(in_features=1000, out_features=500, bias=False) ) ) (generator): Sequential( (0): Linear(in_features=500, out_features=35820, bias=True) (1): Cast() (2): LogSoftmax(dim=-1) ) ) [2020-10-08 14:09:03,135 INFO] encoder: 16506500 [2020-10-08 14:09:03,135 INFO] decoder: 41613820 [2020-10-08 14:09:03,135 INFO] * number of parameters: 58120320 [2020-10-08 14:09:03,137 INFO] Starting training on CPU, could be very slow [2020-10-08 14:09:03,137 INFO] Start training loop and validate every 10000 steps... [2020-10-08 14:09:03,137 INFO] Loading dataset from toy-ende/demo.train.0.pt [2020-10-08 14:09:03,550 INFO] number of examples: 10000 [2020-10-08 14:12:35,629 INFO] Step 50/100000; acc: 3.66; ppl: 420079.26; xent: 12.95; lr: 1.00000; 353/348 tok/s; 212 sec [2020-10-08 14:16:30,224 INFO] Step 100/100000; acc: 3.44; ppl: 157620.36; xent: 11.97; lr: 1.00000; 308/308 tok/s; 447 sec [2020-10-08 14:20:07,964 INFO] Step 150/100000; acc: 4.09; ppl: 13433.41; xent: 9.51; lr: 1.00000; 319/320 tok/s; 665 sec [2020-10-08 14:20:35,304 INFO] Loading dataset from toy-ende/demo.train.0.pt [2020-10-08 14:20:35,793 INFO] number of examples: 10000 [2020-10-08 14:23:49,601 INFO] Step 200/100000; acc: 5.59; ppl: 3317.31; xent: 8.11; lr: 1.00000; 326/323 tok/s; 886 sec [2020-10-08 14:27:48,557 INFO] Step 250/100000; acc: 7.78; ppl: 2303.69; xent: 7.74; lr: 1.00000; 304/301 tok/s; 1125 sec [2020-10-08 14:31:36,504 INFO] Step 300/100000; acc: 9.12; ppl: 1930.14; xent: 7.57; lr: 1.00000; 325/324 tok/s; 1353 sec [2020-10-08 14:32:22,094 INFO] Loading dataset from toy-ende/demo.train.0.pt ……………………
太慢了，换GPU来跑。
$ onmt_train -data toy-ende/demo -save_model demo-model -gpu_ranks 0 [2020-10-09 18:36:10,411 INFO] * src vocab size = 24997 [2020-10-09 18:36:10,411 INFO] * tgt vocab size = 35820 [2020-10-09 18:36:10,411 INFO] Building model... [2020-10-09 18:36:14,786 INFO] NMTModel( (encoder): RNNEncoder( (embeddings): Embeddings( (make_embedding): Sequential( (emb_luts): Elementwise( (0): Embedding(24997, 500, padding_idx=1) ) ) ) (rnn): LSTM(500, 500, num_layers=2, dropout=0.3) ) (decoder): InputFeedRNNDecoder( (embeddings): Embeddings( (make_embedding): Sequential( (emb_luts): Elementwise( (0): Embedding(35820, 500, padding_idx=1) ) ) ) (dropout): Dropout(p=0.3, inplace=False) (rnn): StackedLSTM( (dropout): Dropout(p=0.3, inplace=False) (layers): ModuleList( (0): LSTMCell(1000, 500) (1): LSTMCell(500, 500) ) ) (attn): GlobalAttention( (linear_in): Linear(in_features=500, out_features=500, bias=False) (linear_out): Linear(in_features=1000, out_features=500, bias=False) ) ) (generator): Sequential( (0): Linear(in_features=500, out_features=35820, bias=True) (1): Cast() (2): LogSoftmax(dim=-1) ) ) [2020-10-09 18:36:14,787 INFO] encoder: 16506500 [2020-10-09 18:36:14,787 INFO] decoder: 41613820 [2020-10-09 18:36:14,787 INFO] * number of parameters: 58120320 [2020-10-09 18:36:14,788 INFO] Starting training on GPU: [0] [2020-10-09 18:36:14,788 INFO] Start training loop and validate every 10000 steps... [2020-10-09 18:36:14,788 INFO] Loading dataset from toy-ende/demo.train.0.pt [2020-10-09 18:36:15,097 INFO] number of examples: 10000 [2020-10-09 18:36:21,358 INFO] Step 50/100000; acc: 3.77; ppl: 137786.73; xent: 11.83; lr: 1.00000; 11116/11124 tok/s; 7 sec [2020-10-09 18:36:26,867 INFO] Step 100/100000; acc: 4.36; ppl: 24889.16; xent: 10.12; lr: 1.00000; 12596/12291 tok/s; 12 sec [2020-10-09 18:36:32,897 INFO] Step 150/100000; acc: 6.23; ppl: 6784.35; xent: 8.82; lr: 1.00000; 12054/12176 tok/s; 18 sec [2020-10-09 18:36:33,696 INFO] Loading dataset from toy-ende/demo.train.0.pt [2020-10-09 18:36:33,946 INFO] number of examples: 10000 [2020-10-09 18:36:39,280 INFO] Step 200/100000; acc: 7.56; ppl: 3920.44; xent: 8.27; lr: 1.00000; 11555/11444 tok/s; 24 sec [2020-10-09 18:36:44,861 INFO] Step 250/100000; acc: 8.96; ppl: 2168.38; xent: 7.68; lr: 1.00000; 12336/12185 tok/s; 30 sec [2020-10-09 18:36:50,805 INFO] Step 300/100000; acc: 9.50; ppl: 1994.53; xent: 7.60; lr: 1.00000; 12396/12425 tok/s; 36 sec [2020-10-09 18:36:52,359 INFO] Loading dataset from toy-ende/demo.train.0.pt [2020-10-09 18:36:52,594 INFO] number of examples: 10000 [2020-10-09 18:36:57,142 INFO] Step 350/100000; acc: 9.38; ppl: 1667.39; xent: 7.42; lr: 1.00000; 11536/11542 tok/s; 42 sec [2020-10-09 18:37:02,689 INFO] Step 400/100000; acc: 10.08; ppl: 1478.09; xent: 7.30; lr: 1.00000; 12475/12237 tok/s; 48 sec [2020-10-09 18:37:08,556 INFO] Step 450/100000; acc: 10.88; ppl: 1303.90; xent: 7.17; lr: 1.00000; 12440/12504 tok/s; 54 sec [2020-10-09 18:37:10,973 INFO] Loading dataset from toy-ende/demo.train.0.pt ………… [2020-10-09 21:54:56,217 INFO] Step 99700/100000; acc: 8.97; ppl: 19685.48; xent: 9.89; lr: 0.03125; 11664/11638 tok/s; 11921 sec [2020-10-09 21:55:02,111 INFO] Step 99750/100000; acc: 9.26; ppl: 18087.62; xent: 9.80; lr: 0.03125; 12328/12309 tok/s; 11927 sec [2020-10-09 21:55:07,559 INFO] Step 99800/100000; acc: 9.89; ppl: 15161.89; xent: 9.63; lr: 0.03125; 12524/12399 tok/s; 11933 sec [2020-10-09 21:55:13,534 INFO] Step 99850/100000; acc: 8.78; ppl: 19494.02; xent: 9.88; lr: 0.03125; 12469/12484 tok/s; 11939 sec [2020-10-09 21:55:13,674 INFO] Loading dataset from toy-ende/demo.train.0.pt [2020-10-09 21:55:13,938 INFO] number of examples: 10000 [2020-10-09 21:55:19,731 INFO] Step 99900/100000; acc: 9.34; ppl: 18415.28; xent: 9.82; lr: 0.03125; 11610/11664 tok/s; 11945 sec [2020-10-09 21:55:25,100 INFO] Step 99950/100000; acc: 10.04; ppl: 14772.09; xent: 9.60; lr: 0.03125; 12923/12557 tok/s; 11950 sec [2020-10-09 21:55:31,169 INFO] Step 100000/100000; acc: 8.85; ppl: 19937.46; xent: 9.90; lr: 0.01562; 12166/12255 tok/s; 11956 sec [2020-10-09 21:55:31,169 INFO] Loading dataset from toy-ende/demo.valid.0.pt [2020-10-09 21:55:31,363 INFO] number of examples: 3000 [2020-10-09 21:55:38,832 INFO] Validation perplexity: 31809.2 [2020-10-09 21:55:38,832 INFO] Validation accuracy: 9.18222 [2020-10-09 21:55:39,031 INFO] Saving checkpoint demo-model_step_100000.pt
模型推理
$ onmt_translate -model demo-model_step_100000.pt -src toy-ende/src-test.txt -output pred.txt -replace_unk -verbose ………… [2020-10-09 22:11:54,972 INFO] SENT 416: ['Even', 'the', 'most', 'recent', 'statements', 'from', 'Snowden', 'will', 'not', 'change', 'this', 'fact', '.'] PRED 416: Das hat mein Ziel . PRED SCORE: -7.2980 [2020-10-09 22:18:39,381 INFO] Translating shard 0. [2020-10-09 22:19:17,179 INFO] PRED AVG SCORE: -1.3478, PRED PPL: 3.8491
speech2text
数据准备
wget -O data/speech.tgz http://lstm.seas.harvard.edu/latex/speech.tgz; tar zxf data/speech.tgz -C data/
数据预处理
$ onmt_preprocess -data_type audio -src_dir data/speech/an4_dataset -train_src data/speech/src-train.txt -train_tgt data/speech/tgt-train.txt -valid_src data/speech/src-val.txt -valid_tgt data/speech/tgt-val.txt -shard_size 300 -save_data data/speech/demo [2020-10-08 16:46:42,654 INFO] Extracting features... [2020-10-08 16:46:42,655 INFO] * number of source features: 0. [2020-10-08 16:46:42,655 INFO] * number of target features: 0. [2020-10-08 16:46:42,655 INFO] Building `Fields` object... [2020-10-08 16:46:42,655 INFO] Building & saving training data... …………………… [2020-10-08 16:46:44,859 INFO] * saving 0th valid data shard to data/speech/demo.valid.0.pt.
模型训练
原文中用了4层encoder，本示例中简化成了2层。
onmt_train -model_type audio -enc_rnn_size 512 -dec_rnn_size 512 -audio_enc_pooling 1,1 -dropout 0 -enc_layers 2 -dec_layers 1 -rnn_type LSTM -data data/speech/demo -save_model demo-model -global_attention mlp -gpu_ranks 0 -batch_size 8 -optim adam -max_grad_norm 100 -learning_rate 0.0003 -learning_rate_decay 0.8 -train_steps 100000 --save_checkpoint_steps 2000
输出
[2020-10-09 23:57:46,056 INFO] * tgt vocab size = 31 [2020-10-09 23:57:46,056 INFO] Building model... [2020-10-09 23:57:49,392 INFO] NMTModel( (encoder): AudioEncoder( (W): Linear(in_features=512, out_features=512, bias=False) (batchnorm_0): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (rnn_0): LSTM(161, 512) (pool_0): MaxPool1d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False) (rnn_1): LSTM(512, 512) (pool_1): MaxPool1d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False) (batchnorm_1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (decoder): InputFeedRNNDecoder( (embeddings): Embeddings( (make_embedding): Sequential( (emb_luts): Elementwise( (0): Embedding(31, 500, padding_idx=1) ) ) ) (dropout): Dropout(p=0.0, inplace=False) (rnn): StackedLSTM( (dropout): Dropout(p=0.0, inplace=False) (layers): ModuleList( (0): LSTMCell(1012, 512) ) ) (attn): GlobalAttention( (linear_context): Linear(in_features=512, out_features=512, bias=False) (linear_query): Linear(in_features=512, out_features=512, bias=True) (v): Linear(in_features=512, out_features=1, bias=False) (linear_out): Linear(in_features=1024, out_features=512, bias=True) ) ) (generator): Sequential( (0): Linear(in_features=512, out_features=31, bias=True) (1): Cast() (2): LogSoftmax(dim=-1) ) ) [2020-10-09 23:57:49,393 INFO] encoder: 3747840 [2020-10-09 23:57:49,393 INFO] decoder: 4206763 [2020-10-09 23:57:49,393 INFO] * number of parameters: 7954603 [2020-10-09 23:57:49,394 INFO] Starting training on GPU: [0] [2020-10-09 23:57:49,395 INFO] Start training loop and validate every 10000 steps... [2020-10-09 23:57:49,395 INFO] Loading dataset from data/speech/demo.train.0.pt [2020-10-09 23:57:49,455 INFO] number of examples: 300 [2020-10-09 23:57:52,321 INFO] Loading dataset from data/speech/demo.train.1.pt [2020-10-09 23:57:52,338 INFO] number of examples: 300 [2020-10-09 23:57:53,662 INFO] Step 50/100000; acc: 22.02; ppl: 13.81; xent: 2.63; lr: 0.00030; 93/1318 tok/s; 4 sec [2020-10-09 23:57:56,415 INFO] Loading dataset from data/speech/demo.train.2.pt [2020-10-09 23:57:56,434 INFO] number of examples: 296 [2020-10-09 23:57:59,335 INFO] Step 100/100000; acc: 43.42; ppl: 6.86; xent: 1.93; lr: 0.00030; 70/1602 tok/s; 10 sec [2020-10-09 23:58:00,811 INFO] Loading dataset from data/speech/demo.train.3.pt [2020-10-09 23:58:00,814 INFO] number of examples: 47 [2020-10-09 23:58:01,619 INFO] Loading dataset from data/speech/demo.train.0.pt [2020-10-09 23:58:01,636 INFO] number of examples: 300 [2020-10-09 23:58:03,953 INFO] Step 150/100000; acc: 54.66; ppl: 4.50; xent: 1.50; lr: 0.00030; 86/1478 tok/s; 15 sec …………………… [2020-10-10 02:41:57,632 INFO] Loading dataset from data/speech/demo.train.1.pt [2020-10-10 02:41:57,647 INFO] number of examples: 300 [2020-10-10 02:41:57,865 INFO] Step 100000/100000; acc: 94.06; ppl: 1.19; xent: 0.18; lr: 0.00008; 92/1411 tok/s; 9558 sec [2020-10-10 02:41:57,865 INFO] Loading dataset from data/speech/demo.valid.0.pt [2020-10-10 02:41:57,871 INFO] number of examples: 130 [2020-10-10 02:41:58,077 INFO] Validation perplexity: 68.1276 [2020-10-10 02:41:58,077 INFO] Validation accuracy: 56.2152 [2020-10-10 02:41:58,078 INFO] Saving checkpoint demo-model_step_100000.pt
模型推理
onmt_translate -data_type audio -model demo-model_step_100000.pt -src_dir data/speech/an4_dataset -src data/speech/src-val.txt -output pred.txt -gpu 0 -verbose
结果示例
[2020-10-10 00:07:11,682 INFO] SENT 130: None PRED 130: O N E <space> F I V E <space> T W O <space> O N E <space> S E V E N PRED SCORE: -4.5617 [2020-10-10 00:07:11,682 INFO] PRED AVG SCORE: -0.2203, PRED PPL: 1.2464
很明显，这个阶段的模型效果还是非常差的。