[SOLVED] MachineLearning Homework 5-Sequence to Sequence

Sequence to sequence

Generate a sequence from another sequence

Translation ASR TTS
text to text speech to text text to speech

and more…

Sequence to sequence

Often composed of encoder and decoder

• ●  Encoder: encodes input sequence into a vector or sequence of vectors
• ●  Decoder: decodes a sequence one token at a time, based on 1) encoder

  output and 2) previous decoded tokens

1)

2)

Neural Machine Translation

We will translate from english to traditional chinese ● Cats are so cute. -> 貓咪真可愛。

A sentence is usually translated into another language with different length. Naturally, the seq2seq framework is applied on this task.

Training datasets

• ●  Paired data
  • ○  TED2020: TED talks with transcripts translated by a global community of volunteers to

    more than 100 language

  • ○  We will use (en, zh-tw) aligned pairs
• ●  Monolingual data

○ More TED talks in traditional Chinese

Evaluation

BLEU

• ●  Modified1 n-gram precision (n=1~4)
• ●  Brevity penalty: penalizes short hypotheses

source: Cats are so cute. target:貓咪真可愛。 output: 貓好可愛。

○ c is the hypothesis length, r is the reference length
● The BLEU score is the geometric mean of n-gram precision, multiplied by

brevity penalty

1the precision is clamped to # occurence in reference.

Workflow

1. Preprocessing

2. Training

3. Testing

Workflow

1. Preprocessing

1. download raw data
2. clean and normalize
3. remove bad data (too long/short)
4. tokenization

2. Training

1. initialize a model
2. train it with training data

3. Testing

1. generate translation of test data
2. evaluate the performance

Training tips

• ●  Tokenize data with sub-word units
• ●  Label smoothing regularization
• ●  Learning rate scheduling ● Back-translation

Training tips

● Tokenize data with sub-word units

• ○  For one, we can reduce the vocabulary size (common prefix/suffix)
• ○  For another, alleviate the open vocabulary problem ○ example
  • ▁new ▁ways ▁of ▁making ▁electric ▁trans port ation ▁.
  • new ways of making electric transportation.

Training tips

● Label smoothing regularization

• ○  When calculating loss, reserve some probability for incorrect labels
• ○  Avoids overfitting

Training tips

● Learning rate scheduling

• ○  Linearly increase lr and then decay by inverse square root of steps
• ○  Stablilize training of transformers in early stages

Back-translation (BT)

Leverage monolingual data by creating synthetic translation data

1. Train a translation system in the opposite direction
2. Collect monolingual data in target side and apply machine translation
3. Use translated and original monolingual data as additional parallel data to

train stronger translation systems

translated monoligual data

back-translation

monolingual data

source language

target language

original data

original data

Back-translation

Some points to note about back-translation

1. Monolingual data should be in the same domain as the parallel corpus
2. The performance of the backward model is critical
3. You should increase model capacity (both forward and backward), since

   the data amount is increased.

Requirements

You are encouraged to follow these tips to improve your performance in order to pass the 3 baselines.

1. Train a simple RNN seq2seq to acheive translation
2. Switch to transformer to boost performance
3. Apply back-translation to furthur boost performance

Baseline Guide

Train a simple RNN seq2seq to acheive translation
● Running the sample code should pass the baseline!

Baseline Guide

Switch to transformer to boost performance

1. Change the encoder/decoder architecture to transformer based,

   according to the hints in sample code

   • ○  RNNEncoder -> TransformerEncoder
   • ○  RNNDecoder -> TransformerDecoder
2. Change architecture configurations

○ encoder_ffn_embed_dim -> 1024

○ encoder_layers/decoder_layers -> 4

○ #add_transformer_args(arch_args) -> add_transformer_args(arch_args)

Baseline Guide

Apply back-translation to furthur boost performance

1. Train a backward model by switching languages
   • ○  source_lang = “zh”
   • ○  target_lang = “en”
2. Remember to change architecture to transformer-base
3. Translate monolingual data with backward model to obtain synthetic data
   • ○  complete TODOs in the sample code.
   • ○  all the TODOs can be completed by using commands from earlier cells.
4. Train a stronger forward model with the new data

○ if done correctly, ~30 epochs on new data should pass the baseline.

Expected Run Time

● on colab with Tesla T4

● TA’s training curve https://wandb.ai/george0828zhang/hw5.seq2seq.new

Prediction Submission

• ●  Submit to JudgeBoi
• ●  One example per line, in the original order
• ●  Punctuation will be normalized by JudgeBoi with this script
• ●  Deadline: 4/30 (Fri.) 23:59

Previously… Github Account Survey

We have kindly requested everyone to report your github username and ID. IMPORTANT: You must take this survey in order to submit to JudgeBoi server.

Step 1: Register for Submission

Go to JudgeBoi to login.

Step 2: Sign-in with Github

You need to sign in with the account you reported to us. Or you won’t be able to upload your submissions.

fill in username > fill in password >

Step 3: Submit your Results

You can now submit results to the server and view the leaderboard.

1) click here

3) view leaderboard here 2) Submit result here

Step 4: Select your submissions

• ●  You can select up to 2 submissions.
• ●  If none of your submissions is chosen, we will use your first submission to

  calculate your private score.

• ●  If your selection is successful, you will see a message box as follows:

More about JudgeBoi

• ●  5 submission quota per day, reset at midnight. Users not in whitelist will have no quota.
• ●  Only *.txt file is allowed, filesize should be smaller than 700kB.
• ●  The countdown timer on the homepage is for reference only.
• ●  We do limit the number of connections and request rate for each ip. If you

  cannot access the website temporarily, please wait patiently.

• ●  Please do not attempt to attack JudgeBoi, thank you.
• ●  Every Wednesday and Saturday from 0:00 to 3:00 is our system

  maintenance time. If the website cannot be used during this time, please wait patiently for the completion of the maintenance.

Links

Sample code Colab Colab(chinese version) Parallel data TED2020
Testing data Testdata
Monolingual TED_ZH

Q: My backward (zh-en) model is significantly weaker than forward (en-zh) model, what’s going on?

A: BLEU scores aren’t comparable across languages. However, your backward model should be as strong as possible for BT to work properly.

Q: Larger models or synthetic data requires long training time, but colab has limited usage?

A: The sample code saves model checkpoints each epoch, see next page.

Save checkpoints and data to drive

1. Mount your drive by clicking
2. Save your preprocessed DATA to your drive

         !mkdir -p /content/drive/MyDrive/ML2021-hw5/
         !cp -r ./DATA /content/drive/MyDrive/ML2021-hw5/
   

3. Change checkpoint directory (under config) to your drive

         savedir = "/content/drive/MyDrive/ML2021-hw5/checkpoints/transformer",
   

Next time, load preprocessed data quicky with

 !ln -s /content/drive/MyDrive/ML2021-hw5/DATA/ ./DATA  # symbolic link
 # !cp -r /content/drive/MyDrive/ML2021-hw5/DATA/ ./    # or you can use full copy

Change resume (under config) to following to resume from checkpoint.

 resume="checkpoint_last.pt", # if resume from checkpoint name (under config.savedir)