7 feb 2020

facebook

In this paper the authors check if the unsupervised training using CPC (similar to wav2vec) on english transfers to other languages or no. They find that it works quite well and equal to supervised pre-training.

They introduce some modifications in the CPC training as explained in the supplementary material.

1. using layer norm instead of batch norm
2. Using complex linear predictors i.e, transformers instead of linear layers.