License: Unknown


a common corpus of TED talks which has been translated into many low-resource languages. Under the Open Translation project, TED talks transcripts are available for more than 2400 talks in 109 languages. A histogram plot of language (represented by its ISO Code) vs total number of talks in the original dataset is visualized in the figure below.

TED Talks statistics

To obtain a parallel corpus for experiments, we preprocessed the dataset using Moses tokenizer and used hard punctuation symbols to identify valid sentence boundaries for English language. In order to create train, dev and test sets, we apply a greedy selection algorithm based on the popularity of the talks and selected disjoint talks for each split. We selected talks which had translations in more than 50 languages. Finally, we selected a list of 60 languages that had sufficient data for performing meaningful experiments. The train, test and dev splits for the most common talks are also shown in the table alongside the above figure.


The train, dev and test splits for the above TED talks: ted_talks.tar.gz.

ted_reader.py is a sample python script to read this TED talks data. An example is shown under the "main" attribute of the code.


  author  = {Ye, Qi and Devendra, Sachan and Matthieu, Felix and Sarguna, Padmanabhan and Graham,
  title   = {When and Why are pre-trained word embeddings useful for Neural Machine Translation},
  booktitle = {HLT-NAACL},
  year    = {2018},
Data Summary
Provided by
This is Graham Neubig's lab at the Language Technologies Institute of Carnegie Mellon University. We do research on natural language processing and machine learning, specifically machine translation, multi-lingual NLP, and natural language understanding.
Start Building AI Now