graviti
Products
Resources
About us
tedHrlrTranslate
Text
NLP
|...
License: Unknown

Overview

a common corpus of TED talks which has been translated into many low-resource languages. Under
the Open Translation project, TED talks transcripts
are available for more than 2400 talks in 109 languages. A histogram plot of language (represented
by its ISO Code) vs total number of talks in the original dataset is visualized in the figure
below.

TED Talks statistics

To obtain a parallel corpus for experiments, we preprocessed the dataset using Moses tokenizer
and used hard punctuation symbols to identify valid sentence boundaries for English language.
In order to create train, dev and test sets, we apply a greedy selection algorithm based on
the popularity of the talks and selected disjoint talks for each split. We selected talks which
had translations in more than 50 languages. Finally, we selected a list of 60 languages that
had sufficient data for performing meaningful experiments. The train, test and dev splits for
the most common talks are also shown in the table alongside the above figure.

Instruction

The train, dev and test splits for the above TED talks: ted_talks.tar.gz.

ted_reader.py is a sample python script to read this TED talks data. An example is shown
under the "main" attribute of the code.

Citation

@inproceedings{Ye2018WordEmbeddings,
  author  = {Ye, Qi and Devendra, Sachan and Matthieu, Felix and Sarguna, Padmanabhan and Graham,
Neubig},
  title   = {When and Why are pre-trained word embeddings useful for Neural Machine Translation},
  booktitle = {HLT-NAACL},
  year    = {2018},
  }
Data Summary
Type
Text,
Amount
--
Size
461.27MB
Provided by
NeuLab
This is Graham Neubig's lab at the Language Technologies Institute of Carnegie Mellon University. We do research on natural language processing and machine learning, specifically machine translation, multi-lingual NLP, and natural language understanding.
| Amount -- | Size 461.27MB
tedHrlrTranslate
Text
NLP
License: Unknown

Overview

a common corpus of TED talks which has been translated into many low-resource languages. Under
the Open Translation project, TED talks transcripts
are available for more than 2400 talks in 109 languages. A histogram plot of language (represented
by its ISO Code) vs total number of talks in the original dataset is visualized in the figure
below.

TED Talks statistics

To obtain a parallel corpus for experiments, we preprocessed the dataset using Moses tokenizer
and used hard punctuation symbols to identify valid sentence boundaries for English language.
In order to create train, dev and test sets, we apply a greedy selection algorithm based on
the popularity of the talks and selected disjoint talks for each split. We selected talks which
had translations in more than 50 languages. Finally, we selected a list of 60 languages that
had sufficient data for performing meaningful experiments. The train, test and dev splits for
the most common talks are also shown in the table alongside the above figure.

Instruction

The train, dev and test splits for the above TED talks: ted_talks.tar.gz.

ted_reader.py is a sample python script to read this TED talks data. An example is shown
under the "main" attribute of the code.

Citation

@inproceedings{Ye2018WordEmbeddings,
  author  = {Ye, Qi and Devendra, Sachan and Matthieu, Felix and Sarguna, Padmanabhan and Graham,
Neubig},
  title   = {When and Why are pre-trained word embeddings useful for Neural Machine Translation},
  booktitle = {HLT-NAACL},
  year    = {2018},
  }
0
Start building your AI now
graviti
wechat-QR
Long pressing the QR code to follow wechat official account

Copyright@Graviti