graviti
Products
Resources
About us
bAbI Tasks Data 1-20
Text
NLP
|...
License: Unknown

Overview

This section presents the first set of 20 tasks for testing text understanding and reasoning
in the bAbI project. The tasks are described in detail in the paper:

Jason Weston, Antoine
Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin and Tomas Mikolov.
Towards AI Complete Question Answering: A Set of Prerequisite Toy Tasks,
arXiv:1502.05698.

Please also see the following slides:

Antoine Bordes Artificial Tasks for Artificial Intelligence,
ICLR keynote, 2015.

The aim is that each task tests a unique aspect of text and reasoning,
and hence test different capabilities of learning models. More tasks are planned in the future
to capture more aspects.

Data Collection

Training Set Size: For each task, there are 1000 questions for training, and 1000 for testing.
However, we emphasize that the goal is to use as little data as possible to do well on the
tasks (i.e. if you can use less than 1000 that’s even better) — and without resorting to engineering
task-specific tricks that will not generalize to other tasks, as they may not be of much use
subsequently. Note that the aim during evaluation is to use the same learner across all tasks
to evaluate its skills and capabilities.

Supervision Signal: Further while the MemNN results
in the paper use full supervision (including of the supporting facts) results with weak supervision
would also be ultimately preferable as this kind of data is easier to collect. Hence results
of that form are very welcome. E.g. this paper does include
weakly supervised results.

For the reasons above there are currently several directories:

  1. en/ — the tasks in English, readable by humans.
  2. hn/ — the tasks in Hindi, readable by humans.
  3. shuffled/ — the same tasks with shuffled letters so they are
    not readable by humans, and for existing parsers and taggers cannot be used in a straight-forward
    fashion to leverage extra resources– in this case the learner is more forced to rely on the
    given training data. This mimics a learner being first presented a language and having to learn
    from scratch.
  4. en-10k/ shuffled-10k/ and hn-10k/ — the same tasks in the three formats,
    but with 10,000 training examples, rather than 1000 training examples. Note the results in
    the paper use 1000 training examples.

Data Format

The file format for each task is as follows:

ID text
ID text
ID text
ID question[tab]answer[tab]supporting fact IDS.
...
The IDs for a given “story” start at 1 and increase. When the IDs in a file reset back to 1
you can consider the following sentences as a new “story”. Supporting fact IDs only ever reference
the sentences within a “story”.

For Example:

1 Mary moved to the bathroom.
2 John went to the hallway.
3 Where is Mary? bathroom 1
4 Daniel went back to the hallway.
5 Sandra moved to the garden.
6 Where is Daniel? hallway 4
7 John moved to the office.
8 Sandra journeyed to the bathroom.
9 Where is Daniel? hallway 4
10 Mary moved to the hallway.
11 Daniel travelled to the office.
12 Where is Daniel? office 11
13 John went back to the garden.
14 John moved to the bedroom.
15 Where is Sandra? bathroom 8
1 Sandra travelled to the office.
2 Sandra went to the bathroom.
3 åWhere is Sandra? bathroom 2

Instruction

Some data statistics including overlap between train and test (which is minimal) can be found
here. Code
Code to generate tasks is available on github. We hope this will encourage the machine learning
community to work on, and develop more, of these tasks.

Version: Some small updates since
the original release have been made (see the README in the data download for more details).
You can also get v1.0
and v1.1 here.

Data Summary
Type
Text,
Amount
--
Size
14.99MB
Provided by
Facebook Research
Giving people the power to build community through research and innovation
| Amount -- | Size 14.99MB
bAbI Tasks Data 1-20
Text
NLP
License: Unknown

Overview

This section presents the first set of 20 tasks for testing text understanding and reasoning
in the bAbI project. The tasks are described in detail in the paper:

Jason Weston, Antoine
Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin and Tomas Mikolov.
Towards AI Complete Question Answering: A Set of Prerequisite Toy Tasks,
arXiv:1502.05698.

Please also see the following slides:

Antoine Bordes Artificial Tasks for Artificial Intelligence,
ICLR keynote, 2015.

The aim is that each task tests a unique aspect of text and reasoning,
and hence test different capabilities of learning models. More tasks are planned in the future
to capture more aspects.

Data Collection

Training Set Size: For each task, there are 1000 questions for training, and 1000 for testing.
However, we emphasize that the goal is to use as little data as possible to do well on the
tasks (i.e. if you can use less than 1000 that’s even better) — and without resorting to engineering
task-specific tricks that will not generalize to other tasks, as they may not be of much use
subsequently. Note that the aim during evaluation is to use the same learner across all tasks
to evaluate its skills and capabilities.

Supervision Signal: Further while the MemNN results
in the paper use full supervision (including of the supporting facts) results with weak supervision
would also be ultimately preferable as this kind of data is easier to collect. Hence results
of that form are very welcome. E.g. this paper does include
weakly supervised results.

For the reasons above there are currently several directories:

  1. en/ — the tasks in English, readable by humans.
  2. hn/ — the tasks in Hindi, readable by humans.
  3. shuffled/ — the same tasks with shuffled letters so they are
    not readable by humans, and for existing parsers and taggers cannot be used in a straight-forward
    fashion to leverage extra resources– in this case the learner is more forced to rely on the
    given training data. This mimics a learner being first presented a language and having to learn
    from scratch.
  4. en-10k/ shuffled-10k/ and hn-10k/ — the same tasks in the three formats,
    but with 10,000 training examples, rather than 1000 training examples. Note the results in
    the paper use 1000 training examples.

Data Format

The file format for each task is as follows:

ID text
ID text
ID text
ID question[tab]answer[tab]supporting fact IDS.
...
The IDs for a given “story” start at 1 and increase. When the IDs in a file reset back to 1
you can consider the following sentences as a new “story”. Supporting fact IDs only ever reference
the sentences within a “story”.

For Example:

1 Mary moved to the bathroom.
2 John went to the hallway.
3 Where is Mary? bathroom 1
4 Daniel went back to the hallway.
5 Sandra moved to the garden.
6 Where is Daniel? hallway 4
7 John moved to the office.
8 Sandra journeyed to the bathroom.
9 Where is Daniel? hallway 4
10 Mary moved to the hallway.
11 Daniel travelled to the office.
12 Where is Daniel? office 11
13 John went back to the garden.
14 John moved to the bedroom.
15 Where is Sandra? bathroom 8
1 Sandra travelled to the office.
2 Sandra went to the bathroom.
3 åWhere is Sandra? bathroom 2

Instruction

Some data statistics including overlap between train and test (which is minimal) can be found
here. Code
Code to generate tasks is available on github. We hope this will encourage the machine learning
community to work on, and develop more, of these tasks.

Version: Some small updates since
the original release have been made (see the README in the data download for more details).
You can also get v1.0
and v1.1 here.

0
Start building your AI now
graviti
wechat-QR
Long pressing the QR code to follow wechat official account

Copyright@Graviti