graviti
Products
Resources
About us
SQuAD2.0
Text
NLP
|...
License: CC BY-SA 4.0

Overview

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting
of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every
question is a segment of text, or span, from the corresponding reading passage, or the question
might be unanswerable.
SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000
unanswerable questions written adversarially by crowdworkers to look similar to answerable
ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also
determine when no answer is supported by the paragraph and abstain from answering.

Data Collection

We employed crowd workers on the Daemo crowd-sourcing platform to write unanswerable questions.
Each task consisted of an entire article from SQuAD 1.1. For each paragraph in the article,
workers were asked to pose up to five questions that were impossible to answer based on the
paragraph alone, while referencing entities in the paragraph and ensuring that a plausible
answer is present. As inspiration, we also showed questions from SQuAD 1.1 for each paragraph;
this further encouraged unanswerable questions to look similar to answerable ones.
We removed
questions from workers who wrote 25 or fewer questions on that article; this filter helped
remove noise from workers who had trouble understanding the task, and therefore quit before
completing the whole article. We applied this filter to both our new data and the existing
answerable questions from SQuAD 1.1. To generate train, development, and test splits, we used
the same partition of articles as SQuAD 1.1, and combined the existing data with our new data
for each split. For the SQuAD 2.0 development and test sets, we removed articles for which
we did not collect unanswerable questions. This resulted in a roughly one-to-one ratio of answerable
to unanswerable questions in these splits, whereas the train data has roughly twice as many
answerable questions as unanswerable ones.
To confirm that our dataset is clean, we hired
additional crowd workers to answer all questions in the SQuAD 2.0 development and test sets.
In each task, we showed workers an entire article from the dataset. For each paragraph, we
showed all associated questions; unanswerable and answerable questions were shuffled together.
For each question, workers were told to either highlight the answer in the paragraph, or mark
it as unanswerable. Workers were told to expect every paragraph to have some answerable and
some unanswerable questions. They were asked to spend one minuteper question, and were paid
$10.50 per hour.
To reduce crowd worker noise, we collected multiple human answers for each
question and selected the final answer by majority vote, breaking ties in favor of answering
questions and preferring shorter answers to longer ones. On average, we collected 4.8 answers
per question.

Citation

Please use the following citation when referencing the dataset:

@article{DBLP:journals/corr/abs-1806-03822,
  author    = {Pranav Rajpurkar and
               Robin Jia and
               Percy Liang},
  title     = {Know What You Don't Know: Unanswerable Questions for SQuAD},
  journal   = {CoRR},
  volume    = {abs/1806.03822},
  year      = {2018},
  url       = {http://arxiv.org/abs/1806.03822},
  archivePrefix = {arXiv},
  eprint    = {1806.03822},
  timestamp = {Mon, 13 Aug 2018 16:48:21 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1806-03822.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

License

CC BY-SA 4.0

Data Summary
Type
Text,
Amount
--
Size
44.34MB
Provided by
Stanford
Stanford University, officially Leland Stanford Junior University, is a private research university in Stanford, California.
| Amount -- | Size 44.34MB
SQuAD2.0
Text
NLP
License: CC BY-SA 4.0

Overview

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting
of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every
question is a segment of text, or span, from the corresponding reading passage, or the question
might be unanswerable.
SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000
unanswerable questions written adversarially by crowdworkers to look similar to answerable
ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also
determine when no answer is supported by the paragraph and abstain from answering.

Data Collection

We employed crowd workers on the Daemo crowd-sourcing platform to write unanswerable questions.
Each task consisted of an entire article from SQuAD 1.1. For each paragraph in the article,
workers were asked to pose up to five questions that were impossible to answer based on the
paragraph alone, while referencing entities in the paragraph and ensuring that a plausible
answer is present. As inspiration, we also showed questions from SQuAD 1.1 for each paragraph;
this further encouraged unanswerable questions to look similar to answerable ones.
We removed
questions from workers who wrote 25 or fewer questions on that article; this filter helped
remove noise from workers who had trouble understanding the task, and therefore quit before
completing the whole article. We applied this filter to both our new data and the existing
answerable questions from SQuAD 1.1. To generate train, development, and test splits, we used
the same partition of articles as SQuAD 1.1, and combined the existing data with our new data
for each split. For the SQuAD 2.0 development and test sets, we removed articles for which
we did not collect unanswerable questions. This resulted in a roughly one-to-one ratio of answerable
to unanswerable questions in these splits, whereas the train data has roughly twice as many
answerable questions as unanswerable ones.
To confirm that our dataset is clean, we hired
additional crowd workers to answer all questions in the SQuAD 2.0 development and test sets.
In each task, we showed workers an entire article from the dataset. For each paragraph, we
showed all associated questions; unanswerable and answerable questions were shuffled together.
For each question, workers were told to either highlight the answer in the paragraph, or mark
it as unanswerable. Workers were told to expect every paragraph to have some answerable and
some unanswerable questions. They were asked to spend one minuteper question, and were paid
$10.50 per hour.
To reduce crowd worker noise, we collected multiple human answers for each
question and selected the final answer by majority vote, breaking ties in favor of answering
questions and preferring shorter answers to longer ones. On average, we collected 4.8 answers
per question.

Citation

Please use the following citation when referencing the dataset:

@article{DBLP:journals/corr/abs-1806-03822,
  author    = {Pranav Rajpurkar and
               Robin Jia and
               Percy Liang},
  title     = {Know What You Don't Know: Unanswerable Questions for SQuAD},
  journal   = {CoRR},
  volume    = {abs/1806.03822},
  year      = {2018},
  url       = {http://arxiv.org/abs/1806.03822},
  archivePrefix = {arXiv},
  eprint    = {1806.03822},
  timestamp = {Mon, 13 Aug 2018 16:48:21 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1806-03822.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

License

CC BY-SA 4.0

0
Start building your AI now
graviti
wechat-QR
Long pressing the QR code to follow wechat official account

Copyright@Graviti