Bootstrapping Multiple-Choice Tests with The-Mentor.

Full text

Page 1

Bootstrapping Multiple-Choice Tests with
THE-MENTOR
Ana Cristina Mendes, S´ ergio Curto and Lu´ ısa Coheur
Spoken Language Systems Laboratory - L2F/INESC-ID
Instituto Superior T´ ecnico, Technical University of Lisbon
R. Alves Redol, 9 - 2o– 1000-029 Lisboa, Portugal
{ana.mendes,sergio.curto,luisa.coheur}@l2f.inesc-id.pt
Abstract. It is very likely that, at least once in their lifetime, everyone has an-
swered a multiple-choice test. Multiple-choice tests are considered an effective
technique for knowledge assessment, requiring a short response time and with
the possibility of covering a broad set of topics. Nevertheless, when it comes to
their creation, it can be a time-consuming and labour-intensive task. Here, the
generation of multiple-choice tests aided by computer can reduce these draw-
backs: to the human assessor is attributed the final task of approving or rejecting
the generated test items, depending on their quality.
In this paper we present THE-MENTOR, a system that employs a fully automatic
approach to generate multiple-choice tests. In a first offline step, a set of lexico-
syntactic patterns are bootstrapped by using several question/answer seed pairs
and leveraging the redundancy of the Web. Afterwards, in an online step, the pat-
terns are used to select sentences in a text document from which answers can be
extracted and the respective questions built. In the end, several filters are applied
to discard low quality items and distractors are named entities that comply with
the question category, extracted from the same text.
1 Introduction
Multiple-choice tests are an effective technique for knowledge assessment, requiring a
short response time and with the possibility of covering a broad set of topics. Typically,
these tests consist in a number of test items, each composed by two parts: a question
and a group of suggested answers. Respondents are supposed to identify the correct
answer among the incorrect ones (called distractors). The following is an example of a
multiple-choice test item with one correct answer and two distractors:
Q. “What is the largest ocean?”
1. Atlantic (distractor)
2. Pacific
(correct answer)
3. Indian
(distractor)
The manual creation of multiple-choice test items is a time consuming trial and
error process; in this context, computer aided multiple-choice tests generation can help
reducing the amount of time allocated to this task.

Page 2

In this paper, we hypothesize that the process of generating multiple-choice tests
with only one correct answer per question can rely on the bootstrap of a set of ques-
tion/answer (Q/A) seed pairs. Here, we describe our approach to automatically generate
multiple-choice tests and present THE-MENTOR, a system that generates multi-choice
tests about a free text document. THE-MENTOR is composed by two main components
which perform the following tasks:
Learning lexico-syntactic patterns – A set of Q/A seeds is used to bootstrap pat-
terns that relate questions with answers. Patterns are learned from the Web, and
we exploit its redundancy to create plausible patterns. Moreover, we perform verb
expansion and allow several types of patterns, according to the precision of the
match against the original seeds. The decision of accepting different types of pat-
terns resulted from the fact that, if patterns are too specific (strong patterns), they
will not frequently match and not many tests will be generated; if patterns are too
generic (weak patterns), the quality of the generated tests decreases.
Generation of test items – The retrieved patterns extract sentences where answers can
be found and from which the respective questions can be built. In order to discard
low quality items, several filters are applied. Distractors are named entities that
comply with the question category, extracted from the same text.
Afterwards, the user can evaluate the quality of the generated test items through
a web interface. The multiple-choice test will be composed by the test items the user
considered as having quality.
This paper is organized as follows: in Section 2 we describe the pattern learning
task; in Section 3 we describe how to generate multiple-choice test items. In Section 4
we show the evaluation results; in Section 5 we present related work; in Section 6 we
present our conclusions and point to future work directions.
2 Pattern Learning
The first task in the generation of multiple-choice test items by THE-MENTOR has to
do with learning lexico-syntactic patterns and is performed off-line, that is, before the
user specifies the text document based on which the test will be generated.
Thealgorithmforpatternlearningisbasedonthebootstrappingtechniquepresented
in [1], and involves the following two stages. First, we use a seed pair – composed by a
natural language question and its correct answer – to bootstrap patterns that relate ques-
tions and answers. We call B-PATTERNS to the patterns extracted from the bootstrap
process. Second, the B-PATTERNS are validated using a validation pair – also a natural
language question and its correct answer – as input. The validation pair will help to
remove those patterns that are too specific to the seed pair. Therefore, each seed pair
has a validation pair associated. These pairs are automatically grouped, knowing that al-
though their constituents are lexically distinct, they must share the same syntactic struc-
ture as well as the same category. For example, the seed question “Who is the President
of France?” cannot be validated by the question “What is the capital of France?” since,
although both share the same syntactic structure (WHNP VBZ NP), they have a different

Page 3

focus: the former searches for the name of an individual (HUMAN:INDIVIDUAL) and
the later for the name of a city (LOCATION:CITY).
The syntactic analysis of questions is made using the Berkeley Parser [2] trained
on the QuestionBank [3], a treebank of 4,000 parse-annotated questions. In what con-
cerns the classification of questions, we use a machine learning-based classifier fed with
features derived from a rule-based classifier [4]. Regarding the question categories, we
use Li and Roth’s two-layer taxonomy [5], consisting of a set of six coarse-grained
categories and fifty fine-grained ones.
Findingpatterns Thealgorithmtofind B-PATTERNS startsbygeneratingpermutations
of a set comprising the seed answer, the phrasal nodes of the seed question (excluding
the Wh-phrase), and a wildcard * which stands as a placeholder for one or more words
and adds diversity into the generated patterns. For instance, considering the question
“Who painted the Birth of Venus ?” and the sentence Botticelli has painted the Birth of
Venus, a wildcard is required to match the verb has. Since we do not allow the wildcard
to be the first or the last element in the query, the total number of permutations is n! −
2(n−1)!, in which n is the number of elements to be permuted. In addition, the reason
why we use phrasal nodes instead of question tokens as it is done in [1], is because
they represent a single unit of meaning, and therefore should not be broken down into
parts (except for verb phrases). For example, considering the previous question “Who
painted the Birth of Venus ?”. it does not make sense to divide the noun-phrase the
Birth of Venus, since it would generate several meaningless permutations, like Birth the
painted Botticelli * of Venus.
After the permutations have been created, each is enclosed in double quotes and sent
to Google search1. The double quotes ensure that each search result contains the exact
quoted permutation, with every word in the exact same order in which it appears in
the original query. The snippets retrieved from the search engine are then broken down
into sentences, and if there exists a sentence that matches the respective permutation,
we rewrite it as a pattern. Consider again the question “[WHNPWho] [VBDpainted]
[NPthe Birth of Venus]”, and suppose a sentence Botticelli has painted the Birth of
Venus that matches the permutation Botticelli * painted the Birth of Venus. The resulting
pattern would then be “{ANSWER} has VBD NP”, created by replacing each phrasal
node with the respective tag, and the seed answer with the tag “{ANSWER}”.
Pattern Validation While many of the learned patterns are generic enough to be ap-
plied to other questions, there are others specific to the seed pair. For instance, the
pattern “NP was VBD around 1486 by {ANSWER}”, extracted from the sentence The
Birth of Venus was painted around 1486 by Botticelli. Since this pattern only works for
the seed question (and possibly for a small number of works of art of that same year), it
should be filtered-out. To eliminate these elements, we use a different algorithm which
requires the use of a validation pair.
The validation algorithm works by testing each generated pattern against the val-
idation pair, and calculating its precision. The precision is considered to be the ratio
1In this work we use Google as the search engine. However, there is no technical reason that
prohibits this system to use another search engine.

Page 4

between the number of times the pattern matched the retrieved snippets and the number
of times the pattern was expected to match (that is, the maximum number of snippets
retrieved by the search engine). For example, using the aforementioned pattern and the
validation pair “Who painted Guernica?”/Picasso, the query Guernica was painted
around 1486 by Picasso would be issued, resulting in zero results – and thus zero pre-
cision. This would cause the pattern to be ruled-out, as the algorithm dictates that each
pattern must have precision larger than a threshold in order to be retained.
2.1 Handling Verbs
Our method for learning patterns implies that every phrase (except the Wh-phrase)
must be present in the B-PATTERNS. From now on, we call strong patterns to the
B-PATTERNS that contain all phrases (and their contents) of the seed question. Whereas
this is the expected behaviour for Noun- and Prepositional-phrases that should be stated
ipsis verbis in the sentence fragments that will generate the patterns as they are in the
seed question, the same does not apply for Verb-phrases. The pattern generator should
be flexible enough to capture a pattern in the sentence Botticelli finished painting The
Birth of Venus in 1485., even if the surface word that corresponds to the verb is not the
same as the one present in “Who painted the Birth of Venus?”.
Beingso,weallowtheverbsinthepatterntobeinadifferentinflexionthanthemain
verb in the seed question. Moreover, in case the seed question has an auxiliary verb, the
sentence fragment does not need to contain it, since these are most probable to appear in
interrogative sentences than on declaratives ones. To create these patterns, we pick the
main verb of the question and conjugate it in its multiple inflexions. Afterwards, a new
query is sent to the search engine with the several inflexions, and without the presence
of the auxiliary verb (if it exists in the question).
The B-PATTERNS generated by verb inflection are named inflected patterns.
2.2 Allowing Weak Patterns
There are, however, some patterns that should not be disregarded, even if they do not
contain all question phrases and cannot be handled by allowing the multiple inflexion
of verbs. These patterns arise from sentence that, despite not completely rephrasing
the question, capture the existing relation between it and the answer. For instance, the
pattern “NP, by {ANSWER}” should be recovered from the sentence The Birth of Venus,
by Botticelli, even if it does not include the verb (in this case, painted). These patterns
are different from the strong and inflected patterns, not only because of how
they were created, but also because they will trigger distinct strategies in the test item
generation. To generate this type of patterns (called weak patterns), the procedure is
similar as referred, just we do not allow the Verb Phrases in the question to be present
on the permutations.
Table 1 shows a set of patterns2generated from, and validated with, the snippets
retrieved by the search engine, for questions with flatten syntactic structure WHNP
VBD NP and category HUMAN:INDIVIDUAL.
2Here, as well as throughout the entire paper, the Penn Treebank II Tags [6] are used.

Page 5

Table 1. Example of extracted patterns with respective precision and type.
Question HUMAN:INDIVIDUAL-WHNP VBD NP
Precision Pattern Type
0.625
0.25
0.625
{ANSWER}’s NP
{ANSWER} began VBG NP
NP VBD by {ANSWER}
W
I
S
3 Building Mutiple-Choice Tests
The next task in the generation of multiple-choice test items by THE-MENTOR is done
online. The user specifies the target documents and the system parses the text and ap-
plies the learned patterns in order to obtain Q/A pairs (as well as distractors). This
method also involves several strategies for filtering the obtained Q/A pairs in order to
discard low quality pairs.
3.1Extracting Question/Answer Pairs
Our algorithm for extracting Q/A pairs relies on matching lexico-syntactic information
from the B-PATTERNS against the parsed sentences of the target document. Each match
is done at two levels in the sentence parse tree: at the word level, since most of the
patterns include tokens to separate the syntactic components, and at the syntactic level.
For that purpose, we have developed a tree matching algorithm (out of the scope of this
paper) to find all the occurrences of a given pattern on the syntactic tree of the parsed
sentences.
After the extraction of the sentence fragments where questions and their respec-
tive answers are stated, we apply a set of filters to refine the proceeding generation of
multiple-choice test items, namely:
Forcing Question/Answer Category Matching The extracted fragments in which the
answer does not comply with the expected category can be discarded. Consider, for
example, the Q/A pair: “Who was Franc ¸ois Rabelais?”-An important 16th century
writer. Here, the answer agrees with the semantic class expected by the question
(HUMAN:INDIVIDUAL), indicated by the word writer.
Thus, we test the answer in order to check if at least one of its words belongs to the
question category. By using WordNet’s lexical hierarchy, a word is associated with
a higher-level semantic concept, which represents itself a question category. To do
so, we have manually grouped a set of WordNet synsets into fifty clusters, each
representing a question category. For example the category HUMAN:INDIVIDUAL
is related with the synsets person, individual, someone, somebody and
mortal. The words actor, leader and writer are hyponyms of (at least) one of
these synsets. Since WordNet can be seen as a directed acyclic graph, with synsets
as vertices and lexical relations – such as hypernym – as edges, we employ a
breadth-first search on the translated synset’s hypernym tree, in order to find a
synset that pertains to any of the pre-defined clusters.