Skip to main content

Table 1 Framework of item flaws and their potential effect

From: The future of online testing and assessment: question quality in MOOCs

Technical item flaws or unadvisable formats

Potential effect

Outmoded item formats

 True / false format

True-false questions require that examinees decide if a statement is true or false – at times a difficult decision, in a world where absolutes are rare. In addition, the examinee may also have to make a value judgment as to what extent or degree an option is correct; this may be straightforward, or may involve also trying to anticipate what the examiner had in mind when phrasing the question. (Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

 Overly complex, or K-type, questions; e.g. choose option A if statements 1 and 2 are correct, choose option B if statements 1 and 3 are correct etc.

This format introduces unnecessary complexity to the format, increasing reading time, construct irrelevant variance, and reducing validity. (Case & Swanson, 2002, Downing, 2002a, 2002b, Jozefowicz et al., 2002, Haladyna et al., 2002, Tarrant et al., 2006a, 2006b)

 Fill in the blank

These questions may be linguistically difficult to write, without giving grammatical clues to the examinee, and so are best avoided. (Downing, 2002a, 2002b)

Question ambiguity or obscurity

 Gratuitous information in stem

Inclusion of irrelevant information introduces unnecessary complexity to the format, increasing reading time, construct irrelevant variance, and reducing validity. Stems should be focussed, and only information relevant to answering the question should be included. (Case & Swanson, 2002, Downing, 2002a, 2002b, Tarrant et al., 2006a, 2006b)

 Ambiguous or unclear information

Poorly worded questions can confuse examinees, even those of high ability, and are particularly problematic for non-native speakers. (Downing, 2002a, 2002b, Tarrant et al., 2006a, 2006b)

 Unfocussed stem

Questions should be clear and explicit, with a definitive question (Haladyna et al., 2002, Tarrant et al., 2006a, 2006b)

 Absolute terms

Elimination of options containing the words “always” and “never” greatly improves examinees’ chances of choosing the correct option by chance. In addition, even supposedly absolute terms such as “always” or “never” may be interpreted differently, and means that examinees must make a value judgment as to what the writer means by the term in this context. (Holsgrove & Elzubeir, 1998, Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

 Vague frequency terms

Frequency terms are interpreted very differently by individuals, and their use means that examinees must make a value judgment as to what the writer means by a given frequency term in an individual question context. (Case, 1994, Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

 Negatively worded stem

Some writers advocate that negative stems may occasionally be used, so long as care is taken to phrase them simply and unambiguously. Others hold that high-quality negative MCQs are difficult to write well, and that their inclusion among otherwise positively-phrased questions may be confusing for examinees. (Haladyna et al., 2002, Jozefowicz et al., 2002, Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

Structural or logical flaws

 Options & Stem

Problem is in the options not in the stem

The problem or question of the MCQ should be in the stem, not within the options. Inclusion of the problems within the options instead reduces the format to a true / false, or even a K-type complex format, with all the problems inherent within (above). (Haladyna et al., 2002, Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

Logical cues in stem & correct option

Logical cues (grammatical or numerical) in the stem and options may enable examinees to guess the correct option without any content knowledge. (Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

Word repeats in stem & correct answer

Similar wording in the stem and options enables examinees to guess the correct option without any content knowledge. (Case & Swanson, 2002, Downing, 2002a, 2002b, Tarrant et al., 2006a, 2006b)

 Options

Longest option is correct

Writers often have an inherent bias to take extreme care in making the correct option with exact information and precise grammar, increasing the length; examinees may guess the correct option by assuming that the correct option is also the longest. (Case & Swanson, 2002, Downing, 2002a, 2002b, Tarrant et al., 2006a, 2006b)

Implausible disctractors

Elimination of implausible distractors greatly improves examinees’ chances of choosing the correct option by chance. (Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

More than one, or no correct answer

A move away from the one-best-answer approach instead reduces the format to a true / false, or even a K-type complex format, with all the problems inherent within (above). (Haladyna et al., 2002, Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

Use of all of the above

The use of “all of the above” means that students may correctly eliminate this option by identifying at least one other response as being incorrect. (Haladyna et al., 2002, Tarrant et al., 2006a, 2006b)

Use of none of the above

The use of “none of the above” means that students may correctly eliminate this option by identifying at least one other response as being correct. (Haladyna et al., 2002, Pachai et al., 2015, Case & Swanson, 2002, Tarrant et al., 2006a, 2006b)

Option order / position of the correct option

Listing options in a consistent order on printed examination papers avoids bias on the part of the examiner towards edge aversion, i.e. a reluctance to place the correct option in the first or last positions. In a five option MCQ, this may result in option C being correct more often than would be expected by chance. Online examinations may be programmable to randomise option order, and so avoid this issue. (Case, 1994, Tarrant et al., 2006a, 2006b)

Convergence cues

Writers often have an inherent bias to write distracters derived from the correct option, altering minor words or components; examinees may guess the correct option by choosing the one in which most option components appear together (Case & Swanson, 2002, Tarrant et al., 2006a, 2006b).

  1. Technical flaws or unadvisable