A Comparison of the Quality of Multiple-Choice Questions from CPA Exams and Textbook Test Banks
Abstract
Multiple-choice test questions have substantial support as a reliable and valid
evaluation method. In practice, however, instructors may prepare or select
multiple-choice questions based on content and without adequate attention to
item quality. This paper has two specific objectives: (1) to familiarize accounting
educators with item-writing “rules� (commonly accepted guidelines intended to
achieve validity); and (2) to provide evidence on the relative rule compliance of
test bank questions and CPA exam questions (“quality�). We selected random
samples of 100 multiple-choice questions from each source and employed
judges to evaluate them for rule compliance. Because not all rules apply to all
questions, we categorize the questions into eight types based on their format
(question versus completion), numerical calculation requirements, and the
presence or absence of words in the responses. The test bank questions showed
consistently lower quality than CPA exam questions across the six main question
types. Violations were about 4.6 times more frequent for test bank questions
(6.42% versus 1.41%). The analysis indicates that some types of questions are
especially prone to rule violation.
evaluation method. In practice, however, instructors may prepare or select
multiple-choice questions based on content and without adequate attention to
item quality. This paper has two specific objectives: (1) to familiarize accounting
educators with item-writing “rules� (commonly accepted guidelines intended to
achieve validity); and (2) to provide evidence on the relative rule compliance of
test bank questions and CPA exam questions (“quality�). We selected random
samples of 100 multiple-choice questions from each source and employed
judges to evaluate them for rule compliance. Because not all rules apply to all
questions, we categorize the questions into eight types based on their format
(question versus completion), numerical calculation requirements, and the
presence or absence of words in the responses. The test bank questions showed
consistently lower quality than CPA exam questions across the six main question
types. Violations were about 4.6 times more frequent for test bank questions
(6.42% versus 1.41%). The analysis indicates that some types of questions are
especially prone to rule violation.