A Comparison of the Quality of Multiple-Choice Questions from CPA Exams and Textbook Test Banks

Abstract

Multiple-choice test questions have substantial support as a reliable and valid evaluation method. In practice, however, instructors may prepare or select multiple-choice questions based on content and without adequate attention to item quality. This paper has two specific objectives: (1) to familiarize accounting educators with item-writing “rules� (commonly accepted guidelines intended to achieve validity); and (2) to provide evidence on the relative rule compliance of test bank questions and CPA exam questions (“quality�). We selected random samples of 100 multiple-choice questions from each source and employed judges to evaluate them for rule compliance. Because not all rules apply to all questions, we categorize the questions into eight types based on their format (question versus completion), numerical calculation requirements, and the presence or absence of words in the responses. The test bank questions showed consistently lower quality than CPA exam questions across the six main question types. Violations were about 4.6 times more frequent for test bank questions (6.42% versus 1.41%). The analysis indicates that some types of questions are especially prone to rule violation.