Getting it right: Crafting the CPA Exam

If you could assign a flavor to the questions on the Uniform CPA Examination, it would undoubtedly be pure vanilla.

Tales of adventure, romance, and violence can be found at the bookstore or in the movie theater, but not on the CPA Exam, which is carefully designed to avoid topics that could unnecessarily distract CPA candidates from the actual purpose of the exam.

“We try to give the candidates the opportunity to provide clear evidence of their mastery of accounting-related knowledge, skills, and abilities (KSAs), and we try to exclude factors that have nothing to do with those KSAs,” said Jessica Lucas, an associate manager for the AICPA Examinations team. “For example, we’re not trying to assess their knowledge of unusual vocabulary words unless those words are commonly used in the field of accounting or any of its specialized fields. So a word like opine, which is part of the everyday vocabulary of auditors, can and does appear on the exam; but a word like blandishment does not. We don’t test advanced reading skills either. Questions are designed to assess essential content only, with no embellishment. We never try to trick the candidates in any way. Each sentence is designed to be as straightforward as possible and to have one and only one meaning in context. The meaning has to be clear to all readers.”

The CPA Exam is one of the most respected professional licensure examinations in the world. What’s not so well known is that the rigor associated with exam development procedures is as intense as the review process that many candidates undertake in their efforts to earn a passing score. Each potential question undergoes scrutiny during multiple rounds of review by specialists evaluating it from different points of view. If the question survives those reviews, the next step is a pretest phase to perform a statistical analysis of actual test takers’ responses to the question. (Pretest items don’t count toward a candidate’s score.) If the question fails to meet statistical standards, it is rejected or revised.

As laws, regulations, and other types of guidance change, the exam also needs to change. It takes about eight months from the time the Examinations team identifies a need for more questions to the point when they are ready to be pretested on the exam; in certain cases, the time elapsed can be as short as four months.

In general, each question is designed to be a successful act of communication. Linguistic philosopher Paul Grice observed that successful, cooperative communications typically follow four basic rules:

• Be truthful.
• Be clear.
• Be relevant.
• Be appropriately informative.

The text of the CPA Exam obeys the rules and conventions of international standard English so that the questions will be easy for English speakers around the world to understand. Likewise, staff members use their knowledge of logic and linguistics to make sure that the questions assess only accounting content, not advanced skills in logic or linguistics.

“We design the text of each question to be easy to process. We avoid needlessly complicated wordings and logical structures,” said Timothy Habick, Ph.D., a test development manager for the AICPA Examinations team. “Now some people might wonder how logic could be an issue relevant to the CPA Exam in the first place, and that is the point. We purposely craft the text so that candidates will not need to use logic to understand the intended meaning of any sentence in context. Our goal is to create clear, coherent, and cooperative communications. This tends to make the text of our questions somewhat unremarkable, but that is our intention.

“Candidates are supposed to focus on the essential accounting content, not on our use of language. For the same reason, we tend to avoid using names for individuals and companies that could attract the candidates’ attention. So whenever possible, we refer to ‘a taxpayer,’ ‘an auditor,’ or ‘a company’ without using realistic names. Proper names can contain allusions to irrelevant historical events and individuals, which could be distracting. We’re here to assess the candidates’ accounting-related skills, not to entertain them. We try to include only content that the candidates need to demonstrate what they know. Everything else is removed in order to create a smooth testing experience.”

THE PROCESS

The drafting of a CPA Exam question begins with the CPA Exam Blueprints, which contain detailed information on all the subjects that have been specified for use on the exam, as well as the level of understanding of the material that may be tested. The Blueprints are based on surveys of practicing CPAs regarding the knowledge that newly licensed CPAs should possess. The AICPA maintains a bank of over 10,000 approved questions, classified according to topic and various statistical features. The bank is kept full by external subject-matter experts. The volunteers are experienced CPAs who have been trained to write exam questions in their areas of expertise. All volunteers are carefully screened and are bound by a confidentiality agreement.

As these experts write questions, they work with AICPA staff and mentors who review questions for fairness and test development principles (see the sidebar “Exam Questions: What Are the Rules?”). Once satisfied, they send the questions to a review group composed of content experts with knowledge of specialized areas. The review group votes to accept, reject, or revise each question. The questions that are accepted then receive additional extensive internal reviews from AICPA staff members, who ensure that questions assess topics and skills specified in the Blueprints and review the questions for technical accuracy, readability, clarity, logic, fairness, and simplicity of wording.

“When you consider how much time, research, and care goes into every single question on the exam, it is quite amazing,” Lucas said.

TRIAL RUN

Once the questions pass the internal review, they are ready to be tested by actual candidates. They are first included in the exam as pretest questions, which have no effect on candidate scoring. There is no noticeable difference to candidates taking the exam between pretest questions and questions that count toward the calculation of scores. The statistical performance of a pretest question on the exam provides test developers with valuable information, which is used to further ensure the fairness of the exam.

“We might have thought that a question was perfect in terms of its carefully constructed wordings and content, but if you put it in front of 1,000 candidates and there’s still some problem with it, the statistical results of that trial run will generally highlight the issue,” said Renée Warias, senior manager for the AICPA Examinations team.

An ideal question will be answered correctly by a large percentage of candidates of high ability (as determined by their performance on the test as a whole) and will be answered incorrectly by a large percentage of candidates with lower demonstrated ability. When that situation occurs, the question plays a useful role in distinguishing between candidates who have the required accounting skills and those who do not.

“If 99% of the candidates answer a particular question correctly, the question does not provide useful information about the candidates’ relative abilities,” Warias said. “The same situation occurs if only 5% answer correctly. The question is either too easy or too difficult to provide useful information.”

If the metrics on a pretest question meet statistical standards, the question is sent to a committee of CPAs for a final vote on whether it merits a place on the exam. Questions that receive final approval are added to the bank of questions that can be selected for use as a scored item on the CPA Exam.

HIGH-STAKES COMMUNICATIONS

Throughout the development process, the Examinations team treats exam questions as “high-stakes communications” because there is a lot on the line with professional certification exams. Unclear sentences or wordings in questions could prevent qualified candidates from passing the exam. That’s why the development and review process for CPA Exam questions is so painstaking.

“The high stakes of the CPA Exam mean that we have a duty to do our jobs perfectly,” Habick said. “We are not aiming for 99%. No. It has to be 100% perfect, and there can be nothing wrong with the questions by the time they’re ready for the candidates. That’s our duty, and we take it seriously.”

Managing the whole question-development process is a complicated endeavor. It is a huge job to recruit, train, and manage the external subject-matter experts — whose expertise and willingness to help are extremely valuable. Managing the “bank” of test questions is complicated, too. It’s not a good idea to keep a reserve of too many questions because the ever-changing regulations and accounting standards could render them obsolete before they actually get used on the exam. At the same time, moving questions through a multistep review process can be time-consuming.

Nonetheless, getting it right can be extremely rewarding — providing a pathway for qualified candidates to serve the public as CPAs.

Exam questions: What are the rules?

The rules and principles that guide the creation of CPA Exam questions are meant to ensure fairness to CPA candidates and an effective evaluation of their skills. Here are some of the rules and principles that guide the creation of the exam:

• Multiple-choice answer options are written in grammatically parallel structures. If one answer option is a complete sentence, then all of them are complete sentences. Using parallel structures in this way makes the questions easier to process.
• Questions are designed to be comprehensible by all speakers of standard English, regardless of their regional or social dialects. Care also is taken to make the language accessible to speakers of major dialects of international standard English because the exam is offered at testing centers throughout the world.
• Humor, irony, and satire are not included in exam questions.
• Political points of view and commercial agendas are not promoted in questions. Although mention of religious institutions is sometimes necessary to test understanding of the tax treatment of charitable contributions or the status of certain charities under the Internal Revenue Code, exam questions do not refer to particular religions.
• Situations involving suffering, war, weaponry, or acts of violence are not mentioned on the exam because they could engender unpleasant emotions. If a reference to an accident is necessary in a question to test an accounting concept, only accounting-relevant details about the accident are mentioned.

“Of course, we don’t know the candidates personally, but we still need to communicate effectively with them,” said Timothy Habick, Ph.D., a test development manager for the AICPA Examinations team. “We have no idea what dialect they speak, what their worldview might be, or what personal issues they might be experiencing on the testing date. All we know is that they were approved to take the CPA Exam. So our job is to make their testing experience as smooth, fair, and effective as possible.”

About the author

Ken Tysiac is the JofA’s editorial director. To comment on this article or to suggest an idea for another article, contact him at Kenneth.Tysiac@aicpa-cima.com or 919-402-2112.