Video transcript:

John Mattar, AICPA Director of Psychometrics and Research

What we do in psychometrics on the exams team is really divided into three main areas. First, we want to ensure that we’ve properly defined what’s going to be assessed on the exam, and there’s a lot of professional standards and best practices in what we call practice analysis to define what’s going to be assessed on any test, especially a licensure test.

Secondly, we look at empirical and qualitative evidence that shows that the test questions that are written and administered are measuring what’s been defined for measurement. So, if in a practice analysis we find out we want to measure this content and these type of skills, we have to gather a lot of additional evidence that the questions we write are actually doing that.

And then the third phase of what we do is we look at specifically the performance of all the candidates that take the test and we analyze their results to find out how things are performing, are questions are performing as we think they should?  Is there anything that looks askew? And we then have to score the questions and provide candidates their scores.
 

Josh Stopek, CPA, AICPA Technical Manager for Research

As part of getting new concepts and new ideas onto the exam, we do a lot of research once we’ve created some prototype items to study how candidates are going to perform on those. We have an extensive network that we’ve been building over the last few years of universities and firms to provide us with that type of information.

So, we create these prototype items and we send them over to the universities, and to the firms, and say, “Okay. Show us how you can do on them.” And through that, we get to understand how those items are performing, how long these things take, and “Are the candidates thinking about them the way we want them to?”

So, they provide us with information about whether they’re realistic simulations, about whether the directions make sense, about whether the technology makes sense. And this could be whether the technology is new or whether the format is new. We do research regardless of whether it’s a new technology or not, because anything that’s new and different we want to ensure that we’re creating the most appropriate content and being as fair as possible to every candidate.
 

Rich Gallagher, CPA, CPA Exam Director of Content

I’ve been on board with the AICPA for just a little bit over a year now, and one of the things I’ve learned about the exam is the efforts that the AICPA takes in maintaining the exam as a very fair assessment of the knowledge and skills of candidates.

The number of people who are involved in ensuring both the quality of the questions from a content and technological perspective, as well as beyond that; looking even linguistically to make sure that there’s nothing that’s going to trick a candidate as they read a question—the efforts that are undertaken in this regard are absolutely amazing.