Managing the risks associated with models

Models are everywhere. Many organizations use models for facilitating the decision-making process, for accounting and tax purposes, and for managing day-to-day operations. In fact, organizations are increasingly running their businesses with models for a range of life cycle purposes, such as valuation, loan decisions, inventory management, reserving, acquisitions, customer behavior, and other financial and nonfinancial decisions. It’s important for accountants to understand the model-use environment in their organizations because models can significantly affect the financial statements.

Managing “model risk,” which is the risk of financial loss, erroneous financial statements, improper managerial decisions, or damaged organizational reputation, resulting from poorly built, used, or controlled models, can be costly and challenging. For example, if the input data or assumptions used in an allowance model for estimating reserves are incomplete or inaccurate, the resulting model output could cause an accounting error. Alternatively, if the model theory underpinning the credit valuation adjustment portion of a derivative instrument’s fair value estimate is unsound, the financial statements and fair value footnote could be misstated.

Model risk management (MRM) is a branch of risk management that addresses these concerns. It is a structured approach that defines roles, responsibilities, policies, procedures, and controls to mitigate the potential adverse impact of the model-use environment. It was brought into sharper focus in the past decade, as unmitigated model risk associated with collateralized debt obligations and associated default swaps may have contributed to the 2008 financial crisis, and because regulators have indicated that models were at least partially to blame for other widely publicized financial institution failures.

Regulatory guidance applicable to the banking sector and some insurance companies issued by the Office of the Comptroller of the Currency and the Federal Reserve, such as SR 11-7, Supervisory Guidance on Model Risk Management, has precipitated the recent evolution of MRM leading practices. However, while financial services organizations have established the most formally implemented MRM programs, the lessons learned and practices developed are relevant for other sectors. For instance, the principles that drive the control activities that manage the risk associated with a bank’s credit risk models or an insurance company’s pricing or reserving models can be applied to the controls and risk management activities related to a retail company’s predictive analytics model for customer behavior, a manufacturer’s supply chain model, or an oil and gas company’s price optimization or demand forecast models. Moreover, the accounting life cycle often depends on models used across industry lines, such as tax, goodwill impairment, fair value, and reserving models.

MRM FUNDAMENTALS

Oversight by the board of directors and involvement of senior management serve as the linchpin in a sound MRM environment. Their ownership and involvement in critical MRM decisions, such as those that could materially impact the organization, set the tone that MRM is taken seriously by the highest levels of the organization. Defining roles and responsibilities, outlining mandates and guidance within formal policies and procedures, and establishing internal controls to mitigate model risk are the initial steps in establishing the foundation of an MRM governance framework. Many organizations that have implemented formal MRM programs use a “three lines of defense” approach to managing model risk, where organizational stakeholders are tasked with checks and balances to reduce model risk.

First line of defense

Model developers, who often are also the line-of-business model users, represent the first line of defense and are responsible for developing models based on formal internal standards and requirements; testing the models, their inputs, and implementation; documenting the theoretical underpinnings of the models and the rationale for choices regarding inputs and assumptions; and monitoring the models’ performance once they are in use. Off-the-shelf models, which are purchased from vendors and then customized for the company’s model-use environment, are typically assigned to a model developer who is responsible for performing a similar set of procedures as for an internally developed model. As such, model developers are often the first ones in the organization to identify model nonperformance and the respective potential impacts to the financial statements.

Second line of defense

Before use, and on a rotational basis thereafter, models are generally subject to validation by an independent party, called a “validator,” with sufficient independence from the developer. Procedures are usually performed by an individual within the organization’s risk function to effectively challenge the conceptual soundness and overall quality of the model, including the use of its outputs. Sample testing or replication of model output is performed depending on the model’s riskiness.

Validators accept or reject models and can identify issues that impact the financial statements. In addition to performing validations, the second line of defense is also typically responsible for formally defining what is considered a model for the enterprise (e.g., as opposed to a tool or calculation) and maintaining an inventory of models subject to governance and validation. Since the second line of defense is usually centralized within an organization, it also generally creates and distributes reporting and analytics packages to the board of directors and senior management, who analyze the amount of model risk the organization has, individually and in the aggregate.

Third line of defense

Internal audit generally serves as the third line of defense and is tasked with confirming that proper attention is dedicated to MRM by the board and senior management and that a sound MRM framework has been established and implemented. Through routine and ad hoc inspections, internal audit usually has specific responsibilities focused on the stability of the MRM program as a whole and the foundational components such as the existence of clearly defined roles and responsibilities and policies and procedures. Internal audit’s primary focus may be summarized in three categories:

• Independence: Confirmation that an appropriate level of separation exists between model developers and validators.
• Program assessment: Inspection of the elements of the MRM program to determine if critical control features are designed and implemented (such as the existence of an accurate model inventory).
• Communication: Serve as a “tie-breaker” between the first and second lines of defense when disagreements arise, and facilitate open communication with the board and its committees for matters of critical MRM importance.

GETTING INVOLVED IN MRM

Accountants can contribute significantly to the overarching goals of MRM and help to mitigate the risk of financial misstatement by integrating themselves into the MRM framework and understanding the methods of measuring model risk:

Integrate into the MRM framework: There are many ways for accountants to identify potential model risk that may impact the financial statements. Studying the model inventory and classifying models that impact the financial statements is an easy first step. This can be performed in conjunction with a broader internal controls over financial reporting (ICFR) framework. If a model inventory does not yet exist, consider creating an inventory of models that potentially impact the financial statements and an ongoing process to keep the inventory up-to-date. This will likely require detailed conversations with business leaders from across the organization. Moreover, if a model is the source of a significant journal entry, then an accountant can proactively obtain the model’s development documentation, or have a detailed conversation with stakeholders closest to the design, implementation, and use of the model, to gain a deeper understanding of known model limitations or identified risk, and the modeler’s confidence in the model outputs. This analysis can also be performed in conjunction with activities designed and implemented for ICFR.

Additionally, organizations often prepare and distribute standardized and ad hoc reporting packages to board members associated with models in use, and the related levels of model risk. Even if specific MRM reports aren’t distributed, other reports to the board’s committees can provide valuable insight into where models are being used. By reading these reporting packages, accountants can identify potential model risk to the financial statements and tailor specific activities, such as contacting model owners for additional details or increasing the monitoring of accounting balances or reporting disclosures generated by the related models.

Understand available processes for measuring model risk: Model risk exists for several reasons. For instance, data or assumptions may be questionable, mathematical approaches unsound, or uses limited to certain scenarios. Once model risk is identified, however, it is important to quantify what the impact may be to the financial statements, if any, so a determination can be made with respect to the appropriateness of an adjustment to the books and records. For example, it might be considered appropriate to record a reserve when a nonperforming model is identified. Several techniques can be used to measure model risk:

• Sensitivity analysis: Requesting that a modeler perform sensitivity analyses, whereby the model is run with different input/scenario combinations for a given variable or model feature, is a classic approach to quantifying model risk when a specific variable or feature of the model is known to include uncertainty. This approach provides a narrow estimate of potential model risk since it assumes the remaining aspects of the model are relatively certain. When multiple variables or features of the model include uncertainty, a modeler can determine the upper and lower bounds of model output for use as a gauge of potential risk in model output.
• Challenger models: Developing a “challenger model” is another approach to measuring potential model risk. With this technique, a new model is developed, sometimes on the basis of an alternative model theory or different data set, for comparison to the “champion model” (i.e., the primary model being used). The unexplainable difference may reflect the quantified model risk.
• Analytical procedures: Performing a high-level analytical procedure that compares model output to an intuitive expectation can sometimes be used as a technique for quantifying model risk. This approach should be used only by people who have developed an intuitive understanding of the output being modeled, in the context of its related business circumstance.

APPLYING THE ACCOUNTANT’S PERSPECTIVE

The risk of financial loss, erroneous financial statements, inappropriate decisions, or damaged organizational reputation due to poorly built, used, or controlled models is one that many organizations face regardless of their industry, size, or ownership structure. Great strides in MRM have been made recently by financial services organizations that can be used by model-dependent organizations in any industry. By establishing MRM programs that include roles and responsibilities, governance oversight, and checks-and-balances practices, model risk can be managed in a structured and effective manner.

Accountants can play an integral role in managing model risk. They can gain a deeper understanding of the models that affect the financial statements and serve as a catalyst to measuring and recording needed adjustments to the financial statements. Preparing financial statements affords accountants a broad-based visibility across the organization, unique to their function. It is this view that allows accountants to play such a meaningful role in facilitating model risk management.

About the author

Clifford D. Goss (clifforddeangoss@yahoo.com) is a professor of accounting and economics and has served clients with attest and advisory services for 13 years.

To comment on this article or to suggest an idea for another article, contact Neil Amato, senior editor, at Neil.Amato@aicpa-cima.com or 919-402-2187.

AICPA resources

Articles

• “Money Laundering: Combating a Global Threat,” Sept. 2016
• “Becoming a Strategic Risk Adviser,” Feb. 2015