Internal Model Oversight Framework - Guideline (2019)

Consultation status: Closed

Consultation closed August 30, 2019. We'll keep this draft on the site until the final guideline is released.

This guideline outlines OSFI's expectations for insurersFor the purposes of this guideline, insurers refers to federally regulated property and casualty insurance companies that are not mortgage insurance companies and Canadian branches of foreign property and casualty insurance companies. when establishing and maintaining an oversight framework with policies and procedures that identify, assess, and manage risks of internal models used to determine regulatory capital requirements in accordance with the OSFI Minimum Capital Test (MCT) guideline (internal model).

1.0 Scope of Application

This guideline applies to insurers that have received approval to use an internal model.This guideline does not apply to third party earthquake models used for regulatory capital purposes or to internal models used for other than regulatory capital purposes. An internal model developed by an insurer and used for determining MCT regulatory capital requirements captures the risks faced by the particular insurer more precisely than a non-customized standard approach. Insurer-specific elements of the model could include model inputs, model form, modeling technique or choice of parameters.

Insurers should satisfy the expectations of this guideline on an ongoing basis and demonstrate compliance upon request.Failure to satisfy these expectations on an ongoing basis may result in supervisory action, which could include a temporary requirement to hold additional capital or withdrawal of approval to use an internal model. The withdrawal of approval would require the insurer to use (revert to) the standard approach for determining its regulatory capital requirements.

2.0 Model Oversight Framework

2.1 General

An insurer should align the oversight framework surrounding the use of internal models, as appropriate, within its broader corporate governance framework.<p>OSFI's <a href="/node/592"><em>Corporate Governance Guideline</em></a> articulates OSFI's principles and expectations with respect to corporate governance of institutions.</p> The model oversight framework should articulate, through policies and procedures, how the insurer identifies and manages internal model risk.

Internal model risk is the risk of adverse financial (e.g., capital, losses, revenue) and reputational consequences arising from the design, development, implementation and/or use of an internal model. It can originate from, among other things, inappropriate specifications; incorrect parameter estimates; flawed hypotheses and/or assumptions; mathematical computation errors; inaccurate, inappropriate or incomplete data; inappropriate, improper or unintended usage; and inadequate monitoring and/or controls.

OSFI expects the oversight framework to include, among other things:

1. clearly defined roles and responsibilities, including their powers and authority;
2. separate key roles and responsibilities in order to preclude conflicts of interest;
3. separate production and testing environments, with different staff in each;
4. verification of the integrity of model results and their appropriate use; and
5. periodic reviews to assess compliance with established policies and procedures.

The insurer should have an oversight framework that covers internal model data and each of the following internal model life cycle phasesAn insurer's model process may have additional phases or sub-phases, wherein some of the expected policies and procedures may apply.: initial development or subsequent modification, objective vetting, approval or rejection, ongoing and objective validation, and decommissioning.

2.2 Documentation

Insurers should document their internal model oversight framework. The documentation should include:

1. Roles and responsibilities – individuals within the insurer responsible for verifying that policies and procedures related to control risks in the development, change and use of the model are functioning well;
2. Objective vetting and validation – includes a description of the objective vetting and objective validation processes and the evidence of their performance; and
3. Findings and recommendations – includes findings that require further investigation, the manner in which issues should be resolved and the tracking and verification of changes made.

OSFI expects insurers with internal models to review and update their documentation on a regular basis so it is current, accurate and complete.

3.0 Assessment of Oversight Framework

To assess that the policies and procedures established under the oversight framework are operating as intended, an insurer should implement a process to verify on an ongoing and periodic basis that tasks are completed in accordance with the policies and procedures, hereafter referred to as the Internal Model Risk Control (IMRC) process. An IMRC process should be established for both data risk and internal model risk.

3.1 Data Risk

The IMRC process to assess the appropriateness, accuracy, completeness and timeliness of the data used in the internal model should include:

1. Data quality assessment - identification and verification of the features data should possess in order to produce credible estimates. Performing a fitness for use assessment of the data against business rules.
2. Data quality monitoring - regular and periodic monitoring and verification of data quality. This involves, in particular, monitoring the performance of systems and channels used to collect, store, transmit and process data.
3. Identification and resolution of problems/opportunities - timely identification and resolution of problems, including opportunities for making improvements in data processes (e.g. collection, storage and processing), with a goal to increasing the quality of existing and future data.
4. Identification of data limitations – identification of data limitations, given consideration of its nature, characteristics, quality, and updated for any unresolved quality problems.

3.2 Model Risk

The IMRC process to assess that the internal model risk policies and procedures are operating as intended should include the following elements:

1. an audit trail of actions related to the use and validation of the model;
2. user access controls to prevent unauthorized changes to the model;
3. objective validation and vetting of the model and its results;
4. an assessment of the controls over model risk; and
5. processes for tracking and resolving concerns, differences of opinion, issues and deficiencies in the model or its use, including any findings of the assessment of controls over model risk.

3.2.1 Risk Control Officer/Committee

As part of the IMRC process, an insurer should identify a person or a committee: the Risk Control Officer/Committee (RCO/C) who has the responsibility for the initial vetting that model control processes are effective and the ongoing objective validation that the internal model is working as intended. The RCO/C should be separate from both the business functions (e.g. underwriting and claims reserving) and the internal model development group.

In discharging its responsibilities, the RCO/C should challenge the model's appropriateness. The challenge function must be effective and must be able to elevate concerns to an appropriate level. The RCO/C should reside within the Canadian operations of the insurer and have sufficient authority and stature within the insurer to have any issues and deficiencies addressed in a timely and substantive manner. The RCO/C should report to an individual who is (a) separate from the business functions and the internal model development group, (b) not the model executiveDepending on the size and the complexity of the insurer, it may be acceptable for an insurer to combine the role of model executive and the RCO/C as long as there is no potential conflict of interest and objectivity is maintained., defined in section 3.2.2.3 below, and (c) a member of or have direct access to the Board of Directors or a committee thereof.

In discharging its vetting and validation responsibilities, the RCO/C can use the work of internal objective reviewers (e.g., at the parent or home office) and objective third party expert resources. An objective reviewer or expert should not be or have been responsible for or actively involved in developing, maintaining or using the internal model.

3.2.2 Model Phases

An insurer should subject each of the internal model life cycle phases to its IMRC process. The following describes elements that insurers should consider in each model phase.

3.2.2.1 Initial Development or Subsequent Modification

Prior to the development or material modification of an internal model, the relevant business area (e.g., internal model users) should identify an economic or business rationale for developing a new or revised internal model. For all new internal models and material modifications, the insurer should document the modelling choices, the information/evidence and other considerations used in making the decision, including an assessment of the suitability of the selection in relation to the intended purpose.

After deciding to proceed with a new or revised internal model, an insurer should document the process it intends to follow for model development. This should serve as a control tool and will aid other parties, including the RCO/C, in understanding the internal model/ modification. This will help, for example, in the construction of suitable benchmarks for comparison or for vetting of the internal model. The documentation should include:

1. the modelling techniques adopted;
2. any assumptions and approximations employed (including justifications and/or reasonability assessments for key assumptions, covering both judgmental and qualitative aspects);
3. the data sources and data proxies utilized; and
4. any relevant model weaknesses and limitations.

Insurers should articulate what constitutes a material internal model modification and establish a process for managing and documenting the modifications. This process should consider, for example: a series of controls governing authorizations to change internal model components; a record of validation sign-offs since the internal model inception; and a record of empirical test results to assess whether or not internal model results have changed. The process should identify the personnel permitted, or the authority needed, to make changes to the model. Change control and verification should prevent any divergence between the approved internal model and the one used in operation.

Modifications to an existing model may require OSFI approval before the insurer can use it to determine regulatory capital requirements.

3.2.2.2 Objective vetting

Vetting is an objective review of the theory underlying the model, the model assumptions and inputs, and any software required to put the model into production. The RCO/C should vetFor the purposes of this guideline, we distinguish the expression "vetting" from "validation". We use vetting to identify a discrete activity, occurring as a pre-defined step in a process (e.g., the creation of a new internal model or the making of material modifications of an existing internal model). In contrast, validation is an ongoing monitoring activity (e.g., ongoing assessment of model performance or related user processes). a new or materially revised (design or assumptions) internal model before it is used. The vetting process should constitute an effective challenge. It should also make an objective assessment on whether the internal model is sound and fit for its intended purposes. The vetting should review, among other things, the rationale and information supporting model development team's recommendation for model approval. The vetting process should also include:

1. Verification and assessment: includes checking that all documentation is current and available; assessing the selection of the internal model, or material modification choice, relative to other options; and, evaluating the three components – inputs, computation processes, and reporting processes – of the development process; and
2. Secondary assessment: includes assessing the appropriateness of the model and parameters to evaluate the model's predictive capacity over a range of assumptions; and identifying weaknesses and limitations.

When an insurer makes a material modification to an internal model, it should apply the same level of rigour to vetting the modification as that involved in vetting a new internal model.

The RCO/C should document the results from the internal model vetting process and make a separate objective recommendation for the approval/rejection of the model along with any conditions on usage.

3.2.2.3 Approval or Rejection

Insurers should not approve internal models for operational use without first undergoing an objective vetting process.

Insurers should have a well-defined and documented process for approving/rejecting requests for the use of internal models, including the identification of a model executive. The model executive is the individual and/or committee responsible for assessing the RCO/C's findings and recommendations and making a decision regarding the approval, use and/or limitation of use of any new model or changes to pre-existing models.

The model executive should not be responsible for the development of the internal model. In addition, there should be a clear separation between the model executive and,

1. the person(s), within the insurer, having authority to recommend that a particular internal model be used with particular assumptions or that changes to the internal model or assumptions be made; and
2. the RCO/C vetting the internal model.Depending on the size and the complexity of the insurer, it may be acceptable for an insurer to combine the role of model executive and the RCO/C as long as there is no potential conflict of interest and objectivity is maintained.

In its review, the model executive should assess the RCO/C's findings and recommendations and make a final determination with respect to the use and/or limitations of use of the new model or changes to the existing model.

3.2.2.4 Ongoing and Objective Validation

With the passage of time, developments (e.g., changes to markets, regulations, theoretical advancements, and insurers' policies) can alter the level of risk of an internal model. An insurer's ongoing and objective validation should consider these developments, re-examine the level of internal model risk and determine whether the model continues to perform as intended.

Validation constitutes a review to monitor model performance to confirm that the model remains fit for use and it is producing valid results. The ongoing and objective validation processes should include actions such as:

1. reaffirming the completeness of existing documentation,
2. reviewing the assumptions and data chosen as well as the effects of any modification.

Ongoing validation - Model users and developers are responsible for ongoing validation of the internal model.

Objective validation - The RCO/C is responsible for performing a periodic objective validation of the model. The RCO/C should determine whether all the prescribed steps in a particular process were performed and that the prescribed steps were performed properly, the model remains fit for use, the results were explained correctly and are consistent or contrasted with expectations, and that any tracked issues were addressed in a timely manner. In addition, the RCO/C should:

1. conduct benchmarking analysis;
2. re-examine any noted internal model limitations or documented weaknesses;
3. backtest the model results; and
4. perform sensitivity analysis.

The RCO/C should document its findings and recommendations and report any disagreements to senior management, including the model executive. The insurer should have a process to track and assess the resolution of past RCO/C findings and recommendations.

An objective validation should occur at a frequency that is consistent with internal assessments of model risk materiality. However, objective validations should occur at least annually.

3.2.2.5 Decommissioning

Insurers may decommission an internal model due to its poor performance or obsolescence.

Insurers should have policies and procedures, including documentation standardsDocumentation should be available at the time of model decommission., for decommissioning internal models. The process should include prior notification to OSFI and all other key stakeholders. Where it does not intend to replace a model, the insurer should document the reasons for not replacing the internal model.

3.3 Documentation

The IMRC process documentation should provide evidence that the insurer is complying with its data and model risk control policies and procedures and the occurrence of any exceptions/findings along with the actions being taken to correct deficiencies, if any.

IMRC documentation may include:

1. Systems – maintained electronically when internal models are accessed and when changes are made. This documentation includes records of error messages from internal model runs.
2. Process – includes instructions for how users should set parameters and run internal models. This documentation also describes internal model processes in place (such as sensitivity testing), how to perform these processes and the desired outputs for use in reporting to stakeholders.
3. Vetting and validation – includes a description of the vetting and validation processes, evidence of their performance, including findings that require further investigation, the manner in which issues should be resolved and the tracking and verification that changes were actually made.

OSFI expects insurers with internal models to review and update their documentation on a regular basis so it is current, accurate and complete.

4.0 Role of Internal Audit

Internal audit is an objective function within an insurer. It should promote effective internal model risk oversight and control. OSFI expects that insurers will establish general and specific requirements with respect to the periodic review of internal model oversight to assess compliance with established policies and procedures. To remain objective, individuals conducting the review and making the assessment should not have been involved in internal model development, its objective vetting/validation or use.

Internal audit should assess the overall adequacy of the model oversight framework, including its compliance with the data and model control policies and procedures, as well as the effectiveness of the IMRC process. The internal audit review and assessment should include:

1. Effective controls for model risk:
   • sufficiency and adequacy of model oversight framework, the data and model control policies and procedures, including the controls for each model phase, as applicable;
   • sufficiency and adequacy of controls and processes around internal model risk materiality; and,
   • sufficiency, adequacy and clarity of authorizations around the internal model change control process and that these are appropriately separated.
2. Effectiveness of the IMRC process:
   • determining whether vetting and validation work conducted by the RCO/C is objective, effective and occurring on schedule; and
   • determining whether the exception, issues and escalation reporting processes are consistent with established policies and procedures.
3. Documentation:
   • accuracy and completeness of documentation and reporting.