The role of the Model Validation function to manage and mitigate model risk

Alberto Elices

DOI: https://doi.org/10.48550/arXiv.1211.0225

2012-10-21

Risk Management

Abstract:This paper describes the current taxonomy of model risk, ways for its mitigation and management and the importance of the model validation function in collaboration with other departments to design and implement them.

What problem does this paper attempt to address?

Model Validation Practice in Banking: A Structured Approach for Predictive Models

Agus Sudjianto,Aijun Zhang

2024-11-11

Abstract:This paper presents a comprehensive overview of model validation practices and advancement in the banking industry based on the experience of managing Model Risk Management (MRM) since the inception of regulatory guidance SR11-7/OCC11-12 over a decade ago. Model validation in banking is a crucial process designed to ensure that predictive models, which are often used for credit risk, fraud detection, and capital planning, operate reliably and meet regulatory standards. This practice ensures that models are conceptually sound, produce valid outcomes, and are consistently monitored over time. Model validation in banking is a multi-faceted process with three key components: conceptual soundness evaluation, outcome analysis, and on-going monitoring to ensure that the models are not only designed correctly but also perform reliably and consistently in real-world environments. Effective validation helps banks mitigate risks, meet regulatory requirements, and maintain trust in the models that underpin critical business decisions.

Computers and Society,Applications
Model Validation Via Uncertainty Propagation and Data Transformations

W Chen,L Baghdasaryan,T Buranathiti,J Cao

DOI: https://doi.org/10.2514/1.491

IF: 2.624

2004-01-01

AIAA Journal

Abstract:Model validation has become a primary means to evaluate accuracy and reliability of computational simulations in engineering design. Because of uncertainties involved in modeling, manufacturing processes, and measurement systems, the assessment of the validity of a modeling approach must be conducted based on stochastic measurements to provide designers with confidence in using a model. A generic model validation methodology via uncertainty propagation and data transformations is presented. The approach reduces the number of physical tests at each design setting to one by shifting the evaluation effort to uncertainty propagation of the computational model. Response surface methodology is used to create metamodels as less costly approximations of simulation models for the uncertainty propagation. Methods for validating models with both normal and nonnormal response distributions are proposed. The methodology is illustrated with the examination of the validity of two finite element analysis models for predicting springback angles in a sample flanging process.
Model Risk in Credit Portfolio Models

Christian Meyer

DOI: https://doi.org/10.48550/arXiv.2111.14631

2021-11-23

Risk Management

Abstract:Model risk in credit portfolio models is a serious issue for banks but has so far not been tackled comprehensively. We will demonstrate how to deal with uncertainty in all model parameters in an all-embracing, yet easy-to-implement way.
Measuring the model risk-adjusted performance of machine learning algorithms in credit default prediction

Andrés Alonso Robisco,José Manuel Carbó Martínez

DOI: https://doi.org/10.1186/s40854-022-00366-1

IF: 6.793

2022-07-14

Financial Innovation

Abstract:Implementing new machine learning (ML) algorithms for credit default prediction is associated with better predictive performance; however, it also generates new model risks, particularly concerning the supervisory validation process. Recent industry surveys often mention that uncertainty about how supervisors might assess these risks could be a barrier to innovation. In this study, we propose a new framework to quantify model risk-adjustments to compare the performance of several ML methods. To address this challenge, we first harness the internal ratings-based approach to identify up to 13 risk components that we classify into 3 main categories—statistics, technology, and market conduct. Second, to evaluate the importance of each risk category, we collect a series of regulatory documents related to three potential use cases—regulatory capital, credit scoring, or provisioning—and we compute the weight of each category according to the intensity of their mentions, using natural language processing and a risk terminology based on expert knowledge. Finally, we test our framework using popular ML models in credit risk, and a publicly available database, to quantify some proxies of a subset of risk factors that we deem representative. We measure the statistical risk according to the number of hyperparameters and the stability of the predictions. The technological risk is assessed through the transparency of the algorithm and the latency of the ML training method, while the market conduct risk is quantified by the time it takes to run a post hoc technique (SHapley Additive exPlanations) to interpret the output.

social sciences, mathematical methods,business, finance
Practice-relevant model validation: distributional parameter risk analysis in financial model risk management

Mark Cummins,Fabian Gogolin,Fearghal Kearney,Greg Kiely,Bernard Murphy

DOI: https://doi.org/10.1007/s10479-022-04574-x

IF: 4.82

2022-03-04

Annals of Operations Research

Abstract:Abstract An objective of model validation within organisations is to provide guidance on model selection decisions that balance the operational effectiveness and structural complexity of competing models. We consider a practice-relevant model validation scenario where a financial quantitative analysis team seeks to decide between incumbent and alternative models on the basis of parameter risk. We devise a model risk management methodology that gives a meaningful distributional assessment of parameter risk in a setting where market calibration and historical estimation procedures must be jointly applied. Such a scenario is typically driven by data constraints that preclude market calibration only. We demonstrate our proposed methodology in a natural gas storage modelling context, where model usage is necessary to support profit and loss reporting, and to inform trading and hedging strategy. We leverage our distributional parameter risk approach to devise an accessible technique to support model selection decisions.

operations research & management science
Lessons From Model Risk Management in Financial Institutions for Academic Research

Mahmood Alaghmandan,Olga Streltchenko

2024-06-21

Abstract:In this paper, we discuss aspects of model risk management in financial institutions which could be adopted by academic institutions to improve the process of conducting academic research, identify and mitigate existing limitations, decrease the possibility of erroneous results, and prevent fraudulent activities.

Risk Management
AI model risk: What the current model risk management framework can teach us about managing the risks of AI models

Catarina Souza

DOI: https://doi.org/10.69554/sokx4074

2023-01-01

Abstract:The rapid adoption of Artificial Intelligence (AI) among financial institutions in recent years creates several opportunities, but also presents significant risks that require adequate risk management. Despite advances in recent years, AI regulation remains fragmented. This creates a challenge for financial institutions when looking for guidance on how to address the emerging risks presented by the use of AI. Given the complexity and speed of revision, AI models tend to propagate and amplify existing model risk. This grants them the potential to be more harmful, and raises important model ethics concerns. This paper discusses how the existing model risk management framework can offer important lessons for financial institutions on how to tackle these emerging risks. Additionally, the paper explores possible enhancements to the model risk management framework in order to address the unique challenges posed by AI models. These include adapting governance and policies, including model ethics considerations; enhancing model risk identification and classification; and updating model life cycles, with an emphasis on data management, model development, validation and monitoring. While the author agrees that AI risks are diverse in nature, the focus of the paper is on the risks derived from the use and development of AI models.
Overview of model validation for survival regression model with competing risks using melanoma study data

Zhongheng Zhang,Giuliana Cortese,Christophe Combescure,Roger Marshall,Minjung Lee,Hyun Lim,Bernhard Haller

DOI: https://doi.org/10.21037/atm.2018.07.38

IF: 3.616

2018-01-01

Annals of Translational Medicine

Abstract:The article introduces how to validate regression models in the analysis of competing risks. The prediction accuracy of competing risks regression models can be assessed by discrimination and calibration. The area under receiver operating characteristic curve (AUC) or Concordance-index, and calibration plots have been widely used as measures of discrimination and calibration, respectively. One-time splitting method can be used for randomly splitting original data into training and test datasets. However, this method reduces sample sizes of both training and testing datasets, and the results can be different by different splitting processes. Thus, the cross-validation method is more appealing. For time-to-event data, model validation is performed at each analysis time point. In this article, we review how to perform model validation using the riskRegression package in R, along with plotting a nomogram for competing risks regression models using the regplot() package.
Operational Risk Working Party - Validating Operational Risk Models

Patrick O. J. Kelliher,Madhu Acharyya,Andrew J. Couper,Edward N. V. Maguire,Choong A. Pang,Christopher M. Smerald,Jennifer K. Sullivan,Paul M. Teggin

DOI: https://doi.org/10.1017/s1357321723000168

2024-01-16

British Actuarial Journal

Abstract:Operational Risk is one of the most difficult risks to model. It is a large and diverse category covering anything from cyber losses to mis-selling fines; and from processing errors to HR issues. Data is usually lacking, particularly for low frequency, high impact losses, and consequently there can be a heavy reliance on expert judgement. This paper seeks to help actuaries and other risk professionals tasked with the challenge of validating models of operational risks. It covers the loss distribution and scenario-based approaches most commonly used to model operational risks, as well as Bayesian Networks. It aims to give a comprehensive yet practical guide to how one may validate each of these and provide assurance that the model is appropriate for a firm's operational risk profile.
A New Validation Metric for Models with Correlated Responses Using Limited Experimental and Simulation Data

Weifei Hu,Tongzhou Zhang,Zhenyu Liu,Jin Cheng,Yukun Lou,Xiaoyu Deng,Jianrong Tan

DOI: https://doi.org/10.1115/1.4054877

2022-01-01

Journal of Mechanical Design

Abstract:Model validation is the process of determining the degree to which a model is an accurate representation of the real object. Most of the existing model verification metrics rely on massive data, which are expensive to obtain in complex engineering problems. This paper first proposes a new enhanced Mahalanobis distance (EMD) metric by multiplying the original Mahalanobis distance with a direction angle to incorporate the correlation information using limited experimental and simulation data. Combining with EMD, an angle metric is developed as an alternative of the area metric to reduce the misjudgment rate of model validation. In order to quantify the uncertainty due to insufficient experimental and simulation data, the angle metric is further extended to a new interval angle metric, namely the interval EMD-pooling angle metric as the ultimate metric, for validating models at multiple sites. The proposed interval EMD-pooling angle metric is compared with other existing metrics through three numerical case studies to demonstrate its advantages when both experimental and simulation data are insufficient. An engineering example regarding ultrasonic welding is also provided to demonstrate the effectiveness of the proposed metric for practical model validation problems.
Assessing financial model risk

Pauline Barrieu,Giacomo Scandolo

DOI: https://doi.org/10.1016/j.ejor.2014.10.032

IF: 6.4

2015-04-01

European Journal of Operational Research

Abstract:Model risk has a huge impact on any risk measurement procedure and its quantification is therefore a crucial step. In this paper, we introduce three quantitative measures of model risk when choosing a particular reference model within a given class: the absolute measure of model risk, the relative measure of model risk and the local measure of model risk. Each of the measures has a specific purpose and so allows for flexibility. We illustrate the various notions by studying some relevant examples, so as to emphasize the practicability and tractability of our approach.

operations research & management science
Minimise model risk management oversight for cyber security solutions

Liming Brotcke

DOI: https://doi.org/10.69554/xfxm3841

2023-09-01

Abstract:Adoption of artificial intelligence (AI) and machine learning (ML)-powered cyber security tools and models by financial institutions has received considerable attention in the model risk management community. In parallel, developing trustworthy AI that is more explainable, fair, robust, private and transparent has also received considerable research and regulatory attention. Appropriate governing of cyber security models is inevitable. The prevailing thought at present is to have the model risk management function to oversee the development, implementation and use of such cyber security tools and models. This study first demonstrates two primary challenges of executing this oversight and then offers a few practical suggestions to ensure a reasonable application.
The Dali Model in Risk-Management Practice: The Case of Financial Services Firms

M. Kizilkaya,Rebecca Dalli Gonzi,S. Grima,Jonathan Spiteri

DOI: https://doi.org/10.20944/preprints201910.0047.v1

2019-10-04

Journal of Risk and Financial Management

Abstract:Originality/value—this model contributed to the vast literature on models of change and risk management within organisations, but was not validated empirically for reliability of the factors, and on financial services providers within small jurisdictions. Therefore, the significance and importance of such a study lies firstly on the premise that testing on small countries can be deemed as small laboratories for more complex politics, regulations and policies of larger countries and secondly, the importance of financial services as essential for prosperity in a country’s economy. This model will provide an empirically tested proactive model in a specific environment for managing organisational risks to arrive at their objectives with minimal setbacks.

Business,Economics
A Novel Approach to Quantification of Model Risk for Practitioners

Zuzana Krajcovicova,Pedro Pablo Perez-Velasco,Carlos Vazquez

DOI: https://doi.org/10.48550/arXiv.1705.05572

2017-05-16

Abstract:Models continue to increase their already broad use across industry as well as their sophistication. Worldwide regulation oblige financial institutions to manage and address model risk with the same severity as any other type of risk, which besides defines model risk as the potential for adverse consequences from decisions based on incorrect and misused model outputs and reports. Model risk quantification is essential not only in meeting these requirements but for institution's basic internal operative. It is however a complex task as any comprehensive quantification methodology should at least consider the data used for building the model, its mathematical foundations, the IT infrastructure, overall performance and (most importantly) usage. Besides, the current amount of models and different mathematical modelling techniques is overwhelming. Our proposal is to define quantification of model risk as a calculation of the norm of some appropriate function that belongs to a Banach space, defined over a weighted Riemannian manifold endowed with the Fisher--Rao metric. The aim of the present contribution is twofold: Introduce a sufficiently general and sound mathematical framework to cover the aforementioned points and illustrate how a practitioner may identify the relevant abstract concepts and put them to work.

Risk Management
Model Validation in Ontology Based Transformations

Jesús M. Almendros-Jiménez,Luis Iribarne

DOI: https://doi.org/10.48550/arXiv.1210.6111

2012-10-23

Logic in Computer Science

Abstract:Model Driven Engineering (MDE) is an emerging approach of software engineering. MDE emphasizes the construction of models from which the implementation should be derived by applying model transformations. The Ontology Definition Meta-model (ODM) has been proposed as a profile for UML models of the Web Ontology Language (OWL). In this context, transformations of UML models can be mapped into ODM/OWL transformations. On the other hand, model validation is a crucial task in model transformation. Meta-modeling permits to give a syntactic structure to source and target models. However, semantic requirements have to be imposed on source and target models. A given transformation will be sound when source and target models fulfill the syntactic and semantic requirements. In this paper, we present an approach for model validation in ODM based transformations. Adopting a logic programming based transformational approach we will show how it is possible to transform and validate models. Properties to be validated range from structural and semantic requirements of models (pre and post conditions) to properties of the transformation (invariants). The approach has been applied to a well-known example of model transformation: the Entity-Relationship (ER) to Relational Model (RM) transformation.
Validation of machine learning based scenario generators

Solveig Flaig,Gero Junike

2023-11-06

Abstract:Machine learning methods are getting more and more important in the development of internal models using scenario generation. As internal models under Solvency 2 have to be validated, an important question is in which aspects the validation of these data-driven models differs from a classical theory-based model. On the specific example of market risk, we discuss the necessity of two additional validation tasks: one to check the dependencies between the risk factors used and one to detect the unwanted memorizing effect. The first one is necessary because in this new method, the dependencies are not derived from a financial-mathematical theory. The latter one arises when the machine learning model only repeats empirical data instead of generating new scenarios. These measures are then applied for an machine learning based economic scenario generator. It is shown that those measures lead to reasonable results in this context and are able to be used for validation as well as for model optimization.

Risk Management
Benchmarking Quantitative Default Risk Models: A Validation Methodology

Roger M. Stein,S. Keenan,J. Soběhart

Abstract:WRITTEN CONSENT. All information contained herein is obtained by MOODY'S from sources believed by it to be accurate and reliable. Because of the possibility of human or mechanical error as well as other factors, however, such information is provided " as is " without warranty of any kind and MOODY'S, in particular, makes no representation or warranty, express or implied, as to the accuracy, timeliness, completeness, merchantability or fitness for any particular purpose of any such information. Under no circumstances shall MOODY'S have any liability to any person or entity for (a) any loss or damage in whole or in part caused by, resulting from, or relating to, any error (negligent or otherwise) or other circumstance or contingency within or outside the control of MOODY'S or any of its directors, officers, employees or agents in connection with the procurement, collection, compilation, analysis, interpretation, communication, publication or delivery of any such information, or (b) any direct, indirect, special, consequential, compensatory or incidental damages whatsoever (including without limitation, lost profits), even if MOODY'S is advised in advance of the possibility of such damages, resulting from the use of or inability to use, any such information. The credit ratings, if any, constituting part of the information contained herein are, and must be construed solely as, statements of opinion and not statements of fact or recommendations to purchase, sell or hold any securities. opinion must be weighed solely as one factor in any investment decision made by or on behalf of any user of the information contained herein, and each such user must accordingly make its own study and evaluation of each security and of each issuer and guarantor of, and each provider of credit support for, each security that it may consider purchasing, holding or selling. Pursuant to Section 17(b) of the Securities Act of 1933, MOODY'S hereby discloses that most issuers of debt securities (including corporate and municipal bonds, debentures, notes and commercial paper) and preferred stock rated by MOODY'S have, prior to assignment of any rating, agreed to pay to MOODY'S for appraisal and rating services rendered by it fees ranging from $1,000 to $1,500,000. Overview Many of the world's largest financial institutions have developed advanced quantitative credit risk models that help to measure, monitor and manage credit risk across their business lines. However, the Basel Committee on Banking Supervision recently identified credit model validation as one of the most challenging …

Business,Economics
Knowledge-based model validation using a custom metric

Nicola Henkelmann,Stephan Rhode,Johannes von Keler

2024-12-10

Abstract:Vehicle models have a long history of research and as of today are able to model the involved physics in a reasonable manner. However, each new vehicle has its new characteristics or parameters. The identification of these is the main task of an engineer. To validate whether the correct parameter set has been chosen is a tedious task and often can only be performed by experts. Metrics known commonly used in literature are able to compare different results under certain aspects. However, they fail to answer the question: Are the models accurate enough? In this article, we propose the usage of a custom metric trained on the knowledge of experts to tackle this problem. Our approach involves three main steps: first, the formalized collection of subject matter experts' opinion on the question: Having seen the measurement and simulation time series in comparison, is the model quality sufficient? From this step, we obtain a data set that is able to quantify the sufficiency of a simulation result based on a comparison to corresponding experimental data. In a second step, we compute common model metrics on the measurement and simulation time series and use these model metrics as features to a regression model. Third, we fit a regression model to the experts' opinions. This regression model, i.e., our custom metric, can than predict the sufficiency of a new simulation result and gives a confidence on this prediction.

Computational Engineering, Finance, and Science
Uncertainty Reduction Through Data Management in the Development, Validation, Calibration, and Operation of a Hurricane Vulnerability Model

Jean-Paul Pinelli,Josemar Da Cruz,Kurtis Gurley,Andres Santiago Paleo-Torres,Mohammad Baradaranshoraka,Steven Cocke,Dongwook Shin

DOI: https://doi.org/10.1007/s13753-020-00316-4

IF: 4.5

2020-11-20

International Journal of Disaster Risk Science

Abstract:Abstract Catastrophe models estimate risk at the intersection of hazard, exposure, and vulnerability. Each of these areas requires diverse sources of data, which are very often incomplete, inconsistent, or missing altogether. The poor quality of the data is a source of epistemic uncertainty, which affects the vulnerability models as well as the output of the catastrophe models. This article identifies the different sources of epistemic uncertainty in the data, and elaborates on strategies to reduce this uncertainty, in particular through identification, augmentation, and integration of the different types of data. The challenges are illustrated through the Florida Public Hurricane Loss Model (FPHLM), which estimates insured losses on residential buildings caused by hurricane events in Florida. To define the input exposure, and for model development, calibration, and validation purposes, the FPHLM teams accessed three main sources of data: county tax appraiser databases, National Flood Insurance Protection (NFIP) portfolios, and wind insurance portfolios. The data from these different sources were reformatted and processed, and the insurance databases were separately cross-referenced at the county level with tax appraiser databases. The FPHLM hazard teams assigned estimates of natural hazard intensity measure to each insurance claim. These efforts produced an integrated and more complete set of building descriptors for each policy in the NFIP and wind portfolios. The article describes the impact of these uncertainty reductions on the development and validation of the vulnerability models, and suggests avenues for data improvement. Lessons learned should be of interest to professionals involved in disaster risk assessment and management.

geosciences, multidisciplinary,meteorology & atmospheric sciences,water resources
A "Toy" Model for Operational Risk Quantification using Credibility Theory

Hans Bühlmann,Pavel V. Shevchenko,Mario V. Wüthrich

DOI: https://doi.org/10.48550/arXiv.0904.1772

2009-04-11

Abstract:To meet the Basel II regulatory requirements for the Advanced Measurement Approaches in operational risk, the bank's internal model should make use of the internal data, relevant external data, scenario analysis and factors reflecting the business environment and internal control systems. One of the unresolved challenges in operational risk is combining of these data sources appropriately. In this paper we focus on quantification of the low frequency high impact losses exceeding some high threshold. We suggest a full credibility theory approach to estimate frequency and severity distributions of these losses by taking into account bank internal data, expert opinions and industry data.

Risk Management

The role of the Model Validation function to manage and mitigate model risk

Model Validation Practice in Banking: A Structured Approach for Predictive Models

Model Validation Via Uncertainty Propagation and Data Transformations

Model Risk in Credit Portfolio Models

Measuring the model risk-adjusted performance of machine learning algorithms in credit default prediction

Practice-relevant model validation: distributional parameter risk analysis in financial model risk management

Lessons From Model Risk Management in Financial Institutions for Academic Research

AI model risk: What the current model risk management framework can teach us about managing the risks of AI models

Overview of model validation for survival regression model with competing risks using melanoma study data

Operational Risk Working Party - Validating Operational Risk Models

A New Validation Metric for Models with Correlated Responses Using Limited Experimental and Simulation Data

Assessing financial model risk

Minimise model risk management oversight for cyber security solutions

The Dali Model in Risk-Management Practice: The Case of Financial Services Firms

A Novel Approach to Quantification of Model Risk for Practitioners

Model Validation in Ontology Based Transformations

Validation of machine learning based scenario generators

Benchmarking Quantitative Default Risk Models: A Validation Methodology

Knowledge-based model validation using a custom metric

Uncertainty Reduction Through Data Management in the Development, Validation, Calibration, and Operation of a Hurricane Vulnerability Model

A "Toy" Model for Operational Risk Quantification using Credibility Theory