A report launched by Mazars in 2014 called ‘The Dirty Dozen’ explored 12 modelling horror stories and spreadsheet disasters to have taken place over the last couple of decades. A $240m USD fine for global investment manager firm, AXA Rosenberg, for the cover up of a spreadsheet error that caused an over-estimation of client investment losses was one of the largest losses in this list. This event is one of many examples found readily available not only in the public domain but also throughout ORIC International’s peer risk event data.
Following on from our recent findings in the Capital Benchmarking Survey, which found that ‘Model Risk’ was considered one of the most material operational risk scenarios by capital allocation, ‘Model Errors’ clearly have an inherent need for comprehensive risk governance and oversight. Models themselves should be subject to scepticism and model inputs and outputs need detailed analytical review, challenge and validation.
Of the 151 ‘Model error’ risk events identified in the ORIC International dataset, causal analysis reveals a disparity between modellers and both the quality and availability of the data input. Furthermore, incorrect underlying assumptions during the parameterisation of the model and subsequent inaccuracy in implementation, ultimately drives errors in the modelled risk. Events such as these cost firms on average £1.4m GBP and for near miss events that were quantified, the average exposure, i.e. what a firm could have lost, had the event materialised, was £9.4m GBP.
In addition to this, ‘Human Error’ and ‘Poor Process Design’ were flagged as the most common primary causal types for ‘Model Errors’ and clearly underlines the importance for firms to adopt clearer and more robust processes and documentation, as well the need to implement a more systematic and rigorous review and challenge process. Furthermore, knowledge gaps of those who, potentially are less technical than the modellers are often key contributors to mistakes being missed and lead to ineffective implementation. It is therefore imperative for firms to continually re-assess their processes, controls and most importantly models to ensure they are calculating as intended.
Much like any other risk, model risk requires sufficient risk-mapping and monitoring through the use of key risk indicators and there should be regular engagement between model developers, risk owners and appropriate risk committees in order to enhance the risk identification process for current and emerging risks. Members of ORIC have access to a wealth of key risk indicators as part of their membership and subscription to ORIS (ORIC International’s data sharing platform).
We’d love to get your feedback on this trend and whether this is something that resonates with your firm. For more information please contact Ciaran.