Insurance multi-firm review of outcomes monitoring under the Consumer Duty

Some firms had carefully thought about the suite of metrics and data required to monitor customer outcomes in the context of their business and mapped these against each of the four Duty outcomes. In one example, a firm explained how they had first identified possible good and poor outcomes across all business areas and then considered:

• the type and significance of harm if the poor outcomes materialised 
• Management Information (MI) that would best inform them of whether the good or poor outcome has been experienced (regardless of whether the data/MI was currently available)

Under the Duty, firms are required to avoid causing foreseeable harm to retail customers and some firms had clearly articulated their approach to identification. Where firms formally documented a range of key foreseeable harms relating to a product or service, this led them to better determine what data they required to monitor outcomes to avoid these harms.

Some firms appeared to be reliant on metrics which confirmed the number/percentage of product reviews or value assessments, or other processes being completed (such as a communications review of a sales document). We sometimes saw such metrics were reported with limited, or no insight into key findings of the process or any learnings or actions to be taken. In the example of a communications review, completion of a document review alone does not provide assurance that customers are equipped with the right information to make effective, timely and properly informed decisions.

While most insurers showed they were monitoring the timeliness of claims and number/percentage of declined claims, we saw little monitoring of whether overall good outcomes were being achieved in relation to claim settlements. We only saw a few examples where firms demonstrated claim settlements being comprehensively monitored, tested, assessed, and understood. This included data such as: analysis of overall settlement values, benchmarking of claim settlements, claims complaints, root cause analysis of declined or withdrawn claims, and internal deep-dive reviews (such as internal audits).

This aligns with the findings of our review into insurers' valuation of vehicles[1] which found most firms did not collect basic data on motor total loss claims, such as the number, scale, and reasons for increases to initial settlement offers.

Some firms took the data examples in FG22/5[8] as a starting point to consider what data they required to monitor the specific outcomes of their customers. Firms then determined what data was of direct relevance and then considered what additional data (whether readily available or not) would help them gain further insight. This usually led to a more varied, holistic mix of data types; qualitative and quantitative, RAG-rated data, key indicators, trend data and more. 

Some firms increased emphasis on testing customer outcomes by better understanding customer journeys. We saw examples of clear, risk-based approaches to testing customer journeys (including those for different customer types) which encompass the overall processes and impact on a customer’s outcome. 

This was sometimes developed as a new initiative, or sometimes adapted from existing Quality Assurance frameworks. In the good examples we saw, the analysis from such testing was clearly and directly linked to the four Duty outcomes. We saw evidence that a clear testing framework led to good identification of poor outcomes, and such an approach allowed root causes to be identified and rectified.

In our January 2023 publication on Consumer Duty implementation plans[1], we set out the following as an example of poor practice: 

'In some cases, firms’ data strategies seemed to be largely based on repackaging existing data, with limited consideration of gaps or the outcomes it is intended to monitor. If firms assume they can ‘get by’ largely with repackaging or supplementing existing data, then they risk not thinking deeply or afresh about the types and granularity of data that they will actually need to monitor and evidence outcomes under the Duty effectively.'

We were disappointed to find evidence that this approach has continued in some firms. While we would expect firms to make use of existing data capabilities, we also expect firms to appropriately consider whether the data they have is sufficient for the customer outcome monitoring required by the Duty. 

Some firms had clearly articulated tolerances in place for all key metrics, with formal governance established to ensure that they were set at an appropriate level and scrutinised annually.

The Consumer Duty rules require that customer outcomes are a central focus of firms’ risk control arrangements and internal audit function. We saw some firms integrate second-line scrutiny of how customer outcomes are monitored. Where this approach was taken, we saw good evidence of challenges being provided to the first-line owners of customer outcomes, including challenges on the type and granularity of data being used, its interpretation, and the thresholds applied, especially where reporting was focussed on RAG-ratings.

We observed the following examples of data which we believe may not effectively facilitate scrutiny or challenge:   

• Numbers presented with limited narrative to interpret the data and with no trend data or other context (eg, the number of complaints alone, with no details on total customer numbers, the issues complained about, or trend information). 
• Thresholds/targets which appear to be arbitrary, set too high/low, or are insufficiently granular (eg, loss ratios or claim timeliness targets which were identical for a wide range of products where performance would normally vary). 
• Tolerances set at a level that the firm was unlikely to breach and therefore may permanently report as ‘Green’.   
• Combined RAG rating thresholds which continually report ‘Green’ ratings even though there is a negative trend, or underlying data which should indicate potentially poor customer outcomes. For example, firm-wide customer end-to-end journeys may be on target, but specific key journeys, or elements of a journey, may be materially outside tolerance. 
• Thresholds which are set by using industry averages without justification (eg, using response time targets in line with industry-wide averages, even though the product target market is likely to contain a high percentage of customers with characteristics of vulnerability). 
• Thresholds which appeared to be reduced/softened without justification. 

We will consider taking action where we see examples of the above poor practices occurring and are particularly likely to act where we see evidence of poor outcomes for customers which could have been prevented or reduced with adequate monitoring.

We saw some firms monitor outcomes for customers with characteristics of vulnerability as a subset of data across a wide range of their overall customer outcomes monitoring metrics. This included data on loss ratios, complaints, claim settlements, and service levels. 

Where firms identified potentially poorer outcomes for customers with characteristics of vulnerability compared to other customers, we saw detailed analysis taking place and, in some instances, actions being taken to improve outcomes for this group of customers. In one example we saw a firm identify that customers with characteristics of vulnerability were having claims declined more frequently than other customers. The firm investigated and made changes to its claims processes and training for claims handlers to deliver better outcomes. 

We also saw some firms analyse outcomes of customers with different characteristics of vulnerability to better understand where issues stemmed from.

Some firms showed limited monitoring of outcomes for different groups of customers, including a lack of monitoring of outcomes between customers with characteristics of vulnerability and other customers.

Even where firms could sometimes distinguish between the outcomes of customers with characteristics of vulnerability and other customers, most did not demonstrate they had developed comprehensive approaches for monitoring different groups of customers across their suite of outcome monitoring data. This is consistent with the FCA’s Product Governance Thematic Review work (this review is ongoing, an update was issued to relevant manufacturers on 26 February 2024) which found firms were unable to demonstrate how they assessed whether products were delivering fair value to all customers, including customers with characteristics of vulnerability, or outlier groups of customers potentially receiving significantly less value from the product.

We also saw evidence of this in our review of motor insurers’ valuation of vehicles[7]. For example, we saw firms were not able to demonstrate that groups of customers who may be less able to challenge or complain did not receive systematically worse outcomes.

Some firms were able to show that a wide range of data was reported and where outliers were seen, deeper analysis was conducted in a risk-based manner to allow the firm to identify where improved outcomes may be delivered. Once an action had been determined, progress was tracked, including what impact the action had on outcomes customers received. 

In one example, we saw a firm use survey feedback, complaint analysis and customer journey time MI to identify poor service outcomes for a product. The firm created a dedicated team to improve the customer experience on this product and enhanced digital service capabilities to improve the customer experience.

When changes were made following identification of poor outcomes, some firms were unable to demonstrate how improvements to customer outcomes were being monitored after that change. For example, where policy/plan documents had been through a communications review process, firms did not demonstrate how they monitored whether the changes made had improved customer understanding. Where a change has been made in response to the identification of poor outcomes, firms should have a clear approach to testing the impact of such change.

We saw some firms combine a wide range of metrics and data to better understand whether good product and service outcomes were being achieved. This included insights such as:  

• findings from various internal product reviews 
• complaints data relating to products and services including root cause analysis 
• file reviews, quality assurance and/or call monitoring 
• claims data, such as frequency or severity of claims 
• metrics or data on whether the products or individual features were being used as intended 
• policy administration data, such as not-taken-up rates, lapse rates, mid-term adjustments. 
• data on whether products were performing as expected (eg, whether claims were paid out as expected and in a timely manner, or investments were performing as expected) 
• benchmarking information, such as comparison of loss ratios or fund performance  
• various points of customer feedback, surveys, and research 
• insights from other parties within the distribution chain 

For example, we saw a firm use a range of data, including claim decline rates, customer feedback, complaints analysis and call monitoring, to identify that policies were sometimes not paying out in line with customer expectations. The firm used this information to identify improvements which were made in the approach of a third-party claim handler and worked closely with that third party to improve standards and track improvements.

We saw firms taking assurance that completion of, or lack of material findings from product reviews or fair value assessments, automatically indicated good outcomes were being achieved. In line with the FCA’s Product Governance Thematic Review (this review is ongoing, an update was issued to relevant manufacturers on 26 February 2024), firms should be wary in taking undue comfort solely from a review having been completed. Overall, firms should ensure that their monitoring enables them to identify any risk that they are not acting to deliver good outcomes for retail customers.

We saw some firms combine a wide range of metrics and data, broken down to product (or product group level) to better understand whether good outcomes on price and value were being achieved. This included insights such as: 

• findings from various value assessments 
• product performance data (expected and actual), such as profitability, loss ratios, fund performance  
• analysis of charges, commission or other costs incurred through the distribution chain 
• analysis of actual operational costs of the product 
• premium loading data
• complaints data in relation to price and value including root cause analysis 
• various points of customer feedback and research regarding price and value 
• price and value insights from other parties within the distribution chain 
• operational data which has an impact on service delivered 
• insights into market conditions which may have an impact on short- or long-term customer outcomes 

For example, we saw a firm use a range of data, including claim ratios, claim decline rates, customer feedback and complaints analysis, in relation to ancillary products, to identify policies which may not be offering fair value. The firm took actions such as reducing the price and increasing the level of policy benefits. The firm monitored the changes made to determine whether improved customer outcomes were achieved.

We saw firms which did not make distinctions between products when setting standards or tolerances. For example, we saw loss ratio tolerances which were set at the same level for different products, despite actual loss ratios of the products being vastly different. This meant that for some products the loss ratio would likely never fall below a level which would prompt investigation or action by a firm. Where used, such standards or tolerances would need to be scrutinised and set at meaningful levels if they are to help effectively monitor customer outcomes.

We observed some firms placing what we believe may be undue reliance on complaints data. Particularly for some areas – such as complaints explicitly about the price and value of a product - we believe the frequency and nature of complaints may be less likely to allow firms to gain meaningful insights into customer outcomes. While complaints data can be a relevant consideration to monitor the price and value outcome, the responsibility is with firms to ensure fair value. Focusing too much on complaints pushes responsibility on to customers. Overall, it is the responsibility of firms to ensure that they have sufficient data to be able to identify whether they are delivering good outcomes for customers.

We also saw complaints data used to present the number of complaints only, with no details of root cause analysis conducted. Such analysis is likely to be the most meaningful element a firm can use to understand customer outcomes, rather than raw numbers alone. 

We saw some firms combine a wide range of metrics and data to better understand whether good outcomes were being achieved. This included insights such as: 

• findings from communications testing  
• testing involving both internal assessment (eg, whether the firm meets the requirements of the firm’s communications framework, feedback from employees), and external assessment (eg, feedback from customers and working with third-party communication experts). 
• testing at targeted points during a customer journey (eg, at point of claim)  
• testing considered against analysis of foreseeable harms 
• testing across different types of customer communications – such as call centre scripts – and not just written communications  
• findings from quality assurance work
• other feedback from customers, including customer surveys  
• feedback from other parts of the distribution chain (eg, sales advisors) 
• various behavioural monitoring, including customer response rates following communications  
• claims rates and claim decline rates in relation to customer understanding
• complaints data in relation to customer understanding including root cause analysis

Some firms had made considerable efforts to build an approach to monitoring under this heading, which included data points from a number of sources (as per above). The approach often included firms adapting their internal Quality Assurance (QA) processes and methodology to have a clear focus on monitoring customer understanding (amongst other Duty outcomes). This included clearly mapping a wide range of customer journeys so that the consumer understanding outcome could be tested methodically across different parts of the journey. 

In such examples, we saw positive evidence of actions being taken to make meaningful adaptations to communication approaches because of the monitoring. This included firms thinking about when and how to communicate with a customer – sometimes requiring a redesign of a customer journey rather than just adapting what is being communicated. We then saw firms demonstrate the process in place to test whether the change is effective.

As we observed elsewhere in this review, firm monitoring was sometimes process-driven. For example, reporting whether a communications review has occurred or whether a renewal letter is sent on time. Some firms were reliant on complaints numbers/trends alone to form a view on customer understanding. These are important aspects to monitor under this heading, but we expect firms to monitor the outcomes customers receive clearly and proactively – which may include whether customers respond appropriately to the information provided.

Some firms were highly reliant on net promoter scores and did not give evidence of many meaningful insights from their monitoring. More developed approaches were able to show a wide range of sources, including tailored customer surveys and using focus groups. Such insights were considered alongside outcomes testing work, and we saw evidence from firms making changes to processes, communications, or staff training in response to such insights.

We saw some firms combine a wide range of metrics and data to better understand whether good outcomes on customer support were being achieved. This included insights from: 

• performance against various SLAs 
• analysis of whether SLAs meet customer requirements 
• claims rates, claim acceptance/decline rates, including analysis 
• other product and service usage rates and analysis, such as claims experience 
• customer behavioural data, such as lapse rates and requests for assistance  
• internal quality assurance findings, or other first-/second-/third-line analysis 
• customer journey timeliness, including first contact resolution  
• customer feedback including satisfaction survey feedback and net promoter scores  
• complaints data including root cause analysis

We saw a firm extensively map all key customer journeys and monitor various service data across them, including timeliness of each engagement and end-to-end transactions. The firm complemented this data with QA findings, customer feedback and complaints, and identified points where service improvements could be made to enhance the customer support offered. The firm also assessed elements of the customer journey for different types of customers to identify specific barriers, in particular for customers with characteristics of vulnerability.