Are current approaches to measuring and analysing resident satisfaction fit for purpose?

By Frances Harkin and Mary-Kathryn Rallings Adams - on 25/08/2015

Ahead of the launch of an exciting project, Frances Harkin and Mary-Kathryn Rallings Adams reflect on the scoping study that explored current approaches to measuring resident satisfaction, completed over the summer. HACT and Simetrica uncovered a range of issues with current approaches and are preparing to begin a project to develop a new approach that addresses these challenges, improves the evidence base around resident satisfaction, and facilitates robust Value for Money calculations.

Using surveys to measure resident satisfaction has become one of the key ways to assess the effectiveness of housing providers. Results from resident surveys have the potential to provide insights about their levels of satisfaction with particular services, as well as with their landlords in general. This data is used to assess performance and inform the improvement of different service areas. In providing a basis for comparison with other housing providers, resident satisfaction data is also widely used within the sector to demonstrate Value for Money as part of regulatory requirements. However, social housing providers are beginning to question whether current approaches to measuring and analysing resident satisfaction reflect best practice for generating insights and assessing business performance.  

HACT embarked on a scoping study, in partnership with Simetrica, earlier this year to explore issues relating to current approaches in more detail. Drawing upon in-depth interviews and workshop discussions with eighteen housing providers who took part in the project, we considered what it is that housing providers are trying to measure with satisfaction surveys and how data is being used within organisations. This investigation uncovered three sets of issues with current approaches to capturing resident satisfaction and analysing data. 

Firstly, the majority of study participants highlighted practical issues related to their own approaches to gathering information about resident satisfaction, including determining appropriate methods, gathering data and applying insights. There is often no overriding vision for what the data will be used for once it has been collected, which results in the collection of surplus data and survey fatigue for residents. In addition to this, survey responses are impressionistic and do not necessarily reflect business performance.  A key concern for many was the inconsistency of data gathered and the implications of using a particular survey mode, leading to results that are potentially inaccurate or that lack methodological rigour. 

Secondly, methodological challenges, including a range of survey biases, emerge when analysing satisfaction scores when benchmarking with current methods. Data collected using current resident satisfaction surveys and the statistical methods used to analyse this data only provide descriptive indications of residents’ satisfaction, rather than in-depth insights about the lives of residents. Likewise, the data gathered and analysis performed are not able to allow housing providers to infer the drivers of changes in satisfaction.

Thirdly, the strength and validity of current benchmarking outputs was raised as a key concern for the majority of housing providers participating in the scoping study. Critically, we are not currently comparing housing providers on a like-for-like basis: comparing the satisfaction scores of two different organisations is meaningless unless the residents of the two different organisations were similar before any intervention or service was administered. 

As a result of all of the practical concerns and challenges, as well as the clear methodological issues with current approaches, housing providers have become cautious about using results to inform decision-making. They are also questioning the extent to which resident satisfaction surveys are actually providing the critical information needed to inform business decisions and build high quality services. With the scoping study complete, many housing providers have expressed both need and appetite to develop a new model for measuring and analysing resident satisfaction. 

HACT is currently finalising the full project specification, detailing the co-creation of a new approach for measuring and analysing resident satisfaction, addressing current challenges and responding to the needs of the sector. Critically, the new approach is intended to facilitate a robust understanding of the extent to which a housing provider is able to influence satisfaction and the tools it has at its disposal to do so; by comparing an understanding of the impact of activities to their costs, housing providers will be able to make far more evidence-informed decisions about which activities represent value for money, and be able to provide a robust evidence base for decisions to invest (or disinvest) in activities in their Value for Money self-assessments. 

This approach will enable housing providers to make decisions based on the evidence and insights collected, as well as understanding how those decisions will impact their businesses and the lives of their residents. The project will also improve the evidence base about the methods used to create and analyse satisfaction scores.  If you are interested in taking part, please get in touch with Frances (frances.harkin@hact.org.uk) or MK (mary-kathryn.rallings@hact.org.uk).

 

Tags: 
Resident satisfaction; HACT research; scoping study

Add new comment