The English star rating system - failure of theory or practice?
P. Shekelle
DOI: https://doi.org/10.1177/135581960501000102
2005-01-01
Abstract:All health care systems are dealing with the vexing problem of trying to improve quality of care. The evidence documenting the gaps between what we know should be done and what is done is conclusive. The question is: how can this state of affairs be made better? Many things have been tried – professional education, practice guidelines, disease management programmes, audit and feedback, to name a few. None of these efforts has been as successful as we would like. Recently, two separate but related concepts have become popular in both the US and the UK – the use of financial incentives linked to specific measures of quality and the public release of quality information. Pay for performance (sometimes abbreviated to P4P) arrangements are popping up all over the US.1 The new general practitioner contract in England is the most ambitious pay for performance programme anywhere in the world.2 Americans now have available a plethora of information on quality of care.3 As for public release of information, the star rating system is the first programme in England to make data on quality of providers’ performance publicly available. Both pay for performance systems and the public release of information on quality of care use incentives to encourage providers to meet certain measurable, specific aspects of the health care system. As a major new policy, evaluations of the effects of the star rating system are necessary. In this issue, Mannion, Davies and Marshall present the results of a series of interviews with key participants at six NHS acute hospital trusts, purposively selected to include four that received low star ratings and two that received high ratings.4 They assessed how these trusts viewed the star rating system in terms of its fairness as a measure of performance, and the positive and negative responses the trust had made in response to their rating. In general, they found that the low rated trusts thought the system was unfair and promoted dysfunctional responses while the high rated trusts thought it was great and had resulted in improved morale, albeit with some reservations. These results are perhaps not surprising – similar responses were seen in the US when the first league tables comparing hospital death rates were published.5 In the US there is the confusing situation where different ratings are given to the same hospital by different groups. Clearly, both high and low ratings to the same hospital can not both be right. Given all this, is it time to pull the plug on the star rating system? I think to do so would be premature. First, the effects reported by Mannion, Davies, and Marshall are not measures of the primary purpose of the star rating system. That purpose is to improve quality of care. Ideally, the star rating system would both improve quality of care and improve the outcomes reported by Mannion, Davies, and Marshall, but improving quality of care has to be seen as a more meaningful measure of effect than how workers at the hospitals view the system. In the USA, similar negative views have been voiced about the use of publicly available rating systems, but quality has also demonstrably improved – at least for those aspects that are measured.6,7 Second, the theory behind the star rating system states – and the American experience supports – that if you measure and publicly report on quality of health care, organizations will respond to try and improve performance on those aspects that are measured and reported. The key, then, is to choose those measures wisely. Choose poorly, by choosing process measures that are not meaningfully linked to good health outcomes, or choosing measures that are difficult to reliably ascertain or easy to game, and the result may be more harm than good. But chosen well, measures of the quality of health care that are assessed and publicly reported can lead to meaningful gains in health, in terms of lives saved and morbidity averted. My view is that the star rating system is not flawed in theory but may need changes in its implementation, particularly if evaluations of the rating system’s effect on quality are not promising. In particular, one change that could be accomplished without collecting new data would be the incorporation into the global ratings data from sophisticated, specialized databases that already exist.8 This could provide meaningful information on comparative performance in terms of outcomes. The research agenda proposed by Mannion, Davies, and Marshall is a good one. The way forward is for health services researchers to conduct rigorous evaluations of the star rating system and policy-makers be willing to make changes based on the results. Paul G Shekelle Senior Scientist Health Sciences Program RAND, 1776 Main Street Santa Monica, CA90407 2138, USA