The performance of NHS trusts is monitored through the star ratings system, which the Department of Health introduced in September 2001. The Commission for Health Improvement was given responsibility for assessment in 2002-3.
Researchers from Newcastle upon Tyne analysed the rating system after their trust was downgraded from three stars (the highest rating) in 2001-2 to two stars in 2002-3.
The assessment showed that Newcastle had significantly underachieved on one key target (outpatient waiting times). The purpose of this target was to measure a trust's performance throughout the year, but only four quarter end days were used. With this method, a trust could have a considerable number of breaches during a quarter yet show none on the final day.
To achieve the target, trusts had to have had no more than five breaches during the year. Newcastle undoubtedly breached the target but other trusts with more breaches were judged to have met the target because their figures on the four quarter end days fell within requirements. This method is flawed, argue the authors.
The target also measured the number of breaches irrespective of the size and activity of the trust. Clearly it is easier for a small trust to meet the threshold of five or fewer breaches a year than it is for trusts with a large referral base, they add.
A three star rating brings appreciable benefits to a trust. With such high stakes, the performance assessments should be beyond reproach, yet this was clearly not the case in 2002-3, say the authors. These concerns have also been reinforced by a recent Royal Statistical Society report on performance monitoring in the public sector.
More statistical rigour is needed to ensure trusts' confidence in the ratings system, they conclude.
Everyone has always agreed that star ratings are not perfect but also that performance assessment can be beneficial, writes Anna Walker of the Healthcare Commission, in an accompanying commentary.
She points out that the statistical approach adopted was well established and known to the NHS including Newcastle. However the new healthcare inspectorate is not opposed to improvement of the performance ratings system.
She explains that the Commission wants a system of performance assessment that is more accessible to the public, that drives improvement in the NHS, that is seen as relevant and fair by the service and clinicians, and that is more comprehensive in how it measures organisations.
"We have and will make improvements, she says. "We will be looking to engage with doctors, nurses, and others across the NHS about how this system might work."