Researchers looked at all available measures of quality and risk for surgical procedures in California in 1999. Data was available for 21 procedures – only 12% of those carried out in the state.
There were no measures of the quality of individual surgeons, and none relating to functional assessments, for example ability to walk after a hip replacement. The length of reporting periods meant that most of the measures reflected care delivered between two and five years ago.
The researchers conclude that the shortage and inadequacy of the data available means that consumers cannot yet choose between providers on the basis of quality.
In a commentary, Professor Martin McKee argues that similar problems apply to the publication of healthcare outcomes in the U.K. The Californian example shows that even with high levels of investment in information technology, the published data can only capture a tiny amount of the overall work of healthcare providers.
There are also problems incorporating quality measures into the system in a way that leads to genuine improvements, he argues. The culture of target-setting in the NHS has already led to examples of providers distorting clinical priorities in order to improve what is reported.
He suggests that healthcare policy makers consider a sign that Einstein kept on his wall: "Not everything that counts can be counted; not everything that can be counted counts."