Can quality indicators predict inspection ratings? Cross-sectional study of general practices in England

Research output: Contribution to journalArticle

Abstract

Background
The Care Quality Commission regulates, inspects and rates general practice providers in England. Inspections are costly and infrequent, and are supplemented by a system of routine quality indicators, measuring patient satisfaction and the management of chronic conditions, which can be used to prioritise or target inspections.
Aim
To determine whether this set of indicators can be used to predict the ratings from subsequent inspections.
Design
The indicators and first-inspection ratings were used to build models to predict inspection outcomes on the four-level rating system (Outstanding, Good, Requires Improvement, and Inadequate), for the Overall and domain ratings.
Setting
We consider the first-inspection cycle (2014 to 2017), a dataset of 6860 general practice providers.
Method
We built ordered logistic regression models and assessed predictive accuracy using the percentage of correct predictions and a measure of agreement, weighted kappa.
Results
The model correctly predicted 79.7% of Overall practice ratings. However 78.8% of all practices were in fact rated Good on Overall, and the kappa measure of agreement is was very low (0.097) meaning predictions were little-better than chance. This lack of predictive power wasis also found for each of the rating domains.
Conclusion
The poor power of performance indicators to predict subsequent inspection ratings may call into question the validity and reliability of the indicators, inspection ratings, or both. We suggest a number of changes to the way performance indicators are collected and used, which could improve their predictive value, and recommend that assessments of predictive power should be undertaken prospectively when sets of indicators are being designed and selected by regulators.

Bibliographical metadata

Original languageEnglish
JournalBritish Journal of General Practice
Publication statusAccepted/In press - 7 Aug 2019