Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group: a challenge for clinical trial design

Introduction: The reliability of clinician grading of systemic sclerosis (SSc)-related digital ulcers (DUs) has been reported to be poor to moderate at best, which has important implications for clinical trial design. The aim of this study was to examine the reliability of new proposed UK Scleroderma Study Group DU definitions amongst UK clinicians with an interest in SSc.

Methods: Raters graded (through a custom-built interface) 90 images (80 unique and 10 repeat) of a range of digital lesions collected from patients with SSc. Lesions were graded on an ordinal scale of severity: ‘no ulcer’, ‘healed ulcer’, or ‘DU’.

Results: Twenty-three clinicians: 18 rheumatologists, 3 dermatologists, one hand surgeon and one specialist rheumatology nurse, completed the study. A total of 2070 (1840 unique + 230 repeat) image gradings were obtained. For intra-rater reliability, across all images the overall weighted kappa coefficient was high (0.71) and was moderate (0.55) when averaged across individual raters. Overall inter-rater reliability was poor (0.15).

Conclusion: Although our proposed DU definitions had high intra-rater reliability, the overall inter-rater reliability was poor. Our study highlights the challenges of DU assessment by clinicians with an interest in SSc, and provides a number of useful insights for future clinical trial design. Further research is warranted to improve the reliability of DU definition/rating as an outcome measure in clinical trials, including examining the role for objective measurement techniques, and the development of DU patient reported outcome measures.