Inter-rater agreement between 13 otolaryngologists to diagnose otitis media in Aboriginal and Torres Strait Islander children using a telehealth approach

Int J Pediatr Otorhinolaryngol. 2023 Mar 13;168:111494. doi: 10.1016/j.ijporl.2023.111494. Online ahead of print.

ABSTRACT

INTRODUCTION: Telehealth programs are important to deliver otolaryngology services for Aboriginal and Torres Strait Islander children living in rural and remote areas, where distance and access to specialists is a critical factor.

OBJECTIVE: To evaluate the inter-rater agreement and value of increasing levels of clinical data (otoscopy with or without audiometry and in-field nurse impressions) to diagnose otitis media using a telehealth approach.

DESIGN: Blinded, inter-rater reliability study.

SETTING: Ear health and hearing assessments collected from a statewide telehealth program for Indigenous children living in rural and remote areas of Queensland, Australia.

PARTICIPANTS: Thirteen board-certified otolaryngologists independently reviewed 80 telehealth assessments from 65 Indigenous children (mean age 5.7 ± 3.1 years, 33.8% female).

INTERVENTIONS: Raters were provided increasing tiers of clinical data to assess concordance to the reference standard diagnosis: Tier A) otoscopic images alone, Tier B) otoscopic images plus tympanometry and category of hearing loss, and Tier C) as B plus static compliance, canal volume, pure-tone audiometry, and nurse impressions (otoscopic findings and presumed diagnosis). For each tier, raters were asked to determine which of the four diagnostic categories applied: normal aerated ear, acute otitis media (AOM), otitis media with effusion (OME), and chronic otitis media (COM).

MAIN OUTCOME MEASURES: Proportion of agreement to the reference standard, prevalence-and-bias adjusted κ coefficients, mean difference in accuracy estimates between each tier of clinical data.

RESULTS: Accuracy between raters and the reference standard increased with increased provision of clinical data (Tier A: 65% (95%CI: 63-68%), κ = 0.53 (95%CI: 0.48-0.57); Tier B: 77% (95%CI: 74-79%), 0.68 (95%CI: 0.65-0.72); C: 85% (95%CI: 82-87%), 0.79 (95%CI: 0.76-0.82)). Classification accuracy significantly improved between Tier A to B (mean difference:12%, p < 0.001) and between Tier B to C (mean difference: 8%, p < 0.001). The largest improvement in classification accuracy was observed between Tier A and C (mean difference: 20%, p < 0.001). Inter-rater agreement similarly improved with increasing provision of clinical data.

CONCLUSIONS: There is substantial agreement between otolaryngologists to diagnose ear disease using electronically stored clinical data collected from telehealth assessments. The addition of audiometry, tympanometry and nurse impressions significantly improved expert accuracy and inter-rater agreement, compared to reviewing otoscopic images alone.

PMID:37003013 | DOI:10.1016/j.ijporl.2023.111494