The effect of rater training on the reliability and validity of technical skill assessments: a randomized control trial
Loading...
Date
2016
Authors
Maniar, Reagan
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Rater training (RT) improves the reliability of assessment tools, but has not been well studied for technical skills. This study assessed whether RT improved the psychometric properties of surgical skill assessments.
Surgeons (N=47) were randomized to RT or non-training groups. The RT group underwent frame-of-reference training. Participants assessed trainees performing a suturing and knot-tying task using four assessment tools. Inter-rater reliability, initial and delayed rater agreement, and construct validity were assessed between groups.
There was no significant effect of RT on the assessment tools’ reliability and validity. Reliability and validity were most robust for the global rating scale.
Although there were trends towards improved reliability and validity with RT, confidence intervals were wide and overlapping. Reliability remained below the minimum desired level of 0.8 required for high-stakes testing. Although RT may represent a way to improve reliability, further study is needed to determine effective training methods.
Description
Keywords
Surgical education, Rater training, Technical skills, Technical skill assessments