Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol

Simple item page
dc.contributor.authorJeyaraman, Maya M
dc.contributor.authorAl-Yousif, Nameer
dc.contributor.authorRobson, Reid C
dc.contributor.authorCopstein, Leslie
dc.contributor.authorBalijepalli, Chakrapani
dc.contributor.authorHofer, Kimberly
dc.contributor.authorFazeli, Mir S
dc.contributor.authorAnsari, Mohammed T
dc.contributor.authorTricco, Andrea C
dc.contributor.authorRabbani, Rasheda
dc.contributor.authorAbou-Setta, Ahmed M
dc.date.accessioned2020-03-01T04:32:31Z
dc.date.issued2020-02-12
dc.date.updated2020-03-01T04:32:31Z
dc.description.abstractAbstract Background A new tool, “risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE),” was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus—evaluator burden). Methods Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet’s AC1 statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall’s tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively. Discussion The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.
dc.identifier.citationSystematic Reviews. 2020 Feb 12;9(1):32
dc.identifier.urihttps://doi.org/10.1186/s13643-020-01291-z
dc.identifier.urihttp://hdl.handle.net/1993/34556
dc.language.rfc3066en
dc.rightsopen accessen_US
dc.rights.holderThe Author(s)
dc.titleInter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol
dc.typeJournal Article
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
13643_2020_Article_1291.pdf
Size:
356.8 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
2.24 KB
Format:
Item-specific license agreed to upon submission
Description:
Download
Collections
University of Manitoba Scholarship
Rady Faculty of Health Sciences Scholarly Works