Paul MM van Haard Paul MM van Haard - 3 months ago 44
R Question

Bootstrapping Krippendorff's Alpha

I should like to calculate - by bootstrapping Krippendorff's Alpha outcomes - a 95% confidence interval for Krippendorff's Alpha coefficient of Raters Reliability using R package irr.

Let's use "C data from Krippendorff" in the package irr and the R script for calculating Krippendorff's Alpha once:

# the "C" data from Krippendorff
#rater per row; rated subject per column; NAs allowed
library(irr)
nmm<-matrix(c(1,1,NA,1,2,2,3,2,3,3,3,3,3,3,3,3,2,2,2,2,1,2,3,4,4,4,4,4,
1,1,2,1,2,2,2,2,NA,5,5,5,NA,NA,1,1,NA,NA,3,NA),nrow=4)
kripp.alpha(nmm,"ordinal")

Answer

You can use the boot function from the boot package to bootstrap values. Here I'll bootstrap the set of subjects but keep the raters fixed:

library(boot)
library(irr)
ka <- function(data, indices) kripp.alpha(nmm[,indices], "ordinal")$value
b <- boot(seq(ncol(nmm)), ka, 1000)

Now you can use the boot.ci function to compute a 95% confidence interval for the bootstrapped value; I'll use the percentile confidence interval, but others are available (check out ?boot.ci):

boot.ci(b, type="perc")
# BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
# Based on 1000 bootstrap replicates
# 
# CALL : 
# boot.ci(boot.out = b, type = "perc")
# 
# Intervals : 
# Level     Percentile     
# 95%   ( 0.4297,  1.0000 )  
# Calculations and Intervals on Original Scale
Comments