Reliability of coders
Intra class correlation
To calculate the reliability of the coders in the Classic Expression research group their Intra Class Correlation (ICC) is calculated.
This is to know if coders score consistent and similar using a coding construct.
It is not an absolute number, but a percentage, based on the relative similarity between coders.
It means for example that the highest score of coder 1 is the same child as the highest score of coder 2.
ICC is poor/good if:
-many scores are (not) the same
-the differences are smaller/bigger
-the order in grading is different/similar
The coders of the Classic Expression coding group that have passed the last test in August 2021 have rates between 82 and 92.
These are considered good to excellent by the guidelines for interpretation of ICC by Cicchetti and Koo and Li.
Cicchetti (1994):
•Less than 0.40—poor.
•Between 0.40 and 0.59—fair.
•Between 0.60 and 0.74—good.
•Between 0.75 and 1.00—excellent
Koo and Li (2016):
•below 0.50: poor
•between 0.50 and 0.75: moderate
•between 0.75 and 0.90: good
•above 0.90: excellent
One coder did not yet pass the test yet and will not code in the research.