A measure of reliability using qualitative data

In many research activities, the data is unordered or qualitative. In such circumstances, inter-rater reliability is usually measured by calculating a percentage of agreement score between judges. The present paper expands on an equation first introduced by Goodman and Kruskal for obtaining a reliability measure of one item. This formula determines inter-rater reliability for a series of items across many subjects. The statistic that results is easily interpreted and in many ways is analogous to the conventional reliability for quantitative data.