What is IRR?

In the simplest terms, Inter-Rater Reliability is the measurement of consistency, or agreement level, among multiple raters.  You may have engaged in multi-rater assessments where each assessor produces a separate result.  Or you may have seen them done during some Olympic games.  IRR is about how consistent the evaluations among multiple raters are, and closely the evaluations agree with one another.  The closer the better, and the better the more reliable your assessment is.

Let’s say 5 raters are asked to evaluate a single piece of work.  Let’s say they use a generic rubric with 3 criteria: Writing, Knowledge, and Presentation.  The 5 raters score the work independently from each other, and we notice they mostly agreed on the “Writing” criterion, however, they had more disagreements on “Knowledge” and even more disagreements on “Presentation”.  Simply put, inter-rater reliability is about calculating the level of agreement/disagreement.

What is Reliability in assessment and why is it important?

Reliability of an assessment helps to ensure that the results are accurate and trustworthy.  In addition, reliable assessments are unbiased, consistent, and valid.

For example, a Lykert scale where raters evaluate on a scale of 1 to 5 is more subjective and less reliable than using a rubric with clear and concise descriptors.  Here’s a quick test: Score yourself on a scale of 1 (worst) to 5 (best) on your leadership skills.  On a good day, you may score yourself higher and on a bad day lower.  However, if you were to use a well-written rubric, you would get more consistent results.

Why is Inter-Rater Reliability important:

One method to ensure reliable assessments is to measure Inter-Rater Reliability, i.e., would multiple raters assessing the same item using the same assessment tool come up with the same or closely agreed results?  IRR helps measure the reliability of assessments and helps fine-tune your assessment tools to become more reliable, hence, more consistent, unbiased, accurate and overall valid and effective. 

For example, one of the best methods in refining your rubrics is to measure its IRR, and modify the rubrics and its descriptors to increase the level of agreement among multiple raters.  We will talk about this in details in other articles.

Best setup for IRR:

Simply put, for IRR to work, there needs to be multiple ratings on the same item using the same assessment tool.

To be able to measure IRR, you need to have some basic ingredients.  First, there needs to be multiple raters.  Single rater assessments cannot produce any results.  Second, the raters need to use the same assessment tool.  Having each rater using a separate rubric or any other assessment tool will not help you generate any results.  Third, the raters have to evaluate the same item, so their results can be compared.  Fourth, it only works in subjective areas, but that’s without saying.

How to measure IRR:

There are multiple statistical methods to measure IRR which are beyond the scope of this writing.   Measuring IRR by hand is doable at a very small scale, however, for constant monitoring and measurements, a software application capable of measuring IRR would be the best.  There are statistical analysis tools out there, but you need to format the data to feed into them.  iRubric is a comprehensive tool that is used to do the actual assessment, and can produce IRR with a single click at any time.  

Hope you find this article useful.  To learn more about IRR or to see how iRubric can help you with this complex and time-consuming task, or if you have any feedback about this article, feel free to contact us.