Home »
Python »
Python Programs
How to calculate Cohen's kappa in Python?
By Shivang Yadav Last updated : November 21, 2023
Cohen's Kappa
Cohen's Kappa or Kappa is a statistical value that is used to measure the level of agreement between two or more raters or classifiers when they are assigning categorical ratings to items. It is a widely used statistic in various fields, including psychology, machine learning, and inter-annotator agreement in natural language processing (NLP).
Cohen's Kappa accounts for the possibility of agreement occurring by chance and measures the agreement beyond what would be expected by random chance. The formula for Cohen's Kappa is as follows:
k = (p0 - pe) / (1 - pe)
Where:
- k is Cohen's Kappa.
- po is the observed agreement between raters or classifiers.
- pe is the expected agreement that would occur by chance.
Typical guidelines for interpreting Cohen's Kappa values are as follows:
- 81-1.00: Almost perfect agreement.
- 61-0.80: Substantial agreement.
- 41-0.60: Moderate agreement.
- 21-0.40: Fair agreement.
- 00-0.20: Slight agreement.
- Below 0.00: Poor agreement or disagreement.
Calculation for Cohen's Kappa
A simple and direct way to calculate such a value is by directly applying the formula of the values. But Python provides a huge variety of built-in methods that perform the given task for a program to build higher-level AI and ML applications using these functions. To calculate Cohen's Kappa for two raters, use the cohen_kappa_socre() method which is present in the sklearn library of Python.
Syntax
cohen_kappa_score(raterArr1, raterArr2)
Python program to calculate Cohen's Kappa
from sklearn.metrics import cohen_kappa_score
# Define the two arrays
array1 = [1, 0, 0, 1, 1, 0, 1, 1]
array2 = [0, 1, 1, 1, 1, 1, 0, 0]
# Calculate Cohen's Kappa
kappa = cohen_kappa_score(array1, array2)
print("Cohen's Kappa:", kappa)
Output
The output of the above program is:
Cohen's Kappa: -0.6000000000000001
This value of Cohen's Kappa shows disagreement.
Python SciPy Programs »