Cohen's Kappa score

Þ ¥bonds€¬cell_resultsŒÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ(update! (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7e´°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00¹depends_on_disabled_cellsÂ§runtimeÎ ?Ìµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6cŠ¦queuedÂ¤logs§runningÂ¦output†¤body°KappaCoefficient¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7[ÎŽ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c¹depends_on_disabled_cellsÂ§runtimeÎ¸8vµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1efc1b34-ebf2-11ec-04f5-315c43e9338eŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ,kappa_score (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7'Îš°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1efc1b34-ebf2-11ec-04f5-315c43e9338e¹depends_on_disabled_cellsÂ§runtimeÎ ï …µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$063e664a-9b0d-4cf1-9b5f-1ec925a55353Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM2Œ¥j°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$063e664a-9b0d-4cf1-9b5f-1ec925a55353¹depends_on_disabled_cellsÂ§runtimeÎ 1.µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$649bde36-c7d6-4f6d-ba98-44cd16419bdeŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ<<div class="markdown"><h3>An online version ðŸŒœ</h3>
</div>¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÙÃM2Œ¿H°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$649bde36-c7d6-4f6d-ba98-44cd16419bde¹depends_on_disabled_cellsÂ§runtimeÎ ägµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcceŠ¦queuedÂ¤logs§runningÂ¦output†¤body³0.39999999999999997¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7µËó°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce¹depends_on_disabled_cellsÂ§runtimeÍÀ¿µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857fŠ¦queuedÂ¤logs§runningÂ¦output†¤body³0.39999999999999997¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7E®ï°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857f¹depends_on_disabled_cellsÂ§runtimeÍO!µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙî<div class="markdown"><h1>Cohen&#39;s Kappa score</h1>
<p class="tex">$$\kappa &#61; \frac&#123;p_0 - p_e&#125;&#123;1 - p_e&#125;$$</p>
<p class="tex">$$p_e &#61; \frac 1&#123;N^2&#125;\sum_k n_&#123;k1&#125;n_&#123;k2&#125;$$</p>
</div>¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÙÃM2Œ‘y°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59¹depends_on_disabled_cellsÂ§runtimeÎ ûÏµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ-kappa_score (generic function with 2 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7(÷°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4¹depends_on_disabled_cellsÂ§runtimeÎ ñäµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8ef1af36-9d02-4c8e-9663-22517ea95718Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ&value (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7lh4°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8ef1af36-9d02-4c8e-9663-22517ea95718¹depends_on_disabled_cellsÂ§runtimeÎ áµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$e4f625cd-f846-4fba-843c-3ffc4a4c563aŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyƒ¨elements’’’©0.0710383ªtext/plain’’¨0.039548ªtext/plain¤type¥Tuple¨objectid°540d1d58b5667429¤mimeÙ!application/vnd.pluto.tree+object¬rootassigneeÀ²last_run_timestampËAÙÃM7¥ûç°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$e4f625cd-f846-4fba-843c-3ffc4a4c563a¹depends_on_disabled_cellsÂ§runtimeÎ ¦Áµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c737a28b-2d90-4298-ac8e-91390ef43478Š¦queuedÂ¤logs§runningÂ¦output†¤body¤-1.0¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÙÃM7FŸÇ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c737a28b-2d90-4298-ac8e-91390ef43478¹depends_on_disabled_cellsÂ§runtimeÍKµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂ±cell_dependenciesŒÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00„´precedence_heuristic	§cell_idÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00´downstream_cells_map§update!’Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563aÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce²upstream_cells_map‰°KappaCoefficient‘Ù$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c¦length¡:§@assert¥throw®AssertionError¡+¥count¢==Ù$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c„´precedence_heuristic	§cell_idÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c´downstream_cells_map°KappaCoefficient”Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563aÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcceÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00Ù$8ef1af36-9d02-4c8e-9663-22517ea95718²upstream_cells_map„£Int¥zeros¦Matrix«Base.@kwdefÙ$1efc1b34-ebf2-11ec-04f5-315c43e9338e„´precedence_heuristic	§cell_idÙ$1efc1b34-ebf2-11ec-04f5-315c43e9338e´downstream_cells_map«kappa_score“Ù$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857fÙ$c737a28b-2d90-4298-ac8e-91390ef43478Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563a²upstream_cells_mapŒ¡:§@assert¦length¡-¥throw¡/¡^¡+®AssertionError¥count¢==¡*Ù$063e664a-9b0d-4cf1-9b5f-1ec925a55353„´precedence_heuristic	§cell_idÙ$063e664a-9b0d-4cf1-9b5f-1ec925a55353´downstream_cells_map€²upstream_cells_map€Ù$649bde36-c7d6-4f6d-ba98-44cd16419bde„´precedence_heuristic	§cell_idÙ$649bde36-c7d6-4f6d-ba98-44cd16419bde´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce„´precedence_heuristic	§cell_idÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce´downstream_cells_map€²upstream_cells_mapƒ°KappaCoefficient‘Ù$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c§update!‘Ù$426b93da-bdfd-4ef7-a076-2e6ca773be00¥value‘Ù$8ef1af36-9d02-4c8e-9663-22517ea95718Ù$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857f„´precedence_heuristic	§cell_idÙ$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857f´downstream_cells_map€²upstream_cells_map«kappa_score’Ù$1efc1b34-ebf2-11ec-04f5-315c43e9338eÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4Ù$ffe10202-04d0-41a0-8cde-dd9fe46c4e59„´precedence_heuristic	§cell_idÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4„´precedence_heuristic	§cell_idÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4´downstream_cells_map«kappa_score“Ù$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857fÙ$c737a28b-2d90-4298-ac8e-91390ef43478Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563a²upstream_cells_mapƒ¦length¦unique¥unionÙ$8ef1af36-9d02-4c8e-9663-22517ea95718„´precedence_heuristic	§cell_idÙ$8ef1af36-9d02-4c8e-9663-22517ea95718´downstream_cells_map¥value’Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563aÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce²upstream_cells_map‡°KappaCoefficient‘Ù$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c£sum¡-¡/¡^§eachrow¤prodÙ$e4f625cd-f846-4fba-843c-3ffc4a4c563a„´precedence_heuristic	§cell_idÙ$e4f625cd-f846-4fba-843c-3ffc4a4c563a´downstream_cells_map€²upstream_cells_mapŠ°KappaCoefficient‘Ù$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c¡:«kappa_score’Ù$1efc1b34-ebf2-11ec-04f5-315c43e9338eÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4¥@view£div¢Ã·¡+§update!‘Ù$426b93da-bdfd-4ef7-a076-2e6ca773be00¥value‘Ù$8ef1af36-9d02-4c8e-9663-22517ea95718¤randÙ$c737a28b-2d90-4298-ac8e-91390ef43478„´precedence_heuristic	§cell_idÙ$c737a28b-2d90-4298-ac8e-91390ef43478´downstream_cells_map€²upstream_cells_map«kappa_score’Ù$1efc1b34-ebf2-11ec-04f5-315c43e9338eÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4´cell_execution_orderœÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59Ù$1efc1b34-ebf2-11ec-04f5-315c43e9338eÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4Ù$063e664a-9b0d-4cf1-9b5f-1ec925a55353Ù$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857fÙ$c737a28b-2d90-4298-ac8e-91390ef43478Ù$649bde36-c7d6-4f6d-ba98-44cd16419bdeÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6cÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00Ù$8ef1af36-9d02-4c8e-9663-22517ea95718Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563aÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce´last_hot_reload_timeË        ©shortpath´cohen_kappa_score.jl®process_status¥ready¤pathÙ:/home/runner/work/Notebooks/Notebooks/cohen_kappa_score.jl­pluto_version¨v0.19.47®last_save_timeËAÙÃM2‹Ý×ªcell_orderœÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59Ù$1efc1b34-ebf2-11ec-04f5-315c43e9338eÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4Ù$063e664a-9b0d-4cf1-9b5f-1ec925a55353Ù$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857fÙ$c737a28b-2d90-4298-ac8e-91390ef43478Ù$e4f625cd-f846-4fba-843c-3ffc4a4c563aÙ$649bde36-c7d6-4f6d-ba98-44cd16419bdeÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcceÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6cÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00Ù$8ef1af36-9d02-4c8e-9663-22517ea95718±published_objects€¥nbpkgŠ¯install_time_nsÏ   Ó‡ç¬instantiatedÃ²installed_versions€°terminal_outputsªnbpkg_syncÚ@
[0m[1mInstantiating...[22m
[90m===[39m

[0m[1mResolving...[22m
[90m===[39m
[32m[1m  No Changes[22m[39m to `/tmp/jl_VHsF3F/Project.toml`
[32m[1m  No Changes[22m[39m to `/tmp/jl_VHsF3F/Manifest.toml`

[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m  Activating[22m[39m project at `/tmp/jl_VHsF3F`§enabledÃ·restart_recommended_msgÀ´restart_required_msgÀ­busy_packages¶waiting_for_permissionÂÙ,waiting_for_permission_but_probably_disabledÂ«cell_inputsŒÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00„§cell_idÙ$426b93da-bdfd-4ef7-a076-2e6ca773be00¤codeÚfunction update!(Îº::KappaCoefficient, y1, y2)
	@assert(length(y1) == length(y2))

	Îº.n_observations += length(y1)
	Îº.n_agreed += count(y1 .== y2)
	for cls in 1:Îº.n_classes
		Îº.preds[cls,:] .+= [count(y1 .== cls), count(y2 .== cls)]
	end

	return Îº.n_observations
end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c„§cell_idÙ$216cfb08-1159-4e69-ab34-bcfcfd7d3f6c¤codeÙ›Base.@kwdef mutable struct KappaCoefficient
	n_classes::Int

	n_agreed::Int = 0
	n_observations::Int = 0
	preds::Matrix{Int} = zeros(Int, n_classes, 2)
end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1efc1b34-ebf2-11ec-04f5-315c43e9338e„§cell_idÙ$1efc1b34-ebf2-11ec-04f5-315c43e9338e¤codeÚ(function kappa_score(y1, y2, n_categories)
	@assert length(y1) == length(y2)

	n_observations = length(y1)

	pâ‚‘ = 0.
	for i in 1:n_categories
		pâ‚‘ += count(y1 .== i) * count(y2 .== i)
	end
	pâ‚‘ /= n_observations ^ 2
	pâ‚’ = count(y1 .== y2) / n_observations 

	(pâ‚’ - pâ‚‘) / (1 - pâ‚‘)
end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$063e664a-9b0d-4cf1-9b5f-1ec925a55353„§cell_idÙ$063e664a-9b0d-4cf1-9b5f-1ec925a55353¤code ¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$649bde36-c7d6-4f6d-ba98-44cd16419bde„§cell_idÙ$649bde36-c7d6-4f6d-ba98-44cd16419bde¤codeÙ$md"""
### An online version ðŸŒœ
"""¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce„§cell_idÙ$cd6a99f3-8676-4a8d-9575-3b3c3499dcce¤codeÙlet
	Îº = KappaCoefficient(n_classes = 2)

	update!(Îº, [2], [2])
	update!(Îº, [1], [2])
	update!(Îº, [1], [1])

	value(Îº)
end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857f„§cell_idÙ$fe3e9afa-fbc0-45db-ba0f-03f2ff6d857f¤codeÙ!kappa_score([2, 1, 1], [2, 2, 1])¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59„§cell_idÙ$ffe10202-04d0-41a0-8cde-dd9fe46c4e59¤codeÙ‚md"""
# Cohen's Kappa score

```math
\kappa = \frac{p_0 - p_e}{1 - p_e}
```

```math
p_e = \frac 1{N^2}\sum_k n_{k1}n_{k2}
```
"""¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4„§cell_idÙ$50727c67-86e8-4ed6-88a5-7880b9a597d4¤codeÙIkappa_score(y1, y2) =
	kappa_score(y1, y2, length(unique(union(y1, y2))))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8ef1af36-9d02-4c8e-9663-22517ea95718„§cell_idÙ$8ef1af36-9d02-4c8e-9663-22517ea95718¤codeÙ³function value(Îº::KappaCoefficient)
	pâ‚’ = Îº.n_agreed / Îº.n_observations
	pâ‚‘ = sum(prod, eachrow(Îº.preds)) / (Îº.n_observations ^ 2)

	return (pâ‚’ - pâ‚‘) / (1 - pâ‚‘)
end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$e4f625cd-f846-4fba-843c-3ffc4a4c563a„§cell_idÙ$e4f625cd-f846-4fba-843c-3ffc4a4c563a¤codeÚ/let
	N = 20
	y1 = rand(1:10, N)
	y2 = rand(1:10, N)

	y11, y12 = @view(y1[begin:NÃ·2]), @view(y1[NÃ·2+1:end])
	y21, y22 = @view(y2[begin:NÃ·2]), @view(y2[NÃ·2+1:end])

	Îº = kappa_score(y1, y2)
	Îº2 = KappaCoefficient(n_classes = 10)

	update!(Îº2, y11, y21)
	update!(Îº2, y12, y22)

	Îº, value(Îº2)
end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$c737a28b-2d90-4298-ac8e-91390ef43478„§cell_idÙ$c737a28b-2d90-4298-ac8e-91390ef43478¤code»kappa_score([2, 1], [1, 2])¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂ«notebook_idÙ$af10b314-8a3e-11ef-2d1b-df45447cd920«in_temp_dirÂ¨metadata€