Short expansion of conditional mutual information (#387)

* wip... * Reorganize into separate file for the SECMI test * tests, docs * changelog + version * Slightly reorganize tests * fix tests * docs * no need to store shuffles * examples * fix deprecated syntax * reproducible tests * Add cross-references * documentation example for SECMI * Actually show docstrings * docstring * docstring * add relevant imports * reproducible tests * better description * Fix implementation for mu < 0 * CI badge for main branch only * Update changelog and version * Increase sample size to have enough points to get consistent results * better tests for secmi * a comment explaining the marginal selection * Add min/max variables for SECMI * Use SECMITest in oce tests
JuliaDynamics · Nov 20, 2024 · 7bfb8b9 · 7bfb8b9 · kahaaga · Nov 21, 2024
1 parent 1054b8f
commit 7bfb8b9
Show file tree

Hide file tree

Showing 22 changed files with 479 additions and 97 deletions.
diff --git a/Project.toml b/Project.toml
@@ -2,7 +2,7 @@ name = "Associations"
 uuid = "614afb3a-e278-4863-8805-9959372b9ec2"
 authors = ["Kristian Agasøster Haaga <[email protected]>", "Tor Einar Møller <[email protected]>", "George Datseris <[email protected]>"]
 repo = "https://github.com/kahaaga/Associations.jl.git"
-version = "4.3.0"
+version = "4.4.0"
 
 [deps]
 Accessors = "7d9f7c33-5ae7-4f3b-8dc6-eff91059b697"

diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # Associations
 
-[![CI](https://github.com/juliadynamics/Associations.jl/workflows/CI/badge.svg)](https://github.com/JuliaDynamics/Associations.jl/actions)
+[![CI (main)](https://github.com/juliadynamics/Associations.jl/workflows/CI/badge.svg?branch=main)](https://github.com/JuliaDynamics/Associations.jl/actions)
 [![](https://img.shields.io/badge/docs-latest_tagged-blue.svg)](https://juliadynamics.github.io/Associations.jl/stable/)
 [![](https://img.shields.io/badge/docs-dev_(main)-blue.svg)](https://juliadynamics.github.io/Associations.jl/dev/)
 [![codecov](https://codecov.io/gh/JuliaDynamics/Associations.jl/branch/main/graph/badge.svg?token=0b71n6x6AP)](https://codecov.io/gh/JuliaDynamics/Associations.jl)

diff --git a/changelog.md b/changelog.md
@@ -2,6 +2,11 @@
 
 From version v4.0 onwards, this package has been renamed to to Associations.jl.
 
+# 4.4
+
+- New association measure: `SECMI` (`ShortExpansionConditionalMutualInformation`)
+- New independence test: `SECMITest`, which is based on `SECMI`.
+
 # 4.3
 
 - Compatiblity with StateSpaceSets.jl v2.X

diff --git a/docs/refs.bib b/docs/refs.bib
@@ -1333,4 +1333,14 @@ @article{Azadkia2021
   pages={3070--3102},
   year={2021},
   publisher={Institute of Mathematical Statistics}
+}
+
+@article{Kubkowski2021,
+  title={How to gain on power: novel conditional independence tests based on short expansion of conditional mutual information},
+  author={Kubkowski, Mariusz and Mielniczuk, Jan and Teisseyre, Pawe{\l}},
+  journal={Journal of Machine Learning Research},
+  volume={22},
+  number={62},
+  pages={1--57},
+  year={2021}
 }
diff --git a/docs/src/associations.md b/docs/src/associations.md
@@ -107,6 +107,12 @@ EmbeddingTE
 PartialMutualInformation
 ```
 
+### Short expansion of conditional mutual information
+
+```@docs
+ShortExpansionConditionalMutualInformation
+```
+
 ## [Correlation measures](@id correlation_api)
 
 ```@docs

diff --git a/docs/src/examples/examples_associations.md b/docs/src/examples/examples_associations.md
@@ -1093,6 +1093,24 @@ est = MIDecomposition(CMIShannon(base = 2), KSG1(k = 10))
 association(est, x, z, y)
 ```
 
+## [`ShortExpansionConditionalMutualInformation`](@ref)
+
+### [[`JointProbabilities`](@ref) with [`CodifyVariables`](@ref) and [`ValueBinning`](@ref)](@id example_ShortExpansionConditionalMutualInformation_JointProbabilities_CodifyVariables_ValueBinning)
+
+```@example
+using Associations
+using Test
+using Random; rng = Xoshiro(1234)
+n = 20
+x = rand(rng, n)
+y = randn(rng, n) .+ x .^ 2
+z = randn(rng, n) .* y
+
+# An estimator for estimating the SECMI measure
+est = JointProbabilities(SECMI(base = 2), CodifyVariables(ValueBinning(3)))
+association(est, x, z, y)
+```
+
 ### [[`EntropyDecomposition`](@ref) + [`Kraskov`](@ref)](@id example_CMIShannon_EntropyDecomposition_Kraskov)
 
 Any [`DifferentialInfoEstimator`](@ref) can also be used to compute conditional

diff --git a/docs/src/examples/examples_independence.md b/docs/src/examples/examples_independence.md
@@ -469,4 +469,54 @@ connecting `x` and `z`.)
 independence(test, x, z, y)
 ```
 
-The test verifies our expectation.
+The test verifies our expectation.
+## [[`SECMITest`](@ref)](@id example_SECMITEST)
+
+## [[`JointProbabilities`](@ref) estimation on numeric data](@id example_SECMITEST_JointProbabilities_CodifyVariables_ValueBinning)
+
+```@example example_SECMITEst
+using Associations
+using Test
+using Random; rng = Xoshiro(1234)
+n = 25
+x = rand(rng, n)
+y = randn(rng, n) .+ x .^ 2
+z = randn(rng, n) .* y
+
+# An estimator for estimating the SECMI measure
+est = JointProbabilities(SECMI(base = 2), CodifyVariables(ValueBinning(3)))
+test = SECMITest(est; nshuffles = 19)
+```
+
+When analyzing ``SECMI(x, y | z)``, the expectation is to reject the null hypothesis (independence), since `x` and `y` are connected, regardless of the effect of `z`.
+
+```@example example_SECMITEst
+independence(test, x, y, z)
+```
+
+We can detect this association, even for `n = 25`! When analyzing ``SECMI(x, z | y)``, we 
+expect that we can't reject the null (indepdendence), precisely since `x` and `z` are *not* 
+connected when "conditioning away" `y`.
+
+```@example example_SECMITEst
+independence(test, x, z, y)
+```
+
+## [[`JointProbabilities`](@ref) estimation on categorical data](@id example_SECMITEST_JointProbabilities_CodifyVariables_UniqueElements)
+
+Note that this also works for categorical variables. Just use [`UniqueElements`](@ref) to 
+discretize!
+
+```@example example_SECMITest_categorical
+using Associations
+using Test
+using Random; rng = Xoshiro(1234)
+n = 24
+x = rand(rng, ["vegetables", "candy"], n)
+y = [xᵢ == "candy" && rand(rng) > 0.3 ? "yummy" : "yuck" for xᵢ in x]
+z = [yᵢ == "yummy" && rand(rng) > 0.6 ? "grown-up" : "child" for yᵢ in y]
+d = CodifyVariables(UniqueElements())
+est = JointProbabilities(SECMI(base = 2), d)
+
+independence(SECMITest(est; nshuffles = 19), x, z, y)
+```
diff --git a/docs/src/independence.md b/docs/src/independence.md
@@ -50,3 +50,10 @@ JDDTestResult
 CorrTest
 CorrTestResult
 ```
+
+## [`SECMITest`](@ref)
+
+```@docs
+SECMITest
+SECMITestResult
+```