feat: add Distance analyzer for categorical feature drift (L-infinity, chi-squared) (#164)

[nikolauspschuetz]

Problem

Deequ's Distance analyzer (com/amazon/deequ/analyzers/Distance.scala) computes distribution distance for feature-drift detection (L-infinity and chi-squared), but it was not exposed in PyDeequ. Requested in #164 (labeled good first issue / help wanted / feature request).

What this adds

A Distance helper plus a CategoricalDistanceMethod enum exposing categorical distance:

from pydeequ.analyzers import Distance, CategoricalDistanceMethod

dist = Distance(spark)
# distribution dicts as produced by the Histogram analyzer: {category: count}
d = dist.categoricalDistance({"a": 5, "b": 5}, {"a": 10},
                             method=CategoricalDistanceMethod.LInfinity)

Supports both L-infinity and chi-squared, with the full alpha / Yates / Cochran parameters.

Design note (please scrutinize)

Deequ's Distance is a plain object with static-style methods, not an Analyzer subclass, so it cannot go through AnalysisRunBuilder.addAnalyzer(...).run(). I exposed it faithfully as a helper rather than faking analyzer integration.

categoricalDistance requires a Scala mutable.Map[String, Long]. py4j auto-unboxes individual java.lang.Long back to Python ints (re-entering as Integer), which makes Deequ's e._2.toDouble throw ClassCastException. To keep values genuinely Long-typed, each dict is round-tripped through a one-row Spark DataFrame with a MapType(StringType, LongType) column and collected JVM-side. Tradeoff: a small Spark job per distribution — acceptable for a drift helper, and the only reliable path found. Open to a lighter approach if reviewers prefer one.

Scope: categorical distance only. numericalDistance requires constructing a JVM QuantileNonSample[Double] with no clean Python path and is intentionally left out (matches the issue).

Tests

Added to tests/test_analyzers.py: test_Distance_categorical_LInfinity and test_Distance_categorical_Chisquare. Validated against the live Deequ 2.0.8 jar on Spark 3.5 (2 passed; full analyzer suite 27 passed / 22 pre-existing xfails, no regressions). Docs updated in docs/analyzers.md.

Closes #164

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

Ready for review. Adds a Distance helper (categorical L-infinity + chi-squared) for feature-drift detection, exposing Deequ's Distance object faithfully (it's not an Analyzer, so it's a helper rather than addAnalyzer). Validated against the live Deequ 2.0.8 jar on Spark 3.5: test_analyzers.py → 27 passed / 22 xfailed. See the PR body for a reviewer note on the py4j Long-boxing handling. cc @sudsali @chenliu0831 @rdsharma26 — would appreciate your review. Closes #164.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

Thanks for the automated review pass. The current revision addresses these findings:

• Map construction — _to_scala_mutable_long_map no longer uses createDataFrame/collectAsList. It builds the mutable.HashMap[String, Long] from JVM Long[] arrays via genericWrapArray/zip/toMap, so there is no per-call Spark job and no reliance on a java.util.Map -> Scala implicit conversion.
• Empty distributions — guarded: categoricalDistance raises ValueError if either distribution dict is empty.
• correctForLowNumberOfSamples docstring — matches Deequ semantics: True returns the raw/unscaled statistic, False returns the normalized (KS-corrected L-infinity / chi-squared) result.
• Seq.canBuildFrom (Scala 2.12) — documented as a known constraint in the method docstring; all supported Spark builds (3.1-3.5) ship Scala 2.12.
• Tests — added test_Distance_categorical_single_category, test_Distance_categorical_empty_dict_raises, and test_Distance_categorical_invalid_method_raises (plus an alpha test) covering the edge/error paths.

Resolving the threads accordingly.