added support for MapFromEntries by athlcode · Pull Request #21720 · apache/datafusion

athlcode · 2026-04-18T14:18:42Z

Which issue does this PR close?

Closes # (none: prerequisite for apache/datafusion-comet#2706; follow-up to #17779 and #19274).

Rationale for this change

Spark's default mapKeyDedupPolicy is EXCEPTION, raising SparkRuntimeException with error class DUPLICATED_MAP_KEY on duplicate map keys. The existing Spark map_from_entries / map_from_arrays UDFs silently kept the last occurrence, forcing downstream engines like datafusion-comet to fall back to Spark.

What changes are included in this PR?

Duplicate map keys in Spark map_from_entries, map_from_arrays, and str_to_map now raise [DUPLICATED_MAP_KEY] Duplicate map key {key} was found, matching Spark's default behaviour and error class.

Are these changes tested?

Yes, via sqllogictest assertions covering the new error across the affected Spark map UDFs.

Are there any user-facing changes?

Yes. Duplicate keys now raise [DUPLICATED_MAP_KEY] under the default policy instead of silently collapsing to the last occurrence. No new config keys, no API changes.

…darshan7/datafusion into support/MapFromEntries

nuno-faria

Thanks @athlcode. Since this changes the current behavior, I think it would be a good opportunity to add a config that allows switching between exception and last key wins (e.g., datafusion.execution.mapKeyDedupPolicy). Otherwise existing code relying on the last key wins policy won't be able to adapt.

cc: @Jefffrey and @comphead since you both reviewed the original map_from_entries.

coderfender · 2026-04-19T16:37:29Z

https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html

coderfender

The PR looks good @athlcode . Left some comments and I would like to see some more tests covering various edge cases both on. the rust side and SLT side to add more confidence

…ort/MapFromEntries

athlcode · 2026-04-19T18:30:32Z

The PR looks good @athlcode . Left some comments and I would like to see some more tests covering various edge cases both on. the rust side and SLT side to add more confidence

yep fixing comments and will add tests

athlcode · 2026-04-19T18:40:43Z

Thanks @athlcode. Since this changes the current behavior, I think it would be a good opportunity to add a config that allows switching between exception and last key wins (e.g., datafusion.execution.mapKeyDedupPolicy). Otherwise existing code relying on the last key wins policy won't be able to adapt.

cc: @Jefffrey and @comphead since you both reviewed the original map_from_entries.

thanks @nuno-faria. I am looking into this as well

Jefffrey · 2026-04-21T05:22:53Z

Yes I think it would be a good idea to make this configurable instead of just switching the behaviour (otherwise we're in a loop of having TODO item to support the other behaviour)

coderfender · 2026-04-21T17:09:22Z

I agree with @Jefffrey unless you would have an immediate follow up @athlcode ?

athlcode · 2026-04-21T18:21:49Z

@coderfender I agree with @Jefffrey as well, thank you. The way to do this would be to use datafusion.execution.map_key_dedup_policy config as @nuno-faria mentioned earlier.

coderfender · 2026-04-21T18:58:43Z

Sure @athlcode please take a look at the current config naming conventions to keep them consistent

…darshan7/datafusion into support/MapFromEntries

…ort/MapFromEntries

coderfender · 2026-04-28T18:00:32Z

@athlcode , it seems like there are test failures here.

athlcode · 2026-04-29T17:55:33Z

@athlcode , it seems like there are test failures here.

fixing the issues. Will tag once I fix and push, thank you.

Jefffrey · 2026-04-30T05:45:26Z

        /// `false` — ANSI SQL mode is disabled by default.
        pub enable_ansi_mode: bool, default = false

+        /// Policy for handling duplicate keys in Spark-compatible map-construction


I wonder if we should consider having a namespace for Spark specific configs, cc @andygrove @comphead

This makes sense to me. A few weeks ago , I was proposing semantic layering in DF essentially calling spark functions with same name as datafusion , spark level config , ansi support and potentially spark friendly SQL. This should help users migrating from spark to directly move their leverage logic in datafusion

@coderfender @Jefffrey thank you, updated to namespace

Jefffrey · 2026-04-30T05:46:03Z

+        /// - `"LAST_WIN"`: keep the last occurrence of each duplicate key.
+        ///
+        /// Values are case-insensitive. Only affects functions under `datafusion/spark`.
+        pub map_key_dedup_policy: String, default = "EXCEPTION".to_string()


Would prefer having the enum value here along with the parsing, instead of only being a string

Reference

The config values aren't checked #17498

thank you, added MapKeyDedupPolicy enum with Exception (default) + LastWin

Jefffrey · 2026-04-30T06:20:07Z

+        )
+        .unwrap();
+
+        let map = as_map(&result);


Suggested change

let map = as_map(&result);

let map = result.as_map();

Using https://docs.rs/arrow/latest/arrow/array/trait.AsArray.html#method.as_map

thank you for the suggestion, pushed the changes

…ort/MapFromEntries

…darshan7/datafusion into support/MapFromEntries

…ort/MapFromEntries

added support for MapFromEntries

cc7d2cc

github-actions Bot added common Related to common crate spark labels Apr 18, 2026

athlcode and others added 3 commits April 18, 2026 19:49

Merge branch 'main' into support/MapFromEntries

28ab5d2

removed MapKeyDedupPolicy

ba3c933

Merge branch 'support/MapFromEntries' of https://github.com/KrishnaSu…

bc2296f

…darshan7/datafusion into support/MapFromEntries

github-actions Bot added sqllogictest SQL Logic Tests (.slt) and removed common Related to common crate labels Apr 18, 2026

nuno-faria reviewed Apr 19, 2026

View reviewed changes

Comment thread datafusion/spark/src/function/map/utils.rs Outdated

coderfender reviewed Apr 19, 2026

View reviewed changes

Comment thread datafusion/spark/src/function/map/utils.rs Outdated

Comment thread datafusion/sqllogictest/test_files/spark/map/map_from_entries.slt

coderfender mentioned this pull request Apr 19, 2026

issue: 2607- added support for MapFromEntries apache/datafusion-comet#3991

Draft

Merge branch 'main' of https://github.com/apache/datafusion into supp…

85bd989

…ort/MapFromEntries

athlcode and others added 3 commits April 20, 2026 01:00

changed error message

b2a2102

revert unwanted formatting, added tests

db7dddb

Merge branch 'main' into support/MapFromEntries

1b35bc3

athlcode added 3 commits April 27, 2026 22:39

added mapKeyDedupPolicy, LAST_WIN for MapFromEntries

8858003

Merge branch 'support/MapFromEntries' of https://github.com/KrishnaSu…

ba4e1b2

…darshan7/datafusion into support/MapFromEntries

Merge branch 'main' of https://github.com/apache/datafusion into supp…

233792f

…ort/MapFromEntries

github-actions Bot added the common Related to common crate label Apr 27, 2026

Merge branch 'main' into support/MapFromEntries

539b906

Jefffrey reviewed Apr 30, 2026

View reviewed changes

athlcode added 2 commits May 7, 2026 21:55

Merge branch 'main' of https://github.com/apache/datafusion into supp…

9e2a8c0

…ort/MapFromEntries

Merge branch 'support/MapFromEntries' of https://github.com/KrishnaSu…

3b7b6d2

…darshan7/datafusion into support/MapFromEntries

github-actions Bot added the auto detected api change Auto detected API change label May 7, 2026

athlcode added 2 commits May 7, 2026 22:27

added MapKeyDedupPolicy enum, dropped parse_map_key_dedup_policy

b98720d

added map_key_dedup_policy in config docs

ebb1557

github-actions Bot added the documentation Improvements or additions to documentation label May 7, 2026

athlcode added 3 commits May 7, 2026 23:43

changed map_key_dedup_policy to namespace

e0146c0

Merge branch 'main' of https://github.com/apache/datafusion into supp…

e14eb59

…ort/MapFromEntries

fixed formatting

b76e971

github-actions Bot removed the auto detected api change Auto detected API change label May 7, 2026

Conversation

athlcode commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

nuno-faria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderfender commented Apr 19, 2026

Uh oh!

coderfender left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

athlcode commented Apr 19, 2026

Uh oh!

athlcode commented Apr 19, 2026

Uh oh!

Jefffrey commented Apr 21, 2026

Uh oh!

coderfender commented Apr 21, 2026

Uh oh!

athlcode commented Apr 21, 2026

Uh oh!

coderfender commented Apr 21, 2026

Uh oh!

coderfender commented Apr 28, 2026

Uh oh!

athlcode commented Apr 29, 2026

Uh oh!

Jefffrey Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

coderfender May 5, 2026

Choose a reason for hiding this comment

Uh oh!

athlcode May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

athlcode May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

athlcode May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

athlcode commented Apr 18, 2026 •

edited

Loading