[SPARK-57303][SQL] Store-assignment and up-cast rules for nanosecond-precision timestamp types

[MaxGekk]

What changes were proposed in this pull request?

This PR defines a precision-safe store-assignment / up-cast contract for the whole LTZ/NTZ timestamp family - the microsecond types (TIMESTAMP / TIMESTAMP_NTZ) and their nanosecond-precision counterparts (TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p), p in [7, 9]) - using a single notion of effective fractional-second precision (micros = 6, nanos = p).

For any ordered pair of timestamp-family types (including across the LTZ/NTZ boundary, which Spark already treats as a mutual up-cast for the micro types):

• target precision >= source precision: lossless widening -> up-cast (STRICT) and ANSI-store-assignable;
• target precision < source precision: lossy narrowing -> not an up-cast, blocked under ANSI so it can never silently truncate.

DATE <-> nanos is aligned to the micro DATE <-> TIMESTAMP behavior: DATE -> nanos is a lossless widening (up-cast + ANSI-store-assignable), while nanos -> DATE drops the time-of-day (not an up-cast, but still ANSI-store-assignable). LEGACY policy and explicit CAST are unchanged (they still truncate on narrowing). TIME <-> timestamp is unchanged and stays consistent with TIME <-> micros (never an up-cast, ANSI-store-assignable both ways).

Concretely:

• New shared private[sql] object TimestampFamily (sql/api) with fractionalPrecision(dt): Option[Int] plus isLtz / isNtz, reused across the rule sites (no type-hierarchy change, so no MiMa impact).
• UpCastRule.canUpCast: a single lossless-widening arm for the family (subsuming the existing TimestampType <-> TimestampNTZType cases), plus a generalized DATE -> family widening arm.
• Cast.canANSIStoreAssign: replaced the piecemeal per-subtask arms with one family narrowing block built on the shared helper, before the generic DatetimeType arm.
• TypeCoercionHelper.findWiderDateTimeType: refactored onto the shared helper (behavior-preserving) and updated the now-stale comment, since common-type resolution and the cast rules now agree on admissibility.

Why are the changes needed?

Before this change, the nanosecond timestamp types fell through the generic (_: DatetimeType, _: DatetimeType) arm in Cast.canANSIStoreAssign (risking silent sub-microsecond truncation handled only narrowly), and they were absent from UpCastRule.canUpCast, so STRICT store assignment and up-cast resolution rejected even lossless widening. This PR gives the family a complete, precision-safe contract consistent with the microsecond precedent.

Does this PR introduce any user-facing change?

No. The nanosecond-precision timestamp types are unreleased (@Unstable), so this only affects behavior within the unreleased branch.

How was this patch tested?

• Updated the SPARK-57293 / SPARK-57490 / cross-family / micro-boundary contract tests in CastSuiteBase to the precision-safe widening model.
• Added a full-matrix predicate test over all 8 timestamp-family types asserting canUpCast and canANSIStoreAssign are true iff target precision >= source precision, plus DATE and TIME consistency anchors.
• Ran CastSuite, CastWithAnsiOn/Off, TypeCoercionSuite, AnsiTypeCoercionSuite, DataTypeWriteCompatibilitySuite, V2WriteAnalysisSuite, and SQLQueryTestSuite (cast / try_cast / nanos / typeCoercion) - all pass with no golden-file changes.
• Added coverage for two downstream consumers of canUpCast that the predicate-level tests do not reach: a CastSuiteBase test that Cast.nullable's try-cast branch follows up-cast admissibility for the timestamp family (non-null preserved on widening, conservatively nullable on narrowing), and a new GeneratedColumnExpressionSuite asserting GeneratedColumnExpression.validate accepts a lossless widening generation expression and rejects a lossy narrowing.

Note on scope

canUpCast / canANSIStoreAssign feed several consumers beyond up-cast resolution and store assignment (generated-column validation, subquery decorrelation, V2 expression pushdown, Cast try-cast nullability, and the Spark Connect ArrowVectorReader guard). The widening relaxation here is lossless and applies uniformly to all of them. One follow-up item: nanosecond timestamp types are not yet supported over Spark Connect (no ConnectTypeOps / vector reader), so ArrowVectorReader's canUpCast guard no longer fails fast on a micro-vector/nanos-target mismatch; whenever nanos-over-Connect is implemented, that PR should add the reader and re-examine this guard.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor (Claude Opus 4.8)

[dongjoon-hyun]

+1, LGTM.