[SPARK-57681][SQL] Support dynamic table options for DELETE and UPDATE

[anuragmantri]

What changes were proposed in this pull request?

This PR add the dynamic table options syntax to DELETE and UPDATE statements.

Why are the changes needed?

This is a follow up to SPARK-49098 which itself was a follow up to SPARK-36680

The DSv2 API already provides RowLevelOperationInfo.options() and LogicalWriteInfo.options() for this purpose, but they were always empty for DELETE and UPDATE because the rewrite rules hardcoded CaseInsensitiveStringMap.empty().

This issue came up again in the Iceberg repo.

Does this PR introduce any user-facing change?

Yes, it adds new SQL syntax

  DELETE FROM catalog.db.table WITH (`k` = 'v') WHERE id > 10

  UPDATE catalog.db.table WITH (`k` = 'v' )
  SET status = 'archived' WHERE created_at < '2024-01-01'.

How was this patch tested?

• Parser (DDLParserSuite)
• End-to-end (DeleteFromTableSuiteBase, UpdateTableSuiteBase)DataSourceV2Relation
• LogicalWriteInfo (InMemoryRowLevelOperationTable)

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.

[anuragmantri]

@szehon-ho, tagging you since you made the INSERT counterpart :)

[uros-b]

Please consider this concern: silent option-drop on non-rewrite delete paths: for a table implementing both SupportsRowLevelOperations and SupportsDeleteV2 (e.g. Iceberg), a DELETE … WITH(...) WHERE can take the metadata-only/deleteWhere path (OptimizeMetadataOnlyDeleteFromTable / DataSourceV2Strategy), which has no options parameter; so the user's options are silently ignored. Same for the TruncatableTable truncate path. This would be a new user-visible "silently ignored" surprise, and no test currently covers this path.

[uros-b]

So this PR makes INSERT/DELETE/UPDATE accept WITH(...), but how about MERGE?

[uros-b]

Please make sure to either update the SQL reference docs, or file a followup Jira ticket to do so later.