[SPARK-57650][YARN][TESTS][FOLLOW-UP] Reduce YarnClusterSuite flakiness from runner contention

[HyukjinKwon]

What changes were proposed in this pull request?

Two test-only mitigations in BaseYarnClusterSuite to reduce YarnClusterSuite flakiness on busy CI runners:

• Give the mini NodeManager 8GB (yarn.nodemanager.resource.memory-mb) and a matching yarn.scheduler.maximum-allocation-mb, so executor allocation is never starved once the ~1.4GB AM is running.
• Raise the executor→driver connection retry budget (spark.rpc.io.maxRetries=10, spark.rpc.io.retryWait=2s) so a transient accept stall does not permanently fail the executor. These are defaults; individual tests can still override them.

Why are the changes needed?

This is a follow-up to SPARK-57650, which fixed the deterministic ACCEPTED-state hang (maximum-am-resource-percent). After that fix, the master Build / Maven yarn lanes still go red intermittently: YarnClusterSuite tests time out (handle.getState().isFinal() was false) because the AM/executor containers fail to connect to the driver's RPC server on localhost (Connection refused).

The in-JVM mini RM+NM, the driver subprocess, and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses the race exits after the default 3 connection retries, and the application can then never finish.

This is one of the two remaining unmerged fixes keeping the apache/spark master matrix builds red.

Does this PR introduce any user-facing change?

No. Test-only.

How was this patch tested?

resource-managers/yarn module tests on a fork, repeated multiple times to confirm the flakiness is gone (the formerly-failing YarnClusterSuite tests pass consistently).

• Before (still flaky on apache/spark master, after the SPARK-57650 base fix merged): Build / Maven (Scala 2.13, JDK 17) — resource-managers#yarn, YarnClusterSuite times out (handle.getState().isFinal() was false, BaseYarnClusterSuite.scala:220): https://github.com/apache/spark/actions/runs/28174751831
• After (passing on this branch): resource-managers/yarn module green on the fork — https://github.com/HyukjinKwon/spark/actions/runs/28204995015/job/83556094951

Was this patch authored or co-authored using generative AI tooling?

Yes.

This pull request and its description were written by Isaac.

[HyukjinKwon]

Closing as a duplicate of #56785, which carries the same test-only YarnClusterSuite mitigations as a focused draft and is CI-validated (full yarn-module Maven on JDK17 + JDK21). This branch also picked up unrelated commits. Let's consolidate on #56785.