Fix JMX RMI connector startup failure introduced by CVE-2026-46495 hardening

[vharseko]

Summary

The hardening introduced in commit 7e3a75903159153c877daeb2952a552701e38044
(CVE-2026-46495 OpenDJ Unauthenticated RCE via Java Deserialization in JMX RMI)
prevented the JMX RMI connector from starting on the JDK. This PR fixes the
startup regression while keeping the deserialization protection in place.

Problem

RmiConnector.configureJmxDeserializationProtection(...) populated the JMX
environment map with two mutually exclusive properties at the same time:

• jmx.remote.rmi.server.credential.types
• jmx.remote.rmi.server.credentials.filter.pattern

The JDK forbids specifying both. As a result RMIJRMPServerImpl's constructor
threw:

java.lang.IllegalArgumentException: Cannot specify both
  "jmx.remote.rmi.server.credential.types" and
  "jmx.remote.rmi.server.credentials.filter.pattern"
    at javax.management.remote.rmi.RMIJRMPServerImpl.<init>(RMIJRMPServerImpl.java:111)
    at javax.management.remote.rmi.RMIConnectorServer.newJRMPServer(RMIConnectorServer.java:774)
    at javax.management.remote.rmi.RMIConnectorServer.newServer(RMIConnectorServer.java:708)
    at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:449)

Because startConnectorNoClientCertificate() failed, the
jmxRmiConnectorNoClientCertificate reference stayed null, and the JMX test
suite failed during setup:

JmxPrivilegeTestCase.setUp:89 -> JmxTestCase.setUp:60 -> JmxTestCase.getJmxConnectionHandler:86
Expecting actual not to be null

Fix

In RmiConnector:

• Removed the jmx.remote.rmi.server.credential.types environment property
  (and the now-unused JMX_REMOTE_RMI_SERVER_CREDENTIAL_TYPES and
  JMX_CREDENTIAL_TYPES constants).
• Kept only the credentials-scoped JEP 290 deserialization filter
  jmx.remote.rmi.server.credentials.filter.pattern =
  maxdepth=3;maxarray=2;java.lang.String;!*. This filter is stricter than the
  credential-types allowlist because it also constrains array length and
  nesting depth.
• Documented that the two properties are mutually exclusive so the conflict is
  not reintroduced.

In RmiAuthenticatorTest:

• Removed the credentialTypesAreDefensivelyCopied test and the
  credential.types assertions.
• Added an assertion in configuresCredentialDeserializationProtection to
  guarantee jmx.remote.rmi.server.credential.types is not set.

Security impact

The deserialization protection against CVE-2026-46495 is fully preserved:

• The JEP 290 credentials filter allows only String / String[] payloads up
  to length 2 and depth 3, rejecting any other class (!*).
• RmiAuthenticator.authenticate(...) still validates that the credentials are
  a two-element String[] before any bind attempt.

The filter remains scoped to the credentials object only, so legitimate JMX RMI
traffic (MBean operations, notifications) is unaffected.

Testing

• Reproduced the original IllegalArgumentException with an isolated
  RMIConnectorServer start using both properties.
• Verified the connector starts successfully (isActive() == true) with only
  the credentials filter pattern configured.
• RmiAuthenticatorTest and the JMX test suite (JmxPrivilegeTestCase,
  JmxTestCase) compile and run again.