[#648] slow DN.valueOf / AVA normalization for nested DN-syntax values

[vharseko]

Problem

Reported in OpenDJ issue #648.

The following used to take more than 2.5 minutes:

String dnString = "NTLou=   r1oa"
    + "      +2.5.4.1=2.5.4.1=2.5.4.1=...(many)...=0=0\uFFFDoa"
    + "      +2.5.4.1=2.5.4.1=2.";
DN.valueOf(dnString);

and, equivalently, normalizing a single deeply-nested DN-syntax value
(AVA.getOrderingNormalizedValue()).

Root cause

The attribute type 2.5.4.1 is aliasedObjectName, whose syntax is DN and
whose equality matching rule is distinguishedNameMatch. Normalizing such a
value re-parses it as a DN and normalizes it recursively
(DistinguishedNameEqualityMatchingRuleImpl.normalizeAttributeValue →
DN.valueOf(...).toNormalizedByteString()).

Two problems compound each other:

1. Unnecessary normalization during RDN validation. When building a
   multi-valued RDN, RDN.validateAvas() called Arrays.sort(sortedAVAs),
   which uses AVA.compareTo → getOrderingNormalizedValue() → the recursive
   DN normalization. The sort is only used to detect duplicate attribute
   types, which does not need the values at all.

2. Exponential growth of the normalized form. RDN.toNormalizedByteString
   prefixes each RDN with a separator byte (0x00) and AVA.escapeBytes
   escapes the reserved bytes (0x00, 0x01, 0x02) by prefixing them with
   0x02. Each nesting level therefore roughly doubles the number of reserved
   bytes, so a value with N nested DN levels produces a normalized form of
   size ~2^N. At ~30 levels this is ~10^9 bytes → minutes of CPU and
   OutOfMemoryError.

Fix

1. RDN.validateAvas — detect duplicate types without normalizing values

opendj-core/.../ldap/RDN.java

• The default (3+ AVAs) branch now sorts a copy of the AVAs using a new
  ATTRIBUTE_TYPE_COMPARATOR that compares only the attribute types,
  instead of the full AVA.compareTo (which normalizes values).
• This makes DN.valueOf(dnString) from the report fail fast with
  ERR_RDN_DUPLICATE_AVA_TYPES (the top RDN contains 2.5.4.1 twice) instead
  of triggering the expensive normalization.

2. DistinguishedNameEqualityMatchingRuleImpl — bound recursive normalization

opendj-core/.../ldap/schema/DistinguishedNameEqualityMatchingRuleImpl.java

• Adds a per-thread recursion-depth guard (MAX_NESTED_DN_DEPTH = 100) for
  stack safety.
• Adds a guard on the size of the normalized value
  (MAX_NORMALIZED_VALUE_SIZE = 1 MiB). The first nesting level whose
  normalized form exceeds the limit throws a DecodeException; the enclosing
  AVA catches it (existing behaviour in AVA.getEqualityNormalizedValue /
  getOrderingNormalizedValue) and falls back to a byte-wise comparison.
  This caps the total work and memory regardless of the crafted nesting depth.
• New message ERR_ATTR_SYNTAX_DN_MAX_DEPTH added to core.properties.

Legitimate (shallow) nested DN-syntax values continue to normalize exactly as
before; only pathologically deep/large values degrade to byte-wise comparison.

Files changed

• opendj-core/src/main/java/org/forgerock/opendj/ldap/RDN.java
• opendj-core/src/main/java/org/forgerock/opendj/ldap/schema/DistinguishedNameEqualityMatchingRuleImpl.java
• opendj-core/src/main/resources/com/forgerock/opendj/ldap/core.properties
• opendj-core/src/test/java/org/forgerock/opendj/ldap/RDNTestCase.java
• opendj-core/src/test/java/org/forgerock/opendj/ldap/DNTestCase.java
• opendj-core/src/test/java/org/forgerock/opendj/ldap/schema/DistinguishedNameEqualityMatchingRuleTest.java

Tests

New tests (all run under a timeOut so a regression fails instead of hanging):

• RDNTestCase.testDuplicateDnSyntaxAvasAreDetectedQuickly — a multi-valued RDN
  with duplicate DN-syntax types is rejected quickly.
• RDNTestCase.testDistinctDnSyntaxAvasAreValidatedQuickly — a multi-valued RDN
  with distinct DN-syntax types is built quickly.
• DNTestCase.testValueOfWithDuplicateNestedDnSyntaxAvasIsFast — the exact
  reproduction from the report fails fast.
• DistinguishedNameEqualityMatchingRuleTest.testNormalizationOfDeeplyNestedDnValueIsBounded
  — normalizing a deeply nested DN-syntax value is bounded.

Verification

Before the fix, the reproduction normalization ran for minutes and ended with
OutOfMemoryError (even with -Xmx256m). After the fix:

• DN.valueOf(dnString) throws LocalizedIllegalArgumentException immediately.
• Normalizing a ~30-level nested value completes in tens of milliseconds.
• A 200-level nested value completes well under a second.

Affected test classes pass:

mvn -o -pl opendj-core test \
  -Dtest='RDNTestCase,DNTestCase,DistinguishedNameEqualityMatchingRuleTest,AVATestCase'
# All of the tests passed.