NIFI-15930: Re-init Secrets from a Provider with Controller Service

[bobpaulin]

• Wait for Parameter Provider Controller Services to enable they are valid prior to secret resolution
• Enable Working Context to apply any changes in Connector Property Values and Resolved values to the Parameter Context of the Working Flow Context
• Warn when a parameter provider is not found

Summary

NIFI-15930

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[kevdoran]

Thanks for this @bobpaulin - here are a few things worth mentioning, but overall the fixes in this PR are good.

Should fix

1. lastLoggedStatus is a slow leak

ParameterProviderSecretsManager.lastLoggedStatus is only cleaned up when a provider transitions back to VALID. If a provider is created, goes INVALID, and is then deleted before ever becoming VALID, its entry remains in the map for the lifetime of the manager. Over many flow synchronizations this is unbounded.

// Per-ParameterProvider-id deduplication for the WARN log emitted whenever a SecretReference
// resolution is skipped because its backing provider is not VALID. An entry is present for any
// provider id we have already surfaced via WARN; the value is the status that was logged so the
// same WARN is not repeated until the status either changes or the provider returns to VALID.
private final Map<String, ValidationStatus> lastLoggedStatus = new ConcurrentHashMap<>();

Easiest fix: at the start of getSecretProviders(), retain only IDs currently known to flowManager.getAllParameterProviders():

final Set<String> currentProviderIds = new HashSet<>();
for (final ParameterProviderNode node : flowManager.getAllParameterProviders()) {
    currentProviderIds.add(node.getIdentifier());
}
lastLoggedStatus.keySet().retainAll(currentProviderIds);

Consider

2. Move the initializationContext == null early-return above the "recreated" log

recreateWorkingFlowContext emits the "Working Flow Context has been recreated" info log even when there is no flow to refresh. Move the early-return ahead of the log so the log only fires when a refresh actually follows.

private void recreateWorkingFlowContext() {
    destroyWorkingContext();
    workingFlowContext = flowContextFactory.createWorkingFlowContext(...);
    getComponentLog().info("Working Flow Context has been recreated");

    if (initializationContext == null) {
        return;
    }
    final ConnectorConfiguration config = workingFlowContext.getConfigurationContext().toConnectorConfiguration();
    for (final NamedStepConfiguration stepConfig : config.getNamedStepConfigurations()) {
        ...
    }
}

3. Also log an INFO when a provider returns to VALID

Currently lastLoggedStatus.remove(providerId) runs silently when a provider becomes VALID again. Operators searching the log for the Skipping Parameter Provider … because warning have no nearby recovery line. Emitting an INFO at the recovery point would make the WARN/recovery pairs easy to correlate.

[bobpaulin]

Thank you for the review @kevdoran . I've addressed these feedback items. I went with a different approach on the logging since the intent of the message is surface when the working context has been set which includes the startup. This has been useful in debugging in past sessions. I added another message to detail when the logging has refreshed the config.

[kevdoran]

Thanks for those changes, LGTM, will get this merged

[exceptionfactory]

On a quick review, I'm concerned about the use of reflection for verifying lastWarnedStatus in the test class. Are there other options for changing this approach?

[exceptionfactory]

Thanks for streamlining the test and reverting the defensive change to VersionedFlowSynchronizer @bobpaulin. I noticed some of the log messages are more verbose than needed, but those can be addressed in a follow-on issue. The current version looks good, and I will plan to merge on successful build completion.

Thanks for the review @kevdoran!

[bobpaulin]

Thank you for the review @kevdoran and @exceptionfactory . Created https://issues.apache.org/jira/browse/NIFI-15943 to track logging improvements identified.