feat(fluent-bit-output-plugin): Add dynamic CRD detection

[iypetrov]

How to categorize this PR?

/kind enhancement
/area logging

What this PR does / why we need it:
Currently, the Fluent Bit plugin depends on two CRDs to determine the correct log destination based on the environment in which it is deployed:

• Clusters from the extensions.gardener.cloud API group
• OpenTelemetryCollector from the opentelemetry.io API group

Previously, if the plugin was deployed before the required CRD became available, it would fail to start and crash. While this was partially mitigated in the Gardener environment because the fluent-operator continuously restarts failed processes until the CRD appears, relying on restart retries is not a robust solution. In certain scenarios, this behavior could lead to instability and potential log loss.

To address this issue, a CRD availability watch mechanism has been introduced for both dependencies. When the required CRD is not available, the plugin automatically routes logs to a fallback OTLP backend, which corresponds to the seed's logging stack. Once the required CRD becomes available, the plugin dynamically switches traffic to the intended destination without requiring a restart.

This approach significantly improves resilience and reliability by eliminating the dependency on repeated process restarts. It also makes the plugin more flexible across different deployment scenarios, ensuring predictable behavior regardless of the order in which components and CRDs become available.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Improved Fluent Bit resilience by adding dynamic CRD detection and automatic fallback log routing, preventing startup failures when required CRDs are not yet available.

[gardener-prow]

@iypetrov: The label(s) kind/todo cannot be applied, because the repository doesn't have them.

Details

In response to this:

How to categorize this PR?

/kind TODO
/area logging

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[iypetrov]

/cc @nickytd @plkokanov

[gardener-prow]

@iypetrov: GitHub didn't allow me to request PR reviews from the following users: plkokanov.

Note that only gardener members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @nickytd @plkokanov

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nickytd]

/lgtm

[gardener-prow]

LGTM label has been added.

Details

Git tree hash: 659f4f35f3ba8d950a7e4f8ef94f628fb0028214

[nickytd]

/lgtm

[gardener-prow]

LGTM label has been added.

Details

Git tree hash: 5154a154a074349c01c2c0828db349a974dc33d3

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nickytd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [nickytd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment