Strip azure_blob source + GCP impersonation from v0.51-exaforce#33
Closed
sundaresanr wants to merge 1 commit intov0.51-exaforcefrom
Closed
Strip azure_blob source + GCP impersonation from v0.51-exaforce#33sundaresanr wants to merge 1 commit intov0.51-exaforcefrom
sundaresanr wants to merge 1 commit intov0.51-exaforcefrom
Conversation
The exaforce azure_blob queue source and src/azure/mod.rs auth helpers
were written against azure-core 0.21; upstream v0.51 bumped to 0.25, so
this code can't compile on v0.51 without a proper SDK migration. Neither
the azure_blob source nor impersonated_service_account is used in
production Vector configs, so dropping both is the path of least
resistance to getting a v0.51-based vector-base image.
Deletions:
- src/azure/mod.rs (exaforce-added)
- src/internal_events/azure_queue.rs (exaforce-added)
- src/sources/azure_blob/{mod,queue,integration_tests,test}.rs (all
exaforce-added)
Reverted to upstream v0.51 (1e5bc95):
- src/lib.rs, src/internal_events/mod.rs, src/sources/mod.rs
- src/sinks/azure_blob/{config,integration_tests}.rs
- src/sinks/azure_common/config.rs
- src/gcp.rs, src/sinks/gcp/stackdriver/metrics/tests.rs
- scripts/environment/{prepare.sh,bootstrap-ubuntu-24.04.sh},
scripts/integration/azure/compose.yaml (cosmetic rebase leftovers)
Kept (parquet codec):
- lib/codecs/{Cargo.toml, src/encoding/...} — parquet codec + batch
serializer enum
- src/codecs/encoding/config.rs — wiring
- src/sinks/aws_s3/{config,sink}.rs — parquet output
- src/sinks/util/encoding.rs — Encoder<Vec<Event>> for batching
- src/components/validation/resources/mod.rs — Parquet stub
Additional codecs fixups needed on top of exaforce's original parquet
patches (they were broken on v0.51):
- lib/codecs/src/encoding/serializer.rs: add Parquet* to format::*
imports; fix Gelf unit-vs-tuple match arm; add #[cfg(opentelemetry)]
Otlp branch to build_batched().
- lib/codecs/src/encoding/format/parquet.rs: drop `ref mut` (Rust
2024 match ergonomics).
- lib/codecs/src/encoding/mod.rs: re-export BatchSerializer.
- src/codecs/encoding/config.rs: restore vector_lib::configurable::
configurable_component import (exaforce accidentally dropped it).
- src/components/validation/resources/mod.rs: add Parquet arm to the
serializer-to-deserializer match.
- Cargo.lock: pin pulsar back to 6.3.1 (merge had drifted to 6.5.0
which needs ProducerOptions fields the sink doesn't set).
Verified: cargo check -p vector (default features) and cargo check
--tests -p vector --no-default-features --features
azure-integration-tests both green locally.
Author
|
Closing in favor of a cleaner two-PR sequence:
The strip-then-keep approach here worked but leaves ~1500 lines of azure code in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parquet-only v0.51-exaforce base. The azure_blob queue source + `src/azure/mod.rs` auth helpers pulled over from v0.45-exaforce used azure-core 0.21 APIs; upstream v0.51 is on 0.25, so those files can't compile without a proper SDK migration. Since neither feature is used in production Vector configs (confirmed: no `type: azure_blob` sources in any chart, no `impersonated_service_account` in any sink config), dropping them is the quickest path to a working v0.51-based `vector-base` image.
Deletions (6 files, ~1490 lines)
Reverts to upstream v0.51 (1e5bc95)
Kept (parquet codec + fixups)
All the `lib/codecs` parquet pieces, `src/codecs/encoding/config.rs` wiring, `src/sinks/aws_s3/{config,sink}.rs`, `src/sinks/util/encoding.rs` batch encoder trait.
Had to fix issues left over in the merged v0.51-exaforce parquet patches:
Local verification
Both cover the commands the publish workflow runs (`make test-integration-azure` + `make package-aarch64-unknown-linux-gnu-all`).
Downstream
Once merged, the push to `v0.51-exaforce` will trigger `exaforce-publish.yaml` → `vector-base:v0.51-exaforce-` lands in ECR → `operations` repo `goservices/vector_exec_sources/Dockerfile` gets bumped in a separate follow-up PR.