[WIP] Add databricks dbconnect init / sync commands#5690
Draft
rugpanov wants to merge 33 commits into
Draft
Conversation
Brainstormed design for porting the dbconnect-init.sh demo into a real CLI subcommand namespace with init + sync commands, a shared phase pipeline, full target resolution, a surgical TOML merge, and a stable --json schema. Co-authored-by: Isaac
Bite-sized, TDD task breakdown (11 tasks) covering the command scaffold, result types, envKey mapping, constraint fetch+cache, surgical TOML merge, target resolution, uv package manager, the phase pipeline, Cobra wiring, acceptance tests, and changelog. Co-authored-by: Isaac
Regenerate the golden from the built binary; the prior hand-written version showed the command Short text instead of the rendered Long help. Co-authored-by: Isaac
- Remove noise doc comments from Error() and Unwrap() (idiomatic for standard interface methods) - Replace thin NewError doc comment with meaningful info about fmt.Sprintf and nil handling - Remove YAGNI default case from Mode.String(), use if/return instead Co-authored-by: Isaac
- Replace double TrimPrefix calls with simpler strings.TrimPrefix(strings.ToLower(version), "v") - Hoist pythonVersionRe to package-level var to avoid repeated compilation - Remove noise comment that restated the code Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
The PythonMinorFromRequires call happens after a successful network fetch, so wrapping its error with ErrConstraintFetchFailed was a misattribution. Use ErrValidationFailed instead, which correctly signals that the constraint file content failed to parse rather than that the fetch itself failed. Co-authored-by: Isaac
…etch Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
- Add json tags to PipelineError (code/message/-) so --output json emits the documented contract instead of Go field names - Change uv version probe from "version" subcommand to --version flag to avoid project-scoped failure when no pyproject.toml exists in cwd - Guard renderResult against nil res: synthesize a minimal Result with error populated so JSON mode always emits a structured object - Use i+1 for 1-based phase numbering in text output - Add comment explaining why ValidateTargetFlags is kept alongside MarkFlagsMutuallyExclusive Co-authored-by: Isaac
Add acceptance tests for the dbconnect init/sync feature: - flag-conflict: verifies Cobra mutual exclusion of --cluster/--serverless/--job - no-target: verifies error when no compute target is selected - serverless-check: verifies --serverless v4 --check with stubbed constraint server - serverless-json: verifies --output json with full Result struct - cluster-unsupported: verifies constraint fetch failure for unsupported DBR version - help/test.toml: opts out of bundle-engine matrix for the help case Each case stubs the test server via [[Server]] in test.toml and uses DATABRICKS_DBCONNECT_CONSTRAINT_SOURCE=$DATABRICKS_HOST to point the constraint fetch at the local test server. Co-authored-by: Grigory Panov
no-target and cluster-unsupported tests use commands that must fail; musterr asserts this and fails the test if the command unexpectedly succeeds. errcode is for tolerated failures only. Co-authored-by: Isaac
Co-authored-by: Isaac
Also standardize the serverless-json acceptance uv-version replacement regex to the unwrapped form used by the sibling cases. Co-authored-by: Isaac
…d cluster-unsupported scaffolding Co-authored-by: Isaac
…rors Co-authored-by: Isaac
Co-authored-by: Isaac
These are internal process artifacts and don't belong in the databricks/cli tree. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 8f656d9
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / do-not-merge: opened for early review; not ready to merge.
Changes
Adds a new
databricks dbconnectcommand namespace with two subcommands:databricks dbconnect init— create a freshpyproject.tomland provision a matched.venv.databricks dbconnect sync— merge managed dependencies into an existingpyproject.tomland re-provision.From the selected Databricks compute target (serverless / cluster / job), the command derives and provisions a local Python environment matched to the runtime: the right Python version, the right
databricks-connectpin, and dependency constraints so local resolution matches the Databricks runtime. It runs a phase pipeline: discoveruv→ resolve target → fetch the per-environment constraints (configurable base URL, with an offline cache) → plan → apply → ensure Python →uv sync→ seed pip → validate.Implementation notes:
cmd/dbconnect/) over a unit-testable pipeline (libs/dbconnect/), with aPackageManagerinterface seam (uv implemented; pip/conda can follow).pyproject.tomlmerge that touches only three managed regions and preserves the user's comments, ordering, and their own[tool.uv]keys; idempotent.GetByClusterId→ DBR → envKey, serverless, and job compute) with three-state messaging.~/.config/pip/pip.confindex-url→UV_INDEX_URL(uv ignorespip.conf).--checkdry-run prints the plan + diff and changes nothing;--output jsonemits a stable structured schema, and--debugadds diagnostic logging for troubleshooting on machines we can't access.Why
Promotes a proven proof-of-concept shell script into a real CLI command so the VS Code extension (and users directly) can set up a local environment matched to their compute, instead of guessing Python and
databricks-connectversions. Doing the version/constraint resolution from the compute target avoids local/remote drift.Tests
libs/dbconnect/: merge edge cases (single/multi-line arrays, quote styles, CRLF, idempotency, preserving user[tool.uv]keys), envKey mapping + Python-version parsing, target resolution (precedence + three-state), constraint fetch with offline-cache fallback, and pipeline orchestration incl.--checkgating and validation.acceptance/dbconnect/: serverless--check,--output jsonshape, no-target error, cluster-unsupported, flag conflict, and JSON-mode error exit code..venvwithdatabricks-connect17.x and the injected constraints.Out of scope for this first cut: pip/conda package managers (interface only) and the nearest-supported envKey fallback.
This pull request and its description were written by Isaac.