[Experiment] Combine CI Build and Queue Tests into a single invocation#54933
Draft
mmitche wants to merge 3 commits into
Draft
[Experiment] Combine CI Build and Queue Tests into a single invocation#54933mmitche wants to merge 3 commits into
mmitche wants to merge 3 commits into
Conversation
Today the sdk-build.yml job runs two sequential build.{ps1,sh} invocations:
a "Build" step (-restore -build -pack) and a "Queue Tests" step
(-restore -test, scoped to test/UnitTests.proj). The second step re-restores
and recompiles much of the product because UnitTests.proj publishes every
*.Tests.csproj with a RuntimeIdentifier, propagating the RID through all
product ProjectReferences into RID-qualified output folders.
This experimental change merges the two into one invocation to measure the
savings:
- sdk.slnx now includes test/UnitTests.proj (the Helix orchestrator) so test
queuing can happen as part of the same build.
- sdk.slnf is a new default/build-only filter: sdk.slnx minus UnitTests.proj.
Build-only legs (runTests=false) and the dotnet-format integration leg use
it so they neither pull in the orchestrator nor queue Helix work.
- eng/XUnitV3/XUnitV3.Runner.targets guards the in-process RunTests target on
CustomHelixTargetQueue being empty, so building the full solution with -test
does NOT run every *.Tests.csproj on the build agent; UnitTests.proj still
submits them to Helix.
- sdk-build.yml replaces the separate Build + Queue Tests steps with a single
combined "Build and Test" step when runTests=true (full sdk.slnx, -build
-pack -test + Helix props) and a "Build" step otherwise (sdk.slnf).
Known caveats (this is a draft/experiment):
- Build ordering: UnitTests.proj has no ProjectReferences to the product, so a
single solution build may schedule its publish relative to the product in a
new way; the draft CI run is intended to surface this.
- Local `build.cmd`/`build.sh` with no args auto-detects sdk.slnx, which now
includes UnitTests.proj and its heavy RID publish; devs wanting a lean build
should pass `-projects sdk.slnf`.
- sdk.slnf enumerates all projects except UnitTests.proj and must be kept in
sync as projects are added/removed.
- The RID double-compile itself is not eliminated (it originates in
UnitTests.proj's publish); this measures the restore/bootstrap savings from
not running two full invocations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… adding .proj to .slnx The first attempt added test/UnitTests.proj to sdk.slnx and generated sdk.slnf to exclude it. That broke every CI leg: the .slnx solution format (SolutionPersistence) has no registered project type for the .proj extension, so loading sdk.slnx failed with "ProjectType '' not found" (MSB4025) for all consumers of the solution/filter. Instead, keep sdk.slnx unchanged and have the combined "Build and Test" step build both the product solution and the Helix orchestrator in one invocation via /p:Projects=sdk.slnx;test/UnitTests.proj MSBuild builds listed projects in order, so the product is built before UnitTests.proj publishes/queues the tests. Build-only legs build just sdk.slnx, exactly as the former 'Build' step did. This removes sdk.slnf entirely (and its sync fragility) and reverts the sdk.slnx and dotnet-format-integration.yml edits. The RunTests in-process guard in eng/XUnitV3/XUnitV3.Runner.targets is retained so the per-project *.Tests.csproj do not run on the build agent when queuing to Helix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MSBuild's /p: switch uses ';' to separate multiple property assignments, so
passing /p:Projects=sdk.slnx;test/UnitTests.proj unquoted caused MSBuild to
parse the second path as its own (invalid) switch:
MSBUILD : error MSB1006: Property is not valid.
Switch: .../test/UnitTests.proj
Wrap the value in inner double quotes ('/p:Projects="a;b"') so MSBuild treats
the semicolon-separated list as a single property value, matching the original
Queue Tests step. The Windows leg already used the equivalent escaped-quote form.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
Experiment CI results (build 1477997)Plumbing now works (argument parsing fixed,
Root cause: a single Arcade Conclusion: naively merging into one invocation isn't viable without either (a) scoping the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment: combine the CI Build and Queue Tests steps
Background
The SDK build job runs two sequential invocations in the same ADO job:
-restore -build -pack-restore -testscoped totest/UnitTests.projThe second invocation re-restores and recompiles much of the product.
test/UnitTests.projpublishes every*.Tests.csprojwith aRuntimeIdentifier, which propagates the RID down all productProjectReferences into RID-qualified output folders (.../<tfm>/<rid>/). So ~55 product projects and the whole restore graph are processed twice across the two steps.What this PR does
sdk.slnxnow includestest/UnitTests.proj(the Helix orchestrator), so test queuing can happen as part of a single build.sdk.slnf(new) is a default/build-only filter =sdk.slnxminusUnitTests.proj. Build-only legs (runTests=false) and the dotnet-format integration leg use it so they neither pull in the orchestrator nor queue Helix work.eng/XUnitV3/XUnitV3.Runner.targets— the in-processRunTeststarget is now guarded onCustomHelixTargetQueue == ''. Building the full solution with-testtherefore does not run every*.Tests.csprojon the build agent;UnitTests.projstill submits them to Helix.eng/pipelines/templates/jobs/sdk-build.yml— the separate Build + Queue Tests steps are replaced with a single Build and Test step whenrunTests=true(fullsdk.slnx,-build -pack -test+ Helix props), and a Build step otherwise (sdk.slnf).Known caveats (why it's a draft)
UnitTests.projhas noProjectReferences to the product, so a single solution build may schedule its RID publish relative to the product differently. The draft CI run is meant to surface this.build.cmd/build.shwith no args auto-detectssdk.slnx, which now includesUnitTests.projand its heavy RID publish. Lean local builds should pass-projects sdk.slnf.sdk.slnfmaintenance: it enumerates all projects exceptUnitTests.projand must be kept in sync as projects are added/removed (.slnfhas no exclusion syntax).UnitTests.proj's publish). This experiment measures the restore/bootstrap savings from not running two full invocations, not compile elimination.🤖 Generated with assistance from GitHub Copilot.