Average multiple microbenchmarks results#5228
Closed
VincentBu wants to merge 3 commits into
Closed
Conversation
…ks when creating suites add json-trace map and implement AnalyzeForBenchmark Calculate comparison result by benchmark name, rename classes and adjust namespaces present list of microbenchmarkresults rename iteration section to iterations for Run.yaml Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> fix for microbencharmks comparison fix bugs intrduced in previous commit Add json-only comparison extract a shared helper for analyze command take trace type into consideration move MicrobenchmarkResult to GC.Infrastructure.Core.Analysis.Microbenchmarks namespace validate if run is null rename PauseDurationSeconds_SumWhereIsGen1 to PauseDurationMSec_SumWhereIsGen1 add properties for microbenchmarks comparison get value from StatsData if property is not found in GCTraceMetrics filtering to finite values check possible null references add console output when analyzing results assign value for PromotedMB_MeanWhereIsGen1 Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> redesign microbenchmarks result
move presentation to analyze-command improve performance of analyzing stage includes int type properties check key existence and set parallelism degree to 2 * cpu_count check if metricName is a key of StatsData Filter out null GCTraceMetrics instances before calling CompareGCTraceMetric update initialization of OtherMetrics for MicrobenchmarkComparisonResult comment out cpu_columns related code remove unused imports "Microbechmark" (missing 'n') avoid breaking formatting sort improvements in ascending order Convert to ToDictionary(x => x.Item1, x => x.Item2) (or tuple names) for each projection provide key value pair projection sort in order by index take non-replayable enumerables into consideration check if TraceConfigurations is null skip null or empty MicrobenchmarkResult collection skip null or empty MicrobenchmarkResults collection remove Spectre markup tokens from output path Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Check if first baseline/comparand is null Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> apply q1 and q3 on valid collection use run.Name instead of trace path check possible null value check if metric is in OtherMetricsDiffPerc Add a guard for an empty GC list check metric for both baseline and comparand formatting check keys existence don't drop bdn results even gcprocessdata are not found fix possible bugs fix bugs for addtional metrics comparison results presentation Disable NuGet package pruning to keep Microsoft.Extensions.* assemblies app-local (dotnet#5209) Starting with .NET 10, NuGet's package-pruning feature (NU1510) flags direct PackageReferences to packages that are now in-band in the Microsoft.AspNetCore.App shared framework (Microsoft.Extensions.Configuration, .DependencyInjection, .Caching.Memory, .Http, .Logging, etc.) as well as some that have moved into Microsoft.NETCore.App (System.Formats.Cbor). With TreatWarningsAsErrors this fails the build for net10.0+/net11.0+ TFMs. Rather than removing those PackageReferences and adding a FrameworkReference to Microsoft.AspNetCore.App, set RestoreEnablePackagePruning=false (and suppress NU1510). This keeps the assemblies app-local, which is required for BenchmarkDotNet's corerun toolchain: that toolchain points at a CoreRoot under shared/Microsoft.NETCore.App/... which does not contain Microsoft.AspNetCore.App or its assemblies, so an in-band/FrameworkReference approach would break perf runs that use --corerun against a runtime CoreRoot. Verified locally with benchmarks_ci.py -f net11.0 (build) and a smoke run of ActivatorUtilitiesBenchmark.* under --corerun. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Fix GC pipeline (dotnet#5210) * Fix GC pipeline The GC pipeline has been broken for some time, failing with `[error]No image label found to route agent pool Azure Pipelines` * Switch to NetCore-Public with ImageOverride Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Temporary change to trigger the GC pipeline * Update gc-azure-pipelines.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Try to use the vmImage again * Add missing .NET 11 * Switch tests to core unit tests * Try to fix the test task by explicitly specifying .NET 8.0 framework * Fix indent tab and build config * Remove temporary change in .cs file --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Re-enable Cobalt perf runs (dotnet#5212) The Cobalt machines are back online, so re-enable the cobaltMicro, cobaltSveMicro, and cobaltMicroR2RInterpreter jobs that were temporarily disabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Add explicit PackageReferences for transitive Microsoft.Extensions.* … (dotnet#5211) * Add explicit PackageReferences for transitive Microsoft.Extensions.* abstractions For .NET 10+, NuGet package pruning removes transitive Microsoft.Extensions.* abstractions (Logging.Abstractions, DependencyInjection.Abstractions, Options, Primitives) from project.assets.json because they are in-band in Microsoft.AspNetCore.App. Without them, the build fails with CS0246 for types like ILogger, IChangeToken, ObjectFactory, StringSegment, etc. Disabling pruning via RestoreEnablePackagePruning=false (PR dotnet#5209) only preserves DIRECT PackageReferences. Transitive prunable deps are still removed. Add them as explicit PackageReferences so they remain app-local for BenchmarkDotNet's corerun toolchain (which targets a Microsoft.NETCore.App CoreRoot that does not include the AspNetCore.App shared framework). Also re-enable Microsoft.Extensions.Primitives for net10+ (PR dotnet#5196 wrongly removed it assuming it was in NETCore.App; it is actually in AspNetCore.App). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Also add Microsoft.Extensions.Configuration.Abstractions explicit ref Same root cause: NuGet pruning removes this transitive dep on net10+, causing CS0246 for IConfiguration in ConfigurationBinderBenchmarks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Also add Microsoft.Extensions.Caching.Abstractions explicit ref Same root cause: NuGet pruning removes this transitive dep on net10+, causing CS0012/CS0246 for ICacheEntry and IMemoryCache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Shield SDK resolver from upstream global.json with paths entry Write an empty global.json at the workspace root (parent of the perf repo) before invoking dotnet-install. This stops the SDK resolver's upward walk from finding dotnet/runtime's global.json, which contains a 'paths' entry pointing at a '.dotnet' directory holding an older SDK than the one perf installs into tools/dotnet/x64. Without this shield, ci_setup.py's 'dotnet --info' resolves to the wrong SDK and propagates an outdated DOTNET_VERSION to Helix work items, breaking builds on test machines whose ref pack predates recent runtime API changes (e.g. the Sve API rename). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strip sdk.paths instead of writing empty global.json Writing {} broke arcade's bootstrap because it requires tools.dotnet to be present in global.json. Instead, parse the upstream global.json and remove only the sdk.paths field, preserving all other keys. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Exclude Perf_String.Split_Csv from WASM runs (dotnet#5214) The Split_Csv benchmark reads CSV test data files via File.ReadAllLines from AppContext.BaseDirectory. These files are not bundled into the WASM publish output, so CsvCorpus() throws DirectoryNotFoundException when BDN extracts arguments in the spawned WASM benchmark process. Add [BenchmarkCategory(Categories.NoWASM)] to skip the benchmark on WASM, matching the existing pattern used for other filesystem-dependent benchmarks in this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Add test coverage for ResultsComparer and CLI helper paths (dotnet#5199) This expands automated coverage in the repo’s tooling-focused test surface without large production refactors. * Add coverage for ResultsComparer and CLI helpers * Fix Base classification in matrix comparisons * Run tooling tests in CI * Skip tooling tests on main * Limit tooling tests to public non-main runs * Fix brittle matrix comparison test * Restore culture after invoking ResultsComparer * Bump Newtonsoft.Json for tooling tests * Update src/tools/ResultsComparer.Tests/DataTests.cs --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Rename MicroBenchmarks.Serializers.BinaryData to BinaryDataPayload (dotnet#5216) The serialization-test helper class `MicroBenchmarks.Serializers.BinaryData` shadowed the in-box `System.BinaryData` type (introduced in net6+ via `System.Memory.Data`, brought in transitively by Azure.Core). Inside `DataGenerator.cs` (namespace MicroBenchmarks.Serializers) the unqualified `typeof(BinaryData)` always bound to the local helper class. Inside `ReadJson.cs` / `WriteJson.cs` (namespace System.Text.Json.Serialization.Tests) C# enclosing-namespace lookup found `System.BinaryData` first, so the `[GenericTypeArguments(typeof(BinaryData))]` attribute resolved to a DIFFERENT type than the `Generate<T>()` switch was checking. The mismatch was latent on main (where System.Memory.Data wasn't compile-visible to ReadJson.cs) but surfaced as `System.NotImplementedException` in `DataGenerator.Generate<T>()` whenever the reference closure changed (e.g. retargeting BenchmarkDotNet.Extensions from netstandard2.0 to net8.0). Renaming the helper class to `BinaryDataPayload` removes the shadowing once and for all. No behavior change on main; future-proofs against reference-closure churn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Bump urllib3 from 2.6.3 to 2.7.0 (dotnet#5218) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.6.3 to 2.7.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.6.3...2.7.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.7.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump cryptography from 46.0.3 to 46.0.7 (dotnet#5207) Bumps [cryptography](https://github.com/pyca/cryptography) from 46.0.3 to 46.0.7. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@46.0.3...46.0.7) --- updated-dependencies: - dependency-name: cryptography dependency-version: 46.0.7 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [main] Update dependencies from dotnet/android (dotnet#5206) * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26215.1+azdo.13844220 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.59 * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26216.1+azdo.13861683 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.60 * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26220.1+azdo.13883894 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.61 * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26228.1+azdo.13957946 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.62 * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26254.1+azdo.14005692 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.63 * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26254.1+azdo.14008404 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.64 * Update dependencies from https://github.com/dotnet/android build 10.0.70-ci.main.26261.1+azdo.14066865 On relative base path root Microsoft.Android.Sdk.Windows From Version 36.1.55 -> To Version 36.1.65 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> [main] Update dependencies from dotnet/dotnet (dotnet#5205) * Update dependencies from https://github.com/dotnet/dotnet build 20260414.3 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26214.103 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26214.103 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26214.103 * Update dependencies from https://github.com/dotnet/dotnet build 20260415.1 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26215.101 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26215.101 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26215.101 * Update dependencies from https://github.com/dotnet/dotnet build 20260415.8 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26215.108 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26215.108 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26215.108 * Update dependencies from https://github.com/dotnet/dotnet build 20260415.14 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26215.114 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26215.114 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26215.114 * Update dependencies from https://github.com/dotnet/dotnet build 20260415.21 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26215.121 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26215.121 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26215.121 * Update dependencies from https://github.com/dotnet/dotnet build 20260412.2 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26212.102 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26212.102 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26212.102 * Update dependencies from https://github.com/dotnet/dotnet build 20260411.2 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26211.102 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26211.102 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26211.102 * Update dependencies from https://github.com/dotnet/dotnet build 20260422.42 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26222.142 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26222.142 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26222.142 * Update dependencies from https://github.com/dotnet/dotnet build 20260423.12 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26223.112 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26223.112 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26223.112 * Update dependencies from https://github.com/dotnet/dotnet build 20260424.4 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26224.104 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.4.26224.104 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.4.26224.104 * Update dependencies from https://github.com/dotnet/dotnet build 20260424.23 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26224.123 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26224.123 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26224.123 * Update dependencies from https://github.com/dotnet/dotnet build 20260426.11 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26226.111 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26226.111 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26226.111 * Update dependencies from https://github.com/dotnet/dotnet build 20260427.4 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26227.104 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26227.104 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26227.104 * Update dependencies from https://github.com/dotnet/dotnet build 20260427.24 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26227.124 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26227.124 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26227.124 * Update dependencies from https://github.com/dotnet/dotnet build 20260427.31 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26227.131 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26227.131 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26227.131 * Update dependencies from https://github.com/dotnet/dotnet build 20260428.2 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26228.102 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26228.102 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26228.102 * Update dependencies from https://github.com/dotnet/dotnet build 20260428.23 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26228.123 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26228.123 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26228.123 * Update dependencies from https://github.com/dotnet/dotnet build 20260429.13 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26229.113 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26229.113 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26229.113 * Update dependencies from https://github.com/dotnet/dotnet build 20260430.1 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26230.101 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26230.101 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26230.101 * Update dependencies from https://github.com/dotnet/dotnet build 20260501.10 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26251.110 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26251.110 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26251.110 * Update dependencies from https://github.com/dotnet/dotnet build 20260501.12 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26251.112 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26251.112 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26251.112 * Update dependencies from https://github.com/dotnet/dotnet build 20260504.3 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26254.103 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26254.103 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26254.103 * Update dependencies from https://github.com/dotnet/dotnet build 20260504.12 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26254.112 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26254.112 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26254.112 * Update dependencies from https://github.com/dotnet/dotnet build 20260505.1 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26255.101 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26255.101 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26255.101 * Update dependencies from https://github.com/dotnet/dotnet build 20260505.6 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26255.106 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26255.106 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26255.106 * Update dependencies from https://github.com/dotnet/dotnet build 20260506.1 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26256.101 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26256.101 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26256.101 * Update dependencies from https://github.com/dotnet/dotnet build 20260506.5 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26256.105 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26256.105 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26256.105 * Update dependencies from https://github.com/dotnet/dotnet build 20260507.13 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26257.113 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26257.113 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26257.113 * Update dependencies from https://github.com/dotnet/dotnet build 20260508.7 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26258.107 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26258.107 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26258.107 * Update dependencies from https://github.com/dotnet/dotnet build 20260508.10 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26258.110 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26258.110 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26258.110 * Update dependencies from https://github.com/dotnet/dotnet build 20260511.1 On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk From Version 11.0.0-beta.26210.111 -> To Version 11.0.0-beta.26261.101 Microsoft.Extensions.Logging , Microsoft.NET.ILLink , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Runtime.Emscripten.3.1.56.Node.win-x64 , Microsoft.NETCore.App.Ref , Microsoft.WindowsDesktop.App.Ref , System.Threading.Channels From Version 11.0.0-preview.4.26210.111 -> To Version 11.0.0-preview.5.26261.101 Microsoft.NET.Sdk From Version 11.0.100-preview.4.26210.111 -> To Version 11.0.100-preview.5.26261.101 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Parker Bibus <parkerbibus@microsoft.com> Drop daily quality from release/10.0 and 10.0 channels (dotnet#5219) After .NET 10 GA, `-Quality daily` for these channels resolves to internal-only servicing SDK builds (e.g. 10.0.9) whose matching `Microsoft.NETCore.App.Runtime.<rid>` runtime pack has not yet shipped publicly on the dotnet10 / dotnet-public NuGet feeds. That breaks the CertHelper `--self-contained` publish in run_performance_job.py with NU1102. With no quality specified, dotnet-install hits the public `aka.ms/dotnet/10.0/` GA endpoint (currently SDK 10.0.203 / runtime 10.0.7), whose runtime packs are publicly available, so the publish step succeeds. main (net11) and nativeaot* channels are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Update bdn 0.16 binarydata and async (dotnet#5217) * Enable runtime-async for net11.0+ * Update BenchmarkDotNet to 0.16.0-nightly.20260518.1249 Bumps BenchmarkDotNet to the latest master nightly, built from dotnet/BenchmarkDotNet@c7632225 ("Disassembly follow jump trampolines (dotnet#3136)") and pushed to the benchmark-dotnet-prerelease internal feed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Migrate BenchmarkDotNet.Extensions to BDN 0.16.0 APIs BDN 0.16.0 ('Async Refactor', PR dotnet#2958) made breaking API changes that the harness still consumed: - ExporterBase.ExportToLog(Summary, ILogger) was removed; the new abstract ExportAsync uses an internal CancelableStreamWriter that subclasses can't satisfy. PerfLabExporter now implements IExporter directly. - IValidator.Validate returning IEnumerable<ValidationError> was replaced with ValidateAsync returning IAsyncEnumerable<ValidationError>. The four validators (UniqueArguments, TooManyTestCases, NoWasm, MandatoryCategory) use .ToAsyncEnumerable() (transitively from BDN's System.Linq.AsyncEnumerable dep), not async + yield return - the latter produces an AsyncIteratorMethodBuilder state machine that deadlocks with BDN's BenchmarkSynchronizationContext. Switched BenchmarkDotNet.Extensions from netstandard2.0 to net8.0. The netstandard2.0 target was only used to enable an opt-in net472 path, but PERFLAB_TARGET_FRAMEWORKS=net472 is no longer exercised in CI and the new BDN APIs would otherwise require polyfill packages. Removed the net472 package conditional from MicroBenchmarks.csproj and the net472/.NET Framework references from docs/benchmarkdotnet.md and the micro README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix nullability errors surfaced by net8.0 retarget Switching BenchmarkDotNet.Extensions from netstandard2.0 to net8.0 brought in stricter nullable annotations from the BCL and BDN package, which combined with TreatWarningsAsErrors=true (set in src/Directory.Build.props) turned previously-hidden nullability warnings into build errors. - ValuesGenerator.Dictionary<TKey, TValue>: add 'where TKey : notnull' constraint required by Dictionary<,>. - UniqueArgumentsValidator.BenchmarkArgumentsComparer.Equals: match the IEqualityComparer<T>.Equals(T?, T?) interface signature; handle nulls. - TooManyTestCasesValidator.SkipValidation: parameter must be MemberInfo? since MemberInfo.DeclaringType returns Type?. - DiffableDisassemblyExporter: add null-forgiving (!) on reflection lookups whose return values are intentionally trusted by this type (it's a copy of internal BDN code that operates on known types). - PerfLabExporter: BuildJson can return null; use string?. Verified by running the exact CI commands locally (dotnet restore + build with --framework net11.0) using the SDK from global.json. Build succeeds with 0 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Switch micro entrypoint to RunAsync to avoid discovery-time deadlock BDN 0.16's sync entrypoints (BenchmarkSwitcher.Run / BenchmarkRunner.Run) install BenchmarkDotNetSynchronizationContext (a single-threaded message pump) before benchmark discovery. Discovery executes [ParamsSource] and [ArgumentsSource] callbacks; some perf-repo callbacks do sync-over-async (notably SslStreamTests.GetTls13Support, which calls HandshakeAsync(...) .GetAwaiter().GetResult()). Sync-over-async deadlocks on the single-threaded SyncCtx because the awaited continuation is queued back to a pump that the caller is blocking. Switch Program.Main to async Task<int> and await BenchmarkSwitcher.RunAsync. The async entrypoint never installs BenchmarkDotNetSynchronizationContext on the caller, so discovery runs on the default context and sync-over-async in source callbacks no longer deadlocks. Real benchmark execution still gets the SyncCtx semantics it needs because BDN installs it inside the per-benchmark execute path, not on the entrypoint thread. This is the supported BDN-recommended fix for the discovery-deadlock symptom; no BDN code change is required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Gate runtime-async behind 'runtimeasync' experiment PR dotnet#5195 enabled the runtime-async language feature unconditionally for net11.0+ via `<Features>`. To avoid affecting baseline measurement runs, gate it behind the existing experiment infrastructure so it only takes effect in the dedicated experiment lane. - src/Directory.Build.targets: only set `runtime-async=on` when the `EnableRuntimeAsync` MSBuild property is `true` (in addition to the existing TFM check). - scripts/ci_setup.py: when `--experiment-name=runtimeasync` is passed, emit `EnableRuntimeAsync=true` as an env var so MSBuild picks it up as a property (matches the pattern used by the `jitoptrepeat` experiment). Verified locally: BDN dry runs against the BinaryDataPayload tests succeed both with and without the env var (18/18 benchmarks each). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Andy Gocke <andy@commentout.net> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> remove empty constructor and unused imports avoid printing dup command line args
remove dup console output in presentation stage remove dup lines from markdown table fix bug for run command
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR modernizes the performance tooling and infra to align with newer .NET/BenchmarkDotNet behaviors (async validators/entrypoints), improves comparison correctness & testability, and updates build/CI infrastructure (Arcade template sync, SDK/pipeline adjustments).
Changes:
- Update harness/tooling to async-friendly BenchmarkDotNet patterns (async validators/exporter/entrypoint) and newer TFMs.
- Add/extend test coverage for ResultsComparer and harness command-line helpers.
- Refactor GC microbenchmark analysis/presentation and update CI/Arcade infrastructure + dependency versions.
Reviewed changes
Copilot reviewed 106 out of 108 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/tools/ScenarioMeasurement/Startup.Tests/StartupTests.cs | Tighten OS gating and skip ETW startup tests without admin privileges. |
| src/tools/ResultsComparer/TwoInputsComparer.cs | Adjust equivalence/noise filtering logic and fix threshold variable naming. |
| src/tools/ResultsComparer/Stats.cs | Trim simplified OS name output for consistent aggregation. |
| src/tools/ResultsComparer/ResultsComparer.sln | Add ResultsComparer test project and additional build configurations. |
| src/tools/ResultsComparer/Properties/AssemblyInfo.cs | Expose internals to new ResultsComparer.Tests assembly. |
| src/tools/ResultsComparer/Program.cs | Harden filter parsing against null input. |
| src/tools/ResultsComparer/MultipleInputsComparer.cs | Align equivalence handling for “Base” conclusions and naming consistency. |
| src/tools/ResultsComparer/Directory.Packages.props | Update Newtonsoft.Json version for ResultsComparer tool. |
| src/tools/ResultsComparer/Data.cs | Improve moniker parsing and handle null/empty keys. |
| src/tools/ResultsComparer.Tests/StatsTests.cs | Add unit tests for Stats output and OS name simplification. |
| src/tools/ResultsComparer.Tests/ResultsComparerTestData.cs | Add helpers to generate BenchmarkDotNet JSON test payloads. |
| src/tools/ResultsComparer.Tests/ResultsComparer.Tests.csproj | New ResultsComparer test project configuration. |
| src/tools/ResultsComparer.Tests/ProgramTests.cs | Add CLI-level tests for ResultsComparer program behavior. |
| src/tools/ResultsComparer.Tests/HelperTests.cs | Add tests for file discovery and JSON deserialization helpers. |
| src/tools/ResultsComparer.Tests/DataTests.cs | Add tests for nested archive decompression and version selection. |
| src/tools/ResultsComparer.Tests/ConsoleOutputCollection.cs | Serialize console-output tests by disabling parallelization. |
| src/tools/Reporting/Directory.Packages.props | Update Newtonsoft.Json version for Reporting tool. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/UniqueValuesGeneratorTests.cs | Add generic constraints to match updated generator APIs. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/UniqueArgumentsValidatorTests.cs | Update validator tests to async validation surface. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/PartitionFilterTests.cs | Improve nullability handling around parsed config. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/CommandLineOptionsTests.cs | Add tests for CLI argument parsing helpers. |
| src/tests/harness/BenchmarkDotNet.Extensions.Tests/BenchmarkDotNet.Extensions.Tests.csproj | Force MicroBenchmarks reference to a specific target framework for tests. |
| src/harness/BenchmarkDotNet.Extensions/ValuesGenerator.cs | Add notnull constraint for dictionary keys. |
| src/harness/BenchmarkDotNet.Extensions/UniqueArgumentsValidator.cs | Switch validator to async enumerable to avoid sync-over-async deadlocks. |
| src/harness/BenchmarkDotNet.Extensions/TooManyTestCasesValidator.cs | Switch validator to async enumerable and align nullability. |
| src/harness/BenchmarkDotNet.Extensions/PerfLabExporter.cs | Rework exporter to async file output with custom naming/lifecycle. |
| src/harness/BenchmarkDotNet.Extensions/NoWasmValidator.cs | Switch validator to async enumerable. |
| src/harness/BenchmarkDotNet.Extensions/MandatoryCategoryValidator.cs | Switch validator to async enumerable. |
| src/harness/BenchmarkDotNet.Extensions/DiffableDisassemblyExporter.cs | Apply null-forgiving to reflection results for nullable safety. |
| src/harness/BenchmarkDotNet.Extensions/CommandLineOptions.cs | Fix missing-parameter handling and file formatting. |
| src/harness/BenchmarkDotNet.Extensions/BenchmarkDotNet.Extensions.csproj | Raise TFM to net8.0 and align source-reference TFMs. |
| src/benchmarks/micro/libraries/System.Text.Json/Serializer/WriteJson.cs | Avoid BinaryData type collision by switching to renamed payload type. |
| src/benchmarks/micro/libraries/System.Text.Json/Serializer/ReadJson.cs | Avoid BinaryData type collision by switching to renamed payload type. |
| src/benchmarks/micro/libraries/System.Runtime/Perf.String.cs | Mark CSV benchmark as NoWASM due to filesystem constraints. |
| src/benchmarks/micro/Serializers/DataGenerator.cs | Rename BinaryData model to prevent System.BinaryData collisions; update generator/serializers. |
| src/benchmarks/micro/README.md | Update documented supported TFMs to current set. |
| src/benchmarks/micro/Program.cs | Use async Main + RunAsync to avoid deadlocks during discovery/params sources. |
| src/benchmarks/micro/MicroBenchmarks.csproj | Disable package pruning and ensure app-local deps for corerun toolchain; adjust refs. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/RunSuiteCommand.cs | Fix suite path validation and use MarkupLine for error output. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/RunCommand.cs | Align microbenchmark suite key name. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/CreateSuiteCommand.cs | Rename microbenchmark suite folder/key and add per-suite iteration configuration. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/BaseSuite/MicrobenchmarksToRun.txt | Remove one benchmark from default microbenchmark suite list. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/BaseSuite/Microbenchmarks.yaml | Rename iteration -> iterations field. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/Microbenchmark/MicrobenchmarkCommand.cs | Update microbenchmark execution loop to use iterations and new analysis/presentation flow. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/Microbenchmark/MicrobenchmarkAnalyzeCommand.cs | Factor analysis into reusable ExecuteAnalysis/Present methods. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Presentation.cs | Remove old presentation wrapper in favor of new analysis flow. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Markdown.cs | Update table generation to support additional metrics and escaped names; adjust thresholds. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Json/JsonOutput.cs | Remove unused JSON output type. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Json/Json.cs | Adjust JSON output API to new grouped results model. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/MarkdownReportBuilder.cs | Deduplicate repro commands by commandline args. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Configurations/Microbenchmarks.Configuration.cs | Rename iterations with YAML alias back-compat (iteration). |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Configurations/InputConfiguration.cs | Add top-level iterations configuration map. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResultsAnalyzer.cs | Remove older analyzer implementation in favor of new comparison flow. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResults.cs | Reshape BDN JSON model types used by GC infra microbenchmark analysis. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResultComparison.cs | New pipeline for loading/analyzing/comparing benchmark results (incl. trace mapping). |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResult.cs | New microbenchmark result container including derived metrics. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkComparisonResults.cs | Adjust regression/improvement threshold inclusivity and ordering. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkComparisonResult.cs | Rework comparison to support multiple runs/iterations and compute aggregated metrics. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/GCTraceMetrics.cs | New GC trace metric extraction/aggregation container. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/GCTraceMetricComparisonResult.cs | New comparison result type for trace metrics with outlier filtering. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/GCTraceMetricComparison.cs | New helper for constructing trace metric comparisons. |
| src/benchmarks/gc/GC.Infrastructure/GC.Analysis.API/Statistics.cs | Add outlier removal helper using IQR filtering. |
| src/benchmarks/gc/GC.Infrastructure/Configurations/Run.yaml | Add iterations config in YAML. |
| src/Directory.Build.targets | Gate runtime-async feature flag via CI experiment variable. |
| src/Directory.Build.props | Formatting-only whitespace normalization in props file. |
| scripts/dotnet.py | Add “global.json shielding” to prevent SDK resolver picking parent repo sdk.paths. |
| scripts/ci_setup.py | Add runtimeasync experiment environment variable emission. |
| scripts/channel_map.py | Adjust channel map metadata for 10.0 branches. |
| requirements.txt | Bump cryptography and urllib3 versions. |
| global.json | Update SDK/tooling versions and Arcade/Helix SDK pins. |
| gc-azure-pipelines.yml | Update GC infra build/test pipeline job and add additional unit test runs. |
| eng/pipelines/sdk-perf-jobs.yml | Add Windows tooling tests job for PR validation. |
| eng/pipelines/runtime-perf-jobs.yml | Re-enable cobalt jobs by default. |
| eng/common/tools.sh | Improve dotnet isolation env vars, toolset download behavior, and MSBuild diagnostics/workload resolver handling. |
| eng/common/tools.ps1 | Improve dotnet isolation env vars, remove xcopy MSBuild fallback, toolset download behavior, and MSBuild workload resolver handling. |
| eng/common/templates/steps/component-governance.yml | Remove local CG shim template (Arcade sync). |
| eng/common/templates/job/job.yml | Update job template parameters and artifact publishing behavior (pipeline artifacts). |
| eng/common/templates-official/steps/publish-pipeline-artifacts.yml | Make sbomEnabled/isProduction parameter forwarding conditional on non-empty args. |
| eng/common/templates-official/steps/component-governance.yml | Remove local CG shim template (Arcade sync). |
| eng/common/templates-official/job/job.yml | Update official job template artifact publishing (pipeline artifacts) and remove SBOM shim steps. |
| eng/common/template-guidance.md | Remove references to SBOM shims removed from templates. |
| eng/common/sdk-task.ps1 | Simplify VS msbuild acquisition to rely on installed Visual Studio. |
| eng/common/post-build/sourcelink-validation.ps1 | Remove obsolete SourceLink validation script. |
| eng/common/post-build/redact-logs.ps1 | Use --source rather than --add-source for tool install. |
| eng/common/generate-sbom-prep.sh | Remove deprecated SBOM prep script (Arcade sync). |
| eng/common/generate-sbom-prep.ps1 | Remove deprecated SBOM prep script (Arcade sync). |
| eng/common/dotnet-install.sh | Allow overriding dotnet install root via DOTNET_GLOBAL_INSTALL_DIR. |
| eng/common/dotnet-install.ps1 | Allow overriding dotnet install root via DOTNET_GLOBAL_INSTALL_DIR. |
| eng/common/darc-init.sh | Use --source rather than --add-source for tool install. |
| eng/common/darc-init.ps1 | Use --source rather than --add-source for tool install. |
| eng/common/cross/toolchain.cmake | Adjust Haiku toolchain selection based on CCC_CC to support clang toolchains. |
| eng/common/core-templates/steps/source-build.yml | Mark logs as non-production artifacts via isProduction=false. |
| eng/common/core-templates/steps/publish-logs.yml | Publish logs via pipeline artifacts with attempt-suffixed name. |
| eng/common/core-templates/steps/generate-sbom.yml | Deprecate SBOM generation shim with warning step. |
| eng/common/core-templates/steps/component-governance.yml | Remove core CG template (Arcade sync). |
| eng/common/core-templates/post-build/setup-maestro-vars.yml | Switch release config download to pipeline artifacts task. |
| eng/common/core-templates/post-build/post-build.yml | Add publishing infra v4 path for artifact download, remove SourceLink validation job. |
| eng/common/core-templates/post-build/common-variables.yml | Remove unused SourceLink CLI version variable. |
| eng/common/core-templates/jobs/jobs.yml | Thread publishingVersion parameter into job template usage. |
| eng/common/core-templates/job/renovate.yml | Mark logs as non-production artifacts via isProduction=false. |
| eng/common/core-templates/job/publish-build-assets.yml | Publish ReleaseConfigs via pipeline artifacts and mark as non-production. |
| eng/common/core-templates/job/onelocbuild.yml | Add optional custom PowerShell script parameter to xLoc build. |
| eng/common/core-templates/job/job.yml | Remove componentGovernanceSteps parameter and add enablePublishing flag. |
| eng/common/AGENTS.md | Document Arcade ownership/overwrites for eng/common. |
| eng/Versions.props | Update .NET/BDN package versions used by the repo. |
| eng/Version.Details.xml | Update dependency versions/SHA pins (dotnet, arcade, android). |
| docs/benchmarkdotnet.md | Remove net4x runtimes listing and remove obsolete “Private CLR Build” section. |
| NuGet.config | Update Darc-managed package source feeds. |
Comments suppressed due to low confidence (6)
src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkComparisonResult.cs:1
double.NaNis never equal to itself, so this check will never be true. Usedouble.IsNaN(OtherMetricsDiff[kvp.Key])(or compute/store a local and checkdouble.IsNaN) to correctly propagate NaN and avoid emitting misleading percentage deltas.
src/harness/BenchmarkDotNet.Extensions/PerfLabExporter.cs:1summary.Titlecan contain characters that are invalid in file names (especially on Windows) and can also be excessively long. This can cause exporter failures when writing{Title}-perf-lab-report.json. Consider sanitizing with the same folder/file helper used elsewhere (e.g.,FolderNameHelper.ToFolderName(summary.Title)or an equivalent safe filename helper) and potentially truncating to a safe length.
scripts/dotnet.py:1- This mutates a
global.jsonone directory above the repo root, which is an external side-effect that can impact other checkouts/tools in the same workspace and can be surprising/hard to debug (especially if multiple jobs share the workspace). A safer approach is to avoid rewriting the parent file (e.g., rundotnetwith an explicit working directory that has the intendedglobal.json, write a temporaryglobal.jsonin a scoped directory, or write and restore the original content after the operation).
src/tools/ResultsComparer/Data.cs:1 - Switching from
StartsWithtoContainscan cause incorrect moniker detection when the input contains multiple moniker-like substrings (e.g.,nativeaot10.0-preview...or paths/titles containing bothnet10.0and other tokens). Consider parsing more precisely (e.g., regex with boundaries, or checking preferred tokens in a deterministic order usingIndexOf+ validation) so inputs don’t get misclassified.
eng/common/core-templates/post-build/post-build.yml:1 - The template now allows
publishingInfraVersion: 4(and contains V4 download logic earlier), butpublish-using-darc.ps1is always invoked with-PublishingInfraVersion 3. This makes the parameter ineffective and can break V4 publishing expectations. Pass through${{ parameters.publishingInfraVersion }}instead of hardcoding 3 (or remove the parameter/other V4 logic if V4 is not supported here yet).
parameters:
src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/RunSuiteCommand.cs:1
- Correct spelling/grammar in the exception message: change 'doesn't exists' to 'doesn't exist'.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Open
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR aims at calculating average value of multiple microbenchmarks results. The work revolves around: