Add dotnet test integration tests for all Apple platforms

[jonathanpeppers]

This PR adds end-to-end dotnet test integration tests for all Apple platforms (iOS, tvOS, Mac Catalyst, macOS) using the MTP (Microsoft Testing Platform) test templates.

Changes

• tests/dotnet/UnitTests/DotNetTestTest.cs: New parameterized test that creates a project from dotnet new <template>, replaces the generated tests with a single passing test, and runs dotnet test.
• tests/common/DotNet.cs: Added test verb support to DotNet.Execute.

The test reads the MLAUNCH_PATH environment variable to override the mlaunch binary for mobile platforms (iOS/tvOS), enabling testing with the MTP-enabled mlaunch build.

Screenshot

Related

Depends on mlaunch MTP support: https://devdiv.visualstudio.com/DevDiv/_git/macios-mlaunch/pullrequest/734978

[vs-mobiletools-engineering-service2]

🔥 Unable to find the contents for the comment: D:\a\1\s\change-detection\results\gh-comment.md does not exist :fire

Pipeline on Agent
Hash: 49cb27a0cdbc3a63b8b1dd1de66629e6a108c6ae [PR build]

[vs-mobiletools-engineering-service2]

✅ [CI Build #0950911] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 0950911a792eef175d344bb1712c3952aa2dd14a [PR build]

[vs-mobiletools-engineering-service2]

🔥 [CI Build #49cb27a] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

0 tests crashed, 4 tests failed, 201 tests passed.

Failures

❌ dotnettests tests (iOS) [attempt 2]

1 tests failed, 0 tests passed.

Failed tests

• DotNet tests: Failed (Execution failed with exit code 1)
  • Xamarin.Tests.DotNetTestTest.DotNetTest(iOS,"iostest"): 'dotnet test' failed with exit code 1.
    Binlog: /Users/builder/azdo/_work/1/s/macios/tests/dotnet/UnitTests/bin/Debug/net11.0/t...

Html Report (VSDrops) Download

❌ dotnettests tests (tvOS) [attempt 2]

1 tests failed, 0 tests passed.

Failed tests

• DotNet tests: Failed (Execution failed with exit code 1)
  • Xamarin.Tests.DotNetTestTest.DotNetTest(TVOS,"tvostest"): 'dotnet test' failed with exit code 1.
    Binlog: /Users/builder/azdo/_work/1/s/macios/tests/dotnet/UnitTests/bin/Debug/net11.0/t...

Html Report (VSDrops) Download

❌ linker tests (tvOS) [attempt 2]

1 tests failed, 10 tests passed.

Failed tests

• dont link/tvOS - simulator/Release: BuildFailure

Html Report (VSDrops) Download

❌ monotouch tests (iOS) [attempt 2]

1 tests failed, 25 tests passed.

Failed tests

• monotouch-test/iOS - simulator/Release (link sdk): Failed

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ introspection: All 9 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ linker (iOS): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 11 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 30 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 23 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 26 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 49cb27a0cdbc3a63b8b1dd1de66629e6a108c6ae [PR build]

[rolfbjarne]

Looks like mlaunch can't be found? I'll have a look.

[rolfbjarne]

OK, not our bug: https://gist.github.com/rolfbjarne/f807a74fa6a933295e2e602468e58c8f

[rolfbjarne]

Filed a dotnet/sdk issue: dotnet/sdk#54917

[vs-mobiletools-engineering-service2]

✅ [PR Build #f16e7f4] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: f16e7f4c1ae3357afc9d110fbad3976aedb81416 [PR build]

[vs-mobiletools-engineering-service2]

✅ [PR Build #f16e7f4] Build passed (Build packages) ✅

Pipeline on Agent
Hash: f16e7f4c1ae3357afc9d110fbad3976aedb81416 [PR build]

[vs-mobiletools-engineering-service2]

✅ [CI Build #be77eb7] Prepare .NET Release succeeded ✅

📦 Published NuGet packages (32 packages)

iOS

• Microsoft.iOS.Ref.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Runtime.ios-arm64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Runtime.ios.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Runtime.iossimulator-arm64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Runtime.iossimulator-x64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Sdk.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Templates.26.5.11710-net11-p6.nupkg
• Microsoft.iOS.Windows.Sdk.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.NET.Sdk.iOS.Manifest-11.0.100-preview.6.26.5.11710-net11-p6.nupkg

MacCatalyst

• Microsoft.MacCatalyst.Ref.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.MacCatalyst.Runtime.maccatalyst-arm64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.MacCatalyst.Runtime.maccatalyst-x64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.MacCatalyst.Runtime.maccatalyst.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.MacCatalyst.Sdk.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.MacCatalyst.Templates.26.5.11710-net11-p6.nupkg
• Microsoft.NET.Sdk.MacCatalyst.Manifest-11.0.100-preview.6.26.5.11710-net11-p6.nupkg

macOS

• Microsoft.macOS.Ref.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.macOS.Runtime.osx-arm64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.macOS.Runtime.osx-x64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.macOS.Runtime.osx.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.macOS.Sdk.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.macOS.Templates.26.5.11710-net11-p6.nupkg
• Microsoft.NET.Sdk.macOS.Manifest-11.0.100-preview.6.26.5.11710-net11-p6.nupkg

tvOS

• Microsoft.NET.Sdk.tvOS.Manifest-11.0.100-preview.6.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Ref.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Runtime.tvos-arm64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Runtime.tvos.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Runtime.tvossimulator-arm64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Runtime.tvossimulator-x64.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Sdk.net11.0_26.5.26.5.11710-net11-p6.nupkg
• Microsoft.tvOS.Templates.26.5.11710-net11-p6.nupkg

Other

• Sharpie.Bind.Tool.26.5.0.710-net11-p6.nupkg

Pipeline on Agent
Hash: be77eb7c90419111af5c4d26cdedae32feda38c4 [PR build]