Skip to content

[WIP] Dogfood dotnet test for device tests#11224

Draft
jonathanpeppers wants to merge 5 commits intomainfrom
jonathanpeppers/dogfood-dotnet-test
Draft

[WIP] Dogfood dotnet test for device tests#11224
jonathanpeppers wants to merge 5 commits intomainfrom
jonathanpeppers/dogfood-dotnet-test

Conversation

@jonathanpeppers
Copy link
Copy Markdown
Member

@jonathanpeppers jonathanpeppers commented Apr 27, 2026

Summary

Migrate device test infrastructure from legacy NUnitLite stack to dotnet test with Microsoft Testing Platform (MTP), following the pattern established in commit 031246d.

Changes

New: MTP-based TestInstrumentation base class

  • TestRunner.Core/TestInstrumentation.cs now uses MTP + NUnit (AddNUnit() from NUnit3TestAdapter 5.0+)
  • Follows the same pattern as the androidtest template
  • Tracks test results (passed/failed/skipped) and reports TRX path via instrumentation bundle

Updated test projects

  • Mono.Android.NET-Tests: Uses new TestInstrumentation base, stock NUnit 4.3.2, removed NUnitLite/TestRunner.NUnit deps
  • JcwGen-Tests: Same migration, simplified TestInstrumentation and MainActivity

Deleted (~41K lines)

  • tests/TestRunner.NUnit/ - NUnitTestRunner/NUnitTestInstrumentation
  • tests/TestRunner.xUnit/ - XUnitTestRunner/XUnitTestInstrumentation
  • src/Xamarin.Android.NUnitLite/ - NUnitLite wrapper, TestSuiteInstrumentation, GUI
  • src-ThirdParty/NUnitLite/ - Embedded NUnitLite source code

CI updates

  • apk-instrumentation.yaml: Uses dotnet test --no-build instead of -t:RunTestApp
  • Test results format changed from NUnit to VSTest (TRX)

Package upgrades

  • NUnit: 3.13.3 → 4.3.2
  • NUnit3TestAdapter: 4.4.2 → 5.0.0

Out of scope

  • EmbeddedDSO and Locale-Tests (old-style csproj, need SDK migration first)
  • Xamarin.Legacy.NUnitLite NuGet package (customer-facing, kept as-is)

jonathanpeppers and others added 5 commits April 27, 2026 16:42
@jonathanpeppers
Migrate device tests to use dotnet test with Microsoft Testing Plat…
10eed35 
…form (MTP)

- Revamp `TestRunner.Core/TestInstrumentation.cs` to use MTP + NUnit
  (following the androidtest template pattern with `AddNUnit()`)
- Delete `TestRunner.NUnit/`, `TestRunner.xUnit/`,
  `Xamarin.Android.NUnitLite/`, `src-ThirdParty/NUnitLite/`
- Update `Mono.Android.NET-Tests` and `JcwGen-Tests` to use new
  `TestInstrumentation` base class with stock NUnit 4.3.2
- Update CI YAML to use `dotnet test --no-build` instead of
  `-t:RunTestApp`
- Update `NUnitReferences.projitems` to NUnit 4.3.2 + adapter 5.0.0
- Clean up `Directory.Build.props`, installer targets, and solution
  files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Revert NUnit to 3.13.3 - no upgrade needed for MTP
ee374b6 
The NUnit 4.x upgrade broke all host-side test projects because NUnit 4
removed classic assert APIs (Assert.IsTrue, FileAssert, etc.).

Only NUnit3TestAdapter 5.0.0 is needed (for AddNUnit() MTP support) in
the device test projects. NUnit itself stays at 3.13.3, and the shared
NUnitReferences.projitems is restored to its original versions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Remove NUnitLite entry from THIRD-PARTY-NOTICES.TXT
92b4eab 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Fix test result publishing - add --logger trx to dotnet test
4c1e796 
The publish step was skipped because testResultsFiles was empty.
Add --logger trx and --results-directory to produce TRX files,
and use a deterministic file path for PublishTestResults@2.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Fix review issues and TRX publishing
7b8a944 
- Restore XA4313 AndroidError for Xamarin.Android.NUnitLite in Common.targets
- Restore NUnitLite in XA4313 test Values in BuildTest.cs
- Restore GenerateAssemblyInfo=false and AssemblyInfo.cs in TestRunner.Core
- Delete unused MainActivities and Main.axml layout
- Move MyFragment to its own file (still needed by FragmentFixup tests)
- Fix TRX publishing: copy device-pulled TRX to expected path instead
  of relying on --logger trx which doesn't work with Microsoft.Android.Run

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers jonathanpeppers force-pushed the jonathanpeppers/dogfood-dotnet-test branch from 31e7d49 to 7b8a944 Compare April 27, 2026 21:45
simonrozsival
simonrozsival reviewed Apr 27, 2026
View reviewed changes
Comment thread ...s/Mono.Android-Tests/Mono.Android-Tests/Xamarin.Android.RuntimeTests/NUnitInstrumentation.cs
Comment on lines -33 to -89
ExcludedTestNames = new [] {
// net.dot.jni.test.CallVirtualFromConstructorDerived Java class not in APK
"Java.InteropTests.InvokeVirtualFromConstructorTests",

// net.dot.jni.internal.JavaProxyObject Java class not in APK — fixture setup fails (16 tests)
"Java.InteropTests.JavaObjectArray_object_ContractTest",

// net.dot.jni.internal.JavaProxyObject Java class not in APK
"Java.InteropTests.JniValueMarshaler_object_ContractTests.JniValueMarshalerContractTests`1.CreateArgumentState",
"Java.InteropTests.JniValueMarshaler_object_ContractTests.JniValueMarshalerContractTests`1.CreateGenericArgumentState",
"Java.InteropTests.JniValueMarshaler_object_ContractTests.JniValueMarshalerContractTests`1.CreateGenericObjectReferenceArgumentState",
"Java.InteropTests.JniValueMarshaler_object_ContractTests.JniValueMarshalerContractTests`1.CreateGenericValue",
"Java.InteropTests.JniValueMarshaler_object_ContractTests.JniValueMarshalerContractTests`1.CreateObjectReferenceArgumentState",
"Java.InteropTests.JniValueMarshaler_object_ContractTests.JniValueMarshalerContractTests`1.CreateValue",
"Java.InteropTests.JniValueMarshaler_object_ContractTests.SpecificTypesAreUsed",

// No generated JavaPeerProxy for java/lang/Object with IJavaPeerable target type
"Java.InteropTests.JniValueMarshaler_IJavaPeerable_ContractTests.JniValueMarshalerContractTests`1.CreateGenericValue",
"Java.InteropTests.JniValueMarshaler_IJavaPeerable_ContractTests.JniValueMarshalerContractTests`1.CreateValue",

// net.dot.jni.internal.JavaProxyThrowable — proxy throwable creation fails
"Java.InteropTests.JavaExceptionTests.InnerExceptionIsNotAProxy",

// IJavaInterfaceInvoker ctor trimmed / missing JavaPeerProxy for test types
"Java.InteropTests.JavaPeerableExtensionsTests.JavaAs",
"Java.InteropTests.JavaPeerableExtensionsTests.JavaAs_Exceptions",
"Java.InteropTests.JavaPeerableExtensionsTests.JavaAs_InstanceThatDoesNotImplementInterfaceReturnsNull",

// JNI method remapping not supported in trimmable typemap
"Java.InteropTests.JniPeerMembersTests.ReplaceInstanceMethodName",
"Java.InteropTests.JniPeerMembersTests.ReplaceInstanceMethodWithStaticMethod",
"Java.InteropTests.JniPeerMembersTests.ReplacementTypeUsedForMethodLookup",
"Java.InteropTests.JniPeerMembersTests.ReplaceStaticMethodName",

// net.dot.jni.test.GenericHolder Java class not in APK
"Java.InteropTests.JniTypeManagerTests.CannotCreateGenericHolderFromJava",

// JniPrimitiveArrayInfo lookup fails for JavaBooleanArray
"Java.InteropTests.JniTypeManagerTests.GetType",

// net.dot.jni.test.GetThis — cannot register native members
"Java.InteropTests.JavaObjectTest.DisposeAccessesThis",

// NotSupportedException instead of InvalidCastException — no generated JavaPeerProxy
"Java.InteropTests.JavaObjectExtensionsTests.JavaCast_BadInterfaceCast",
"Java.InteropTests.JavaObjectExtensionsTests.JavaCast_BaseToGenericWrapper",
"Java.InteropTests.JavaObjectExtensionsTests.JavaCast_CheckForManagedSubclasses",
"Java.InteropTests.JavaObjectExtensionsTests.JavaCast_InvalidTypeCastThrows",

// Open generic type handling differs from non-trimmable
"Java.InteropTests.JnienvTest.NewOpenGenericTypeThrows",

// Throwable subclass registration
"Java.InteropTests.JnienvTest.ActivatedDirectThrowableSubclassesShouldBeRegistered",

// Instance identity after JNI round-trip
"Java.LangTests.ObjectTest.JnienvCreateInstance_RegistersMultipleInstances",
Copy link
Copy Markdown
Member

@simonrozsival simonrozsival Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test exclusion mechanism was very handy for skipping tests from the Java.Interop submodule which are not in this repo and so applying attributes to the individual tests is not practical. What would the new way of listing these tests as skipped be?

@jonathanpeppers
Copy link
Copy Markdown
Member Author

jonathanpeppers commented Apr 27, 2026 •
edited
Loading

With MTP, test filtering works via command-line args passed to TestApplication.CreateBuilderAsync(). NUnit's MTP adapter supports --treenode-filter which accepts NUnit filter expressions.

I've now added this support to TestInstrumentation. Subclasses can override:

  • ExcludedCategories — to exclude NUnit categories
  • ExcludedTestNames — to exclude tests by fully-qualified name (for Java.Interop submodule tests where adding attributes isn't practical)

The old am instrument extras still work too:

  • -e exclude "Cat1,Cat2" — exclude categories at runtime
  • -e include "Cat1" — include categories at runtime
  • -e noexclusions true — skip built-in exclusions

Everything gets translated to a --treenode-filter expression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@simonrozsival simonrozsival simonrozsival left review comments

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jonathanpeppers @simonrozsival