Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ updates:
- "/src/r8"
- "/src/manifestmerger"
- "/src/proguard-android"
- "/tests/CodeGen-Binding/Xamarin.Android.LibraryProjectZip-LibBinding/java/JavaLib"
schedule:
interval: "weekly"
- package-ecosystem: "gitsubmodule"
Expand Down
19 changes: 14 additions & 5 deletions .github/instructions/gradle.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,22 @@ Test the CI path locally: `$env:RunningOnCI='true'` (PowerShell) or `RunningOnCI

## When CI fails 401 on a Dependabot bump

The new package isn't cached in the feed yet. One-time setup, then ingest:
The new package isn't cached in the dnceng `dotnet-public-maven` feed yet. CI agents only do anonymous reads, so someone has to authenticate once locally to make the feed pull the package (and its transitive deps) from upstream.

1. `iex "& { $(irm https://aka.ms/install-artifacts-credprovider.ps1) }"` (or the `.sh` equivalent)
2. `$env:RunningOnCI='true'; ./build-tools/gradle/gradlew.bat --project-dir src/<project> build` — sign in via the device-flow prompt; the feed proxies + caches the package.
3. Re-run CI on the Dependabot PR. No PR edit needed.
Use the helper script — it runs the build, parses any 401 URLs out of the log, re-fetches each one with an Azure DevOps bearer token (so the feed mirrors it), and loops until the build succeeds:
Comment thread
jonathanpeppers marked this conversation as resolved.

The credprovider plugin is a no-op when no AzDO repos are configured (i.e. local builds without `RunningOnCI`).
```powershell
az login # one-time, corp account with MFA satisfied

pwsh ./eng/gradle/mirror-dependencies.ps1 `
-ProjectDir <path-to-failing-gradle-project> `
-Task <gradle-task-CI-runs> `
-AndroidHome <path-to-Android-SDK> # required for any com.android.* project
```

The mirror must run in the project that actually needs the new package — a sibling project's build won't trigger a mirror for someone else's deps. Typical convergence is 2-5 iterations as the resolver walks the dep graph breadth-first.

After it succeeds, just re-run the failed CI job. No PR edits needed — the packages are now anonymous-readable forever.

## Don'ts

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.12-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-9.4.1-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
141 changes: 141 additions & 0 deletions eng/gradle/mirror-dependencies.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
#!/usr/bin/env pwsh
<#
.SYNOPSIS
Mirrors a gradle project's dependencies into the dnceng dotnet-public-maven
Azure Artifacts feed so CI can resolve them anonymously.

.DESCRIPTION
When Dependabot bumps a gradle dependency (or its transitive graph changes),
CI fails with 401 errors because the new package(s) haven't been pulled
from upstream into the dnceng feed yet. CI agents only do anonymous reads,
so a developer has to authenticate locally once to seed the feed.

This script does that by running the requested gradle build in a loop:
1. Run gradle with RunningOnCI=true so it points at the dnceng feed.
2. Parse any 'Could not GET' URLs out of the build log.
3. Re-fetch each failing URL with an Azure DevOps OAuth bearer token
(obtained via `az account get-access-token`). The feed's upstream
connector then pulls the package and caches it for anonymous reads.
4. Repeat until the build succeeds or no more 401s appear.

After the loop converges, no PR edits are needed — just re-run the failing
CI job, since the packages are now anonymous-readable.

.PARAMETER ProjectDir
Path to the gradle project (the one containing the failing dependency).
Mirroring must run in the project that actually requires the package;
a sibling project's build won't trigger a mirror for someone else's deps.

.PARAMETER Task
Gradle task(s) to run. Should be one that resolves the new dependency
graph (e.g. 'assembleDebug', 'build', 'extractProguardFiles').

.PARAMETER AndroidHome
Optional path to the Android SDK. Required when the gradle build needs it
(any project using the com.android.* plugins). Defaults to the value of
`$env:ANDROID_HOME` if set.

.PARAMETER MaxIterations
Cap on build/mirror cycles. Default 15. Typical convergence is 2-5
iterations as the resolver walks the dep graph breadth-first.

.EXAMPLE
pwsh ./eng/gradle/mirror-dependencies.ps1 `
-ProjectDir tests/CodeGen-Binding/Xamarin.Android.LibraryProjectZip-LibBinding/java/JavaLib `
-Task assembleDebug `
-AndroidHome D:\android-toolchain\sdk

.EXAMPLE
pwsh ./eng/gradle/mirror-dependencies.ps1 -ProjectDir src/proguard-android -Task extractProguardFiles
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true)]
[string] $ProjectDir,

[Parameter(Mandatory=$true)]
[string] $Task,

[string] $AndroidHome = $env:ANDROID_HOME,

[int] $MaxIterations = 15
)

$ErrorActionPreference = 'Stop'
$repoRoot = Resolve-Path (Join-Path $PSScriptRoot '../..') | Select-Object -ExpandProperty Path
$projectDirAbs = Resolve-Path (Join-Path $repoRoot $ProjectDir) -ErrorAction Stop | Select-Object -ExpandProperty Path
$gradlew = if ($IsWindows -or $env:OS -eq 'Windows_NT') {
Join-Path $repoRoot 'build-tools/gradle/gradlew.bat'
} else {
Join-Path $repoRoot 'build-tools/gradle/gradlew'
}
if (-not (Test-Path $gradlew)) { throw "gradlew not found at $gradlew" }

# Azure DevOps resource id — same for every AzDO tenant.
$azDevOpsResource = '499b84ac-1321-427f-aa17-267ca6975798'

function Get-AzDevOpsToken {
$token = az account get-access-token --resource $azDevOpsResource --query accessToken -o tsv 2>$null
if ([string]::IsNullOrEmpty($token)) {
throw "Could not get an Azure DevOps access token. Run 'az login' first."
}
return $token
}

function Invoke-Mirror($logPath) {
$urls = Select-String -Path $logPath -Pattern "Could not GET 'https://pkgs\.dev\.azure\.com/dnceng/[^']+'" -AllMatches |
ForEach-Object { $_.Matches } |
ForEach-Object { $_.Value -replace "^Could not GET '", "" -replace "'$", "" } |
Sort-Object -Unique
if ($urls.Count -eq 0) { return 0 }
$token = Get-AzDevOpsToken
$headers = @{ Authorization = "Bearer $token" }
$ok = 0; $fail = 0
foreach ($u in $urls) {
try {
$r = Invoke-WebRequest -Uri $u -Headers $headers -SkipHttpErrorCheck -ErrorAction Stop
if ($r.StatusCode -eq 200) { $ok++ } else { $fail++; Write-Host " $($r.StatusCode) $u" -ForegroundColor Yellow }
} catch {
$fail++
Write-Host " ERR $u : $_" -ForegroundColor Yellow
}
}
Write-Host " -> mirrored OK=$ok, not-found=$fail (of $($urls.Count))" -ForegroundColor Cyan
return $urls.Count
}

Write-Host "Repo root: $repoRoot"
Write-Host "Project: $projectDirAbs"
Write-Host "Task: $Task"
if ($AndroidHome) { Write-Host "ANDROID_HOME: $AndroidHome" }

# Verify az is available and authenticated up front so we fail fast.
Get-AzDevOpsToken | Out-Null

if ($AndroidHome) { $env:ANDROID_HOME = $AndroidHome }
$env:RunningOnCI = 'true'

Push-Location $projectDirAbs
try {
for ($i = 1; $i -le $MaxIterations; $i++) {
Write-Host "`n=== iteration $i ===" -ForegroundColor Green
$log = Join-Path ([IO.Path]::GetTempPath()) "gradle-mirror-iter-$i.log"
& $gradlew $Task --no-daemon --refresh-dependencies *>&1 | Tee-Object -FilePath $log | Out-Null
if (Select-String -Path $log -Pattern 'BUILD SUCCESSFUL' -SimpleMatch -Quiet) {
Write-Host "`nBUILD SUCCESSFUL after $i iteration(s). The feed now has the packages CI needs." -ForegroundColor Green
return
}
$count = Invoke-Mirror $log
if ($count -eq 0) {
Write-Host "`nGradle failed but no 401s to mirror — see $log" -ForegroundColor Red
Get-Content $log -Tail 30
exit 1
}
}
Write-Host "`nExhausted $MaxIterations iterations without success. Last log:" -ForegroundColor Red
Get-Content $log -Tail 30
exit 1
}
finally {
Pop-Location
}
33 changes: 5 additions & 28 deletions eng/gradle/plugin-repositories.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -15,34 +15,11 @@
// reads work forever after. CI therefore does NOT need credentials — it just
// reads anonymously from packages already cached in the feed.
//
// =================== TESTING / INGESTING LOCALLY ===================
//
// To exercise the CI code path locally (or to ingest a new package that
// Dependabot brought in but isn't yet cached in the feed):
//
// 1. Install the Azure Artifacts credential provider (one-time):
//
// PowerShell: iex "& { $(irm https://aka.ms/install-artifacts-credprovider.ps1) }"
// bash: wget -qO- https://aka.ms/install-artifacts-credprovider.sh | bash
//
// 2. Flip the switch and run the gradle build that needs the package:
//
// PowerShell: $env:RunningOnCI='true'; ./build-tools/gradle/gradlew.bat --project-dir src/r8 build
// bash: RunningOnCI=true ./build-tools/gradle/gradlew --project-dir src/r8 build
//
// On first authenticated request, you'll get a device-flow login prompt
// pointing at https://aka.ms/devicelogin — sign in with your Microsoft
// account. The credprovider caches the token; the feed caches the
// package; future CI runs read it anonymously and pass.
//
// =================== WORKFLOW FOR DEPENDABOT PRs ===================
//
// 1. Dependabot opens a PR bumping a Gradle dep (uses public repos, so it
// always sees the latest upstream version).
// 2. CI runs with RunningOnCI=true, hits the feed, and fails with 401 if
// the new package version isn't ingested yet.
// 3. A maintainer follows the steps above to ingest the package, then
// re-runs CI. No PR edit is required.
// When a Dependabot PR's CI fails with 401 because a new package isn't yet
// cached in the feed, run the helper script described in
// .github/instructions/gradle.instructions.md (TL;DR: `az login` once, then
// `pwsh ./eng/gradle/mirror-dependencies.ps1 -ProjectDir <path> -Task <task>`).
// After it succeeds, just re-run the failed CI job — no PR edit is needed.

repositories {
// Anonymous public Azure Artifacts feed that hosts the
Expand Down
2 changes: 1 addition & 1 deletion src/proguard-android/build.gradle
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
plugins {
id 'com.android.application' version '8.7.0'
id 'com.android.application' version '9.2.1'
}

android {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<LibraryProjectZip Include="$(OutputPath)JavaLib.aar" />
</ItemGroup>
<ItemGroup>
<InputJar Include="java\JavaLib\library\build\intermediates\aar_main_jar\debug\classes.jar">
<InputJar Include="java\JavaLib\library\build\libs\classes.jar">
<Link>Jars\classes.jar</Link>
</InputJar>
</ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,26 +1,10 @@
// Top-level build file where you can add configuration options common to all sub-projects/modules.

buildscript {
repositories {
google()
mavenCentral()
}
dependencies {
classpath 'com.android.tools.build:gradle:7.4.2'

// NOTE: Do not place your application dependencies here; they belong
// in the individual module build.gradle files
}
}

allprojects {
repositories {
google()
mavenCentral()
}
plugins {
id 'com.android.library' version '9.2.1' apply false
}

task clean(type: Delete) {
delete rootProject.buildDir
delete rootProject.layout.buildDirectory
}

Original file line number Diff line number Diff line change
@@ -1,25 +1,40 @@
apply plugin: 'com.android.library'
plugins {
id 'com.android.library'
}

android {
compileSdkVersion 25
namespace 'com.example.javalib'
compileSdk 35

defaultConfig {
minSdkVersion 19
targetSdkVersion 25
minSdk 21
targetSdk 35
versionCode 1
versionName "1.0"

testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"

}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
}
}
}

dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar'])
}

// Extract classes.jar from the AAR to a stable path that the binding
// project (../../../Xamarin.Android.LibraryProjectZip-LibBinding.csproj)
// can reference without depending on AGP intermediates/ layout, which
// Google reorganizes between major AGP versions.
tasks.register('extractClassesJar', Copy) {
dependsOn 'assembleDebug'
def aar = layout.buildDirectory.file('outputs/aar/library-debug.aar')
from({ zipTree(aar) }) {
include 'classes.jar'
}
into layout.buildDirectory.dir('libs')
}

tasks.matching { it.name == 'assembleDebug' }.configureEach { finalizedBy 'extractClassesJar' }
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.javalib">
<manifest xmlns:android="http://schemas.android.com/apk/res/android">

</manifest>
Original file line number Diff line number Diff line change
@@ -1 +1,15 @@
// See: eng/gradle/plugin-repositories.gradle, eng/gradle/dependency-repositories.gradle
pluginManagement {
apply from: "${rootDir}/../../../../../eng/gradle/plugin-repositories.gradle", to: pluginManagement
}

plugins {
id 'com.microsoft.azure.artifacts.credprovider' version '1.1.1'
}

dependencyResolutionManagement {
apply from: "${rootDir}/../../../../../eng/gradle/dependency-repositories.gradle", to: dependencyResolutionManagement
}

rootProject.name = 'JavaLib'
include ':library'
Loading