Skip to content

Speed up mobile PR Android and iOS CI builds - #3717

Open
nic-olo wants to merge 6 commits into
devfrom
ci/faster-mobile-pr-builds
Open

Speed up mobile PR Android and iOS CI builds#3717
nic-olo wants to merge 6 commits into
devfrom
ci/faster-mobile-pr-builds

Conversation

@nic-olo

@nic-olo nic-olo commented Aug 19, 2026

Copy link
Copy Markdown
Member

Summary

ci-gate-dev is only as fast as the slowest gated builder. On mobile PRs that is Mobile App Android Build: last successful run spent ~17 min in assembleRelease on a 4 vCPU / 16 GB box compiling all four ABIs. iOS was ~10 min (plus Mac-mini queue). This PR cuts that wall clock without moving ABI coverage onto main.

Production mobile artifacts come from staging-builds.yml on pushes to staging, via bun run release:android. That is the full-ABI lane. This compile-check workflow stays arm64-only.

  • Bigger runner. Android 4 vCPU / 16 GB8 vCPU / 32 GB. The Expo plugin already sets -Xmx8192m; that heap was fighting the 16 GB machine.
  • ABI policy (Isaiah's review).
    • This workflow (PRs/pushes to dev and main): arm64-v8a only
    • Staging production (staging-builds.ymlrelease-android.mjs): all four ABIs
    • release-android.mjs no longer overwrites a caller-provided ORG_GRADLE_PROJECT_reactNativeArchitectures (defaults to arm64 for local releases)
    • :lc3Lib honors reactNativeArchitectures the same way Silero does, so CMake is not still building v7a/x86 on an arm64 job
  • Lint. Do not -x lintVitalAnalyzeRelease / -x lintVitalReportRelease. assembleRelease still runs lintVitalRelease, which then fails looking for those output files (this PR's red Android run). Android lint stays on this job; Mobile App Quality Checks is TypeScript/Jest only.
  • Drop pre-checkout rm / disk-prune on ephemeral Blacksmith. iOS: drop -jobs $(sysctl -n hw.ncpu) on the shared Mac mini (oversubscribes concurrent branch builds); give the retry path a 90 min timeout.

Expected Android dev-PR wall clock: ~19 min → ~8–12 min. Gate wait is max(Android, iOS+queue).

Pricing (Blacksmith Ubuntu x64)

We already pay Blacksmith for this job; this is a size bump, not a vendor change. iOS stays on the self-hosted Mac mini ($0 Blacksmith). Staging full-ABI is existing production traffic, not a new bill on every dev PR.

4 vCPU (today) 8 vCPU (this PR)
List price $0.008 / min $0.016 / min (2×)
RAM / disk 16 GB / 80 GB 32 GB / 160 GB
Typical dev PR (arm64) 19.5 min → **$0.16** 8–12 min → **$0.13–$0.19**
Staging production (full ABI, existing) unchanged lane still all four ABIs, not this workflow

Monthly (this compile-check workflow only)

Last 15 days: 200 runs (~13.3/day) → ~400 runs/month. Almost all of that is dev PRs (arm64 after this PR). Staging full-ABI is a handful of production builds, billed on staging-builds.yml, not here.

Scenario Est. monthly $ vs today
Today (4 vCPU, all ABIs, every compile check) ~$45
Expected (8 vCPU, arm64, lint still on) ~$40–60 about even to +$15/mo
Best (speedup + fewer cancels) ~$30–35 −$10–15/mo
Miss (8 vCPU, little ABI win) ~$70–90 +$25–45/mo

Engineer wait is the expensive line: ~20 min blocked per mobile dev PR × ~350 PR builds/month.

Test plan

  • A PR into dev builds with ORG_GRADLE_PROJECT_reactNativeArchitectures=arm64-v8a and does not compile :lc3Lib for v7a/x86
  • assembleRelease gets past :app:lintVitalRelease (no missing lintVitalReportRelease inputs)
  • A push to staging sets all four ABIs and release-android.mjs logs that value instead of overwriting to arm64
  • Local bun run release:android with no env still defaults to arm64
  • Sideload a dev PR APK on an arm64 phone and launch Mentra
  • iOS compile check still greens on the self-hosted Mac without -jobs

Note

Cursor Bugbot is generating a summary for commit a60adb0. Configure here.

PR Android was the long pole (~17 min assembleRelease on 4 vCPU, all ABIs). Use 8 vCPU, arm64-only for PRs, and skip lint/test on this compile job so ci-gate-dev settles faster.
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

📋 PR Review Helper

📱 Mobile App Build

Ready to test! (commit a60adb0)

📥 Download APK

🕶️ ASG Client Build

Waiting for build...


🔀 Test Locally

gh pr checkout 3717

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

PR Agent Orchestrator State

{
  "cycle": 6,
  "fixRound": 0,
  "totalReviewerRuns": 6,
  "consecutiveNoNewReviews": 1,
  "openFindings": [],
  "resolvedFindings": [],
  "nitFindings": [],
  "phase": "discovery",
  "status": "in_progress",
  "lastPair": [
    "standards",
    "depth"
  ],
  "stagnationFixRounds": 0,
  "lastOpenCount": 0,
  "fingerprintReopenCounts": {},
  "mutedFingerprints": [],
  "revision": 17
}

@github-actions

Copy link
Copy Markdown
Contributor

bugbot run

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

🤖 PR Agent Review — cycle 6

✅ No blocking findings · 0 blocking · 0 nits
Reviewers this cycle: standards, depth

Claude — standards — ✅ approve

Commander, recon complete. Full sweep of the PR against MentraOS doctrine is done. Here's the after-action report.

Review

Scope: This PR is pure CI/build infrastructure — Android/iOS workflow tuning, ABI-filter plumbing in two native build.gradle modules, and a defensive guard in release-android.mjs. No application runtime logic (BLE, pairing, transcription) is touched.

Standards check — all clear:

  • Secrets hygiene — Every credential (MAPBOX_DOWNLOADS_TOKEN, RNMAPBOX_MAPS_DOWNLOAD_TOKEN, upload keystore, GH_TOKEN, Play/Sentry) is injected via ${{ secrets.* }}. No hardcoded tokens committed. The dual-name Mapbox token mapping is consistent across the primary build, the retry build, and the staging release step. Compliant with the AGENTS.md Mapbox two-credential doctrine (sk.… build-time only, never shipped).
  • ABI-filter logicfindProperty("reactNativeArchitectures") correctly resolves the ORG_GRADLE_PROJECT_* env var Gradle maps to a project property. The split/trim/filter chain is sound, and abiFilters(*abiList) spreads correctly. Default fallback (armeabi-v7a,arm64-v8a,x86,x86_64) preserves prior behavior for Maven publish (lc3Lib) and local builds where the property is unset — no regression for the sonatypeCentral publish path.
  • release-android.mjs guard — The if (!process.env.ORG_GRADLE_PROJECT_reactNativeArchitectures) no-clobber pattern is the correct fix; staging passes all four ABIs, local/dev defaults to arm64. Logging the resolved value is a good operator touch.
  • Testing guidelines — AGENTS.md requires unit + E2E coverage only for pairing/BLE/transcription changes. This PR touches none of that runtime surface; missing tests is not a finding here.
  • Naming conventions — N/A for this diff (no Java/TS source). Groovy locals (abiList) and JS follow idiomatic style.
  • Commit/PR hygiene — Focused single-concern scope (CI ABI + runner tuning). Nothing in the diff shows AI co-author trailers or unrelated churn.

Notes (non-blocking, not findings):

  • The comment "The Expo plugin already sets -Xmx8192m, which is tight on the 16 GB box" refers to the old 4vCPU/16GB runner as justification for moving to 8vCPU/32GB — reads slightly ambiguously but is accurate context, not an issue.

No open findings to re-test (orchestrator reported clean). No new blocking issues, no regressions, no nits worth raising. Mission verdict: approve.


Claude — depth — ✅ approve

Commander, recon complete. I traced the ABI-filter property through both native modules, the release script, and all three workflows, and verified the runtime property-resolution chain. Here is my after-action report.

Depth review — PR #3717

Change summary: CI/build infra only. Bumps the PR Android runner to 8 vCPU, adds timeouts, filters native-module ABIs by reactNativeArchitectures, and stops release-android.mjs from clobbering a caller-supplied ABI list so staging can produce all-ABI store artifacts.

Traced and verified:

  • Property resolution is sound. Gradle project-property precedence is env ORG_GRADLE_PROJECT_* > gradle.properties. The Expo-generated mobile/android/gradle.properties ships all four ABIs, but the job-level ORG_GRADLE_PROJECT_reactNativeArchitectures env var in each workflow correctly overrides it. PR build → arm64-v8a; staging → all four. Confirmed the same property drives both the app (RN plugin) and the two library modules, so no app-packages-4/lib-builds-1 mismatch that would ship an APK with missing .so files.
  • No regression for local/dev. The ?: fallback in both build.gradle files defaults to all four ABIs, matching silero's previous hardcoded list and lc3Lib's previous no-filter (all-ABI) behavior. android.mjs / android-release.mjs / android-internal.mjs still pin arm64, now honored by the modules — consistent.
  • release-android.mjs guard is correct and is actually a fix. Previously it unconditionally forced arm64-v8a, meaning the Play internal AAB from staging was arm64-only. The new if (!process.env...) lets staging's all-four value survive through prebuild → assembleRelease/bundleRelease. expo prebuild regenerates gradle.properties but the inherited env still wins. Good.
  • Retry step got the missing RNMAPBOX_MAPS_DOWNLOAD_TOKEN. This closes a latent gap where the clean-cache retry would re-add the rnmapbox Maven repo unauthenticated → "Could not find navigationcore." Now matches the first build step.
  • -x test / lint tasks: verified the build only excludes unit test, not the lintVital* chain the comment warns about. assembleReleaselintVitalRelease still finds its inputs.
  • Removed cleanup/disk steps: Blacksmith runners are ephemeral single-use VMs, so dropping "Clean build artifacts" and "Free up disk space" is safe (no stale state across runs); Gradle cache is still restored via actions/cache. arm64-only build is smaller, so disk pressure is lower than the old 4-ABI job.
  • Empty-string edge case (ORG_GRADLE_PROJECT_reactNativeArchitectures=) would yield an empty abiFilters() (no filtering → all ABIs), but no workflow sets it empty, so it's not reachable here — noting for completeness only.

Edge cases considered: disconnect/partial-failure paths in staging (pin gate, partial upload) are untouched by this PR; AAB/Play lane now gets full ABI coverage; iOS change is timeout + debug-step removal only.

No blocking logic issues found. openFindings is empty and nothing regressed.

Updated automatically by the PR Agent Orchestrator each review cycle. Nits do not block merge.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread .github/workflows/mentra-app-ios-build.yml Outdated
@github-actions

Copy link
Copy Markdown
Contributor

bugbot run

Dev PRs and pushes stay arm64-only so ci-gate-dev is not waiting on emulator/32-bit NDK. Multi-ABI coverage runs on the production promotion path.
Pushes to main already passed that check on the promotion PR, so keep them arm64-only like dev.
Dev stays arm64-only. Main keeps the emulator/32-bit check after merge as well as on the promotion PR.
@github-actions

Copy link
Copy Markdown
Contributor

bugbot run

@github-actions

Copy link
Copy Markdown
Contributor

bugbot run

@github-actions

Copy link
Copy Markdown
Contributor

PR Agent Orchestrator failed. See run: https://github.com/Mentra-Community/MentraOS/actions/runs/32235996491

Copy link
Copy Markdown
Member

The main ABI condition seems wired to the wrong lane.

Our production mobile artifacts are built on pushes to staging by staging-builds.yml, not by PRs/pushes to main. This workflow does not run on staging, and mobile/scripts/release-android.mjs currently forces ORG_GRADLE_PROJECT_reactNativeArchitectures=arm64-v8a. So as written, full-ABI coverage happens on unrelated main checks while the actual production build remains arm64-only. The PR description's claim that multi-ABI coverage moves to the production promotion path is therefore not accurate.

I think the intended policy should be:

  • dev/ordinary PR compile checks: arm64-v8a only
  • staging production build: all four ABIs
  • make release-android.mjs preserve a caller-provided architecture value instead of unconditionally overwriting it

There are also two concrete Android issues:

  1. The current job fails because it excludes lintVitalAnalyzeRelease and lintVitalReportRelease, but assembleRelease still runs lintVitalRelease, which then cannot find their output files. Both attempts in the current Actions run failed this way. Also, Mobile App Quality Checks only runs TypeScript/Jest; it does not replace Android release lint coverage.
  2. lc3Lib still ignores reactNativeArchitectures. The arm64 CI run visibly compiled armeabi-v7a, x86, and x86_64 CMake targets for :lc3Lib, so the dev build is not actually arm64-only yet. It needs the same ABI filtering treatment as Silero.

The 8-vCPU runner change and making Silero honor the ABI property look good; the routing/filtering and lint invocation need adjustment before merge.

Full ABI belongs on staging production, not main. Stop excluding lintVital subtasks that assembleRelease still needs, honor reactNativeArchitectures in lc3Lib, and drop iOS -jobs on the shared Mac.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 20, 2026

Copy link
Copy Markdown

Deploying mentra-live-ota-site with  Cloudflare Pages  Cloudflare Pages

Latest commit: a60adb0
Status: ✅  Deploy successful!
Preview URL: https://1f47c1c0.mentra-live-ota-site.pages.dev
Branch Preview URL: https://ci-faster-mobile-pr-builds.mentra-live-ota-site.pages.dev

View logs

@nic-olo

nic-olo commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

Addressed Israelov's review on this push:

ABI routing. This compile-check workflow is now always arm64-v8a. Full ABI is on the actual production lane: staging-builds.yml sets ORG_GRADLE_PROJECT_reactNativeArchitectures to all four ABIs, and release-android.mjs keeps a caller-provided value instead of overwriting it (still defaults to arm64 for local releases).

Lint. Dropped the -x lintVitalAnalyzeRelease / -x lintVitalReportRelease skips. Those left :app:lintVitalRelease looking for files that were never produced — that was the red Android run. Lint stays on this job; Quality Checks is still TS/Jest only.

lc3Lib. Same reactNativeArchitecturesabiFilters treatment as Silero, so CMake is not still building v7a/x86 on an arm64 job.

Also took Cubic's iOS note: dropped -jobs $(sysctl -n hw.ncpu) on the shared Mac mini (oversubscribe), and gave the retry path a 90 min timeout.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying mentra-store-dev with  Cloudflare Pages  Cloudflare Pages

Latest commit: a60adb0
Status: ✅  Deploy successful!
Preview URL: https://d4427a2b.augmentos-appstore-2.pages.dev
Branch Preview URL: https://ci-faster-mobile-pr-builds.augmentos-appstore-2.pages.dev

View logs

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying dev-augmentos-console with  Cloudflare Pages  Cloudflare Pages

Latest commit: a60adb0
Status: ✅  Deploy successful!
Preview URL: https://4f505932.dev-augmentos-console.pages.dev
Branch Preview URL: https://ci-faster-mobile-pr-builds.dev-augmentos-console.pages.dev

View logs

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying prod-augmentos-account with  Cloudflare Pages  Cloudflare Pages

Latest commit: a60adb0
Status: ✅  Deploy successful!
Preview URL: https://2473d67e.augmentos-e84.pages.dev
Branch Preview URL: https://ci-faster-mobile-pr-builds.augmentos-e84.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants