Consolidate CI pipeline jobs and add NuGet caching#8418
Open
Consolidate CI pipeline jobs and add NuGet caching#8418
Conversation
* copy layout files for installer build. enable all steps in PR for now * add restore package
… error Moved installer steps into the build job — BoM generation, installer build (Setup/dirs.proj), vsts drop upload, and bootstrapper creation now run as conditional steps inside the build job instead of a separate installer job. This fixes VSSDK1025: Could not find file 'raw\obj\VCDebugLauncher\*.pkgdef' caused by the separate job only downloading raw/binaries/ but missing intermediate outputs in raw/obj/. Removed separate installer job — eliminates redundant checkout, microbuild install, package restore, artifact download/copy overhead. Removed raw and SBOM artifact outputs — only existed for the installer job or diagnostics; saves storage. Added package restore to nuget job — needed for MicroBuild.Core used by signing. Temporarily enabled PublishNugetPackageAsBuildArtifact for PR builds. Kept buildInstaller parameter — allows opt-in installer build on PR builds when manually queuing.
…parate nuget job has been removed. Here's what changed:
NuGet steps added to build job (lines 268-280) — build_nuget_package.yml template + PublishBuildArtifacts for the pkg artifact, under the condition ${{ if or(notin(variables['Build.Reason'], 'PullRequest'), eq(variables['PublishNugetPackageAsBuildArtifact'], true)) }}
Eliminated duplicate work — No more checkout, MicroBuild plugin install, package restore, binary download, or binary copy steps that the separate job required
Removed separate nuget job entirely — No more waiting for a build machine
Member
Author
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Updated restore_packages.yml to: Add a Cache@2 step caching $(Pipeline.Workspace)/npm-cache (keyed on package.json + the pylance params). Pass npm env vars into the Restore packages task so PreBuild.ps1’s npm install reuses cached downloads (npm_config_cache, npm_config_prefer_offline, and disables audit/fund/progress). How to verify in CI You should see a new step “Cache npm” with a hit/miss message. On subsequent runs, the npm install part of “Restore packages” should download far less and run noticeably faster.
… node_modules when that switch is set. Updated restore_packages.yml to: Add a Cache node_modules step caching $(Build.SourcesDirectory)\node_modules (keyed on OS + package.json + pylance params). Pass -preserveNodeModules when invoking PreBuild.ps1 in CI. How to tell it’s working in the next run “Cache node_modules” shows a cache hit on the second run. In “Restore packages” logs you should see Preserving node_modules, and the Pylance portion should be much faster (often near-instant if versions didn’t change). Caveat: node_modules cache can be large; if the cache service starts evicting or misses frequently, we can narrow it to @pylance (or similar) instead.
The NPM_CONFIG_CACHE error was because Windows env vars are case-insensitive, so Azure Pipelines treats npm_config_cache and NPM_CONFIG_CACHE as the same key and flags the second as a duplicate. I removed NPM_CONFIG_CACHE from restore_packages.yml. npm_config_cache alone is enough. The “stages parameter is not a valid StageList” was very likely a cascading template-expansion error caused by that duplicate env-var key. After removing it, both: azure-pipelines.yml and restore_packages.yml parse cleanly as YAML locally.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The pipeline was split into 4 separate jobs (build, installer, test, nuget) that each had to wait for a build machine, do a fresh checkout, restore packages, and install MicroBuild plugins. This caused:
raw/binaries/but the VSSDKCopyPkgDeftarget needs intermediate outputs fromraw/obj/VCDebugLauncher/nugetjob spent most of its time waiting for a build machine and duplicating setup work already done by the build jobChanges
azure-pipelines.yml
buildInstallerparameter is setPublishNugetPackageAsBuildArtifactis setbuildInstallerparameter (default:false) — allows opt-in installer builds on PR runsinstallerandnugetjobs — all work now runs in the single build jobraw,Layout, andSBOMdiagnostic artifact outputs — no longer neededrestore_packages.yml
Cache@2task keyed onpackages.configcontent hash to cache the NuGet HTTP cache directory between runs, avoiding redundant package downloadsNet effect