Skip to content

[Infrastructure] Make npm restore and build incremental #62466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 27, 2025
+139 −7
Merged

[Infrastructure] Make npm restore and build incremental #62466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
60e6853
Make npm restore incremental
javiercn Jun 25, 2025
26ecb5b
Exclude package.json files from inside node_modules as inputs
javiercn Jun 25, 2025
7e92ba4
Avoid modifying dependencies during build if already correct
javiercn Jun 25, 2025
d72b54b
More incremental fixes
javiercn Jun 25, 2025
221fb2d
Build incrementality
javiercn Jun 25, 2025
01e4d29
Common properties
javiercn Jun 25, 2025
cf1e423
Add a specific file list for all projects
javiercn Jun 25, 2025
822fc73
Fix local validation build
javiercn Jun 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 124 additions & 4 deletions eng/Npm.Workspace.nodeproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,131 @@
<_NpmAdditionalEnvironmentVariables>PUPPETEER_SKIP_DOWNLOAD=1</_NpmAdditionalEnvironmentVariables>
</PropertyGroup>

<Target Name="Restore">
<PropertyGroup>
<NpmWorkspaceRoot>$(MSBuildThisFileDirectory)..</NpmWorkspaceRoot>
<NodeModulesMarkerFile>$(NpmWorkspaceRoot)\node_modules\.package-lock.json</NodeModulesMarkerFile>
<!-- Common exclude patterns for NPM build input/output files -->
<NpmCommonExcludePatterns>
**\pkg-version.ts;
$(NpmWorkspaceRoot)\src\**\node_modules\**\*;
$(NpmWorkspaceRoot)\src\**\bin\**\*;
$(NpmWorkspaceRoot)\src\**\obj\**\*
</NpmCommonExcludePatterns>
</PropertyGroup>

<!-- Input files for incremental restore check -->
<ItemGroup>
<NpmInputFiles Include="$(NpmWorkspaceRoot)\package.json" />
<NpmInputFiles Include="$(NpmWorkspaceRoot)\package-lock.json" Condition="Exists('$(NpmWorkspaceRoot)\package-lock.json')" />
<NpmInputFiles Include="$(NpmWorkspaceRoot)\src\**\package.json" Exclude="$(NpmWorkspaceRoot)\src\**\node_modules\**\package.json" />
<NpmInputFiles Include="$(NpmWorkspaceRoot)\.npmrc" Condition="Exists('$(NpmWorkspaceRoot)\.npmrc')" />
</ItemGroup>

<!-- Output files for incremental restore check -->
<ItemGroup>
<NpmOutputFiles Include="$(NodeModulesMarkerFile)" />
</ItemGroup>

<!-- Output files for incremental build check -->
<ItemGroup>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How often might we have to update this list?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven’t changed anything in years I would say.

The list of outputs is not as important as the incrementalism is “all or nothing” for all the us projects.

MSBUILD requires at least one output or else it will not run the task.

The goal is that if you’re not touching the JavaScript. You don’t have to build any part of the JavaScript pipeline during your dev builds.

<!-- SignalR client outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\browser\signalr.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\browser\signalr.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\browser\signalr.min.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\browser\signalr.min.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\cjs\**\*.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\cjs\**\*.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\esm\**\*.d.ts" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\esm\**\*.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\esm\**\*.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\webworker\signalr.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\webworker\signalr.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\webworker\signalr.min.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr\dist\webworker\signalr.min.js.map" />

<!-- SignalR MessagePack protocol outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\browser\signalr-protocol-msgpack.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\browser\signalr-protocol-msgpack.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\browser\signalr-protocol-msgpack.min.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\browser\signalr-protocol-msgpack.min.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\cjs\**\*.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\cjs\**\*.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\esm\**\*.d.ts" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\esm\**\*.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\signalr-protocol-msgpack\dist\esm\**\*.js.map" />

<!-- JSInterop outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\JSInterop\Microsoft.JSInterop.JS\src\dist\src\Microsoft.JSInterop.d.ts" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\JSInterop\Microsoft.JSInterop.JS\src\dist\src\Microsoft.JSInterop.d.ts.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\JSInterop\Microsoft.JSInterop.JS\src\dist\src\Microsoft.JSInterop.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\JSInterop\Microsoft.JSInterop.JS\src\dist\src\Microsoft.JSInterop.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\JSInterop\Microsoft.JSInterop.JS\src\dist\tsconfig.tsbuildinfo" />

<!-- Components Web.JS outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.server.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.server.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.web.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.web.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.webassembly.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.webassembly.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Debug\blazor.webview.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.server.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.server.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.web.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.web.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.webassembly.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.webassembly.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\Web.JS\dist\Release\blazor.webview.js" />

<!-- WebAssembly Authentication outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\WebAssembly\WebAssembly.Authentication\src\Interop\dist\Debug\AuthenticationService.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\WebAssembly\WebAssembly.Authentication\src\Interop\dist\Debug\AuthenticationService.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\WebAssembly\WebAssembly.Authentication\src\Interop\dist\Release\AuthenticationService.js" />

<!-- WebAssembly Authentication MSAL outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\WebAssembly\Authentication.Msal\src\Interop\dist\Debug\AuthenticationService.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\WebAssembly\Authentication.Msal\src\Interop\dist\Debug\AuthenticationService.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\WebAssembly\Authentication.Msal\src\Interop\dist\Release\AuthenticationService.js" />

<!-- Custom Elements outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\CustomElements\src\js\dist\Debug\Microsoft.AspNetCore.Components.CustomElements.lib.module.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\CustomElements\src\js\dist\Debug\Microsoft.AspNetCore.Components.CustomElements.lib.module.js.map" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\Components\CustomElements\src\js\dist\Release\Microsoft.AspNetCore.Components.CustomElements.lib.module.js" />

<!-- FunctionalTests outputs -->
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\FunctionalTests\wwwroot\dist\signalr-functional-tests.js" />
<NpmBuildOutputFiles Include="$(NpmWorkspaceRoot)\src\SignalR\clients\ts\FunctionalTests\wwwroot\dist\signalr-functional-tests.js.map" />
</ItemGroup>

<!-- Input files for incremental build check -->
<ItemGroup>
<!-- Source files -->
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\*.ts" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns)" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\*.js" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns);**\samples\**\wwwroot\lib\**\*;**\FunctionalTests\wwwroot\lib\**\*;$(NpmWorkspaceRoot)\src\**\dist\**\*" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\*.mjs" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns);$(NpmWorkspaceRoot)\src\**\dist\**\*" />

<!-- Configuration files -->
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\package.json" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns);$(NpmWorkspaceRoot)\src\**\node_modules\**\package.json" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\tsconfig*.json" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns)" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\rollup.config.*" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns)" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\webpack.config.*" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns)" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\src\**\jest.config.*" Exclude="@(NpmBuildOutputFiles);$(NpmCommonExcludePatterns)" />

<!-- Root workspace configuration files that affect build -->
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\package.json" />
<NpmBuildInputFiles Include="$(NpmWorkspaceRoot)\tsconfig*.json" />
</ItemGroup>


<Target Name="Restore" Inputs="@(NpmInputFiles)" Outputs="@(NpmOutputFiles)">
<Message Text="Restoring NPM packages..." Importance="high" />
<Exec
Command="npm ci"
WorkingDirectory="$(MSBuildThisFileDirectory).."
WorkingDirectory="$(NpmWorkspaceRoot)"
EnvironmentVariables="$(_NpmAdditionalEnvironmentVariables)" />
</Target>

<Target Name="Build">
<Target Name="Build" DependsOnTargets="Restore" Inputs="@(NpmBuildInputFiles)" Outputs="@(NpmBuildOutputFiles)">
<PropertyGroup>
<PackageVersion>$(VersionPrefix)</PackageVersion>
<PackageVersion Condition="'$(VersionSuffix)' != ''">$(VersionPrefix)-$(VersionSuffix)</PackageVersion>
Expand All @@ -51,7 +167,7 @@
<Exec Command="npm run test" ContinueOnError="true" WorkingDirectory="$(MSBuildThisFileDirectory).." />
</Target>

<Target Name="Pack">
<Target Name="Pack" DependsOnTargets="Build">
<PropertyGroup>
<PackageVersion>$(VersionPrefix)</PackageVersion>
<PackageVersion Condition="'$(VersionSuffix)' != ''">$(VersionPrefix)-$(VersionSuffix)</PackageVersion>
Expand All @@ -60,6 +176,10 @@
<MakeDir Directories="$(PackageOutputPath)" Condition="!Exists('$(PackageOutputPath)')" />
<MakeDir Directories="$(IntermediateOutputPath)" Condition="!Exists('$(IntermediateOutputPath)')" />

<Exec
Command="node $(MSBuildThisFileDirectory)scripts/npm/pack-workspace.mjs --update-versions $(RepoRoot)package.json $(PackageVersion) $(PackageOutputPath) $(IntermediateOutputPath)"
EnvironmentVariables="$(_NpmAdditionalEnvironmentVariables)" />

<Exec
Command="node $(MSBuildThisFileDirectory)scripts/npm/pack-workspace.mjs --create-packages $(RepoRoot)package.json $(PackageVersion) $(PackageOutputPath) $(IntermediateOutputPath)"
EnvironmentVariables="$(_NpmAdditionalEnvironmentVariables)" />
Expand Down
18 changes: 15 additions & 3 deletions eng/scripts/npm/update-dependency-versions.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@ function applyPackageVersion(packagesToPack, defaultPackageVersion) {
const packageName = packageJson.name;
const packageVersion = defaultPackageVersion;
const packageDir = path.dirname(packagePath);

// Check if the version is already correct
if (packageJson.version === packageVersion) {
console.log(`Skipping ${packageName} - version ${packageVersion} already applied.`);
continue;
}

// Run npm version packageVersion --no-git-tag-version
// This will update the package.json version to the specified version without creating a git tag
// Make a backup of the package.json
Expand Down Expand Up @@ -90,11 +97,16 @@ function updateDependencyVersions(packagesToPack) {
// Find the dependency in packagesToPack, load the package.json and update the dependency version
const dependencyPackage = seenPackagesAndVersions.find(([_, packageJson]) => packageJson.name === dependency);
if (dependencyPackage) {
modified = true;
const dependencyPackagePath = dependencyPackage[0];
const dependencyPackageJson = JSON.parse(fs.readFileSync(dependencyPackagePath, 'utf-8'));
dependencies[dependency] = `>=${dependencyPackageJson.version}`;
console.log(`Updated dependency ${dependency} to ${dependencyPackageJson.version} in ${packageJson.name}.`);
const newDependencyVersion = `>=${dependencyPackageJson.version}`;

// Only update if the dependency version is actually different
if (dependencies[dependency] !== newDependencyVersion) {
modified = true;
dependencies[dependency] = newDependencyVersion;
console.log(`Updated dependency ${dependency} to ${dependencyPackageJson.version} in ${packageJson.name}.`);
}
}
}
if (modified) {
Expand Down
Loading