Repository navigation
gomodfs: cache file contents in WinFsp mode - #33
Merged
Merged
Conversation
WinFsp mode fetched a file's contents from the store on the first read of each open, and Windows doesn't keep a file's cached data once no process has it open, so every go command fetched every file it read again. In a Windows VM serving from a remote store, three identical cached builds each fetched the same ~2,880 files. Keep fetched contents in an LRU cache on FS keyed by module version and path, bounded like the NFS contents cache by GetFileCacheSize (the --mem-limit-mb flag, default 2 GiB). The NFS cache is keyed by NFS handle, so WinFsp couldn't use it. In that VM, fully cached go commands (median of 30, p < 1e-10): go build tailscale.com/cmd/tailscale/cli: 2.23s -> 0.98s go run github.com/tc-hib/go-winres: 1.73s -> 0.64s The first command after mounting isn't helped, as everything is a miss then. "go mod verify" of all 75 modules of the build passes over the mount, as does a full build of tailscale.com/cmd/tailscale. Updates tailscale/corp#24037 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WinFsp mode fetched a file's contents from the store on the first read
of each open, and Windows doesn't keep a file's cached data once no
process has it open, so every go command fetched every file it read
again. In a Windows VM serving from a remote store, three identical
cached builds each fetched the same ~2,880 files.
Keep fetched contents in an LRU cache on FS keyed by module version and
path, bounded like the NFS contents cache by GetFileCacheSize (the
--mem-limit-mb flag, default 2 GiB). The NFS cache is keyed by NFS
handle, so WinFsp couldn't use it.
In that VM, fully cached go commands (median of 30, p < 1e-10):
The first command after mounting isn't helped, as everything is a miss
then. "go mod verify" of all 75 modules of the build passes over the
mount, as does a full build of tailscale.com/cmd/tailscale.
Updates tailscale/corp#24037