Cache submodule into git db #16246
Cache submodule into git db #16246
Conversation
rustbot
added
A-git
|
r? @weihanglo rustbot has assigned @weihanglo. Use |
|
It would help reviewers if you break this down into atomic commits. For example, by making a commit for extracting See https://doc.crates.io/contrib/process/working-on-cargo.html#submitting-a-pull-request |
This comment has been minimized.
This comment has been minimized.
- creates GitSource for each submodules - replaces fetch inside update_submodule by GitSource::update_db and db.copy_to - removes recursive update_submodules calls cos db.copy_to already recursive. - adjust git testsuite
avnyu
force-pushed
the
cached-submodules
branch
from
70350b1 to
ffd78e6
Compare
|
This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
|
Done, the "extract code into update_db" part is a single commit now. I think the remains part needs to be in the same commit |
src/cargo/sources/git/utils.rs
Outdated
| .with_context(|| { | ||
| let name = child.name().unwrap_or(""); | ||
| format!("failed to fetch submodule `{name}` from {child_remote_url}",) | ||
| })?; |
There was a problem hiding this comment.
Seems like we should have this also on update_db and copy_to
There was a problem hiding this comment.
I tried and the testsuite failed. Should I also adjust testsuite more? I do try to keep it unchanged as much as possible.
There was a problem hiding this comment.
How about put it only after db.copy_to? I tested and it do pass the testsuite without further modification. The original fetch was replaced by (gitsource create, update_db and db.copy_to), so put the error context to after db.copy_to seem enough?
src/cargo/sources/git/utils.rs
Outdated
| SourceId::from_url(&format!("git+{child_remote_url}#{head}"))?, | ||
| gctx, | ||
| RemoteKind::GitDependency, | ||
| ) |
There was a problem hiding this comment.
What impact does this have on our progress bars?
There was a problem hiding this comment.
Mostly the same? The old fetch will have a single progress bar, which now changes into a progress bar for update_db and a brief progress bar after for db.copy. This progress bars behavior should be the same as when update git source. Most of the time I don't even notice the brief one.
src/cargo/sources/git/utils.rs
Outdated
| db.copy_to(actual_rev, &repo.path(), gctx)?; | ||
|
|
||
| let obj = repo.find_object(head, None)?; | ||
| reset(&repo, &obj, gctx)?; | ||
| update_submodules(&repo, gctx, &child_remote_url) | ||
| reset(&repo, &obj, gctx) |
There was a problem hiding this comment.
Would it be more proper to copy_to after reset?
There was a problem hiding this comment.
If I not wrong, the original fetch create checkout?/source at the submodules path. update_db only create git db at CARGO_HOME/git/db, not at the checkout submodules path, so db.copy_to need to run first before we can git reset.
the original fetch is replace by (GitSource create, update_db, db.copy) so the error context should be put after db.copy
4 participants
What does this PR try to resolve?
My attempt to continue #10279.
This:
Fixes #7987.
How to test and review this PR?
I tested using the original pull method:
and confirmed that the time to do the second update is negligible.
Also test if it can fetch submodule offline using the downloaded git db