c8d/system: Fix race between df and prune#51979
Merged
Merged
Conversation
When running `docker system df` concurrently with `docker system prune` or image removal, `DiskUsage` would fail with "snapshot does not exist" error. This happened because layerDiskUsage walks all snapshots and gets their usage, but a concurrent prune could delete a snapshot between Walk and Usage calls. Handle NotFound errors gracefully by skipping deleted snapshots instead of returning an error. Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
|
I don't know how the cache is handled in Docker, but if there's a walk, can the GC specified by: {
"builder": {
"gc": {
"enabled": true,
"defaultKeepStorage": "40GB"
}
},
}trigger the same problem for |
Contributor
Author
|
The builder GC is handled by Buildkit (https://github.com/moby/buildkit) which handles this differently and doesn't walk all available snapshots and also seems to handle the not found errors gracefully: https://github.com/moby/buildkit/blob/649062d5e7be1785c31e79729a5725c699cf1370/cache/refs.go#L361 So I think it should be fine, but if there are some issues then they need to be handled on buildkit side. |
Member
|
Failure on Oracle 8 is unrelated, and a known flaky test; |
sahilsGit
added a commit
to sahilsGit/moby
that referenced
this pull request
May 22, 2026
When `docker system df` runs concurrently with image prune, a blob can be removed between presentChildrenHandler's store.Info check and the c8dimages.Children call that reads it, failing the API with "NotFound: content digest sha256:...: not found". PR moby#51979 fixed the same race in the snapshotter path; this handles it in the content-store walk. Treat NotFound from c8dimages.Children like the store.Info NotFound branch above it: the content is gone, so it has no children. Other errors still propagate. Fixes moby#52538 Signed-off-by: Sahil Singh <sahiilsiingh37@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docker system dffails with "Error response from daemon: failed to calculate image disk usage" #51978When running
docker system dfconcurrently withdocker system pruneor image removal,DiskUsagewould fail with "snapshot does not exist" error.This happened because layerDiskUsage walks all snapshots and gets their usage, but a concurrent prune could delete a snapshot between Walk and Usage calls.
Handle NotFound errors gracefully by skipping deleted snapshots instead of returning an error.
- What I did
- How I did it
- How to verify it
- Human readable description for the release notes
- A picture of a cute animal (not mandatory but encouraged)