SFAM-2777 fixes - #60
Open
mxsrc wants to merge 3 commits into
Open
Conversation
spdk_delete_rmt_lvol_pg/destroy_xfer_task_tmo freed rmt_lvol, reqs, pdus,
and rings on xfer-task timeout with no check of outstanding_io, unlike the
device-remove path (spdk_wait_for_pg_io_cleanup), which correctly defers
free behind an outstanding_io==0 poller. A late S3 GET or local blob-write
completion then wrote through freed memory in set_req_status_and_queued
(rmt->outstanding_io--, req->status, free_ring enqueue) - a heap UAF that
surfaced as a stalled S3 restore ("outstanding io N, state 13") and, when
completions did arrive late, as glibc heap corruption crashing later on an
unrelated allocation.
- add XFER_REQ_STATUS_IN_FLIGHT, set when helper_xfer_poller claims a req
- spdk_delete_rmt_lvol_pg: if rmt_lvol->outstanding_io > 0, best-effort
abort in-flight reqs via spdk_bdev_abort (matches by existing req cb_arg,
no new bookkeeping) and defer put_io_channel/free behind a new
outstanding_io-gated poller (spdk_wait_for_xfer_pg_io_cleanup_poller),
mirroring the device-remove path instead of freeing synchronously
- xfer->pg[lpg->id] is now only cleared once the rmt_lvol is actually
drained and freed, so destroy_xfer_task_tmo's gate is meaningful
- close a race where device-remove teardown wins for the same rmt_lvol:
its poller now also clears xfer_task->pg[lpg->id], otherwise that flag
would hang forever instead of draining
- set_req_status_and_queued: return early on the (compiled-out-under-
NDEBUG) assert(false) paths instead of falling through and scribbling
further on already-inconsistent state
No behavior change to destroy_xfer_task_tmo itself or to timeouts/retries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds fixes regarding https://simplyblock.atlassian.net/browse/SFAM-2777. In one failure case, a UAF is avoided by deferring freeing transfer structures until all inflight IO has been resolved. The main fix is the introduction of selective retries of IO requests on S3 devices. The root cause of the bug is the AWS SDK swallowing individual requests very rarely, this allows to gracefully handle the problem