Skip to content

Commit b23ffbb

Browse files
WillemKaufStephanDollberg
authored andcommitted
cpu_profiler: avoid taking samples during context switches
If the profiler's signal lands while a `seastar::thread` is switching stacks, backtracing walks inconsistent register state and can crash in libgcc's unwinder. This was observed as a `SIGSEGV` in `uw_update_context` on aarch64 ASan builds while a debug bundle's 30s cpu_profile request sampled admin handlers running under ss::async: glibc's aarch64 swapcontext has a nop-only CFI program but clobbers the link register, so the unwinder computed register-save addresses from a garbage CFA. Check the context-switch flag introduced for the stall detector and drop the sample instead, counting it under a new dedicated stat, the same treatment as samples dropped during exception unwinding.
1 parent ef29e99 commit b23ffbb

2 files changed

Lines changed: 13 additions & 2 deletions

File tree

include/seastar/core/internal/cpu_profiler.hh

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,19 +82,22 @@ struct cpu_profiler_stats {
8282
unsigned dropped_samples_from_exceptions{0};
8383
unsigned dropped_samples_from_buffer_full{0};
8484
unsigned dropped_samples_from_mutex_contention{0};
85+
unsigned dropped_samples_from_context_switches{0};
8586

8687
void clear_dropped() {
8788
dropped_samples_from_manual_disablement = 0;
8889
dropped_samples_from_exceptions = 0;
8990
dropped_samples_from_buffer_full = 0;
9091
dropped_samples_from_mutex_contention = 0;
92+
dropped_samples_from_context_switches = 0;
9193
}
9294

9395
unsigned sum_dropped() const {
9496
return dropped_samples_from_manual_disablement
9597
+ dropped_samples_from_buffer_full
9698
+ dropped_samples_from_exceptions
97-
+ dropped_samples_from_mutex_contention;
99+
+ dropped_samples_from_mutex_contention
100+
+ dropped_samples_from_context_switches;
98101
}
99102
};
100103

src/core/cpu_profiler.cc

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525

2626
#include <seastar/core/internal/cpu_profiler.hh>
2727
#include <seastar/core/scheduling.hh>
28+
#include <seastar/core/thread_impl.hh>
2829
#include <seastar/util/log.hh>
2930

3031
namespace seastar {
@@ -117,13 +118,20 @@ void cpu_profiler::on_signal() {
117118
// is currently being unwound and avoid taking a profiling sample if so.
118119
//
119120
// Note: this only protects against C++ exceptions, therefore foreign
120-
// exceptions or long jumps could still cause segfaults within the profiler.
121+
// exceptions could still cause segfaults within the profiler.
121122
const bool no_uncaught_exceptions = std::uncaught_exceptions() == 0;
122123

123124
if (force_drop_stacktraces) {
124125
_stats.dropped_samples_from_manual_disablement++;
125126
} else if (!no_uncaught_exceptions) {
126127
_stats.dropped_samples_from_exceptions++;
128+
} else if (thread_impl::is_context_switch_in_progress()) {
129+
// While a seastar::thread switches stacks the register state is
130+
// inconsistent with the unwind info (glibc's swapcontext/longjmp
131+
// clobber the link register on aarch64 while their CFI still claims
132+
// the return address lives there), so backtracing from here would
133+
// walk garbage frames and can crash in libgcc's unwinder.
134+
_stats.dropped_samples_from_context_switches++;
127135
} else if (auto guard_opt = _traces_mutex.try_lock(); guard_opt.has_value()) {
128136
// Skip the sample if the main thread is currently reading
129137
// _traces. This case shouldn't happen often though.

0 commit comments

Comments
 (0)