8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp #27885

YaSuenag · 2025-10-19T06:16:56Z

jhsdb jstack --mixed would not work when attaches to the process runs with -Xcomp.

It has been reported by @pchilano in #27728. You can reproduce the problem with Test.java (attached JBS). You can see following stack.

----------------- 646689 -----------------
"Thread-0" #24 prio=5 tid=0x00007f1cec18c890 nid=646689 waiting on condition [0x00007f1cd0158000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
0x00007f1cf3b7f462 __syscall_cancel_arch + 0x32
0x00007f1cf3b7375c __internal_syscall_cancel + 0x5c
0x00007f1cf3b766a8 ___pthread_cond_timedwait + 0x178
0x00007f1cf270e1e9 PlatformEvent::park_nanos(long) + 0x119
0x00007f1cf2005f4c JavaThread::sleep_nanos(long) + 0xfc
0x00007f1cf218789f JVM_SleepNanos + 0x28f
0x00007f1cdb95f299 java.lang.Thread.sleepNanos0(long) + 0x99 (Native method)

Thread.sleepNanos0 is the bottom stack, but actually it has more call frames. You can see them with -XX:+PreserveFramePointer.

----------------- 646841 -----------------
"Thread-0" #24 prio=5 tid=0x00007f4a0018c9e0 nid=646841 waiting on condition [0x00007f49e4fd7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
0x00007f4a0aa29462 __syscall_cancel_arch + 0x32
0x00007f4a0aa1d75c __internal_syscall_cancel + 0x5c
0x00007f4a0aa206a8 ___pthread_cond_timedwait + 0x178
0x00007f4a0970e1e9 PlatformEvent::park_nanos(long) + 0x119
0x00007f4a09005f4c JavaThread::sleep_nanos(long) + 0xfc
0x00007f4a0918789f JVM_SleepNanos + 0x28f
0x00007f49ef961099 java.lang.Thread.sleepNanos0(long) + 0x99 (Native method)
0x00007f49e7f477b4 * java.lang.Thread.sleepNanos(long) bci:33 line:509 (Compiled frame)
0x00007f49e7f41a64 * java.lang.Thread.sleep(long) bci:25 line:540 (Compiled frame)
0x00007f49e7f4037c * Test.run() bci:3 line:6 (Compiled frame)
0x00007f49ef943328 * java.lang.Thread.runWith(java.lang.Object, java.lang.Runnable) bci:5 line:1487 (Compiled frame)
                        * java.lang.Thread.run() bci:19 line:1474 (Compiled frame)
0x00007f49ef3385fd <StubRoutines (initial stubs)>
0x00007f4a08fc247e JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 0x4ce
0x00007f4a08fc2bb3 JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*) + 0x2d3
0x00007f4a08fc31bb JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*) + 0xab
0x00007f4a09185590 thread_entry(JavaThread*, JavaThread*) + 0xd0
0x00007f4a09004206 JavaThread::thread_main_inner() + 0x256
0x00007f4a09c66747 Thread::call_run() + 0xb7
0x00007f4a096fccc8 thread_native_entry(Thread*) + 0x128
0x00007f4a0aa20f54 start_thread + 0x2e4

Java frame might be use the register for frame pointer (RBP in AMD64) as general purpose register, so SA cannot rely it in stack unwinding.

hs_err log has mixed stack trace as "Native frames", it would be unwinded by NativeStackPrinter in HotSpot, and it works as mixed mode. NativeStackPrinter uses frame::next_frame() to find sender frame regardless whether Java frame or C frame, and it leverages sender FP/PC to create sender frame. On the other hand, SA separates CFrame and VFrame to unwind in mixed mode jstack, so sender FP/PC would not propagate to CFrame, thus the frame located at bottom of Java frame might not be shown.

It is difficult to unify unwinder in PStack in SA, so it would be reasonable to propagate sender FP/PC to the sender of CFrame.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp (Bug - P4)

Reviewers

Chris Plummer (@plummercj - Reviewer)
Kevin Walls (@kevinjwalls - Reviewer)

Contributors

Fei Yang <[email protected]>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27885/head:pull/27885
$ git checkout pull/27885

Update a local copy of the PR:
$ git checkout pull/27885
$ git pull https://git.openjdk.org/jdk.git pull/27885/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27885

View PR using the GUI difftool:
$ git pr show -t 27885

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27885.diff

Using Webrev

Link to Webrev Comment

bridgekeeper · 2025-10-19T06:18:18Z

👋 Welcome back ysuenaga! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2025-10-19T06:19:42Z

@YaSuenag This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp

Co-authored-by: Fei Yang <[email protected]>
Reviewed-by: cjplummer, kevinw

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 6 new commits pushed to the master branch:

4cfabd6: 8370974: Command-line Hotspot debugger tests fail on Windows AArch64
05efc1f: 8370578: SystemMapTest.java#normal fails intermittendly on macOS aarch64
e8a1a87: 8369186: HotSpot Style Guide should permit some uses of the C++ Standard Library
... and 3 more: https://git.openjdk.org/jdk/compare/d226023643f90027a8980d161ec6d423887ae3ce...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2025-10-19T06:20:19Z

@YaSuenag The following label will be automatically applied to this pull request:

serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

YaSuenag · 2025-10-19T06:21:51Z

/issue JDK-8370176

openjdk · 2025-10-19T06:23:07Z

@YaSuenag The primary solved issue for a PR is set through the PR title. Since the current title does not contain an issue reference, it will now be updated.

mlbridge · 2025-10-19T07:03:41Z

Webrevs

plummercj · 2025-10-20T19:44:07Z

test/hotspot/jtreg/serviceability/sa/TestJhsdbJstackMixedWithXComp.java

+ * @test
+ * @bug 8370176
+ * @requires vm.hasSA
+ * @requires os.family == "linux"


Do Windows and OSX have a similar problem that should be fixed also?

This problem is in mixed mode (PStack) only, thus we need to skip OSX because you mentioned mixed mode is not supported on OSX.

In Windows, I'm not sure, but I guess we need to consider UNWIND_INFO to unwind call frames correctly like DWARF in Linux, however it hasn't done yet. Thus we can think mixed mode is not supported in Windows too, so I didn't add Windows here.
https://learn.microsoft.com/cpp/build/exception-handling-x64

Actually I could not see all of stacks as following in mixed mode. It works in normal mode (without --mixed) of course. (I tested it on Windows 11 x64, upstream JDK built by VS 2022)

----------------- 13 ----------------- "Reference Handler" #15 daemon prio=10 tid=0x00000207280b9f70 nid=12684 waiting on condition [0x000000aaf6aff000] java.lang.Thread.State: RUNNABLE JavaThread state: _thread_blocked 0x00007fffa6b45844 ntdll!NtWaitForAlertByThreadId + 0x14 0x00000000ffffffff ????????

Mind you that this new test seems to fail even on linux systems without pstack. This is happening on both of my AMD64 machine running Debian 12 and ARM64 machine running Ubuntu 22.04.4.

Can you share .jtr file?

Sure. This is what I got on my amd64 machine:

$ make test TEST="serviceability/sa/TestJhsdbJstackMixedWithXComp.java"

TestJhsdbJstackMixedWithXComp.jtr.txt

I updated this PR to check glibc version in TestJhsdbJstackMixedWithXComp.java added by this PR. It skips the test on Ubuntu 22.04, OTOH it works on Fedora 43. It is expected.
I attempted to add this check to SATestUtils at first, but it seems to be difficult because native access have to be allowed all of SATestUtils users - the impact is too significant.

I will file another issue to apply this check to other tests of jhsdb jstack --mixed user after this PR.

I updated testcase again. It is more scalable for other mixed jstack tests. It works fine on both Fedora 43 and Ubuntu 22.04 .

Still fails on other distros like Debian 13 on AMD64. It has glibc version 2.41. Attached please fine the JTR file.

TestJhsdbJstackMixedWithXComp.jtr.txt

@RealFYang Thanks a lot for sharing JTR file!

I could find out the problem, and I fixed. We need to handle RSP more carefully when parsing DWARF. This PR works fine on Fedora 43 and Ubuntu 22. I believe it works on your Debian/Ubuntu.

But I think the failure what you saw on AArch64 is caused by problem(s) on stack unwinder for Linux AArch64, it is different from AMD64. Currently DWARF is supported on Linux AMD64 only, other platforms would attempt to unwind relies on base pointer. It is traditional, but it might not work on DWARF based binaries. I saw DWARF is contained in AArch64 binary in Fedora Rawhide for AArch64 at least. So the test added by this PR might not work on other platforms includes AArch64, RISC-V. Thus I removed them from @requires in the test. I think we can enable them if the unwinder (e.g. LinuxAARCH64CFrame.java) supports DWARF, but it is not a scope of this bug.

Yes, the latest version now works on my Debian 13 AMD64 machine. Thanks for finding it out. It's good to know the difference.

plummercj

Changes look good. Thanks for fixing this.

RealFYang · 2025-10-23T03:08:51Z

@YaSuenag :
Hi, I tried the test on linux-riscv64 and seems this platform bears the same issue.
Would you mind adding this add-on fix for this platform please? Thanks.
8370176-riscv64.diff.txt

YaSuenag · 2025-10-23T04:43:07Z

@RealFYang Thanks a lot for sharing a patch for RISC-V! Merged to this PR.

YaSuenag · 2025-10-23T04:48:31Z

@plummercj Thanks a lot for your review!

I'm trying to fix mixed mode on Windows. I think we can unwind native stacks with this change, but it is not enough for Java frames - I think we can see all of them if we modify after this PR.

----------------- 13 -----------------
"Reference Handler" #15 daemon prio=10 tid=0x000001a8df240270 nid=4800 waiting on condition [0x0000002207fff000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_blocked
0x00007fff725a5844      ntdll!NtWaitForAlertByThreadId + 0x14
0x00007fff7244c30b      ntdll!RtlSleepConditionVariableCS + 0x14b
0x00007fff6f6ca688      KERNELBASE!SleepConditionVariableCS + 0x38
0x00007ffed580c92d      jvm!PlatformMonitor::wait + 0x3d
0x00007ffed5796f3f      jvm!Monitor::wait + 0x15f
0x00007ffed5448310      jvm!JVM_WaitForReferencePendingList + 0xb0
0x000001a8ce87fa18      <interpreter> native method entry point (kind = native)
0x000001a8df240710              ????????
0x0000002207fffa38              ????????
0x000001a8df240270              ????????
0x000001a8e000bd98              ????????
0x000001a8ce87f39b      <interpreter> native method entry point (kind = native)
0xfffffffffffffff7              ????????
0x000000003e871a28              ????????
0x0000000000000003              ????????
0x000000003e2054f8              ????????

YaSuenag · 2025-10-28T04:23:38Z

@plummercj @RealFYang
Could you review/approve this PR again?

RealFYang · 2025-10-29T01:50:59Z

@plummercj @RealFYang Could you review/approve this PR again?

Hi, Sorry, as a co-author I don't think I can review this. BTW: Do you mind listing me as a co-author? Thanks.

plummercj · 2025-10-29T01:51:15Z

test/hotspot/jtreg/serviceability/sa/TestJhsdbJstackMixedWithXComp.java

+        System.out.println(stdout);
+        System.err.println(out.getStderr());
+
+        out.stderrShouldBeEmptyIgnoreDeprecatedWarnings();


I might want you to change this to stderrShouldBeEmptyIgnoreVMWarnings(). We have issues with our internal testing when using -XX:+UseLargePages that sometimes results in a VM warning on stderr that causes this (and some other tests) to fail. We are still deciding on the best approach to fixing this, but I think the solution is most likely to be switching to stderrShouldBeEmptyIgnoreDeprecatedWarnings(). Hopefully I'll know within the next day.

Ok, I will update with the right way. Let me know what should I do when you know.

I updated the test to use stderrShouldBeEmptyIgnoreVMWarnings(). @plummercj let me know if I should make more changes.

YaSuenag · 2025-10-29T02:11:19Z

/contributor add @RealFYang

openjdk · 2025-10-29T02:12:49Z

@YaSuenag
Contributor Fei Yang <[email protected]> successfully added.

kevinjwalls · 2025-10-31T17:33:33Z

src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/debugger/linux/amd64/LinuxAMD64CFrame.java

-     return (nextCFA != null) &&
-             !nextCFA.lessThan(context.getRegisterAsAddress(AMD64ThreadContext.RSP));
+   private boolean isValidFrame(Address nextCFA, boolean isNative) {
+     // CFA should not be null even if it is Java frame.


This comment means a Java interpreter frame need not have a different CFA/frame pointer?

"native" means we found DWARF. Non-native means Java interpreted or Java compiled.
Java frames just need a non-null CFA.

// CFA should never be null.
// nextCFA must be greater than current CFA, if frame is native.
// Java interpreter frames can share the CFA (frame pointer).

Exactly. I updated the comment. Thanks!

openjdk · 2025-11-01T02:36:04Z

@YaSuenag this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout jhsdb-jstack-bp
git fetch https://git.openjdk.org/jdk.git master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

YaSuenag · 2025-11-01T03:17:32Z

I resolved merge confliction, and it passed all of serviceability/sa tests on Linux AMD64.

kevinjwalls

Thanks, spent quite a while going over this yesterday and think it's good.
Yes was waiting for the conflict with 8369994, that looks like it merged in OK. 8-)

YaSuenag · 2025-11-02T06:29:26Z

@plummercj Can you approve again this PR? Thanks!

plummercj

Looks good.

YaSuenag · 2025-11-03T14:24:21Z

Thanks a lot everyone involved in this PR!

/integrate

openjdk · 2025-11-03T14:25:50Z

Going to push as commit 045018d.
Since your change was applied there have been 20 commits pushed to the master branch:

8dbefc5: 8370822: Improve performance for MemorySegment::fill
ef464d6: 8371142: [BACKOUT] [ubsan] add non-zero offset to nullptr in cds/archiveBuilder.cpp
4520983: 8368882: NPE during text drawing on machine with JP locale
... and 17 more: https://git.openjdk.org/jdk/compare/d226023643f90027a8980d161ec6d423887ae3ce...master

Your commit was automatically rebased without conflicts.

openjdk · 2025-11-03T14:26:00Z

@YaSuenag Pushed as commit 045018d.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

YaSuenag added 3 commits October 17, 2025 21:31

Mixed jstack does not work without -XX:+PreserveFramePointer

58668e6

Add change for AArch64

39274ff

Update bug ID in testcase

9d02325

openjdk bot added the serviceability [email protected] label Oct 19, 2025

openjdk bot changed the title ~~Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp~~ 8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp Oct 19, 2025

YaSuenag marked this pull request as ready for review October 19, 2025 06:59

openjdk bot added the rfr Pull request is ready for review label Oct 19, 2025

Update testcase

a75aa29

plummercj reviewed Oct 21, 2025

View reviewed changes

plummercj approved these changes Oct 22, 2025

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Oct 22, 2025

YaSuenag added 2 commits October 23, 2025 13:37

Remove debug messages

3f7697f

Add the fix for RISC-V

0e7c4cc

openjdk bot removed the ready Pull request is ready to be integrated label Oct 23, 2025

YaSuenag and others added 7 commits October 25, 2025 11:48

Add glibc version check to TestJhsdbJstackMixedWithXComp.java

434e8d2

Update testcase

ff070a3

Update testcase

b08101b

Fix imports

082ee46

Revert testcase changes

0ba1e48

Respect RSP to parse DWARF

879cf19

TestJhsdbJstackMixedWithXComp.java should be run on Linux AMD64 only

16a9a20

plummercj reviewed Oct 29, 2025

View reviewed changes

Use stderrShouldBeEmptyIgnoreVMWarnings

3e0bfd0

kevinjwalls reviewed Oct 31, 2025

View reviewed changes

Improve comments in isValidFrame method

ec5d0e2

openjdk bot added the merge-conflict Pull request has merge conflict with target branch label Nov 1, 2025

Merge branch 'master' into jhsdb-jstack-bp

eb2f0ab

openjdk bot removed the merge-conflict Pull request has merge conflict with target branch label Nov 1, 2025

kevinjwalls approved these changes Nov 1, 2025

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Nov 1, 2025

plummercj approved these changes Nov 3, 2025

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Nov 3, 2025

openjdk bot closed this Nov 3, 2025

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Nov 3, 2025

YaSuenag deleted the jhsdb-jstack-bp branch November 3, 2025 14:26

YaSuenag restored the jhsdb-jstack-bp branch November 5, 2025 08:11

YaSuenag mentioned this pull request Nov 15, 2025

8371194: serviceability/sa/TestJhsdbJstackMixedWithXComp.java failing #28284

Open

3 tasks

8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp #27885

8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp #27885

Uh oh!

Conversation

YaSuenag commented Oct 19, 2025 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Contributors

Reviewing

Uh oh!

bridgekeeper bot commented Oct 19, 2025

Uh oh!

openjdk bot commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YaSuenag commented Oct 19, 2025

Uh oh!

openjdk bot commented Oct 19, 2025

Uh oh!

mlbridge bot commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YaSuenag Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RealFYang Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RealFYang Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plummercj left a comment

Choose a reason for hiding this comment

Uh oh!

RealFYang commented Oct 23, 2025

Uh oh!

YaSuenag commented Oct 23, 2025

Uh oh!

YaSuenag commented Oct 23, 2025

Uh oh!

YaSuenag commented Oct 28, 2025

Uh oh!

RealFYang commented Oct 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YaSuenag commented Oct 29, 2025

Uh oh!

openjdk bot commented Oct 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

YaSuenag commented Oct 19, 2025 •

edited by openjdk bot

Loading

openjdk bot commented Oct 19, 2025 •

edited

Loading

openjdk bot commented Oct 19, 2025 •

edited

Loading

mlbridge bot commented Oct 19, 2025 •

edited

Loading

YaSuenag Oct 21, 2025 •

edited

Loading

RealFYang Oct 23, 2025 •

edited

Loading

RealFYang Oct 23, 2025 •

edited

Loading