8369442: ExitOnOutOfMemoryError should exit more gracefully

[shipilev]

See RFE for more discussion. It seems we have a leeway in defining what "exit" means for -XX:+ExitOnOutOfMemoryError, and this PR does it more akin to JVM_Halt, rather than abrupt os::_exit. This gives VM a chance to shutdown some of its subsystems gracefully.

Comments welcome!

Additional testing:

• Linux x86_64 server fastdebug, tier1
• Linux x86_64 server fastdebug, all

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8369442: ExitOnOutOfMemoryError should exit more gracefully (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27718/head:pull/27718
$ git checkout pull/27718

Update a local copy of the PR:
$ git checkout pull/27718
$ git pull https://git.openjdk.org/jdk.git pull/27718/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27718

View PR using the GUI difftool:
$ git pr show -t 27718

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27718.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back shade! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@shipilev The following label will be automatically applied to this pull request:

• hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (40062dfc)
• 00: Full (238d2901)

[dholmes-ora]

One concern I just realized with this, is that we will now grab the Heap_lock before doing the VM_Exit safepoint op. But we are calling this in the context of a failed allocation, so are we sure this will always be safe and not e.g. cause a deadlock?

I don't know who might use ExitOnOutOfMemoryError, nor in what way, so it is hard to say whether this change might break something

[xmas92]

Are we sure that before_exit can handle being called at any point between set_init_completed and when we start running user byte code.

I know we have internally questioned the validity of the development flag ExitOnFullCodeCache which also can call before_exit very early, albiet it uses the non-halting version which is scarier.

[dholmes-ora]

Are we sure that before_exit can handle being called at any point between set_init_completed and when we start running user byte code.

Good point! More generally if we can trigger OOM during VM init, then some of the tear-down actions in before_exit may not be valid if we haven't completed the corresponding init actions. And some of those "init" actions may happen after set_init_completed.

I know we have internally questioned the validity of the development flag ExitOnFullCodeCache which also can call before_exit very early,

I would also question its validity, but as a develop flag I'd be prepared to accept that unexpected things may happen.