8371647: 7 Integer overflows in mlib_malloc of mlib_sys.c:85

[DamonGuy]

There is a possible overflow when using mlib_alloc(). For example, mlib_alloc(sizeof(mlib_s32) * (m * n)) may overflow if m and n are greater than 46430, since this would be greater than the max value for a signed 32 bit integer. I have added SAFE_TO_ADD and SAFE_TO_MULT in an attempt to amend this issue. CI testing shows all green.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8371647: 7 Integer overflows in mlib_malloc of mlib_sys.c:85 (Bug - P3) ⚠️ Issue is not open.

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28560/head:pull/28560
$ git checkout pull/28560

Update a local copy of the PR:
$ git checkout pull/28560
$ git pull https://git.openjdk.org/jdk.git pull/28560/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28560

View PR using the GUI difftool:
$ git pr show -t 28560

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28560.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back dnguyen! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@DamonGuy The following label will be automatically applied to this pull request:

• client

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (306d24dc)

[prrace]

If mlib_malloc ends up in
void *__mlib_malloc(mlib_u32 size);
which I think it must do, because I can't find anything else,
then that accepts an unsigned 32 bit int, which makes sense because malloc accepts a size_t which is unsigned.

Note that sizeof() returns size_t too, so the multiplication result should be promoted to unsigned in the existing code, and preserved when passed as an arg.

But SAFE_TO_MULT will return a failure on overflow of signed arithmetic. So I think we need something different here so we don't reject cases which are actually OK. ie in at least cases like this, we want to detect overflow of 32 bit unsigned, not 32 bit signed.