Running Wasm-R3-Bench on WAMR

[Michielvk]

When trying to run Wasm-R3-Bench on WAMR we face the following issue for some of the modules:

WASM module load failed: a module with WASI apis must export memory by default

After some digging around, we found that disabling WASI libc and using WAMR's built-in libc solves this issue. This is achieved by compiling the runtime (iwasm in our case) using the following config options:

cmake .. -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_LIBC_BUILTIN=1

And the resulting iwasm binary is able to run the full suite.

Another approach we tried was to modify the Wasm-R3-Bench binaries that had trouble running by adding the following line:

(export "memory" (memory 0))

which also solved the issue.

We don't like the second approach though, because then we would be running binaries that differ from Wasm-R3-Bench, which could make our results less comparable to prior or future results.

On the other hand, we are unsure of the implications of using WAMR's built-in libc. Any thoughts?

[doehyunbaek]

Hello!

On the other hand, we are unsure of the implications of using WAMR's built-in libc. Any thoughts?

I think for me to make a decision, there should be a concrete reproducing repository with the relevant wasm-r3-bench module and the relevant engines. But even then, to actually judge whether what is the "correct" behavior would probably be out of my expertise. For that, you should consult WAMR developers and WASI people IMHO.

I don't think I can spend a lot of time looking at this case but I can take a look if you make a reproducing repository.

We don't like the second approach though, because then we would be running binaries that differ from Wasm-R3-Bench, which could make our results less comparable to prior or future results.

I think if there is no prior work and the modification you make to the module is minor (this case seems to be so in my judgement), I think you can take this approach. Even more so if you properly document the changes you made to the benchmark modules and publish the artifacts. Still, as you pointed out, it might make it more troublesome for future works.

It's your call, but judging from the description you gave me, I would make that modification, reveal the fact, and publish the relevant artifacts.