Add FreeBSD Support

[moonshine-cc]

Running FreeBSD 13.0-RELEASE with current versions of go, rust and gmake installed:

build-rust-release works and finishes without errors (only requires a modification of SHARED_LIB_EXT = "so" for FreeBSD OS detection in the Makefile).

build-go doesn't work. Here is the output:

$ gmake build-go
go build ./...
go: downloading github.com/stretchr/testify v1.7.0
go: downloading github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
# github.com/CosmWasm/wasmvm/api
ld: error: undefined symbol: analyze_code
>>> referenced by cgo-gcc-prolog:56
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_analyze_code)
>>> referenced by cgo-gcc-prolog:609
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_analyze_code)

ld: error: undefined symbol: execute
>>> referenced by cgo-gcc-prolog:90
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_execute)
>>> referenced by cgo-gcc-prolog:639
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_execute)

ld: error: undefined symbol: get_metrics
>>> referenced by cgo-gcc-prolog:113
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_get_metrics)
>>> referenced by cgo-gcc-prolog:658
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_get_metrics)

ld: error: undefined symbol: ibc_channel_close
>>> referenced by cgo-gcc-prolog:146
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_channel_close)
>>> referenced by cgo-gcc-prolog:687
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_channel_close)

ld: error: undefined symbol: ibc_channel_connect
>>> referenced by cgo-gcc-prolog:179
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_channel_connect)
>>> referenced by cgo-gcc-prolog:716
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_channel_connect)

ld: error: undefined symbol: ibc_channel_open
>>> referenced by cgo-gcc-prolog:212
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_channel_open)
>>> referenced by cgo-gcc-prolog:745
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_channel_open)

ld: error: undefined symbol: ibc_packet_ack
>>> referenced by cgo-gcc-prolog:245
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_packet_ack)
>>> referenced by cgo-gcc-prolog:774
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_packet_ack)

ld: error: undefined symbol: ibc_packet_receive
>>> referenced by cgo-gcc-prolog:278
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_packet_receive)
>>> referenced by cgo-gcc-prolog:803
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_packet_receive)

ld: error: undefined symbol: ibc_packet_timeout
>>> referenced by cgo-gcc-prolog:311
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_ibc_packet_timeout)
>>> referenced by cgo-gcc-prolog:832
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_ibc_packet_timeout)

ld: error: undefined symbol: init_cache
>>> referenced by cgo-gcc-prolog:337
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_init_cache)
>>> referenced by cgo-gcc-prolog:854
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_init_cache)

ld: error: undefined symbol: instantiate
>>> referenced by cgo-gcc-prolog:371
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_instantiate)
>>> referenced by cgo-gcc-prolog:884
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_instantiate)

ld: error: undefined symbol: load_wasm
>>> referenced by cgo-gcc-prolog:395
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_load_wasm)
>>> referenced by cgo-gcc-prolog:904
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_load_wasm)

ld: error: undefined symbol: migrate
>>> referenced by cgo-gcc-prolog:428
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_migrate)
>>> referenced by cgo-gcc-prolog:933
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_migrate)

ld: error: undefined symbol: pin
>>> referenced by cgo-gcc-prolog:449
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_pin)
>>> referenced by cgo-gcc-prolog:950
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_pin)

ld: error: undefined symbol: query
>>> referenced by cgo-gcc-prolog:479
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_query)
>>> referenced by cgo-gcc-prolog:976
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_query)

ld: error: undefined symbol: reply
>>> referenced by cgo-gcc-prolog:512
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_reply)
>>> referenced by cgo-gcc-prolog:1017
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_reply)

ld: error: undefined symbol: save_wasm
>>> referenced by cgo-gcc-prolog:536
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_save_wasm)
>>> referenced by cgo-gcc-prolog:1037
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_save_wasm)

ld: error: undefined symbol: sudo
>>> referenced by cgo-gcc-prolog:569
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_sudo)
>>> referenced by cgo-gcc-prolog:1066
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_sudo)

ld: error: undefined symbol: unpin
>>> referenced by cgo-gcc-prolog:590
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_C2func_unpin)
>>> referenced by cgo-gcc-prolog:1083
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_unpin)

ld: error: undefined symbol: release_cache
>>> referenced by cgo-gcc-prolog:991
>>>               $WORK/b047/_x004.o:(_cgo_d3f6466c8ed4_Cfunc_release_cache)

ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
gmake: *** [Makefile:37: build-go] Error 2

[webmaster128]

Have a look at api/link_*.go. There you see a build condition and different link commands depending on the system. I guess linux in both cases makes that both link_*.go are excluded and the linkage lines

// #cgo LDFLAGS: -Wl,-rpath,${SRCDIR} -L${SRCDIR} -lwasmvm
import "C"

is missing.

[moonshine-cc]

@webmaster128 And clue on how to resolve this?

I should have mentioned that I'm not a developer/programmer and that I have no experience with go.

[webmaster128]

@moonshine-cc if that is the case I see very little chance for FreeBSD support any time soon. We have to build the library for each target system individually which is quite hard for Linux/Mac/Windows on x86/ARM already and this is the first time I hear about the desire for building for FreeBSD.

What's your motivation for targetting this system?

[moonshine-cc]

@webmaster128 With FreeBSD it's possible to use Linux binaries and libraries natively thanks to its Linux Binary Compatibility Layer, which makes use of CentOS 7 (if interested in the details, see Chapter 10. Linux Binary Compatibility of the FreeBSD handbook.

I'm running a set of validators on a server powered by FreeBSD, which is an extremely pleasant experience so far (way better then with the usual Linux distributions).

Thanks to the Linux Binary Compatibilty Layer and all the needed components (like go and rust) available natively, I could make almost all validator projects that I have tried run without problems so far, except the latest one that made use of CosmWasm.

[webmaster128]

Oh, that's interesting. Our .so is built on a CentOS 7 system. So it might just work with this diff: #313. Can you try that?

[moonshine-cc]

@webmaster128 Seems to have worked:

$ go build ./...
go: downloading github.com/tendermint/tm-db v0.6.7
go: downloading github.com/stretchr/testify v1.7.0
go: downloading github.com/google/btree v1.0.0
go: downloading github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/golang/snappy v0.0.1

No errors, but now I'm puzzled, what's the result?
(I've just cloned the repo, applied the diff and only ran go build ./... to check what's going to happen)

[moonshine-cc]

@webmaster128 Wow, it really works! 😃

So I have just rebuilt said project with the necessary changes applied and everything indeed turned out fine (though there was some trickery involved, but that's that project's fault). This doesn't even require the FreeBSD Linux Compatibilty Layer, because with go and rust being available natively, libwasmvm.so is build natively for FreeBSD.

So all that needs to be done to make wasmvm compatible with FreeBSD is as follows:

• Your patch for the two api/link_*.go files.
• Make "so" the default in Makefile line 11 (instead of empty) or add freebsd to the OS detection for the "so" file extension in the following lines.

You than can add FreeBSD to the supported builds in the README.md.

This is just awesome, because I have been fiddling around with this several times including giving up in the meantime, only to come back and try again. The major catch with said project was, that it doesn't build the libwasmvm.so but uses the already present binary, which then doesn't work as it's linked to Linux binaries which aren't there on this FreeBSD system (at least not in the right place, seems to be a mixup between FreeBSD and Linuxinator binaries).

[webmaster128]

Glad to hear that.

Now the question is how you want this deployed. The Linux .so is built by the CI and checked into git for every change. See https://github.com/CosmWasm/wasmvm/tree/main/api for the .so and .dylib. I would like to do that for FreeBSD too because it comsumes a lot of space in git and I don't know if I found a FreeBSD CI system. If you don't want to build everything manually and patch the chain for every upgrade using a local version of wasmvm, it's probably better to use the Linux build.

[moonshine-cc]

@webmaster128 Sorry, not really sure what you mean. The already present Linux libwasmvm.so can not be used on FreeBSD, so that library has to be built natively on a FreeBSD system, which is what I just did. I don't know if this can be solved in any better way, but as long as I'm able to build libwasmvm.so locally, just having the mentioned changes works fine by me.

[webmaster128]

The already present Linux libwasmvm.so can not be used on FreeBSD

But didn't you say it can be used via CentOS 7 compatibility mode on FreeBSD?

[moonshine-cc]

But didn't you say it can be used via CentOS 7 compatibility mode on FreeBSD?

That's what I thought as well, but it didn't work. This is what it looks like with the Linux libwasmvm.so:

$ ldd craftd 
craftd:
        libthr.so.3 => /lib/libthr.so.3 (0x803d3e000)
        libwasmvm.so => /home/craftnator/go/pkg/mod/github.com/!cosm!wasm/wasmvm@v1.0.0-soon2.0.20220223181252-4aba468c9bc6/api/libwasmvm.so (0x803e00000)
        libc.so.7 => /lib/libc.so.7 (0x8043c4000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x803d6b000)
        librt.so.1 => /usr/lib/librt.so.1 (0x803d84000)
        libpthread.so.0 => not found (0)
        libm.so.6 => not found (0)
        libdl.so.2 => not found (0)
        libc.so.6 => not found (0)
        ld-linux-x86-64.so.2 (0)

I'm not able to explain what the problem is with this (looks like the Linux binary can't find the needed libraries in the right place, and even with some manual symlinking, it then said something about a wrong glibc version).

This is what it looks like with the FreeBSD libwasmvm.so:

$ ldd craftd
craftd:
        libthr.so.3 => /lib/libthr.so.3 (0x803d3e000)
        libwasmvm.so => /home/craftnator/go/pkg/mod/github.com/!cosm!wasm/wasmvm@v1.0.0-soon2.0.20220223181252-4aba468c9bc6/api/libwasmvm.so (0x803d6b000)
        libc.so.7 => /lib/libc.so.7 (0x804076000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x804487000)
        libm.so.5 => /lib/libm.so.5 (0x8044a0000)

So the only way to go is to build libwasmvm.so natively for FreeBSD.

[moonshine-cc]

@webmaster128 So it's working, but I'm not sure how you could implement the required patches to make it universally usable. The diff for the two api/link_*.go files is OK, but oddly the OS detection for determining the library extension isn't.

I have GNU Make installed on FreeBSD, so I'm compiling the project with gmake build and although it supposedly should work that way, I'm getting "Need an operator" errors on the used ifeq / elseif / endif in lines 12, 14 and 22, which is why I simply have commented the whole checks out and made "so" the default:

SHARED_LIB_EXT = "so" # File extension of the shared library
#ifeq ($(OS),Windows_NT)
#       SHARED_LIB_EXT = dll
#else
#       UNAME_S := $(shell uname -s)
#       ifeq ($(UNAME_S),Linux)
#               SHARED_LIB_EXT = so
#       endif
#       ifeq ($(UNAME_S),Darwin)
#               SHARED_LIB_EXT = dylib
#       endif
#endif

Any ideas?

[webmaster128]

I think it should not be too hard to convert the main Makefile into a set of shell scripts. We hardly use any make feature there anyways. If you want to roll out make build, you can just do:

(cd libwasmvm && cargo build --release)
cp libwasmvm/target/release/libwasmvm.so api
	
cp libwasmvm/bindings.h api

go build ./...

[edjroz]

Manual build seems the way to go. you could also do a modification to the existing makefile with a BSD branch

SHARED_LIB_EXT = "" # File extension of the shared library
ifeq ($(OS),Windows_NT)
       SHARED_LIB_EXT = dll
else
       UNAME_S := $(shell uname -s)
       ifeq ($(UNAME_S),Linux)
               SHARED_LIB_EXT = so
       endif
      ifeq ($(UNAME_S), FreeBSD) # since the library works for both OS
               SHARED_LIB_EXT = so 
       endif
       ifeq ($(UNAME_S),Darwin)
               SHARED_LIB_EXT = dylib
       endif
endif

@moonshine-cc This might lead to underlying issues so thread carefully e.g:
Wasmd offers support for OpenBSD but has to disable ledger support