Merge #6919: backport: merge bitcoin#31100, bitcoin#31626, bitcoin#31840, bitcoin#30584, bitcoin#31982, bitcoin#32086, bitcoin#31998, bitcoin#32505, bitcoin#32568, bitcoin#32690, bitcoin#32731, bitcoin#32716, bitcoin#32837, bitcoin#32266, bitcoin#30095, bitcoin#30137, bitcoin#33580, partial bitcoin#30454 (build backports: part 4)

8e79c7a fix: allow `dbghelp.dll` for PE targets (Kittywhiskers Van Gogh)
f97d3d6 fix: skip `EXPORTED_SYMBOLS` validation on ELF targets (Kittywhiskers Van Gogh)
d5dffb6 merge bitcoin#33580: Use $(package)_file_name when downloading from the fallback (Kittywhiskers Van Gogh)
b7febbe merge bitcoin#30137: Remove `--enable-threadlocal` (Kittywhiskers Van Gogh)
dcf67c7 merge bitcoin#30095: avoid using thread_local variable that has a destructor (Kittywhiskers Van Gogh)
4e57d1a merge bitcoin#32266: Avoid `warning: "_FORTIFY_SOURCE"` redefined for `libevent` (Kittywhiskers Van Gogh)
6e66ef8 merge bitcoin#32837: fix libevent `_WIN32_WINNT` usage (Kittywhiskers Van Gogh)
61f2a23 merge bitcoin#32716: Override host compilers for FreeBSD and OpenBSD (Kittywhiskers Van Gogh)
ca52975 merge bitcoin#32731: Build `qt` package for FreeBSD hosts (Kittywhiskers Van Gogh)
ee9e934 build: check against `$host` instead of `TARGET_OS` in stacktrace search (Kittywhiskers Van Gogh)
44d32a3 merge bitcoin#32690: fix multiprocess build on OpenBSD (apply capnp patch, correct SHA256SUM command) (Kittywhiskers Van Gogh)
8d90c3c merge bitcoin#32568: use "mkdir -p" when installing xproto (Kittywhiskers Van Gogh)
d4bc0aa merge bitcoin#32505: bump to latest config.guess and config.sub (Kittywhiskers Van Gogh)
6020cdc merge bitcoin#31998: patch around PlacementNew issue in capnp (Kittywhiskers Van Gogh)
00350a0 merge bitcoin#32086: Shuffle depends instructions and recommend modern make for macOS (Kittywhiskers Van Gogh)
c8e27a2 merge bitcoin#31982: rename libmultiprocess repository (Kittywhiskers Van Gogh)
86d0a27 merge bitcoin#30584: Make default `host` and `build` comparable (Kittywhiskers Van Gogh)
e6d6d17 merge bitcoin#31840: add missing Darwin objcopy (Kittywhiskers Van Gogh)
8d887c3 merge bitcoin#31626: Use base system's `sha256sum` utility on FreeBSD (Kittywhiskers Van Gogh)
b443c14 partial bitcoin#30454: Introduce CMake-based build system (Kittywhiskers Van Gogh)
67aa238 merge bitcoin#31100: remove dependency install instructions from win docs (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Dependent on #6918

  * Dependency for #6927

  * [dash#6966](#6966) broke Guix build post-compilation binary verification for ELF and PE binaries as

    * For PE binaries, with improved `libbacktrace` detection, we are now building the capability in Windows binaries and that introduces a dependency on `dbghelp.dll` ([source](https://github.com/dashpay/dash/blob/edcb9f265b63693a8e684bd22fba5555434eff62/configure.ac)) that wasn't in the `PE_ALLOWED_LIBRARIES` allowlist, it has since been added.

    * For ELF binaries, now that we use `-export-dynamic` for symbol preservation, the `EXPORTED_SYMBOLS` check now fails. As the diagnostic value of retaining these symbols far exceeds the file size reduction, we have opted to comment out the test entirely.

  ## Breaking Changes

  None expected.

  ## Checklist

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    re-utACK 8e79c7a

Tree-SHA512: 08f904dfcbf4ece848fe805395d78b3615ff40ed5d4df3afa73576de8960ccee5c70547495aefba54cd7b01dceefe0933831640484b0996468189da265dc711b

Author: PastaPastaPasta

build-aux/m4/bitcoin_qt.m4

@@ -148,7 +148,7 @@ AC_DEFUN([BITCOIN_QT_CONFIGURE],[
       _BITCOIN_QT_CHECK_STATIC_PLUGIN([QWindowsIntegrationPlugin], [-lqwindows])
       _BITCOIN_QT_CHECK_STATIC_PLUGIN([QWindowsVistaStylePlugin], [-lqwindowsvistastyle])
       AC_DEFINE([QT_QPA_PLATFORM_WINDOWS], [1], [Define this symbol if the qt platform is windows])
-    elif test "$TARGET_OS" = "linux"; then
+    elif test "$TARGET_OS" = "linux" -o "$TARGET_OS" = "freebsd"; then
       _BITCOIN_QT_CHECK_STATIC_PLUGIN([QXcbIntegrationPlugin], [-lqxcb])
       AC_DEFINE([QT_QPA_PLATFORM_XCB], [1], [Define this symbol if the qt platform is xcb])
     elif test "$TARGET_OS" = "darwin"; then

configure.ac

@@ -237,12 +237,6 @@ AC_ARG_ENABLE([lcov-branch-coverage],
   [use_lcov_branch=yes],
   [use_lcov_branch=no])
 
-AC_ARG_ENABLE([threadlocal],
-  [AS_HELP_STRING([--enable-threadlocal],
-  [enable features that depend on the c++ thread_local keyword (currently just thread names in debug logs). (default is to enable if there is platform support)])],
-  [use_thread_local=$enableval],
-  [use_thread_local=auto])
-
 AC_ARG_ENABLE([zmq],
   [AS_HELP_STRING([--disable-zmq],
   [disable ZMQ notifications])],
@@ -952,8 +946,8 @@ case $host in
    *linux*)
      TARGET_OS=linux
      ;;
-   *bsd*)
-     TARGET_OS=bsd
+   *freebsd*)
+     TARGET_OS=freebsd
      ;;
 esac
 
@@ -1192,45 +1186,6 @@ AC_COMPILE_IFELSE([AC_LANG_SOURCE([
   [AC_MSG_RESULT([no])]
 )
 
-if test "$use_thread_local" = "yes" || test "$use_thread_local" = "auto"; then
-  TEMP_LDFLAGS="$LDFLAGS"
-  LDFLAGS="$TEMP_LDFLAGS $PTHREAD_CFLAGS"
-  AC_MSG_CHECKING([for thread_local support])
-  AC_LINK_IFELSE([AC_LANG_SOURCE([
-    #include <thread>
-    static thread_local int foo = 0;
-    static void run_thread() { foo++;}
-    int main(){
-    for(int i = 0; i < 10; i++) { std::thread(run_thread).detach();}
-    return foo;
-    }
-    ])],
-    [
-     case $host in
-       *mingw*)
-          dnl mingw32's implementation of thread_local has also been shown to behave
-          dnl erroneously under concurrent usage; see:
-          dnl https://gist.github.com/jamesob/fe9a872051a88b2025b1aa37bfa98605
-          AC_MSG_RESULT([no])
-          ;;
-        *freebsd*)
-          dnl FreeBSD's implementation of thread_local is also buggy (per
-          dnl https://groups.google.com/d/msg/bsdmailinglist/22ncTZAbDp4/Dii_pII5AwAJ)
-          AC_MSG_RESULT([no])
-          ;;
-        *)
-          AC_DEFINE([HAVE_THREAD_LOCAL], [1], [Define if thread_local is supported.])
-          AC_MSG_RESULT([yes])
-          ;;
-      esac
-    ],
-    [
-      AC_MSG_RESULT([no])
-    ]
-  )
-  LDFLAGS="$TEMP_LDFLAGS"
-fi
-
 BACKTRACE_FLAGS=
 BACKTRACE_LDFLAGS=
 BACKTRACE_LIBS=
@@ -1240,17 +1195,19 @@ if test "$enable_stacktraces" != "no"; then
       BACKTRACE_LIBS="$BACKTRACE_LIBS -ldbghelp"
     else
       AC_CHECK_HEADERS([execinfo.h], [
-        if test "$TARGET_OS" = "bsd"; then
-          AC_SEARCH_LIBS([backtrace], [execinfo], [
-            BACKTRACE_LIBS="$BACKTRACE_LIBS -lexecinfo"
-          ], [
-            if test "$enable_stacktraces" = "yes"; then
-              AC_MSG_ERROR([--enable-stacktraces was specified but cannot locate libexecinfo])
-            fi
-            AC_MSG_WARN([cannot locate libexecinfo, stacktraces will be disabled])
-            enable_stacktraces=no
-          ])
-        fi
+        case $host in
+          *bsd*)
+            AC_SEARCH_LIBS([backtrace], [execinfo], [
+              BACKTRACE_LIBS="$BACKTRACE_LIBS -lexecinfo"
+            ], [
+              if test "$enable_stacktraces" = "yes"; then
+                AC_MSG_ERROR([--enable-stacktraces was specified but cannot locate libexecinfo])
+              fi
+              AC_MSG_WARN([cannot locate libexecinfo, stacktraces will be disabled])
+              enable_stacktraces=no
+            ])
+            ;;
+        esac
         AX_CHECK_LINK_FLAG([-Wl,-export-dynamic], [BACKTRACE_LDFLAGS="$BACKTRACE_LDFLAGS -Wl,-export-dynamic"], [
           AX_CHECK_LINK_FLAG([-rdynamic], [BACKTRACE_LDFLAGS="$BACKTRACE_LDFLAGS -rdynamic"], [
             if test "$enable_stacktraces" = "yes"; then

contrib/devtools/symbol-check.py

@@ -163,7 +163,8 @@
 'msvcrt.dll', # C standard library for MSVC
 'SHELL32.dll', # shell API
 'WS2_32.dll', # sockets
-'bcrypt.dll',
+'bcrypt.dll', # crypto API
+'dbghelp.dll', # debugging routines
 # bitcoin-qt only
 'dwmapi.dll', # desktop window manager
 'GDI32.dll', # graphics device interface
@@ -285,7 +286,8 @@ def check_ELF_ABI(binary) -> bool:
 CHECKS = {
 lief.EXE_FORMATS.ELF: [
     ('IMPORTED_SYMBOLS', check_imported_symbols),
-    ('EXPORTED_SYMBOLS', check_exported_symbols),
+    # Dash: We export symbols aggressively to aid in backtrace generation
+    # ('EXPORTED_SYMBOLS', check_exported_symbols),
     ('LIBRARY_DEPENDENCIES', check_ELF_libraries),
     ('INTERPRETER_NAME', check_ELF_interpreter),
     ('ABI', check_ELF_ABI),

depends/Makefile

@@ -52,7 +52,6 @@ C_STANDARD ?= c11
 CXX_STANDARD ?= c++20
 
 BUILD = $(shell ./config.guess)
-HOST ?= $(BUILD)
 PATCHES_PATH = $(BASEDIR)/patches
 BASEDIR = $(CURDIR)
 HASH_LENGTH:=11
@@ -61,11 +60,6 @@ DOWNLOAD_RETRIES:=3
 HOST_ID_SALT ?= salt
 BUILD_ID_SALT ?= salt
 
-host:=$(BUILD)
-ifneq ($(HOST),)
-host:=$(HOST)
-endif
-
 ifneq ($(DEBUG),)
 release_type=debug
 else
@@ -75,9 +69,15 @@ endif
 base_build_dir=$(WORK_PATH)/build
 base_staging_dir=$(WORK_PATH)/staging
 base_download_dir=$(WORK_PATH)/download
-canonical_host:=$(shell ./config.sub $(HOST))
 build:=$(shell ./config.sub $(BUILD))
 
+host:=$(build)
+ifneq ($(HOST),)
+host:=$(HOST)
+endif
+HOST ?= $(BUILD)
+canonical_host:=$(shell ./config.sub $(HOST))
+
 build_arch =$(firstword $(subst -, ,$(build)))
 build_vendor=$(word 2,$(subst -, ,$(build)))
 full_build_os:=$(subst $(build_arch)-$(build_vendor)-,,$(build))

depends/README.md

@@ -1,16 +1,59 @@
-### Usage
+# Depends build
+
+This is a system of building and caching dependencies necessary for building
+Dash Core. It supports cross-compilation. For more details see [description.md](description.md),
+as well as [packages.md](packages.md) for how to add packages.
+
+## Usage
+
+### Ubuntu & Debian
+
+    apt install automake bison cmake curl libtool make patch pkg-config python3 xz-utils
 
 To build dependencies for the current arch+OS:
 
     make
 
-To build for another arch/OS:
+### macOS
 
-    make HOST=host-platform-triplet
+Install Xcode Command Line Tools and Homebrew Package Manager,
+see [build-osx.md](../doc/build-osx.md).
 
-For example:
+    brew install cmake make ninja
 
-    make HOST=x86_64-w64-mingw32 -j4
+To build dependencies for the current arch+OS:
+
+    gmake
+
+### FreeBSD
+
+    pkg install bash
+
+Skip the following packages if you don't intend to use the GUI and will build with [`NO_QT=1`](#dependency-options):
+
+    pkg install bison ninja pkgconf python3
+
+To build dependencies for the current arch+OS:
+
+    gmake
+
+### NetBSD
+
+    pkgin install bash gmake
+
+To build dependencies for the current arch+OS:
+
+    gmake
+
+### OpenBSD
+
+    pkg_add bash gmake gtar
+
+To build dependencies for the current arch+OS:
+
+    gmake
+
+## Configuring Dash Core
 
 **Dash Core's `configure` script by default will ignore the depends output.** In
 order for it to pick up libraries, tools, and settings from the depends build,
@@ -23,6 +66,53 @@ created. To use it during compilation:
 The default install prefix when using `config.site` is `--prefix=depends/<host-platform-triplet>`,
 so depends build outputs will be installed in that location.
 
+## Dependency Options
+
+The following can be set when running make: `make FOO=bar`
+
+- `SOURCES_PATH`: Downloaded sources will be placed here
+- `BASE_CACHE`: Built packages will be placed here
+- `SDK_PATH`: Path where SDKs can be found (used by macOS)
+- `FALLBACK_DOWNLOAD_PATH`: If a source file can't be fetched, try here before giving up
+- `C_STANDARD`: Set the C standard version used. Defaults to `c11`.
+- `CXX_STANDARD`: Set the C++ standard version used. Defaults to `c++20`.
+- `NO_BOOST`: Don't download/build/cache Boost
+- `NO_LIBEVENT`: Don't download/build/cache Libevent
+- `NO_QT`: Don't download/build/cache Qt and its dependencies
+- `NO_QR`: Don't download/build/cache packages needed for enabling qrencode
+- `NO_ZMQ`: Don't download/build/cache packages needed for enabling ZeroMQ
+- `NO_WALLET`: Don't download/build/cache libs needed to enable the wallet
+- `NO_BDB`: Don't download/build/cache BerkeleyDB
+- `NO_SQLITE`: Don't download/build/cache SQLite
+- `NO_UPNP`: Don't download/build/cache packages needed for enabling UPnP
+- `NO_NATPMP`: Don't download/build/cache packages needed for enabling NAT-PMP
+- `NO_USDT`: Don't download/build/cache packages needed for enabling USDT tracepoints
+- `ALLOW_HOST_PACKAGES`: Packages that are missed in dependencies (due to `NO_*` option or
+  build script logic) are searched for among the host system packages using
+  `pkg-config`. It allows building with packages of other (newer) versions
+- `MULTIPROCESS`: build libmultiprocess (experimental, requires cmake)
+- `DEBUG`: Disable some optimizations and enable more runtime checking
+- `HOST_ID_SALT`: Optional salt to use when generating host package ids
+- `BUILD_ID_SALT`: Optional salt to use when generating build package ids
+- `LOG`: Use file-based logging for individual packages. During a package build its log file
+  resides in the `depends` directory, and the log file is printed out automatically in case
+  of build error. After successful build log files are moved along with package archives
+- `LTO`: Enable options needed for LTO. Does not add `-flto` related options to *FLAGS.
+- `NO_HARDEN=1`: Don't use hardening options when building packages
+
+If some packages are not built, for example `make NO_WALLET=1`, the appropriate
+options will be passed to Dash Core's configure. In this case, `--disable-wallet`.
+
+## Cross compilation
+
+To build for another arch/OS:
+
+    make HOST=host-platform-triplet
+
+For example:
+
+    make HOST=x86_64-w64-mingw32 -j4
+
 Common `host-platform-triplet`s for cross compilation are:
 
 - `i686-pc-linux-gnu` for Linux 32 bit
@@ -41,13 +131,7 @@ Common `host-platform-triplet`s for cross compilation are:
 - `aarch64-linux-android` for Android ARM 64 bit
 - `x86_64-linux-android` for Android x86 64 bit
 
-The paths are automatically configured and no other options are needed unless targeting [Android](../doc/build-android.md).
-
-### Install the required dependencies: Ubuntu & Debian
-
-#### Common
-
-    apt install automake bison cmake curl libtool make patch pkg-config python3 xz-utils
+The paths are automatically configured and no other options are needed.
 
 #### For macOS cross compilation
 
@@ -60,7 +144,7 @@ For more information, see [SDK Extraction](../contrib/macdeploy/README.md#sdk-ex
 
 #### For Win64 cross compilation
 
-- see [build-windows.md](../doc/build-windows.md#cross-compilation-for-ubuntu-and-windows-subsystem-for-linux)
+    apt install g++-mingw-w64-x86-64-posix
 
 #### For linux (including i386, ARM) cross compilation
 
@@ -88,60 +172,9 @@ For linux S390X cross compilation:
 
     sudo apt-get install g++-s390x-linux-gnu binutils-s390x-linux-gnu
 
-### Install the required dependencies: FreeBSD
-
-    pkg install bash
-
-### Install the required dependencies: OpenBSD
-
-    pkg_add bash gtar
-
-### Dependency Options
-
-The following can be set when running make: `make FOO=bar`
-
-- `SOURCES_PATH`: Downloaded sources will be placed here
-- `BASE_CACHE`: Built packages will be placed here
-- `SDK_PATH`: Path where SDKs can be found (used by macOS)
-- `FALLBACK_DOWNLOAD_PATH`: If a source file can't be fetched, try here before giving up
-- `C_STANDARD`: Set the C standard version used. Defaults to `c11`.
-- `CXX_STANDARD`: Set the C++ standard version used. Defaults to `c++20`.
-- `NO_BOOST`: Don't download/build/cache Boost
-- `NO_LIBEVENT`: Don't download/build/cache Libevent
-- `NO_QT`: Don't download/build/cache Qt and its dependencies
-- `NO_QR`: Don't download/build/cache packages needed for enabling qrencode
-- `NO_ZMQ`: Don't download/build/cache packages needed for enabling ZeroMQ
-- `NO_WALLET`: Don't download/build/cache libs needed to enable the wallet
-- `NO_BDB`: Don't download/build/cache BerkeleyDB
-- `NO_SQLITE`: Don't download/build/cache SQLite
-- `NO_UPNP`: Don't download/build/cache packages needed for enabling UPnP
-- `NO_NATPMP`: Don't download/build/cache packages needed for enabling NAT-PMP
-- `NO_USDT`: Don't download/build/cache packages needed for enabling USDT tracepoints
-- `ALLOW_HOST_PACKAGES`: Packages that are missed in dependencies (due to `NO_*` option or
-  build script logic) are searched for among the host system packages using
-  `pkg-config`. It allows building with packages of other (newer) versions
-- `MULTIPROCESS`: build libmultiprocess (experimental, requires cmake)
-- `DEBUG`: Disable some optimizations and enable more runtime checking
-- `HOST_ID_SALT`: Optional salt to use when generating host package ids
-- `BUILD_ID_SALT`: Optional salt to use when generating build package ids
-- `LOG`: Use file-based logging for individual packages. During a package build its log file
-  resides in the `depends` directory, and the log file is printed out automatically in case
-  of build error. After successful build log files are moved along with package archives
-- `LTO`: Enable options needed for LTO. Does not add `-flto` related options to *FLAGS.
-- `NO_HARDEN=1`: Don't use hardening options when building packages
-
-If some packages are not built, for example `make NO_WALLET=1`, the appropriate
-options will be passed to Dash Core's configure. In this case, `--disable-wallet`.
-
 ### Additional targets
 
     download: run 'make download' to fetch all sources without building them
     download-osx: run 'make download-osx' to fetch all sources needed for macOS builds
     download-win: run 'make download-win' to fetch all sources needed for win builds
     download-linux: run 'make download-linux' to fetch all sources needed for linux builds
-
-
-### Other documentation
-
-- [description.md](description.md): General description of the depends system
-- [packages.md](packages.md): Steps for adding packages

depends/builders/freebsd.mk

@@ -1,5 +1,9 @@
 build_freebsd_CC=clang
 build_freebsd_CXX=clang++
 
-build_freebsd_SHA256SUM = shasum -a 256
+build_freebsd_SHA256SUM = sha256sum
 build_freebsd_DOWNLOAD = curl --location --fail --connect-timeout $(DOWNLOAD_CONNECT_TIMEOUT) --retry $(DOWNLOAD_RETRIES) -o
+
+# freebsd host on freebsd builder: override freebsd host preferences.
+freebsd_CC = clang
+freebsd_CXX = clang++

depends/builders/openbsd.mk

@@ -1,9 +1,13 @@
 build_openbsd_CC = clang
 build_openbsd_CXX = clang++
 
-build_openbsd_SHA256SUM = sha256
+build_openbsd_SHA256SUM = sha256 -r
 build_openbsd_DOWNLOAD = curl --location --fail --connect-timeout $(DOWNLOAD_CONNECT_TIMEOUT) --retry $(DOWNLOAD_RETRIES) -o
 
 build_openbsd_TAR = gtar
 # openBSD touch doesn't understand -h
 build_openbsd_TOUCH = touch -m -t 200001011200
+
+# openbsd host on openbsd builder: override openbsd host preferences.
+openbsd_CC = clang
+openbsd_CXX = clang++