Using the search function crashes kiwix-serve

[ASIXicle]

Running in a Debian container on Proxmox, kiwix-tools installed via apt. Using the 2024_1.zim from Wikipedia-- just trying to serve on LAN using command:

kiwix-serve --port=80xx -b /opt/wiki/data/dumps/wikipedia_en_all_maxi_2024-01.zim

Works swimmingly and search works, until it doesn't. Having a lot of trouble diagnosing why it's crashing out. Best I can sus out is a segfault error in the PVE system log:

Jun 29 10:56:46 pve kernel: MHD-worker[1636621]: segfault at 4 ip 00007835cf77451f sp 00007835cbe12fb0 error 4 in libxapian.so.30.12.3[7835cf693000+173000] likely on CPU 0 (core 0, socket 0)

AFAIK this is a memory issue, but there is significant free memory and it's all fine and dandy with memtest.

I've allocated 8GB to the container but it's yet to use more than 500MB. Maybe this is causing an issue so I'm going to drop it to 2GB.

This error will occur when all other containers are idle, i/o et all at basically nil. Container is on a zpool on a fast SSD.

Any advice?

#EDIT TO ADD:

kiwix-tools 3.3.0

libkiwix 11.0.0

• libzim 8.0.0
• libxapian 1.4.21
• libcurl 7.86.0
• libmicrohttpd 0.9.75
• libz 1.2.13
• libicu 72.1.0
• libpugixml 1.12.0

libzim 8.1.1

• libzstd 1.5.4
• liblzma 5.4.1
• libxapian 1.4.22
• libicu 72.1.0

[kelson42]

@ASIXicle Thank you for your bug report.

The version 3.3.0 is already 3 years ago and we have released multiple new versions of both kiwix-serve and its dependence.

The most important know is to achieve a clear list of reproduction steps. Do you have one?

[ASIXicle]

I just noticed that, wonder why the version in the repo is so out of date.

As far as reproduction steps I'm not exactly sure what you mean but I just ran it with the --verbose flag and here's the output leading to the fault:

======================
Requesting : 
full_url  : /wikipedia_en_all_maxi_2024-01/I/Red_pog.svg.png.webp
method    : GET (0)
version   : HTTP/1.1
request#  : 63
headers   :
 - accept : 'image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5'
 - accept-encoding : 'gzip, deflate'
 - accept-language : 'en-US'
 - connection : 'keep-alive'
 - dnt : '1'
 - host : '100.98.8.6:8036'
 - priority : 'u=5, i'
 - referer : 'http://100.98.8.6:8036/wikipedia_en_all_maxi_2024-01/A/Lake_Marmara'
 - sec-gpc : '1'
 - user-agent : 'Mozilla/5.0 (Android 16; Mobile; rv:140.0) Gecko/140.0 Firefox/140.0'
arguments :
Parsed : 
full_url: /wikipedia_en_all_maxi_2024-01/I/Red_pog.svg.png.webp
url   : /wikipedia_en_all_maxi_2024-01/I/Red_pog.svg.png.webp
acceptEncodingGzip : 1
has_range : 0
is_valid_url : 1
.............
** running handle_content
Found I/Red_pog.svg.png.webp
mimeType: image/webp
Response :
httpResponseCode : 200
headers :
 - Accept-Ranges: 'bytes'
 - Cache-Control: 'max-age=2723040, public'
 - ETag: '"1751215347482162749/c"'
 - Access-Control-Allow-Origin: '*'
 - Content-Type: 'image/webp'
Request time : 0.000292s
----------------------
Found I/Marmara_gölü.jpg.webp
mimeType: image/webp
Response :
httpResponseCode : 200
headers :
 - Accept-Ranges: 'bytes'
 - Cache-Control: 'max-age=2723040, public'
 - ETag: '"1751215347482162749/c"'
 - Access-Control-Allow-Origin: '*'
 - Content-Type: 'image/webp'
Request time : 0.003031s
----------------------
======================
Requesting : 
full_url  : /w/load.php
method    : GET (0)
version   : HTTP/1.1
request#  : 64
headers   :
 - accept : '*/*'
 - accept-encoding : 'gzip, deflate'
 - accept-language : 'en-US'
 - connection : 'keep-alive'
 - dnt : '1'
 - host : '100.98.8.6:8036'
 - referer : 'http://100.98.8.6:8036/wikipedia_en_all_maxi_2024-01/A/Lake_Marmara'
 - sec-gpc : '1'
 - user-agent : 'Mozilla/5.0 (Android 16; Mobile; rv:140.0) Gecko/140.0 Firefox/140.0'
arguments :
 - debug : 1
 - lang : en
 - modules : ext.cite.ux-enhancements|jquery|mediawiki.base
 - skin : vector
 - version : ztntf
Parsed : 
full_url: /w/load.php
url   : /w/load.php
acceptEncodingGzip : 1
has_range : 0
is_valid_url : 1
.............
** running handle_content
Response :
httpResponseCode : 404
headers :
 - Vary: 'Accept-Encoding'
 - Content-Encoding: 'gzip'
 - Cache-Control: 'no-cache, no-store, must-revalidate'
 - Access-Control-Allow-Origin: '*'
 - Content-Type: 'text/html; charset=utf-8'
Request time : 0.000696s
----------------------
======================
Requesting : 
full_url  : /suggest
method    : GET (0)
version   : HTTP/1.1
request#  : 65
headers   :
 - accept : 'application/json, text/javascript, */*; q=0.01'
 - accept-encoding : 'gzip, deflate'
 - accept-language : 'en-US'
 - connection : 'keep-alive'
 - dnt : '1'
 - host : '100.98.8.6:8036'
 - referer : 'http://100.98.8.6:8036/wikipedia_en_all_maxi_2024-01/A/Lake_Marmara'
 - sec-gpc : '1'
 - user-agent : 'Mozilla/5.0 (Android 16; Mobile; rv:140.0) Gecko/140.0 Firefox/140.0'
 - x-requested-with : 'XMLHttpRequest'
arguments :
 - content : wikipedia_en_all_maxi_2024-01
 - term : t
 - userlang : en
Parsed : 
full_url: /suggest
url   : /suggest
acceptEncodingGzip : 1
has_range : 0
is_valid_url : 1
.............
** running handle_suggest
Searching suggestions for: "t"
======================
Requesting : 
full_url  : /suggest
method    : GET (0)
version   : HTTP/1.1
request#  : 66
headers   :
 - accept : 'application/json, text/javascript, */*; q=0.01'
 - accept-encoding : 'gzip, deflate'
 - accept-language : 'en-US'
 - connection : 'keep-alive'
 - dnt : '1'
 - host : '100.98.8.6:8036'
 - referer : 'http://100.98.8.6:8036/wikipedia_en_all_maxi_2024-01/A/Lake_Marmara'
 - sec-gpc : '1'
 - user-agent : 'Mozilla/5.0 (Android 16; Mobile; rv:140.0) Gecko/140.0 Firefox/140.0'
 - x-requested-with : 'XMLHttpRequest'
arguments :
 - content : wikipedia_en_all_maxi_2024-01
 - term : tes
 - userlang : en
Parsed : 
full_url: /suggest
url   : /suggest
acceptEncodingGzip : 1
has_range : 0
is_valid_url : 1
.............
** running handle_suggest
Searching suggestions for: "tes"
Segmentation fault

[kelson42]

@ASIXicle "Reproduction steps" means the exact list of operations you, step by step, do to reproduce the bug. I need that to try to reproduce it on my computer.

[ASIXicle]

Create a Debian (bookworm) LXC on Proxmox. Allocate 4 cores, 2-8GB RAM, 200GB of zpool.

In the new container:
sudo apt udpate
Install and setup Tailscale with allow-lan-access=true
sudo apt-get install aria2
sudo add-apt-repository ppa:kiwixteam/release
sudo apt-get install kiwix-tools
mkdir -p /opt/wiki/data/dumps
cd /opt/wiki/data/dumps
aria2c https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_maxi_2024-01.zim.torrent
kiwix-serve --port=8036 -b --verbose /opt/wiki/data/dumps/wikipedia_en_all_maxi_2024-01.zim

Pretty dead simple. And works fantastic until it doesn't.

Really appreciate your help @kelson42

[ASIXicle]

I suspect it's something to do with Tailscale because I can't get it to crash while using off of 192.168.1.0/24 addresses.

Switching to browsing the Kiwix Wiki on an Android or iPhone over Tailscale almost immediately causes a failure.

======================
Requesting : 
full_url  : /suggest
method    : GET (0)
version   : HTTP/1.1
request#  : 392
headers   :
 - accept : 'application/json, text/javascript, */*; q=0.01'
 - accept-encoding : 'gzip, deflate'
 - accept-language : 'en-US'
 - connection : 'keep-alive'
 - dnt : '1'
 - host : '100.98.8.6:8036'
 - referer : 'http://100.98.8.6:8036/wikipedia_en_all_maxi_2024-01/A/Acceptance_testing'
 - sec-gpc : '1'
 - user-agent : 'Mozilla/5.0 (Android 16; Mobile; rv:140.0) Gecko/140.0 Firefox/140.0'
 - x-requested-with : 'XMLHttpRequest'
arguments :
 - content : wikipedia_en_all_maxi_2024-01
 - term : aga
 - userlang : en
Parsed : 
full_url: /suggest
url   : /suggest
acceptEncodingGzip : 1
has_range : 0
is_valid_url : 1
.............
** running handle_suggest
Searching suggestions for: "aga"
double free or corruption (out)
Aborted

[kelson42]

@ASIXicle So you search (suggestion) anything in wikipedia_en_all_maxi_2024-01.zim and it crashes immediatly?

[ASIXicle]

@kelson42 It may process one search and then crash. Very odd. But yes, every time wikipedia_en_all_maxi_2024-01.zim will crash out after a search or two. Clicking through articles works just fine, very snappy.