Kiwix catalog shows `articleCount` almost 3X higher than the actual number of ZIM (e.g. Wikipedia) articles

[holta]

1. Any idea why https://library.kiwix.org/catalog/v2/entries?count=-1 has in recent times begun reporting incorrect articleCount values?

2. Here is one of several examples from the URL above, incorrectly showing 18,579,023 articles for the 7 Million article wikipedia_en_all from August 2025:

     <entry>
       <id>urn:uuid:1ee73243-001b-33e7-1fb7-466f1b4ffec4</id>
       <title>Wikipedia</title>
       <updated>2025-06-17T00:00:00Z</updated>
       <summary>offline version of Wikipedia in English</summary>
       <language>eng</language>
       <name>wikipedia_en_all</name>
       <flavour>mini</flavour>
       <category>wikipedia</category>
       <tags>wikipedia;_category:wikipedia;_pictures:no;_videos:no;_details:no;_ftindex:yes</tags>
       <articleCount>18579023</articleCount>
       <mediaCount>33612</mediaCount>
       <link rel="http://opds-spec.org/image/thumbnail"
             href="/catalog/v2/illustration/1ee73243-001b-33e7-1fb7-466f1b4ffec4/?size=48"
             type="image/png;width=48;height=48;scale=1"/>
       <link type="text/html" href="/content/wikipedia_en_all_mini_2025-06" />
       <author>
         <name>Wikipedia</name>
       </author>
       <publisher>
         <name>openZIM</name>
       </publisher>
       <dc:issued>2025-06-17T00:00:00Z</dc:issued>
       <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_mini_2025-06.zim.meta4" length="13823501312" />
     </entry>

   In contrast, browsing wikipedia_en_all_maxi_2025-08.zim shows a more meaningful (and more accurate) number:

   

3. What would be the best way(s) to figure this out?

4. Should this problem be reported to some other repo?

[kelson42]

@benoit74 Do we have created a regression in MWoffliner? Have we started to count redirects? I though first this is the same as openzim/libzim#1000... but it seems wrong in an other way!

Remark: we need to be careful, this is a 18 month old code base which run there.

[rgaudin]

In : zim2024 = Archive(Path("/data/download/zim/wikipedia/wikipedia_en_all_maxi_2024-01.zim"))

In : zim2024.all_entry_count
Out: 24908076

In : zim2024.article_count
Out: 6863660

In : zim2024.entry_count
Out: 24908076

In : zim2024.media_count
Out: 7203816

In : zim2024.counters
Out:
{'text/plain': 11,
 'text/css': 40,
 'application/javascript': 35,
 'image/png': 7,
 'text/html': 6863660,
 'image/webp': 6669133,
 'image/svg+xml; profile="https://www.mediawiki.org/wiki/Specs/SVG/1.0.0"': 534337,
 'undefined': 53533,
 'image/gif': 330,
 'application/pdf': 37,
 'image/svg+xml': 9}

####

zim2025 = Archive(Path("/data/download/zim/wikipedia/wikipedia_en_all_maxi_2025-08.zim"))

In : zim2025.all_entry_count
Out: 26659480

In : zim2025.article_count
Out: 18675825

In : zim2025.entry_count
Out: 26659463

In : zim2025.media_count
Out: 7983596

In : zim2025.counters
Out:
{'application/javascript': 4,
 'font/ttf': 1,
 'image/apng': 8,
 'image/gif': 53956,
 'image/jpeg': 2530,
 'image/png': 642,
 'image/svg+xml': 21,
 'image/svg+xml; profile="https://www.mediawiki.org/wiki/Specs/SVG/1.0.0"': 565131,
 'image/webp': 7361308,
 'text/css': 33,
 'text/html': 8284234,
 'text/html; charset=iso-8859-1': 1,
 'text/javascript': 3}

This is clearly a ZIM issue ; should be transfered to mwoffliner.

[kelson42]

This is clearly a ZIM issue ; should be transfered to mwoffliner.

@rgaudin Thank you for the very quick analysis. I will transfer the issue.

[benoit74]

99% chance this is either normal or a libzim issue. Quite sure this 18M is the number of articles including redirects.
The difference is explained by the move to libzim 9 in 2025.

I will confirm tomorrow

[rgaudin]

As mentioned in openzim/mwoffliner#2536, this number is directly the number of FRONT_ARTICLE entries for ZIM with a title index (both 2024 and 2025 have one).
This could be explained by:

• mwoffliner creating redirects as FRONT
• node-libzim doing so without it being requested.
• libzim doing it without it being requested (could be side effect of Remove old (v0) titleindex (title pointer list) openzim/libzim#800)

[Markus-Rost]

mwoffliner recently added support for section redirects, which are using html pages. This could explain the increased number.

[rgaudin]

Simple pylibzim test seems to confirm libzim is working OK when params are set properly and/or hints are sent as an empty map.

from zimscraperlib.zim import Creator, Archive


In : with Creator(Path("test_noredir.zim"), main_path="home").config_dev_metadata() as creator:
    creator.add_item_for("home", "Home", content="welcome home", is_front=True )
    creator.add_item_for("page", "Other page", content="Other page content", is_front=True)

In : zim = Archive(Path("test_noredir.zim"))

In : zim.article_count
Out: 2

In : with Creator(Path("test_redir1.zim"), main_path="home").config_dev_metadata() as creator:
     creator.add_item_for("home", "Home", content="welcome home", is_front=True )
     creator.add_item_for("page", "Other page", content="Other page content", is_front=True)
     creator.add_redirect("bonjour", "page", "bonjour redir", is_front=False)
     creator.add_redirect("hola", "page", "ola redir", is_front=True)
     creator.add_redirect("bismillah", "page", "bismillah redir")

In : zim2 = Archive(Path("test_redir1.zim"))

In : zim2.article_count
Out: 3

[kelson42]

Redirects to FRONT articles been FRONT themselves is good, but they should never be included in Counter metatada.

[rgaudin]

Redirects to FRONT articles been FRONT themselves is good, but they should never be included in Counter metatada.

• Doesn't seem like they are: 18,675,825 in articles_count while Counter says 8,284,234 text/html
• Counter metadata is not used in computation of articles_count as previously mentioned, it's number of entries in TitleIndex (FRONT articles then)

[kelson42]

Counter metadata is not used in computation of articles_count as previously mentioned, it's number of entries in TitleIndex (FRONT articles then)

At this stage this sounds to me buggy, this number has NO meaning IMHO and I wonder why this is here, like this. Worse this looks like a regression.