Fetch InfiniNews CLI

A command-line tool for selectively downloading ruggsea/infini-news-corpus from the Hugging Face Hub — it filters each shard before saving, so you only keep the rows you actually need.

Dataset url: https://huggingface.co/datasets/ruggsea/infini-news-corpus

The dataset was originally published in Lazzaroni, Lasser, & Solovev (2026), Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles, arXiv:2605.18337. It also entertains its API at https://infini-news.uni-graz.at/api-docs

Please cite the original ArXiV paper if you use this data.

Installation

git clone https://github.com/giomagg/Fetch-Infini-News
cd Fetch-Infini-News
pip install -r requirements.txt

Creating a virtual environment before installing requirements is suggested.

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Use

Fetch dataset schema

This can be used to see which columns exist in the dataset. See also dataset docs.

python infininews_cli.py schema

Fetch filtered subset

Let's say we want to get Italian and French news from 2020-2022 mentioning "climate" in the title, and keep only a handful of columns. We would run:

python infininews_cli.py fetch \
    --years 2020-2022 \
    --languages ita fra \
    --title-keywords climate "global warming" \
    --columns url publish_date title sitename lang text \
    --output-dir ./data

This walks every year=/month= shard on HF Hub, downloads it to a temp folder, keeps only rows and columns matching your query, writes one parquet file per shard under ./data/<year>/, and deletes the temp download.

Already-saved shards are skipped on re-runs (use --overwrite to redo them).

Query Flags

1. Language, dates, and sources

Flag	Functioning
--years	e.g. 2020-2022 2024 (ranges and single years, space separated)
--months	e.g. 1 2 3 (default: all 12)
--languages	keep rows whose lang/language column matches any of these codes
--language-columns	which columns to check (default: lang language)
--sources	keep rows whose url_hostname matches any of these
--source-match	type of match: substring (default) or exact match

2. Text-based queries

Flag	Functioning
--title-keywords	keep rows whose title contains any/all of these phrases
--text-keywords	keep rows whose article text contains any/all of these phrases
--keyword-logic	any (default) or all, applied within each keyword list; title and text lists are always OR'd together
--whole-word	match keywords as whole words, not substrings
--case-sensitive	default is case-insensitive
--min-text-length / --max-text-length	filter on article length
--min-publish-date / --max-publish-date	YYYY-MM-DD

3. Saving files

Flag	Functioning
--columns	which columns to keep in the saved files (default: all known columns present in the shard)
--output-dir	where to write parquet files. Falls back to $INFININEWS_OUTPUT_DIR if set, then ./infininews_data
--file-prefix	prefix added to every saved filename
--overwrite	redo shards that already have an output file

4. Run, authentication, and rate-limits specs

Flag	Functioning
--dry-run	list shards / would-be output paths without downloading anything. Good for testing query.
--sleep-between-months	seconds to pause between months (rate-limit courtesy)
--max-retries	retries per shard on rate-limit/network errors before skipping it (default: 5)
--retry-backoff-seconds	wait before the first retry, doubling each attempt (default: 30)
--token / HF_TOKEN env var	Hugging Face auth; omit for a prompt, or use --no-auth

Setting output directory

--output-dir is a normal flag/config field. Nevertheless you can also set a machine-wide default without editing the script or retyping the flag, by exporting an environment variable before running:

export INFININEWS_OUTPUT_DIR="/Users/you/path/to/Data"
python infininews_cli.py fetch --years 2024

Precedence note: highest wins: --output-dir flag > output_dir in --config file > $INFININEWS_OUTPUT_DIR > ./infininews_data.

Query using config file

Put any of the flags above (using underscores, e.g. title_keywords: [climate]) into a YAML or JSON file — see example_query.yaml — and run:

python infininews_cli.py fetch --config example_query.yaml

Note that any CLI flag passed alongside --config overrides the corresponding value in the file, so you can keep a base config and tweak individual runs, e.g.:

python infininews_cli.py fetch --config example_query.yaml --years 2023 --overwrite

Dry runs

Before committing to a big download, try listing every shard and its would-be output path without downloading or filtering

python infininews_cli.py fetch --years 2020-2026 --dry-run

Hugging Face rate limits

Hugging Face rate-limits file downloads (HTTP 429 Too Many Requests), especially for unauthenticated requests or long runs that hammer the Hub with many shards back-to-back.

If you hit this (unlikely):

1. Authenticate. Anonymous downloads share stricter, IP-based limit. Pass --token, set the HF_TOKEN environment variable, or just answer the token prompt — a free Hugging Face account token (https://huggingface.co/settings/tokens, "read" scope is enough).

2. Let the built-in retry logic handle transient 429s. Every shard download is wrapped in exponential backoff: on a rate-limit or network error it waits --retry-backoff-seconds (default 300s), doubling each attempt, up to --max-retries times (default 2) before giving up on that shard and moving on.

3. Re-run to resume. Shards that were already saved are skipped on the next run (unless --overwrite), and shards that were skipped after exhausting retries will simply be picked up again — so if a long job gets rate-limited partway through, just re-run the same command later (ideally with a token and/or a longer --sleep-between-months) and it will pick up where it left off rather than starting over.

If you're getting rate-limited repeatedly, raise the ceiling instead of racing it:

python infininews_cli.py fetch --years 2020-2022 \
    --max-retries 10 --retry-backoff-seconds 400

Notes

• Filtering happens in-memory per shard using pandas, after reading only the columns needed for your filters/output (so shards are never fully materialized with unused columns).

• Already-saved shards are skipped on re-runs (use --overwrite to redo them).

• --languages matches against the lang/language columns; the original script used ISO 639-3-ish codes like ita for Italian — check schema output if you're unsure what values are present in a given shard or refer to the dataset card on hugging face.

• Set HF_HUB_ENABLE_HF_TRANSFER=1 (already set at the top of the script) and install hf_transfer for faster downloads.