Skip to content

Faster Bicleaner AI #1387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ZJaume merged 3 commits into from
Mar 13, 2026
Merged

Faster Bicleaner AI #1387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1c281da
Update bicleaner-ai
ZJaume Mar 2, 2026
5c33ee6
Use fp16 and bigger batch in Bicleaner
ZJaume Mar 2, 2026
9f5bff0
CUDA is now installed along with tensorflow with pip
ZJaume Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions pipeline/bicleaner/bicleaner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ python3 -c "from pipeline.common.marian import assert_gpus_available; assert_gpu

test -v SRC
test -v TRG
test -v CUDA_DIR

corpus_prefix=$1
output_prefix=$2
Expand Down Expand Up @@ -75,18 +74,22 @@ else
# to operate on the CPU very slowly. To guard against this wasting expensive
# GPU time, always check that it can find GPUs.
python3 -c "import tensorflow; exit(0) if tensorflow.config.list_physical_devices('GPU') else exit(75)"
bicleaner-ai-classify --disable_hardrules --require_gpu --scol ${scol} --tcol ${tcol} - - $1
bicleaner-ai-classify --disable_hardrules --require_gpu \
--fp16 --batch_size 512 --block_size 50000 \
--scol ${scol} --tcol ${tcol} - - $1
}
export -f biclean
# {%} is a 1-indexed job slot number from GNU parallel. We use that as the 1-indexed offset in CUDA_VISIBLE_ARRAY
paste <(zstdmt -dc "${corpus_prefix}.${SRC}.zst") <(zstdmt -dc "${corpus_prefix}.${TRG}.zst") |
parallel -j ${#CUDA_VISIBLE_ARRAY[@]} --pipe -k --block 10M biclean "${pack_dir}"/*.yaml {%} |
parallel -j ${#CUDA_VISIBLE_ARRAY[@]} --pipe -k --block 20M biclean "${pack_dir}"/*.yaml {%} |
zstdmt >"${output_prefix}.scored.zst"
else
export BICLEANER_AI_THREADS=${threads}
paste <(zstdmt -dc "${corpus_prefix}.${SRC}.zst") <(zstdmt -dc "${corpus_prefix}.${TRG}.zst") |
bicleaner-ai-classify --disable_hardrules --scol ${scol} --tcol ${tcol} "${threads}" - - "${pack_dir}"/*.yaml |
zstdmt >"${output_prefix}.scored.zst"
bicleaner-ai-classify --disable_hardrules \
--fp16 --batch_size 512 --block_size 50000 \
--scol ${scol} --tcol ${tcol} - - "${pack_dir}"/*.yaml \
| zstdmt >"${output_prefix}.scored.zst"
fi

echo "### Filtering"
Expand Down
5 changes: 3 additions & 2 deletions pipeline/bicleaner/requirements/bicleaner-ai.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
bicleaner-ai==3.4.0
PyICU==2.15.2
bicleaner-ai[and-cuda,transliterate]==3.6.0
tensorflow==2.20.*
PyICU==2.15.2
Loading