feat: MiniSBD support

[pierotofy]

• Adds support for MiniSBD as a sentence boundary detector
• Makes spacy optional
• Make stanza pipeline initialization lazy (initialize on use)
• Allows for a minisbd package folder to contain a .onnx file to use as the MiniSBD tokenizer -or- it downloads the model from GitHub (multi-process/multi-thread safe).

Simple test:

test_trans.py

import argostranslate.translate


text = """Le chat domestique (Felis catus ou Felis silvestris catus) est la forme domestique du chat sauvage Felis silvestris, une espèce de mammifères carnivores, de la famille des Félidés.

Selon les résultats de travaux menés en 2006 et 2007[1], le chat domestique est une sous-espèce du chat sauvage issue d'ancêtres appartenant à la sous-espèce du chat sauvage d'Afrique (Felis silvestris lybica). Les premières domestications ont probablement lieu au Néolithique, entre 6000 et 8000 av. J.-C., dans le Croissant fertile, époque correspondant au début de la culture de céréales et à l'engrangement de réserves susceptibles d'être attaquées par des rongeurs, le chat devenant alors pour l'Homme un auxiliaire utile se prêtant à la domestication.
"""
translatedText = argostranslate.translate.translate(text, "fr", "en")
print(translatedText)

time ARGOS_DEBUG=1 ARGOS_CHUNK_TYPE=MINISBD python3 test_trans.py
real	0m4.647s
user	0m6.631s
sys	0m1.186s

time ARGOS_DEBUG=1 python3 test_trans.py
real	0m4.917s
user	0m6.907s
sys	0m1.244s

Notes

I've removed the fallback behavior of stanza to spacy; it's a strange way to handle missing languages support. I think if a language is missing from stanza (which ones?) we should just initialize the SpacySentencizerSmall directly. I tested all languages in the argos-pm index and only Arabic (ar) is currently giving some issues.

Another problem with the Spacy logic was that the spacy cache was initialized module wise, which was not process/thread safe and causing LibreTranslate/LibreTranslate#882. I've now set the spacy cache to initialize only if CHUNK_TYPE is SPACY (and spacy is installed). This is still a problem if you use Spacy in multi-process environment, but it's now mitigated by having the ability to disable Spacy.

• I will test a bit more in the upcoming days, but thought to open an early PR to encourage early feedback
• Azerbaijani az does not seem to work with Stanza (but works with MiniSBD), maybe it's just my environment

[pierotofy]

Ok it was just my environment. Re-downloading the az model solved the issue.

[pierotofy]

Ok this is ready for review. 🚀

The one thing that has changed compared to the previous version is the fallback behavior of stanza --> spacy, but otherwise the program should work the same unless a person explicitly enables MiniSBD support.

Once merged I can bring this into LibreTranslate.

[ArtanisTheOne]

Since spacy is now optional does it make sense to remove it from requirements.txt as well?

[argosopentech]

This looks good thanks!

[argosopentech]

This has been released on PyPI in Argos Translate 1.11.0!