I was doing some research on stanza's internals and noticed that currently argos-translate instantiates the multi-word-token "mwt" processing step if available:
def get_stanza_processors(lang_code: str, resources: dict) -> str:
"""Get appropriate processors for a language, including MWT if available."""
try:
return "tokenize,mwt" if resources[lang_code].get("mwt") else "tokenize"
except (KeyError, TypeError):
return "tokenize"
I was doing some research on stanza's internals and noticed that currently argos-translate instantiates the multi-word-token "mwt" processing step if available:
However, we only need the sentence boundary detection feature of stanza, which does not seem to require mwt ? https://stanfordnlp.github.io/stanza/tokenize.html#tokenization-and-sentence-segmentation