Skip to content

MWT expansion might not be needed #509

Description

@pierotofy

I was doing some research on stanza's internals and noticed that currently argos-translate instantiates the multi-word-token "mwt" processing step if available:

def get_stanza_processors(lang_code: str, resources: dict) -> str:
    """Get appropriate processors for a language, including MWT if available."""
    try:
        return "tokenize,mwt" if resources[lang_code].get("mwt") else "tokenize"
    except (KeyError, TypeError):
        return "tokenize"

However, we only need the sentence boundary detection feature of stanza, which does not seem to require mwt ? https://stanfordnlp.github.io/stanza/tokenize.html#tokenization-and-sentence-segmentation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions