Upgrade stanza to 1.10.1 + add translation tests - #488
Conversation
- Add language code mapping for Chinese Traditional (zt→zh-hant) and Brazilian Portuguese (pb→pt) - Implement SpaCy fallback for unsupported Stanza languages - Fix segmentation faults by using tokenize_pretokenized=True to prevent MWT processing
…ntheses for multi-line return statement feat(tests): add comprehensive test suite for English translations including basic, directional, and non-English pairs to ensure translation accuracy and reliability
|
@argosopentech, please let me know if you want those tests or not; they will not run in actions unless we add at least one language installation. |
|
This all looks good but I'm not sure about upgrading Stanza. When we've tried to do that in the past it's caused issues. |
|
Well I was one of the people complaining that some of the languages stopped working last time; that's why I added the tests (and they may suck, but it's better than nothing). I'm not going to try and force a solution, but as a community we could really use some guidance, especially since Stanza 1.1.1 support is really maintainable moving forward. I was originally under the assumption we were getting rid of Stanza completely until you merged #460 , but now I'm not so sure anymore? I'm happy to conform this PR to the direction you see the project going; just let us know where. |
|
Figuring out what to do about Stanza/SBD is still a work in progress. For the near term the plan is to support both Stanza and Spacy like in the merged pull request. I'm open to upgrading the Stanza version if you've tested it but I remember in the past upgrading it has broken things. The more tests the better, I'll definitely merge the tests. |
|
For reference, this is the closed ticket from the last upgrade : #400 |
|
I ran this myself and it seems to work. Thanks! Stanza is still kind of a mess but hopefully this will help. |
3 participants
A few fixes here:
Tested on all -> English in our pipelines, throughput is pretty good.