Skip to content

Add STANZA and SPACY sentence boundary detection modes - #490

Merged
argosopentech merged 2 commits into
argosopentech:masterfrom
yudelevi:sbd-mode-configuration
Aug 14, 2025
Merged

Add STANZA and SPACY sentence boundary detection modes#490
argosopentech merged 2 commits into
argosopentech:masterfrom
yudelevi:sbd-mode-configuration

Conversation

@yudelevi

Copy link
Copy Markdown
Contributor

Frankly, I can't see any speed improvement for using spacy only:

❯ pytest tests/test_english_translations.py::TestSystemStatus tests/test_english_translations.py::TestBidirectionalRoundTrip -k "not long_paragraph" -v --durations=0
============================================================================= slowest durations =============================================================================
29.57s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[paragraph-Once upon a time, there was a small village. The people were very friendly. They welcomed all visitors with open arms.]
23.41s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_bidirectional_round_trip
14.99s call     tests/test_english_translations.py::TestSystemStatus::test_languages_available
13.98s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[sentence-The quick brown fox jumps over the lazy dog.]
0.97s setup    tests/test_english_translations.py::TestSystemStatus::test_translation_packages_exist
0.00s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[simple-Hello world]
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[sentence-The quick brown fox jumps over the lazy dog.]
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[paragraph-Once upon a time, there was a small village. The people were very friendly. They welcomed all visitors with open arms.]
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_bidirectional_round_trip
0.00s setup    tests/test_english_translations.py::TestSystemStatus::test_languages_available
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[simple-Hello world]
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[paragraph-Once upon a time, there was a small village. The people were very friendly. They welcomed all visitors with open arms.]
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_bidirectional_round_trip
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[sentence-The quick brown fox jumps over the lazy dog.]
0.00s teardown tests/test_english_translations.py::TestSystemStatus::test_languages_available
0.00s call     tests/test_english_translations.py::TestSystemStatus::test_translation_packages_exist
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[simple-Hello world]
0.00s teardown tests/test_english_translations.py::TestSystemStatus::test_translation_packages_exist
========================================================== 6 passed, 1 deselected, 2 warnings in 87.25s (0:01:27) ===========================================================

> ARGOS_CHUNK_TYPE=SPACY_ONLY pytest tests/test_english_translations.py::TestSystemStatus tests/test_english_translations.py::TestBidirectionalRoundTrip -k "not long_paragraph" -v --durations=0 -v -s
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= slowest durations =============================================================================
122.91s call     tests/test_english_translations.py::TestSystemStatus::test_languages_available
38.83s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_bidirectional_round_trip
25.15s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[paragraph-Once upon a time, there was a small village. The people were very friendly. They welcomed all visitors with open arms.]
19.94s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[sentence-The quick brown fox jumps over the lazy dog.]
0.93s setup    tests/test_english_translations.py::TestSystemStatus::test_translation_packages_exist
0.01s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_bidirectional_round_trip
0.00s call     tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[simple-Hello world]
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[simple-Hello world]
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[paragraph-Once upon a time, there was a small village. The people were very friendly. They welcomed all visitors with open arms.]
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[sentence-The quick brown fox jumps over the lazy dog.]
0.00s teardown tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[simple-Hello world]
0.00s setup    tests/test_english_translations.py::TestSystemStatus::test_languages_available
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[paragraph-Once upon a time, there was a small village. The people were very friendly. They welcomed all visitors with open arms.]
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_round_trip_phrases[sentence-The quick brown fox jumps over the lazy dog.]
0.00s setup    tests/test_english_translations.py::TestBidirectionalRoundTrip::test_bidirectional_round_trip
0.00s teardown tests/test_english_translations.py::TestSystemStatus::test_languages_available
0.00s call     tests/test_english_translations.py::TestSystemStatus::test_translation_packages_exist
0.00s teardown tests/test_english_translations.py::TestSystemStatus::test_translation_packages_exist
========================================================== 6 passed, 1 deselected, 2 warnings in 212.18s (0:03:32) =========================================================

@yudelevi yudelevi changed the title Add STANZA_ONLY and SPACY_ONLY sentence boundary detection modes Add STANZA and SPACY sentence boundary detection modes Aug 13, 2025
@PJ-Finlay

Copy link
Copy Markdown
Collaborator

Frankly, I can't see any speed improvement for using spacy only:

Interesting. Is this with a GPU or a CPU?

@argosopentech
argosopentech merged commit d0cf455 into argosopentech:master Aug 14, 2025
1 check passed
@yudelevi

Copy link
Copy Markdown
Contributor Author

it's CPU

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants