Optimizations

[yudelevi]

Inspired by https://opennmt.net/CTranslate2/performance.html
Add LRU cache for language load (get_installed_languages is slow), quantization support, and configurable batch size.
Also, beam_size is set to be a minimum of 4. I am not sure if that's what we intended.

Quantizations provided a nice little boost:

$ ARGOS_COMPUTE_TYPE="float32" python test_performance_es_to_en.py
Starting performance test: 10 iterations from es to en
Testing 12 phrases, total: 4849 characters
============================================================
Running warmup translation (not counted in timing)...
2025-08-19 04:00:34 WARNING: Unsupported language: az  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 04:00:36 WARNING: Unsupported language: eo  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 04:00:37 WARNING: Unsupported language: ms  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 04:00:38 WARNING: Unsupported language: tl  If trying to add a new language, consider using allow_unknown_language=True
Warmup complete. Sample result: Software development is a complex discipline that requires both technical skills and creativity. Pro...
------------------------------------------------------------
Iteration  1: 1.7180s (12 phrases)
Iteration  2: 1.3710s (12 phrases)
Iteration  3: 1.3931s (12 phrases)
Iteration  4: 1.2430s (12 phrases)
Iteration  5: 1.2463s (12 phrases)
Iteration  6: 1.2482s (12 phrases)
Iteration  7: 1.2442s (12 phrases)
Iteration  8: 1.2464s (12 phrases)
Iteration  9: 1.2449s (12 phrases)
Iteration 10: 1.2437s (12 phrases)
============================================================
PERFORMANCE RESULTS:
Average time:  1.3199s
Median time:   1.2464s
Min time:      1.2430s
Max time:      1.7180s
Std deviation: 0.1511s
Total time:    13.1988s
Per phrase:    0.1100s avg
============================================================
CONSISTENCY CHECK:
Identical translations: 12/12
✅ All translations are consistent!

$ ARGOS_COMPUTE_TYPE="int8" python test_performance_es_to_en.py
Starting performance test: 10 iterations from es to en
Testing 12 phrases, total: 4849 characters
============================================================
Running warmup translation (not counted in timing)...
2025-08-19 03:59:49 WARNING: Unsupported language: az  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 03:59:50 WARNING: Unsupported language: eo  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 03:59:52 WARNING: Unsupported language: ms  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 03:59:53 WARNING: Unsupported language: tl  If trying to add a new language, consider using allow_unknown_language=True
Warmup complete. Sample result: Software development is a complex discipline that requires both technical skills and creativity. Pro...
------------------------------------------------------------
Iteration  1: 0.8528s (12 phrases)
Iteration  2: 0.9388s (12 phrases)
Iteration  3: 0.9670s (12 phrases)
Iteration  4: 0.9757s (12 phrases)
Iteration  5: 0.9645s (12 phrases)
Iteration  6: 0.9738s (12 phrases)
Iteration  7: 0.9926s (12 phrases)
Iteration  8: 0.9770s (12 phrases)
Iteration  9: 0.9725s (12 phrases)
Iteration 10: 0.9594s (12 phrases)
============================================================
PERFORMANCE RESULTS:
Average time:  0.9574s
Median time:   0.9698s
Min time:      0.8528s
Max time:      0.9926s
Std deviation: 0.0393s
Total time:    9.5741s
Per phrase:    0.0798s avg
============================================================
CONSISTENCY CHECK:
Identical translations: 12/12
✅ All translations are consistent!

$ ARGOS_COMPUTE_TYPE="int8_float32" python test_performance_es_to_en.py
Starting performance test: 10 iterations from es to en
Testing 12 phrases, total: 4849 characters
============================================================
Running warmup translation (not counted in timing)...
2025-08-19 04:00:11 WARNING: Unsupported language: az  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 04:00:13 WARNING: Unsupported language: eo  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 04:00:14 WARNING: Unsupported language: ms  If trying to add a new language, consider using allow_unknown_language=True
2025-08-19 04:00:15 WARNING: Unsupported language: tl  If trying to add a new language, consider using allow_unknown_language=True
Warmup complete. Sample result: Software development is a complex discipline that requires both technical skills and creativity. Pro...
------------------------------------------------------------
Iteration  1: 0.8521s (12 phrases)
Iteration  2: 0.9387s (12 phrases)
Iteration  3: 0.9632s (12 phrases)
Iteration  4: 0.9646s (12 phrases)
Iteration  5: 0.9625s (12 phrases)
Iteration  6: 0.9556s (12 phrases)
Iteration  7: 0.9594s (12 phrases)
Iteration  8: 0.9666s (12 phrases)
Iteration  9: 0.9558s (12 phrases)
Iteration 10: 0.9581s (12 phrases)
============================================================
PERFORMANCE RESULTS:
Average time:  0.9477s
Median time:   0.9588s
Min time:      0.8521s
Max time:      0.9666s
Std deviation: 0.0345s
Total time:    9.4765s
Per phrase:    0.0790s avg
============================================================
CONSISTENCY CHECK:
Identical translations: 12/12
✅ All translations are consistent!

Increasing batch size didn't make any change in CPU (at least not in my testing); it might be different in GPU.

[yudelevi]

Reverted beam_size to >=4, results for some languages were horrible.