Detect and auto switch languages

[mbalc]

Related to #147

Auto models have the following cons when it comes to real-life multi-language usage:

• only basic models available
• models too general - often ( = "in my case" :v ), you only need like just 2 or 3 specific languages:
  • you can't get Intermediate Results, even if the specific languages you use do have support for that,
  • at least theoretically - a model specialized in a single language should have better precision than one that's generalized over a set of languages you never really use
    • I think it is reasonable enough to make an assumption that a single listening session will in most cases not mix languages, and if it does, it should be easy to restart listening when you switch
      • this could be less intuitive for some, but I think (at least in my case) - this would better fit my workflow overall
• auto models might be slower, at least that's what's been suggested in comments in STT language auto-detection #147 (comment)

Feature draft:

• say, you have a single "favorite" model selected per each language that's enabled in the app
• you have an extra "language recognition" enumeration model

1. in an "auto" language mode, every time a STT listening session starts, when you start speaking, the language recognition model runs first while buffering input for the STT model
2. after the language is recognized with enough certainty, the choice of the model is made based on language recognized and the "favorite" model that's chosen for that language
3. stuff buffered so far is passed to the STT model

• here we can probably assume that just the last 3-5 seconds maximum will be needed if we can assume the lang recognition model is fast and accurate enough
  • (maybe even the lang recognition model could point out the exact moment the speech actually starts?)

4. the rest of the current listening session proceeds as normal

[mbalc]

bonus - a very similar idea could be applied for auto-read - as right now there is no 'auto' TTS model available in the app

[mbalc]

after messing around a bit with these "auto" models... oh, these actually are the enumerator models? it's not intuitive to use at all, i think there could be better communication by the app around this feature

[mkiol]

Hi. Thanks for the idea.

Currently, "Auto" models only work for STT and only for Whisper. In fact, this works much as you describe in "Feature Draft", but is limited to Whisper, as only Whisper has the ability to detect language. I think it is doable to extend this "automatic language detection" to other model types - in this scenario, the smallest/fastest Whisper model would only be used for language detection and the main processing would be done with the selected model. Theoretically, this can be done.

As for auto lang detection for TTS models, this is not currently implemented. I have not found a suitable library that can determine the language from the text. I probably gave up too quickly. This type of detection would be very useful for both TTS and Translator.

I am adding "automatic language detection" for TTS and Translator to my TO-DO list for future version.

[mbalc]

okay, one thing is to extend it to work for more kinds of models, that's more work

for me personally though, the biggest problem with auto mode is that it often mismatches the language for shorter sentences - if it was possible to limit the set of output languages (ie. filter out any languages from classifier model output that were not whitelisted by the user, pick top guess) would improve overall usability

[mkiol]

for me personally though, the biggest problem with auto mode is that it often mismatches the language for shorter sentences

Make sure that Use Tiny model is disabled. This optimization makes lang detection quicker but may reduce accuracy.

[mbalc]

i do have that disabled -

it's just that some languages are too similar to each other to realistically distinguish, especially on short inputs

when, let's say for example, you just enter a single phrase (in Polish) and the model recognizes Russian - they sound similar for that phrase but they use completely different alphabets and you don't really even know or ever use Russian

I'll provide more feedback once I use it for a bit longer, as so far I was mostly trying to make it fit my general workflow (and i didn't spend much time on that, only now i am satisfie more or less with the setup)