How it's important the LLM used? Just two examples...

[rafpigna]

I have done a test on REFLECT model, trying to recall an information from Hindisght memory

"When the backup script for homelab will be executed"?

I have this information in different places in the memory, explained in different ways.

Using your suggested gpt-oss-20b (and also the 120b version). The reply took about 7s, about 16.000 tokens in input, and it was just a couple of sentences like

"The script is planned to execute at 9AM each day, called via cron. The script is in /opt/backup.sh " and just a couple of other informations.

About 380-420 tokens max output on each test I did, also with other questions.

Then tried DeepSeek V4 Flash (with low reasoning): the reply took 13s.
It provided never less then 1200 tokens reply with a lot of informations, paths, mount points for the backup drive, paths that will be backupped and some other stuff.

Summarizing: the biggest model provide much more informations on the same query and with the same tokens in input!

Now my question and basically my fear is that the quality of "how good memory works" is highly dependant from the LLM model used.
It's true that both models answered the main question, but the larger one provided also other informations that can be useful if in 6 months I will ask my coding agent to work on that script, or just to explain me what it does.

The ridicoulus thing is that nowdays, if you dont run gpt-oss-20b locally, the cloud api pricing is cheaper for Deepseek v4 flash than gpt-oss-20b !

So please, plan to test more updated models because gpt-oss-20b is basically a 2024 model and two years in the AI age is million years in model's capabilities.

Or maybe I'm worrying too much, and please explain me how all this can works fine with a model that provide just a couple of sentences versus a model that provide 5x times longer and detailed reply.

[nicoloboschi]

yes we suggest to use latest model. ds4 is a very nice model to use with Hindsight. we're planning to update the docs and benchmarks page with newer models soon