I just tried autotune and it failed: 
I would like autotune to monitor both ollama (journalctl -u ollama -f) and the Linux kernel (dmesg -W) while it is doing tests, and to look for errors and warnings in those outputs if something fails, and preferably to automatically tune the necessary parameters (e.g. echo 5000 > /sys/class/drm/card1/engine/rcs0/preempt_timeout_ms ,
echo 5000 > /sys/class/drm/card1/engine/rcs0/heartbeat_interval_ms ) (tune those values to be as short as necessary to make sure that ollama does not crash, in the range between the default 640 to 10000, which would be 10 seconds)
It should detect common errors, warnings like these:
[23934.419003] Fence expiration time out i915-0000:00:02.0:llama-server[517649]:7ce4!
[23653.859854] i915 0000:00:02.0: [drm] Resetting rcs0 for stopped heartbeat on rcs0
[16346.089609] i915 0000:00:02.0: [drm] Resetting rcs0 for preemption time out
[16346.089687] i915 0000:00:02.0: [drm] llama-server[316143] context reset due to GPU hang
[16346.852191] i915 0000:00:02.0: [drm] GPU HANG: ecode 9:1:8ed1fff2, in llama-server [316143]
The
PARAMETER num_batch 256
should be tuned in the values from 128, 256, 512, 1024 and 2048
to see which ones are stable (no GPU hangs), and which one is the fastet.
E.g. >=256 seems to cause crashes on my system, so I am currently trying it with 128. But other hardware might be able to do more...
I just tried autotune and it failed:
I would like autotune to monitor both ollama (journalctl -u ollama -f) and the Linux kernel (dmesg -W) while it is doing tests, and to look for errors and warnings in those outputs if something fails, and preferably to automatically tune the necessary parameters (e.g. echo 5000 > /sys/class/drm/card1/engine/rcs0/preempt_timeout_ms ,
echo 5000 > /sys/class/drm/card1/engine/rcs0/heartbeat_interval_ms ) (tune those values to be as short as necessary to make sure that ollama does not crash, in the range between the default 640 to 10000, which would be 10 seconds)
It should detect common errors, warnings like these:
[23934.419003] Fence expiration time out i915-0000:00:02.0:llama-server[517649]:7ce4!
[23653.859854] i915 0000:00:02.0: [drm] Resetting rcs0 for stopped heartbeat on rcs0
[16346.089609] i915 0000:00:02.0: [drm] Resetting rcs0 for preemption time out
[16346.089687] i915 0000:00:02.0: [drm] llama-server[316143] context reset due to GPU hang
[16346.852191] i915 0000:00:02.0: [drm] GPU HANG: ecode 9:1:8ed1fff2, in llama-server [316143]
The
PARAMETER num_batch 256
should be tuned in the values from 128, 256, 512, 1024 and 2048
to see which ones are stable (no GPU hangs), and which one is the fastet.
E.g. >=256 seems to cause crashes on my system, so I am currently trying it with 128. But other hardware might be able to do more...