feat: Add MiniMax M3 Vision Projector Support by jkyamog · Pull Request #2086 · ikawrakow/ik_llama.cpp

jkyamog · 2026-07-05T17:26:52Z

Adds MiniMax M3 vision support to examples/mtmd.

This adds:

PROJECTOR_TYPE_MINIMAX_M3_VL
MiniMax M3 image token markers for mtmd
MiniMax M3 mmproj GGUF converter
MiniMax M3 vision graph:
- Qwen2.5VL-compatible preprocessing path
- two temporal patch embedding convs
- 3-axis NEOX vision RoPE
- MiniMax projector MLP
- 2x2 projected patch merge
- patch merge MLP
ERF GELU CPU execution and CUDA dispatch/support wiring

Notes

MiniMax M3’s HF implementation describes the vision tower preprocessing as Qwen2.5VL-compatible, so this reuses the existing Qwen2.5VL preprocessing path rather than adding a separate MiniMax-specific preprocessor.

The MiniMax projector has two stages:

multi_modal_projector.linear_1/2
patch_merge_mlp.linear_1/2

The patch merge needs to happen after the first projector stage, matching the HF/reference path. The converter writes both projector stages into the mmproj GGUF as mm.0, mm.2, mm.4, and mm.6.

ERF GELU is kept intentionally. Farmadupe’s parity test showed tanh GELU materially diverges from the reference embeddings, while ERF GELU keeps much closer parity.

The custom f32 conv path is also intentional. Testing showed the default conv path had noticeably worse embedding parity, while f32 conv closely matched the reference implementation.

Testing

Tested locally with the generated MiniMax M3 mmproj GGUF and MiniMax M3 Q6 model through the OpenAI-compatible server path.

Image smoke tests passed:

stop sign image: correctly identified STOP sign and readable text
cat image: correctly identified tabby cat and visual details
receipt image: correctly read merchant/header, date, line items, and total

Hermes native image attachment path was also tested after marking the local minimax-m3 provider as vision-capable; it routed images natively to the active MiniMax M3 model and produced correct results for all three images.

Artifact

The tested mmproj artifact is published here:

https://huggingface.co/jkyamog/MiniMax-M3-mmproj-GGUF

File:

mmproj-minimax-m3-f16-qwen-preproc.gguf

jkyamog · 2026-07-05T17:29:12Z

I had this for about a week, I still need to test this properly but I think its good to get this out for testing. Also I am unsure if this the right way go about it, suggestion are welcome. Another thing is where do we publish mmproj-minimax-m3-f16.gguf?

Farmadupe · 2026-07-06T19:49:48Z

I was hoping that someone would pick up the vision side of minimax3!

Some comments:

the hf transformers source for minimaxv3 is stating that preprocessing for the vision tower is a clone of qwen2.5vl.
I think that means in clip.cpp all preprocessing is already implemented, and function minimax_m3_preprocess_image is actually not needed, because we already have all of the preprocessing. Where you see case PROJECTOR_TYPE_QWEN25VL: in clip.cpp, you can add minimax3l too.

Farmadupe · 2026-07-07T01:52:20Z

I had some tokens to burn, so I think this may be a rough cut of the entire feature.

Mostly vibed. But the changes to src aren't that big. Certainly for the parts I understand.
New GGML ops: GGML_UNARY_OP_GELU_ERF
- The LLM mentioned that these ops were already defined in mainline, but I don't know if it lifted them. Lifting would be ideal.
New image rope function: build_rope_vision_neox
Embedding performance metrics relative to reference implementation are provided. The table shows worst token cosim and l2norm per image.
- I ordered the LLM to chase l2 norm divergence quite "low" (it could be better), so it may have gone a bit over the top in implementation. conv_2d_f32 may not actually be necessary.
I did not test with a LM decoder.
Embedding metrics calculated with f32 weights.
tested on CPU and cuda build.

Farmadupe@98a56e3

image	min_cosim	worst_l2
llama.png	0.9999999945	0.0001586593
max_672.png	0.9999998287	0.0006448357
min_56.png	0.9999999986	0.0000550312
small_168.png	0.9999999999	0.0000153180
square_224.png	0.9999999955	0.0002243123
square_280.png	0.9999999895	0.0002836175
tall_224x336.png	0.9999999803	0.0002451776
tall_84x140.png	0.9999999998	0.0000190169
test1.png	0.9999999935	0.0002028798
wide_140x84.png	0.9999999988	0.0000484583
wide_336x224.png	0.9999999855	0.0002174335
wide_448x252.png	0.9999999991	0.0000608255
wide_560x140.png	0.9999999987	0.0000501512

Farmadupe · 2026-07-07T02:23:38Z

I'm not sure that inventing the erf ggml ops was necessary? tanh is basically the same? I can run the benchmarks again with normal gelu and see if ti makes a meaningful difference to the numbers.
I'm not sure if the LLM wrote a erf gelu implementation for cuda. Either way, if tanh will do then we don't need one.

Farmadupe · 2026-07-07T02:34:07Z

apparantly there is an erf gelu in cuda that isn't wired up (presuambly there was no point to do so before the cpu impl existed).

Either way, if the encoder metrics come out good for tanh then it's all good.

Farmadupe · 2026-07-07T02:48:54Z

diff --git a/ggml/src/ggml-cuda.cu b/ggml/src/ggml-cuda.cu
index e006e661c..dc2c86b17 100644
--- a/ggml/src/ggml-cuda.cu
+++ b/ggml/src/ggml-cuda.cu
@@ -3772,6 +3772,9 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
                 case GGML_UNARY_OP_GELU:
                     ggml_cuda_op_gelu(ctx, dst);
                     break;
+                case GGML_UNARY_OP_GELU_ERF:
+                    ggml_cuda_op_gelu_erf(ctx, dst);
+                    break;
                 case GGML_UNARY_OP_SILU:
                     ggml_cuda_op_silu(ctx, dst);
                     break;
@@ -4692,6 +4695,7 @@ GGML_CALL static bool ggml_backend_cuda_supports_op(ggml_backend_t backend, cons
         case GGML_OP_UNARY:
             switch (ggml_get_unary_op(op)) {
                 case GGML_UNARY_OP_GELU:
+                case GGML_UNARY_OP_GELU_ERF:
                 case GGML_UNARY_OP_SILU:
                 case GGML_UNARY_OP_SWIGLU:
                 case GGML_UNARY_OP_SWIGLU_OAI:
diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c
index 53aef9869..66187ccfe 100644
--- a/ggml/src/ggml.c
+++ b/ggml/src/ggml.c
@@ -3316,11 +3316,16 @@ inline static void ggml_vec_hardsigmoid_f32 (const int n, float * y, const float
 static const float GELU_QUICK_COEF = -1.702f;
 static const float GELU_COEF_A     = 0.044715f;
 static const float SQRT_2_OVER_PI  = 0.79788456080286535587989211986876f;
+static const float SQRT_2_INV      = 0.70710678118654752440084436210484f;
 
 inline static float ggml_gelu_f32(float x) {
     return 0.5f*x*(1.0f + tanhf(SQRT_2_OVER_PI*x*(1.0f + GELU_COEF_A*x*x)));
 }
 
+inline static float ggml_gelu_erf_f32(float x) {
+    return 0.5f*x*(1.0f + erff(x*SQRT_2_INV));
+}
+
 inline static float ggml_gelu_quick_f32(float x) {
     return x*(1.0f/(1.0f+expf(GELU_QUICK_COEF*x)));
 }
@@ -3942,6 +3947,13 @@ inline static void ggml_vec_gelu_f32(const int n, float * y, const float * x) {
     }
 #endif
 }
+
+inline static void ggml_vec_gelu_erf_f32(const int n, float * y, const float * x) {
+    for (int i = 0; i < n; ++i) {
+        y[i] = ggml_gelu_erf_f32(x[i]);
+    }
+}
+
 inline static void ggml_vec_mul_gelu_f32(const int n, float * z, const float * x, const float * y) {
     int i = 0;
 #if defined(__AVX512F__) && defined(__AVX512DQ__)
@@ -4158,8 +4170,6 @@ inline static void ggml_vec_geglu_f16(const int n, ggml_fp16_t * y, const ggml_f
     }
 }
 
-static const float SQRT_2_INV = 0.70710678118654752440084436210484f;
-
 inline static void ggml_vec_geglu_erf_f32(const int n, float * y, const float * x, const float * g) {
     for (int i = 0; i < n; ++i) {
         float xi = x[i];
@@ -15578,6 +15588,65 @@ static void ggml_compute_forward_gelu(
     }
 }
 
+// ggml_compute_forward_gelu_erf
+
+static void ggml_compute_forward_gelu_erf_f32(
+        const struct ggml_compute_params * params,
+        struct ggml_tensor * dst) {
+
+    const struct ggml_tensor * src0 = dst->src[0];
+
+    assert(ggml_is_contiguous_1(src0));
+    assert(ggml_is_contiguous_1(dst));
+    assert(ggml_are_same_shape(src0, dst));
+
+    const int ith = params->ith;
+    const int nth = params->nth;
+
+    const int nc = src0->ne[0];
+    const int nr = ggml_nrows(src0);
+
+    // rows per thread
+    const int dr = (nr + nth - 1)/nth;
+
+    // row range for this thread
+    const int ir0 = dr*ith;
+    const int ir1 = MIN(ir0 + dr, nr);
+
+    for (int i1 = ir0; i1 < ir1; i1++) {
+        ggml_vec_gelu_erf_f32(nc,
+                (float *) ((char *) dst->data  + i1*( dst->nb[1])),
+                (float *) ((char *) src0->data + i1*(src0->nb[1])));
+
+#ifndef NDEBUG
+        for (int k = 0; k < nc; k++) {
+            const float x = ((float *) ((char *) dst->data + i1*( dst->nb[1])))[k];
+            UNUSED(x);
+            assert(!isnan(x));
+            assert(!isinf(x));
+        }
+#endif
+    }
+}
+
+static void ggml_compute_forward_gelu_erf(
+        const struct ggml_compute_params * params,
+        struct ggml_tensor * dst) {
+
+    const struct ggml_tensor * src0 = dst->src[0];
+
+    switch (src0->type) {
+        case GGML_TYPE_F32:
+            {
+                ggml_compute_forward_gelu_erf_f32(params, dst);
+            } break;
+        default:
+            {
+                GGML_ABORT("fatal error");
+            }
+    }
+}
+
 // ggml_compute_forward_fill
 
 static void ggml_compute_forward_fill_f32(const struct ggml_compute_params * params, struct ggml_tensor * dst) {
@@ -23143,6 +23212,10 @@ static void ggml_compute_forward_unary(
             {
                 ggml_compute_forward_gelu(params, dst);
             } break;
+        case GGML_UNARY_OP_GELU_ERF:
+            {
+                ggml_compute_forward_gelu_erf(params, dst);
+            } break;
         case GGML_UNARY_OP_GELU_QUICK:
             {
                 ggml_compute_forward_gelu_quick(params, dst);
@@ -26455,6 +26528,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
                 case GGML_UNARY_OP_SIGMOID:
                 case GGML_UNARY_OP_NEG:
                 case GGML_UNARY_OP_GELU:
+                case GGML_UNARY_OP_GELU_ERF:
                 case GGML_UNARY_OP_GELU_QUICK:
                 case GGML_UNARY_OP_SILU:
                 case GGML_UNARY_OP_EXP:

That should restore erf gelu intot he PR branch and enable it for cuda. It would need testing.

jkyamog · 2026-07-07T10:46:19Z

@Farmadupe thanks for the useful comments, we had modified it based on your comments. I will still do more testing.

Farmadupe · 2026-07-07T11:29:35Z

I tested tanh. I'm guessing from these numbers it's required to keep erf

image	erf min_cosim	tanh min_cosim	erf worst_l2	tanh worst_l2	L2 x-worse
llama.png	0.9999999945	0.9827235121	0.0001586593	0.2234916491	1408.6x
max_672.png	0.9999998287	0.9982558270	0.0006448357	0.0618903808	96.0x
min_56.png	0.9999999986	0.9999633498	0.0000550312	0.0125389619	227.9x
small_168.png	0.9999999999	0.9994450485	0.0000153180	0.0453326954	2959.4x
square_224.png	0.9999999955	0.9996551260	0.0002243123	0.0301189439	134.3x
square_280.png	0.9999999895	0.9993377919	0.0002836175	0.0409894723	144.5x
tall_224x336.png	0.9999999803	0.9989652935	0.0002451776	0.0624811351	254.8x
tall_84x140.png	0.9999999998	0.9991583224	0.0000190169	0.0477991916	2513.5x
test1.png	0.9999999935	0.9210481573	0.0002028798	0.3940294307	1942.2x
wide_140x84.png	0.9999999988	0.9996427263	0.0000484583	0.0302280373	623.8x
wide_336x224.png	0.9999999855	0.9991644281	0.0002174335	0.0429059985	197.3x
wide_448x252.png	0.9999999991	0.9856605169	0.0000608255	0.1802130823	2962.8x
wide_560x140.png	0.9999999987	0.9978900656	0.0000501512	0.0723634300	1442.9x

Farmadupe · 2026-07-09T19:46:07Z

SOme further explorations with datatypes:

MiniMax-M3 vision parity results

f16 · default conv

image	min_cosim	worst_l2
llama.png	0.9863607102	0.2082483417
max_672.png	0.9942489856	0.1350791210
min_56.png	0.9999975254	0.0023482821
small_168.png	0.9999964701	0.0043225417
square_224.png	0.9930651279	0.1184444560
square_280.png	0.9999995435	0.0012509715
tall_224x336.png	0.9996626615	0.0268324485
tall_84x140.png	0.9999998498	0.0006528658
test1.png	0.9887208432	0.1549920034
wide_140x84.png	0.9999983867	0.0019891680
wide_336x224.png	0.9957772507	0.1592469987
wide_448x252.png	0.9999826398	0.0070293378
wide_560x140.png	0.9998299816	0.0191456178

f16 · conv2d-f32

image	min_cosim	worst_l2
llama.png	0.9999944844	0.0033443478
max_672.png	0.9999997804	0.0006946645
min_56.png	0.9999999977	0.0000915014
small_168.png	0.9999999156	0.0004582453
square_224.png	0.9999998924	0.0006914206
square_280.png	0.9999992293	0.0015510403
tall_224x336.png	0.9999990533	0.0022347171
tall_84x140.png	0.9999999493	0.0004887953
test1.png	0.9999826981	0.0094499592
wide_140x84.png	0.9999996439	0.0009687577
wide_336x224.png	0.9999997413	0.0007659606
wide_448x252.png	0.9999978248	0.0021019796
wide_560x140.png	0.9999981259	0.0020206456

bf16 · default conv

image	min_cosim	worst_l2
llama.png	0.9853383334	0.2153727116
max_672.png	0.8826839396	0.6562203118
min_56.png	0.9976036090	0.1069909208
small_168.png	0.9999171312	0.0189777190
square_224.png	0.9706941785	0.4257451418
square_280.png	0.9894639645	0.2390405840
tall_224x336.png	0.9857115860	0.2757033073
tall_84x140.png	0.9992205541	0.0433252912
test1.png	0.9193792009	0.3961775201
wide_140x84.png	0.9970593855	0.0849274122
wide_336x224.png	0.9781033226	0.3200531934
wide_448x252.png	0.9992293314	0.0530375304
wide_560x140.png	0.9987250082	0.0580621089

bf16 · conv2d-f32

image	min_cosim	worst_l2
llama.png	0.9977563159	0.0873221880
max_672.png	0.9054384974	0.5409307229
min_56.png	0.9971114910	0.1171114881
small_168.png	0.9999114705	0.0223720026
square_224.png	0.9857464238	0.2817545767
square_280.png	0.9683709027	0.2892729677
tall_224x336.png	0.9866867423	0.3102123013
tall_84x140.png	0.9993401543	0.0396715329
test1.png	0.9193477344	0.3967876877
wide_140x84.png	0.9958698132	0.0992577182
wide_336x224.png	0.9815901657	0.3815293329
wide_448x252.png	0.9995370684	0.0640330260
wide_560x140.png	0.9977508380	0.0670470117

f32 · default conv

image	min_cosim	worst_l2
llama.png	0.9863397369	0.2084866074
max_672.png	0.9942479611	0.1350971379
min_56.png	0.9999975501	0.0023338248
small_168.png	0.9999957232	0.0041612184
square_224.png	0.9930215339	0.1188183509
square_280.png	0.9999995499	0.0009501287
tall_224x336.png	0.9996680531	0.0266149495
tall_84x140.png	0.9999999640	0.0003147060
test1.png	0.9905457761	0.1424212562
wide_140x84.png	0.9999976924	0.0025168472
wide_336x224.png	0.9958144917	0.1584711659
wide_448x252.png	0.9999798237	0.0083642450
wide_560x140.png	0.9998656745	0.0169829593

f32 · conv2d-f32

image	min_cosim	worst_l2
llama.png	0.9999999945	0.0001586593
max_672.png	0.9999998287	0.0006448357
min_56.png	0.9999999986	0.0000550312
small_168.png	0.9999999999	0.0000153180
square_224.png	0.9999999955	0.0002243123
square_280.png	0.9999999895	0.0002836175
tall_224x336.png	0.9999999803	0.0002451776
tall_84x140.png	0.9999999998	0.0000190169
test1.png	0.9999999935	0.0002028798
wide_140x84.png	0.9999999988	0.0000484583
wide_336x224.png	0.9999999855	0.0002174335
wide_448x252.png	0.9999999991	0.0000608255
wide_560x140.png	0.9999999987	0.0000501512

Summary — worst min_cosim

storage	default conv	conv2d-f32
f16	0.9863607102	0.9999826981
bf16	0.8826839396	0.9054384974
f32	0.9863397369	0.9999998287

Summary — worst worst_l2

storage	default conv	conv2d-f32
f16	0.2082483417	0.0094499592
bf16	0.6562203118	0.5409307229
f32	0.2084866074	0.0006448357


Upshot is that bf16 is actually _worse_, so f16 conversion is worthwhile, and the custom f32 implementation of whatever op that wsa custom implmeented, is probably surprisngly necessary.

jkyamog · 2026-07-11T15:51:59Z

@Farmadupe Thank you very much for guiding me (codex) to the right direction. Please kindly test with the latest code and the GGUF I have published on HF (I have updated the PR description).

Farmadupe · 2026-07-11T18:25:33Z

Will do!

FYI the mmproj doesn't have a license, which makes it forbidden for most people to download and use. The usual solution here is to use the same license as the orignal model was released under.

Farmadupe · 2026-07-11T18:32:45Z

It's already been published so it's no a disaster, but the appearence of `qwen` in the name might be a LLMism coming through. This is totally the vision tower of minimax m3 (the reuse of qwen's preprocessing code is an ipmlementation detail))

Farmadupe · 2026-07-11T19:18:47Z

Sorry for the spam, I just wanted to share that Codec's hallucination caused Claude to multiply the hallucination!

Farmadupe · 2026-07-11T19:26:37Z

mmproj	min_cosim	worst_l2
f16	0.9999826981	0.0094499592

I will also run a end to end test with the LM decoder to double check.

Farmadupe · 2026-07-11T20:30:40Z

It can seeeeee!

Farmadupe · 2026-07-11T21:49:31Z

The LLM thinks that the warnings could be solved by swapping some arguments around somewhere

jkyamog marked this pull request as draft July 5, 2026 17:27

jkyamog force-pushed the feature/minimax-m3-vision branch from 281bb42 to c38dd8d Compare July 6, 2026 08:50

jkyamog force-pushed the feature/minimax-m3-vision branch from c38dd8d to 81d6ec2 Compare July 7, 2026 02:15

jkyamog force-pushed the feature/minimax-m3-vision branch from 81d6ec2 to 2d5467d Compare July 7, 2026 02:30

jkyamog force-pushed the feature/minimax-m3-vision branch from 2d5467d to 3389b34 Compare July 7, 2026 12:12

jkyamog force-pushed the feature/minimax-m3-vision branch from 3389b34 to eb1a946 Compare July 10, 2026 06:58

mtmd: add MiniMax M3 vision support

9026d13

jkyamog force-pushed the feature/minimax-m3-vision branch from eb1a946 to 9026d13 Compare July 11, 2026 15:26

jkyamog marked this pull request as ready for review July 11, 2026 15:52

Conversation

jkyamog commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Testing

Artifact

Uh oh!

jkyamog commented Jul 5, 2026

Uh oh!

Farmadupe commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Farmadupe commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Farmadupe commented Jul 7, 2026

Uh oh!

Farmadupe commented Jul 7, 2026

Uh oh!

Farmadupe commented Jul 7, 2026

Uh oh!

jkyamog commented Jul 7, 2026

Uh oh!

Farmadupe commented Jul 7, 2026

Uh oh!

Farmadupe commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MiniMax-M3 vision parity results

f16 · default conv

f16 · conv2d-f32

bf16 · default conv

bf16 · conv2d-f32

f32 · default conv

f32 · conv2d-f32

Summary — worst min_cosim

Summary — worst worst_l2

Uh oh!

jkyamog commented Jul 11, 2026

Uh oh!

Farmadupe commented Jul 11, 2026

Uh oh!

Farmadupe commented Jul 11, 2026

Uh oh!

Farmadupe commented Jul 11, 2026

Uh oh!

Farmadupe commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Farmadupe commented Jul 11, 2026

Uh oh!

Farmadupe commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jkyamog commented Jul 5, 2026 •

edited

Loading

Farmadupe commented Jul 6, 2026 •

edited

Loading

Farmadupe commented Jul 7, 2026 •

edited

Loading

Farmadupe commented Jul 9, 2026 •

edited

Loading

Farmadupe commented Jul 11, 2026 •

edited

Loading