Skip to content

[OpenCL:Bugfix] Use vload_half/vstore_half builtins in buffer_convert_buf - #4805

Merged
Qxinyu merged 1 commit into
masterfrom
feature/opencl-half-buffer-convert
Aug 25, 2026
Merged

[OpenCL:Bugfix] Use vload_half/vstore_half builtins in buffer_convert_buf#4805
Qxinyu merged 1 commit into
masterfrom
feature/opencl-half-buffer-convert

Conversation

@Qxinyu

@Qxinyu Qxinyu commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Fix issue Qwen2.5-Omni-3B/7B 在最新代码上opencl上推理崩溃 #4802: Qwen2.5-Omni crashes on OpenCL at buffer_convert_buf kernel build on devices whose driver advertises cl_khr_fp16 but rejects direct scalar access to __global half* ("loading directly from pointer to type 'const __global half' requires cl_khr_fp16. Use vector data load builtin functions instead").
  • In buffer_convert_buf.cl, route all scalar/vector half element access through vload_half / vstore_half / vload_half4 / vstore_half4 builtins via new MNN_LOAD_INPUT / MNN_STORE_OUTPUT / MNN_LOAD_INPUT4 / MNN_STORE_OUTPUT4 / MNN_LOAD_FLOAT macros. Covers buffer_convert_to_buffer, buffer_copy_to_buffer (incl. PACK_LEAVE tail loop) and the conv/dw filter conversion kernels (20 direct FLOAT* dereferences). Non-half types keep the original direct/vload4 paths unchanged.
  • C++ side defines -DINPUT_IS_HALF / -DOUTPUT_IS_HALF wherever half types are selected (AddBuildOptionOfDataType in BufferConvertor, tensor-derived types in OpenCLRuntime::buildKernelWithCache, which also covers Softmax/ArgMax callers of these kernels).
  • Null-check every buildKernelWithCache result in BufferConvertor so a kernel build failure degrades to a logged error instead of a segmentation fault (the crash path in the issue).
  • Bump the buffer_convert_buf md5 in opencl_source_map.hpp to invalidate stale compiled-kernel caches.

Test plan

  • build_ocl full op suite on OpenCL backend: 248/248 passed (fp16 path active on Apple GPU)
  • softmax / argmax / convolution OpenCL op tests passed (exercise buffer_convert_to_buffer with half NC4HW4 unpack/pack)
  • Needs verification on the reporter's device (星睿 O6) with Qwen2.5-Omni-3B

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Qxinyu
Qxinyu merged commit c425199 into master Aug 25, 2026
18 of 20 checks passed
123xuwu pushed a commit to 123xuwu/MNN that referenced this pull request Aug 25, 2026
* [LLM:Feature] Add Qwen3-ASR and Qwen3-VL-Embedding export and runtime support. (alibaba#4478)

Co-authored-by: UNIkeEN <sunyujie@sjtu.edu.cn>

* [QNN:Bugfix] Add support for convert llm/visual mnn model into same qnn folder (Qnn model) (alibaba#4757)

* [Express:Bugfix] Fix loadMap/load output readMap returning NULL (alibaba#4751)

Fixes alibaba#4750

* [LLM:Feature] Support Qwen3-TTS-12Hz-0.6B-Base (alibaba#4589)

* [CPU:Refact] Refactor the KleidiAI integration code (alibaba#4602)

* [LLM:Bugfix] Align Qwen-VL runtime preprocessing (alibaba#4595)

* [LLM:Feature] DFlash draft shares target's lm_head (no separate lm_head)

The DFlash draft's lm_head is identical to the target's (tie_word_embeddings),
but was baked into dflash.mnn, duplicating the vocab x hidden weights on disk
and in memory (~219 MB for Qwen3-4B int4).

Export: DFlash.forward now outputs post-final-norm hidden_states instead of
logits, so dflash.mnn contains no lm_head. The separate dflash_lmhead.mnn
export path (--dflash_export_separate_lmhead) is removed: the shared
subgraph already covers its use case -- mixed precision (fp16 transformer +
int4 lm_head) -- via the target's own lm_head.

Engine: DFlash load() requires dflash_shared_lmhead_input and loads the
subgraph {input -> logits} from llm.mnn with Module::Config{base=target,
rearrange=true}. The embedded-lmhead and separate-lmhead fallbacks are gone:
a draft without the config field is refused at load time with a re-export
hint. Refusal is enforced through Generation::load(), which now returns
bool; on failure of any module the draft step dereferences (draft graph,
lm_head, fc) Llm::load() aborts and status stays NOT_LOADED, so every
inference entry is fenced by CHECK_LLM_RUNNING instead of reaching a null
module.

Verify (M5 iPad, 2B int4 b8 greedy): acceptance 4.59 -> 4.63, indistinguishable
from the separate-lm_head baseline (difference < 0.04 across 9 prompts).

Discussed-in: Merge-Request 28763320 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/28763320
GitOrigin-RevId: 318842e009b5067f67b1b5aa53b35c44df23536c

* [Test:Bugfix] Relax FP16 loadMap output tolerance

Squashed from 2 internally reviewed commits.

- 64f5b15 [JNI:Bugfix] Fix string array handling (alibaba#4753, alibaba#4754) — wangzhaode <zhaode.wzd@alibaba-inc.com>
- b9cedef [Test:Bugfix] Relax FP16 loadMap output tolerance — wangzhaode <zhaode.wzd@alibaba-inc.com>

Co-authored-by: wangzhaode <zhaode.wzd@alibaba-inc.com>
GitOrigin-RevId: e71e6b3c07041b3e6380974e529cbafff613b790

* [CPU:Feature] Detect SME2 core count

Discussed-in: Merge-Request 29219430 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29219430
GitOrigin-RevId: 8de11463229a9e0fc9004b7122ec59ac73b62b52

* [Hexagon:Perf] Add int4 vrmpy decode GEMV (M=1),  Opitimize FlashAttention and Gemm

GitOrigin-RevId: 49f3d6e2864ca11bcfee7f61d84ced9fab022915

* [LLM:Feature] Add Qwen3.5 video input (alibaba#4577)

* [NPU:rknn] Add initial rknn support for Rockchip NPUs, supporting traditional models. (alibaba#4524)

* [Infra:Bugfix] Replace remaining vector-to-pointer casts for Mali compat

GitOrigin-RevId: f5c45ecb5631ccdbf333cfdd925b73dd238f327e

* [LLM:Perf] DFlash engine: on-device fc/concat, incremental draft KV, on-device argmax

Link: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29237251
* [Metal:Feature] Speculative-block aware LinearAttention and Q4 GEMM
GitOrigin-RevId: 5317f5b246de830c817acaa487973cb1105c4265

* [Metal:Perf] Fused-Q4 M8 tile and K-split for speculative-block decode

The fused Q4 GEMM stage kernel uses an M32 matmul2d tile. A speculative
block is 8 rows, so it runs at quarter occupancy, and for projections
with few output tiles it also dispatches too few threadgroups to
saturate bandwidth.

Add two variants behind FUSED_Q4_REAL_UNPACK:
  - conv1x1_fused_q4_gemm_stage_m8: descriptor sized to M=8.
  - conv1x1_fused_q4_gemm_stage_ksplit{,_m8} + _ksplit_reduce: split K
    four ways across gid.z into fp32 partials, then sum + bias +
    activation in a second pass.

Both are gated on KVMeta::spec_block so nothing outside speculative
decoding changes path.

New switches, all default to auto (unset):
  MNN_METAL_FUSED_Q4_KSPLIT   =1 force on, =0 force off
  MNN_METAL_FUSED_Q4_M8       =0 force off (=1 same as unset, the tile is
  MNN_METAL_FUSED_Q4_KSPLIT_M8    only correct for area <= 8)

K-split reassociates the K accumulation, so output is not
bit-identical: on an iPad tier-9 sweep 8 of 9 prompts match the
baseline verbatim and one diverges on a near-tie greedy argmax.

The area-gate part of this work already landed with the fused-Q4
spec-block gate.

Discussed-in: Merge-Request 29266048 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29266048
GitOrigin-RevId: c465bb4195f3722161c2338440b301f43e052b01

* [LLM:Test] Add multi-LoRA training and runtime samples

Link: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/28972017
GitOrigin-RevId: 083103db5be967f96993d4def4dc7af6ec8d0e28

* [OpenCL:Bugfix] Fix Qwen2.5-Omni token2wav OpenCL crashes (fix issue alibaba#4673)

Link: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29004536
GitOrigin-RevId: f50421bdd3a7429e24d2080a412d34a51d2778ac

* [LLM:Bugfix] Fix LLM build when vision/audio support is disabled

Discussed-in: Merge-Request 29292144 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29292144
GitOrigin-RevId: e1f27b3ad70b8664bdffbf7eddf098e94df1f7ca

* [Hexagon:Feature] Support Qwen3-VL with symmetric and asymmetric block-wise quantization (alibaba#4774)

* [Vulkan:Opt] use coop matrix optimize vulkan attention qk * v

GitOrigin-RevId: 344788e334ab918f39c11d370a81d915e665e8f3

* [Metal:Perf] Optimize Q4 decode loads, lane splits, and RMSNorm

GitOrigin-RevId: 4f3539a1d266f6d6175d1668b64f09756bfc392f

* [OpenCL:Bugfix] Report dynamic pool memory in getSessionInfo(MEMORY) (alibaba#4784)

* [OpenCL:Bugfix] Cap Winograd transform buffers under Memory_Low (fix issue alibaba#4782) (alibaba#4783)

* [OpenCL:Bugfix] Cap Winograd transform buffers under Memory_Low (fix issue alibaba#4782)

* [Infra:Chore] Switch the reported plaintext HTTP URLs to HTTPS

GitOrigin-RevId: 0bc9f0186780621fee19a653537263fc924a61a5

* [Hexagon:Perf] Optimize asymmetric performance, Add Mod BinaryOp

GitOrigin-RevId: 87839f8c5a222e94e4f431691c201c8aa03fa010

* [OpenCL:Bugfix] Use vload_half/vstore_half builtins in buffer_convert_buf (fix issue alibaba#4802) (alibaba#4805)

* [OpenCL:Bugfix] Fix image-mode reduction kernel selection for unsupported shapes. Fix attention clone ignoring output_c4 flag, causing wrong vision results with weight pre-rearrange

GitOrigin-RevId: 9821ac77453745dcd89610d76b917c29a4b26442

---------

Co-authored-by: huangzhengxiang <100776727+huangzhengxiang@users.noreply.github.com>
Co-authored-by: UNIkeEN <sunyujie@sjtu.edu.cn>
Co-authored-by: blueskycoco <dillonhua@gmail.com>
Co-authored-by: fshuang8299 <167897122+fshuang8299@users.noreply.github.com>
Co-authored-by: HenryDen <Shuheng.Deng@arm.com>
Co-authored-by: YYDH <36133064+yydhYYDH@users.noreply.github.com>
Co-authored-by: 花熊 <zhanghuaxiang.zhx@taobao.com>
Co-authored-by: 王召德 <zhaode.wzd@alibaba-inc.com>
Co-authored-by: qianxinyu.qxy <qianxinyu.qxy@alibaba-inc.com>
Co-authored-by: xly <xlycae@126.com>
Co-authored-by: jingbang.yjb <jingbang.yjb@alibaba-inc.com>
Co-authored-by: tangexing.tgx <tangexing.tgx@taobao.com>
Co-authored-by: Qxinyu <37499237+Qxinyu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants