[OpenCL:Bugfix] Use vload_half/vstore_half builtins in buffer_convert_buf - #4805
Merged
Conversation
|
|
ValiantDiligent
approved these changes
Aug 25, 2026
123xuwu
pushed a commit
to 123xuwu/MNN
that referenced
this pull request
Aug 25, 2026
* [LLM:Feature] Add Qwen3-ASR and Qwen3-VL-Embedding export and runtime support. (alibaba#4478) Co-authored-by: UNIkeEN <sunyujie@sjtu.edu.cn> * [QNN:Bugfix] Add support for convert llm/visual mnn model into same qnn folder (Qnn model) (alibaba#4757) * [Express:Bugfix] Fix loadMap/load output readMap returning NULL (alibaba#4751) Fixes alibaba#4750 * [LLM:Feature] Support Qwen3-TTS-12Hz-0.6B-Base (alibaba#4589) * [CPU:Refact] Refactor the KleidiAI integration code (alibaba#4602) * [LLM:Bugfix] Align Qwen-VL runtime preprocessing (alibaba#4595) * [LLM:Feature] DFlash draft shares target's lm_head (no separate lm_head) The DFlash draft's lm_head is identical to the target's (tie_word_embeddings), but was baked into dflash.mnn, duplicating the vocab x hidden weights on disk and in memory (~219 MB for Qwen3-4B int4). Export: DFlash.forward now outputs post-final-norm hidden_states instead of logits, so dflash.mnn contains no lm_head. The separate dflash_lmhead.mnn export path (--dflash_export_separate_lmhead) is removed: the shared subgraph already covers its use case -- mixed precision (fp16 transformer + int4 lm_head) -- via the target's own lm_head. Engine: DFlash load() requires dflash_shared_lmhead_input and loads the subgraph {input -> logits} from llm.mnn with Module::Config{base=target, rearrange=true}. The embedded-lmhead and separate-lmhead fallbacks are gone: a draft without the config field is refused at load time with a re-export hint. Refusal is enforced through Generation::load(), which now returns bool; on failure of any module the draft step dereferences (draft graph, lm_head, fc) Llm::load() aborts and status stays NOT_LOADED, so every inference entry is fenced by CHECK_LLM_RUNNING instead of reaching a null module. Verify (M5 iPad, 2B int4 b8 greedy): acceptance 4.59 -> 4.63, indistinguishable from the separate-lm_head baseline (difference < 0.04 across 9 prompts). Discussed-in: Merge-Request 28763320 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/28763320 GitOrigin-RevId: 318842e009b5067f67b1b5aa53b35c44df23536c * [Test:Bugfix] Relax FP16 loadMap output tolerance Squashed from 2 internally reviewed commits. - 64f5b15 [JNI:Bugfix] Fix string array handling (alibaba#4753, alibaba#4754) — wangzhaode <zhaode.wzd@alibaba-inc.com> - b9cedef [Test:Bugfix] Relax FP16 loadMap output tolerance — wangzhaode <zhaode.wzd@alibaba-inc.com> Co-authored-by: wangzhaode <zhaode.wzd@alibaba-inc.com> GitOrigin-RevId: e71e6b3c07041b3e6380974e529cbafff613b790 * [CPU:Feature] Detect SME2 core count Discussed-in: Merge-Request 29219430 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29219430 GitOrigin-RevId: 8de11463229a9e0fc9004b7122ec59ac73b62b52 * [Hexagon:Perf] Add int4 vrmpy decode GEMV (M=1), Opitimize FlashAttention and Gemm GitOrigin-RevId: 49f3d6e2864ca11bcfee7f61d84ced9fab022915 * [LLM:Feature] Add Qwen3.5 video input (alibaba#4577) * [NPU:rknn] Add initial rknn support for Rockchip NPUs, supporting traditional models. (alibaba#4524) * [Infra:Bugfix] Replace remaining vector-to-pointer casts for Mali compat GitOrigin-RevId: f5c45ecb5631ccdbf333cfdd925b73dd238f327e * [LLM:Perf] DFlash engine: on-device fc/concat, incremental draft KV, on-device argmax Link: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29237251 * [Metal:Feature] Speculative-block aware LinearAttention and Q4 GEMM GitOrigin-RevId: 5317f5b246de830c817acaa487973cb1105c4265 * [Metal:Perf] Fused-Q4 M8 tile and K-split for speculative-block decode The fused Q4 GEMM stage kernel uses an M32 matmul2d tile. A speculative block is 8 rows, so it runs at quarter occupancy, and for projections with few output tiles it also dispatches too few threadgroups to saturate bandwidth. Add two variants behind FUSED_Q4_REAL_UNPACK: - conv1x1_fused_q4_gemm_stage_m8: descriptor sized to M=8. - conv1x1_fused_q4_gemm_stage_ksplit{,_m8} + _ksplit_reduce: split K four ways across gid.z into fp32 partials, then sum + bias + activation in a second pass. Both are gated on KVMeta::spec_block so nothing outside speculative decoding changes path. New switches, all default to auto (unset): MNN_METAL_FUSED_Q4_KSPLIT =1 force on, =0 force off MNN_METAL_FUSED_Q4_M8 =0 force off (=1 same as unset, the tile is MNN_METAL_FUSED_Q4_KSPLIT_M8 only correct for area <= 8) K-split reassociates the K accumulation, so output is not bit-identical: on an iPad tier-9 sweep 8 of 9 prompts match the baseline verbatim and one diverges on a near-tie greedy argmax. The area-gate part of this work already landed with the fused-Q4 spec-block gate. Discussed-in: Merge-Request 29266048 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29266048 GitOrigin-RevId: c465bb4195f3722161c2338440b301f43e052b01 * [LLM:Test] Add multi-LoRA training and runtime samples Link: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/28972017 GitOrigin-RevId: 083103db5be967f96993d4def4dc7af6ec8d0e28 * [OpenCL:Bugfix] Fix Qwen2.5-Omni token2wav OpenCL crashes (fix issue alibaba#4673) Link: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29004536 GitOrigin-RevId: f50421bdd3a7429e24d2080a412d34a51d2778ac * [LLM:Bugfix] Fix LLM build when vision/audio support is disabled Discussed-in: Merge-Request 29292144 , URL: https://code.alibaba-inc.com/AliNN/AliNNPrivate/codereview/29292144 GitOrigin-RevId: e1f27b3ad70b8664bdffbf7eddf098e94df1f7ca * [Hexagon:Feature] Support Qwen3-VL with symmetric and asymmetric block-wise quantization (alibaba#4774) * [Vulkan:Opt] use coop matrix optimize vulkan attention qk * v GitOrigin-RevId: 344788e334ab918f39c11d370a81d915e665e8f3 * [Metal:Perf] Optimize Q4 decode loads, lane splits, and RMSNorm GitOrigin-RevId: 4f3539a1d266f6d6175d1668b64f09756bfc392f * [OpenCL:Bugfix] Report dynamic pool memory in getSessionInfo(MEMORY) (alibaba#4784) * [OpenCL:Bugfix] Cap Winograd transform buffers under Memory_Low (fix issue alibaba#4782) (alibaba#4783) * [OpenCL:Bugfix] Cap Winograd transform buffers under Memory_Low (fix issue alibaba#4782) * [Infra:Chore] Switch the reported plaintext HTTP URLs to HTTPS GitOrigin-RevId: 0bc9f0186780621fee19a653537263fc924a61a5 * [Hexagon:Perf] Optimize asymmetric performance, Add Mod BinaryOp GitOrigin-RevId: 87839f8c5a222e94e4f431691c201c8aa03fa010 * [OpenCL:Bugfix] Use vload_half/vstore_half builtins in buffer_convert_buf (fix issue alibaba#4802) (alibaba#4805) * [OpenCL:Bugfix] Fix image-mode reduction kernel selection for unsupported shapes. Fix attention clone ignoring output_c4 flag, causing wrong vision results with weight pre-rearrange GitOrigin-RevId: 9821ac77453745dcd89610d76b917c29a4b26442 --------- Co-authored-by: huangzhengxiang <100776727+huangzhengxiang@users.noreply.github.com> Co-authored-by: UNIkeEN <sunyujie@sjtu.edu.cn> Co-authored-by: blueskycoco <dillonhua@gmail.com> Co-authored-by: fshuang8299 <167897122+fshuang8299@users.noreply.github.com> Co-authored-by: HenryDen <Shuheng.Deng@arm.com> Co-authored-by: YYDH <36133064+yydhYYDH@users.noreply.github.com> Co-authored-by: 花熊 <zhanghuaxiang.zhx@taobao.com> Co-authored-by: 王召德 <zhaode.wzd@alibaba-inc.com> Co-authored-by: qianxinyu.qxy <qianxinyu.qxy@alibaba-inc.com> Co-authored-by: xly <xlycae@126.com> Co-authored-by: jingbang.yjb <jingbang.yjb@alibaba-inc.com> Co-authored-by: tangexing.tgx <tangexing.tgx@taobao.com> Co-authored-by: Qxinyu <37499237+Qxinyu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
buffer_convert_bufkernel build on devices whose driver advertisescl_khr_fp16but rejects direct scalar access to__global half*("loading directly from pointer to type 'const __global half' requires cl_khr_fp16. Use vector data load builtin functions instead").buffer_convert_buf.cl, route all scalar/vector half element access throughvload_half/vstore_half/vload_half4/vstore_half4builtins via newMNN_LOAD_INPUT/MNN_STORE_OUTPUT/MNN_LOAD_INPUT4/MNN_STORE_OUTPUT4/MNN_LOAD_FLOATmacros. Coversbuffer_convert_to_buffer,buffer_copy_to_buffer(incl. PACK_LEAVE tail loop) and the conv/dw filter conversion kernels (20 directFLOAT*dereferences). Non-half types keep the original direct/vload4 paths unchanged.-DINPUT_IS_HALF/-DOUTPUT_IS_HALFwherever half types are selected (AddBuildOptionOfDataTypein BufferConvertor, tensor-derived types inOpenCLRuntime::buildKernelWithCache, which also covers Softmax/ArgMax callers of these kernels).buildKernelWithCacheresult in BufferConvertor so a kernel build failure degrades to a logged error instead of a segmentation fault (the crash path in the issue).buffer_convert_bufmd5 inopencl_source_map.hppto invalidate stale compiled-kernel caches.Test plan
build_oclfull op suite on OpenCL backend: 248/248 passed (fp16 path active on Apple GPU)buffer_convert_to_bufferwith half NC4HW4 unpack/pack)