支持用MiniCPM-o-4_5-awq 吗

[wodsoe]

支持用MiniCPM-o-4_5-awq 吗

[bokesyo]

正在研究，如有进展会同步在这里

[bokesyo]

您好 @wodsoe ，awq模型初步跑通了，可以使用，峰值显存由28G降低到15G，模型显存占用由22G降低到12G。

但由于之前的校准集覆盖度不全，因此目前版本的awq权重存在精度问题，为了达到最好的效果，会准备一份更好的校准集重新量化一次，争取能本周末前放出来，如有更新，会更新在此处

[cuixg]

效果挺棒的，坐等更好的量化模型

[bokesyo]

@cuixg 感谢您的反馈！🫶 临时有些额外的事情有些耽搁，下周能有量化的模型~

[fengxiqaq]

gguf 呢？能用吗？这个 pytorch demo？

[tc-mb]

@cuixg 我们之前已经提供量化版本模型，hf和ms的路径在这里。
精度会比bf16会略有掉点，我们也会继续做量化优化，争取q4精度损失尽可能小，可以先用这个。
https://huggingface.co/openbmb/MiniCPM-o-4_5-awq
https://modelscope.cn/models/OpenBMB/MiniCPM-o-4_5-awq

[tc-mb]

@fengxiqaq gguf仓库在这里，https://huggingface.co/openbmb/MiniCPM-o-4_5-gguf

[lllyyy264]

==================================================
MiniCPM-o 4.5 Service (Docker)

GPUs: 0,1,2,3 (4)
Gateway: http://0.0.0.0:8006 Workers: localhost:22400 ~ localhost:22403==================================================
[Worker 0] Starting on GPU 0, port 22400...
[Worker 1] Starting on GPU 1, port 22401...
[Worker 2] Starting on GPU 2, port 22402...
[Worker 3] Starting on GPU 3, port 22403...
Waiting for Workers to load models (~30-90s)...
[Worker 0] FAILED to start! Log:
2026-03-25 05:16:38,240 [INFO] config: 配置已加载: model=/workspace/models/MiniCPM-o-4_5-awq, attn_implementation=auto, gateway_port=8006, playback_delay_ms=200, chat_vocoder=token2wav
2026-03-25 05:16:38,241 [INFO] worker: Starting Worker on port 22400, GPU 0
INFO: Started server process [93]
INFO: Waiting for application startup.
2026-03-25 05:16:38,264 [INFO] worker: [GPU 0] Loading model from /workspace/models/MiniCPM-o-4_5-awq...2026-03-25 05:16:38,264 [INFO] core.processors.unified: Loading unified model: /workspace/models/MiniCPM-o-4_5-awq/usr/local/lib/python3.10/dist-packages/transformers/models/auto/image_processing_auto.py:604: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use slow_image_processor_class, or fast_image_processor_class instead
warnings.warn(
2026-03-25 05:16:40,848 [INFO] core.processors.unified: [Attention] auto -> sdpa (flash-attn unavailable, using PyTorch built-in SDPA. For flash_attention_2, install: MAX_JOBS=16 pip install 'flash-attn>=2.6' --no-build-isolation)
2026-03-25 05:16:40,848 [INFO] core.processors.unified: Quantized model detected
2026-03-25 05:16:40,851 [INFO] MiniCPMO45.configuration_minicpmo: vision_config is None, using default vision config
ERROR: Traceback (most recent call last):
2026-03-25 05:26:53,695 [INFO] MiniCPMO45.configuration_minicpmo: vision_config is None, using default vision config
ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 638, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/app/worker.py", line 554, in lifespan
await asyncio.to_thread(worker.load_model)
File "/usr/lib/python3.10/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/app/worker.py", line 172, in load_model
self.processor = UnifiedProcessor(
File "/app/core/processors/unified.py", line 1249, in init
super().init(model_path=model_path, device=device)
File "/app/core/processors/base.py", line 126, in init
self._load_model()
File "/app/core/processors/unified.py", line 1358, in _load_model
self.model = MiniCPMO.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4215, in from_pretrained
config.quantization_config = AutoHfQuantizer.merge_quantization_configs(
File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/auto.py", line 198, in merge_quantization_configs
quantization_config = AutoQuantizationConfig.from_dict(quantization_config)
File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/auto.py", line 128, in from_dict
return target_cls.from_dict(quantization_config_dict)
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/quantization_config.py", line 119, in from_dict
config = cls(**config_dict)
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/quantization_config.py", line 872, in init
self.post_init()
File "/usr/local/lib/python3.10/dist-packages/transformers/utils/quantization_config.py", line 931, in post_init
raise ValueError(
ValueError: You current version of autoawq does not support module quantization skipping, please upgrade autoawq package to at least 0.1.8

按照README的Docker部署方式部署的，请问是什么原因?