Skip to content

Commit 4b7f701

Browse files
RaQiuRaQiu
committed
[feat]: merge upstream kvcache-ai/ktransformers main
- Merged upstream changes including GPTQ expert loading support - Resolved conflict in loader.py: kept both our mmap methods and upstream GPTQSafeTensorLoader.load_experts()
2 parents 0931bb7 + 7a02daa commit 4b7f701

22 files changed

Lines changed: 3546 additions & 17 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project has evolved into **two core modules**: [kt-kernel](https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel/) and [kt-sft](https://github.com/kvcache-ai/ktransformers/tree/main/kt-sft).
1717

1818
## 🔥 Updates
19-
19+
* **Mar 26, 2026**: Support AVX2-only CPU backend for KT-Kernel inference. ([Tutorial](./doc/en/kt-kernel/AVX2-Tutorial.md))
2020
* **Feb 13, 2026**: MiniMax-M2.5 Day0 Support! ([Tutorial](./doc/en/MiniMax-M2.5.md))
2121
* **Feb 12, 2026**: GLM-5 Day0 Support! ([Tutorial](./doc/en/kt-kernel/GLM-5-Tutorial.md))
2222
* **Jan 27, 2026**: Kimi-K2.5 Day0 Support! ([Tutorial](./doc/en/Kimi-K2.5.md)) ([SFT Tutorial](./doc/en/SFT_Installation_Guide_KimiK2.5.md))

‎doc/SUMMARY.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
- [For kt-kernel](en/kt-kernel/kt-kernel_intro.md)
66
- [For kt-sft](en/SFT/KTransformers-Fine-Tuning_User-Guide.md)
77

8-
# Tutorial
8+
# Tutorial
99
- [kt-sft part](en/SFT/README.md)
1010
- [Injection Tutorial](en/SFT/injection_tutorial.md)
1111
- [kt-sft developer tech notes](en/SFT/KTransformers-Fine-Tuning_Developer-Technical-Notes.md)
@@ -19,6 +19,8 @@
1919
- [Makefile Usage](en/makefile_usage.md) -->
2020
- [kt-kernel part](en/kt-kernel/README.md)
2121
- [kt-cli](en/kt-kernel/kt-cli.md)
22+
- [AVX2 Backend Tutorial](en/kt-kernel/AVX2-Tutorial.md)
23+
- [AVX2 后端教程（中文）](zh/AVX2-Tutorial_zh.md)
2224
# FAQ
2325
- [FAQ](en/FAQ.md)
2426
<!-- # V3 Reproduction

‎doc/en/kt-kernel/AVX2-Tutorial.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Running KTransformers on AVX2 CPUs
2+
3+
This tutorial explains how to run KTransformers on machines that only support AVX2 (without AVX512 or AMX).
4+
5+
## Table of Contents
6+
7+
- [Supported Precision Formats](#supported-precision-formats)
8+
- [Hardware Requirements](#hardware-requirements)
9+
- [Installation](#installation)
10+
- [Verification](#verification)
11+
- [Starting the Inference Server](#starting-the-inference-server)
12+
- [Example: Qwen3-30B-A3B (BF16)](#example-qwen3-30b-a3b-bf16)
13+
- [Example: Qwen3.5-35B-A3B-FP8 (FP8)](#example-qwen35-35b-a3b-fp8-fp8)
14+
- [Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)](#example-qwen3-30b-a3b-gptq-int4-gptq_int4)
15+
- [Sending Requests](#sending-requests)
16+
- [Performance Tuning](#performance-tuning)
17+
- [FAQ](#faq)
18+
19+
## Supported Precision Formats
20+
21+
| `--kt-method` | Precision | Description |
22+
|---------------|-----------|-------------|
23+
| `BF16` | BF16 native precision | Zero precision loss, uses BF16 weights directly |
24+
| `FP8` | FP8 block quantization | |
25+
| `GPTQ_INT4` | INT4 GPTQ | |
26+
27+
28+
## Hardware Requirements
29+
30+
- **CPU**: x86-64 + AVX2 + FMA (Intel Haswell 2013+ / AMD Zen+)
31+
- **GPU**: NVIDIA 24GB+ VRAM (RTX 3090/4090/5090, etc.)
32+
- **Memory**: At least the size of the model weights (e.g., Qwen3-30B-A3B BF16 requires 64GB+)
33+
- **OS**: Linux
34+
35+
## Installation
36+
37+
Build and install from source (one-click install for kt-kernel + SGLang):
38+
39+
```bash
40+
git clone https://github.com/kvcache-ai/ktransformers.git
41+
cd ktransformers
42+
git submodule update --init --recursive
43+
44+
# One-click install
45+
./install.sh
46+
```
47+
48+
On AVX512 or AMX machines, you can also manually force AVX2 compilation:
49+
50+
```bash
51+
export CPUINFER_CPU_INSTRUCT=AVX2
52+
export CPUINFER_ENABLE_AMX=OFF
53+
./install.sh kt-kernel --manual
54+
```
55+
56+
57+
58+
## Verification
59+
60+
```bash
61+
# Check if the CPU supports AVX2
62+
lscpu | grep -i avx2
63+
64+
# Check the loaded kt-kernel variant
65+
python -c "import kt_kernel; print(kt_kernel.__cpu_variant__)"
66+
# Expected output: avx2
67+
68+
# System diagnostics
69+
kt doctor
70+
```
71+
72+
## Starting the Inference Server
73+
74+
Use `--kt-method BF16`, `FP8`, or `GPTQ_INT4`. KT-Kernel will **automatically detect** the CPU and fall back to the AVX2 backend when AVX512/AMX is unavailable.
75+
76+
### Example: Qwen3-30B-A3B (BF16)
77+
78+
```bash
79+
# Download the model
80+
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /path/to/Qwen3-30B-A3B
81+
82+
# Check physical core count and NUMA node count
83+
lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core|NUMA node\(s\)"
84+
85+
# Start the server (adjust kt-cpuinfer and kt-threadpool-count based on your hardware)
86+
python -m sglang.launch_server \
87+
--host 0.0.0.0 --port 30000 \
88+
--model /path/to/Qwen3-30B-A3B \
89+
--kt-weight-path /path/to/Qwen3-30B-A3B \
90+
--kt-cpuinfer 16 \
91+
--kt-threadpool-count 1 \
92+
--kt-num-gpu-experts 32 \
93+
--kt-method BF16 \
94+
--attention-backend flashinfer \
95+
--trust-remote-code \
96+
--mem-fraction-static 0.80 \
97+
--chunked-prefill-size 8192 \
98+
--max-running-requests 2 \
99+
--served-model-name Qwen3 \
100+
--enable-mixed-chunk \
101+
--tensor-parallel-size 1 \
102+
--enable-p2p-check \
103+
--disable-shared-experts-fusion
104+
```
105+
106+
### Example: Qwen3.5-35B-A3B-FP8 (FP8)
107+
108+
```bash
109+
# Download the model
110+
huggingface-cli download Qwen/Qwen3.5-35B-A3B-FP8 --local-dir /path/to/Qwen3.5-35B-A3B-FP8
111+
112+
# Start the server
113+
python -m sglang.launch_server \
114+
--host 0.0.0.0 --port 30000 \
115+
--model /path/to/Qwen3.5-35B-A3B-FP8 \
116+
--kt-weight-path /path/to/Qwen3.5-35B-A3B-FP8 \
117+
--kt-cpuinfer 16 \
118+
--kt-threadpool-count 1 \
119+
--kt-num-gpu-experts 2 \
120+
--kt-method FP8 \
121+
--kt-gpu-prefill-token-threshold 400 \
122+
--attention-backend triton \
123+
--trust-remote-code \
124+
--mem-fraction-static 0.85 \
125+
--chunked-prefill-size 4096 \
126+
--max-running-requests 1 \
127+
--max-total-tokens 32000 \
128+
--enable-mixed-chunk \
129+
--tensor-parallel-size 1 \
130+
--disable-shared-experts-fusion
131+
```
132+
133+
### Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)
134+
135+
```bash
136+
# Download the model
137+
huggingface-cli download Qwen/Qwen3-30B-A3B-GPTQ-Int4 --local-dir /path/to/Qwen3-30B-A3B-GPTQ-Int4
138+
139+
# Start the server
140+
python -m sglang.launch_server \
141+
--host 0.0.0.0 --port 30000 \
142+
--model /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
143+
--kt-weight-path /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
144+
--kt-cpuinfer 16 \
145+
--kt-threadpool-count 1 \
146+
--kt-num-gpu-experts 2 \
147+
--kt-method GPTQ_INT4 \
148+
--attention-backend triton \
149+
--trust-remote-code \
150+
--mem-fraction-static 0.85 \
151+
--chunked-prefill-size 4096 \
152+
--max-running-requests 1 \
153+
--max-total-tokens 32000 \
154+
--enable-mixed-chunk \
155+
--tensor-parallel-size 1 \
156+
--disable-shared-experts-fusion
157+
```
158+
159+
### Sending Requests
160+
161+
```bash
162+
# Interactive chat
163+
kt chat
164+
165+
# OpenAI-compatible API
166+
curl http://localhost:30000/v1/chat/completions \
167+
-H "Content-Type: application/json" \
168+
-d '{"model":"Qwen3","messages":[{"role":"user","content":"Hello"}],"stream":true}'
169+
```
170+
171+
172+
173+
## Performance Tuning
174+
175+
- `--kt-cpuinfer`: set to the number of **physical cores**
176+
- `--kt-threadpool-count`: set to the number of **NUMA nodes**
177+
- `--kt-num-gpu-experts`: higher values reduce CPU load but increase GPU VRAM usage
178+
- Memory bandwidth is often the bottleneck; high-frequency DDR5 memory helps significantly
179+
180+
## FAQ
181+
182+
183+
184+
**GPU OOM**
185+
- Reduce `--kt-num-gpu-experts`, `--chunked-prefill-size`, `--max-total-tokens`
186+
- Lower `--mem-fraction-static`
187+
188+
For more questions, see [FAQ](../FAQ.md).

0 commit comments

Comments
 (0)