mlx-anywhere

Convert HuggingFace models to MLX quantized format on any hardware — Linux, Windows, Mac x86, or Apple Silicon.

No Apple Silicon required for conversion. Just numpy + safetensors.

The Problem

The official mlx_lm.convert requires Apple Silicon (Metal GPU). If you want to convert a large model like Gemma 4 26B on a Linux cloud VM with lots of RAM, you're out of luck.

mlx-anywhere solves this by implementing MLX's affine quantization in pure NumPy, producing output identical to what mlx_lm.convert would produce on a Mac.

It also converts one shard at a time, so peak RAM usage is roughly the size of the largest shard (~3-5 GB for most models), not the full model.

Install

pip install mlx-anywhere

Or from source:

git clone https://github.com/msabty/mlx-anywhere
cd mlx-anywhere
pip install -e .

Usage

# Convert Gemma 4 26B to 4-bit MLX format
mlx-anywhere convert \
  --hf-path google/gemma-4-26B-A4B-it \
  --mlx-path ./gemma-4-26b-mlx-4bit \
  --bits 4

# Convert and upload to HuggingFace
mlx-anywhere convert \
  --hf-path google/gemma-4-26B-A4B-it \
  --mlx-path ./gemma-4-26b-mlx-4bit \
  --bits 4 \
  --upload-repo your-username/gemma-4-26B-A4B-it-mlx-4bit

Python API

from mlx_anywhere import convert

convert(
    hf_path="google/gemma-4-26B-A4B-it",
    mlx_path="./output",
    bits=4,
    group_size=64,
)

Supported Models

Model family	Status
Gemma 4 (31B dense, 26B MoE)	✅
Qwen 2.5 / 3.5	✅
Llama 3.x	✅
Mistral	✅
Generic (any transformer)	✅

How It Works

MLX uses affine quantization with:

• bias = max(group)
• scale = (min - max) / (2^bits - 1) ← negative
• q = round((w - bias) / scale)
• Packed as uint32 with 8 × 4-bit values per element (LSB first)
• Scales/biases stored as bfloat16 (uint16)

mlx-anywhere replicates this in NumPy, including bfloat16 precision rounding, producing weights loadable by mlx-lm and mlx-vlm.

For Gemma 4, it also handles:

• Key renaming (strip model. prefix, remap language_model.X → language_model.model.X)
• MoE expert splitting (gate_up_proj → gate_proj + up_proj)
• 3D expert tensor quantization with reshape

Memory Usage

Model	BF16 size	Peak RAM during conversion
Gemma 4 26B MoE	52 GB	~4 GB (one shard)
Gemma 4 31B	62 GB	~8 GB (one shard)
Qwen 2.5 72B	~144 GB	~10 GB (one shard)

Contributing

PRs welcome — especially:

• New model sanitizers (add to mlx_anywhere/sanitize.py)
• mxfp4 quantization mode
• Parallel shard processing
• Tests