Metadata-only plan for a local coding assistant using MiniCPM5 2B GPTQ AWQ 4-bit, with portable GGUF and Apple Silicon MLX alternatives. Selects source-backed packages and supported platform routes.
| Machine | MiniCPM5 2B GPTQ (AWQ GEMM 4-bit) | MiniCPM5 2B Heretic Abliterated Q4_K_M GGUF | MiniCPM5 2B CRACK JANG_8M MLX |
|---|---|---|---|
| NVIDIA CUDA GPU, 8 GB VRAM | NVIDIA CUDA 8 GB VRAM Linux: vLLM AWQ route; 2.1 GB weights plus runtime. | NVIDIA CUDA 8 GB VRAM: llama.cpp GGUF route; 1.56 GiB weights plus KV cache. | Unsupported on NVIDIA; MLX JANG is Apple Silicon. Use GGUF or AWQ package. |
| AMD ROCm GPU, 8 GB VRAM | AMD 8 GB VRAM: exact AWQ vLLM route not evidenced on ROCm; use GGUF package. | AMD 8 GB VRAM: llama.cpp GGUF if ROCm/Vulkan backend is evidenced in build. | Unsupported on AMD; MLX JANG is Apple Silicon. Use GGUF package. |
| Intel Arc GPU, 8 GB VRAM | Intel Arc 8 GB VRAM: exact AWQ route not evidenced; vLLM XPU/AWQ loading unconfirmed. Use GGUF. | Intel Arc 8 GB VRAM: llama.cpp GGUF if SYCL/Vulkan backend is evidenced in build. | Unsupported on Intel Arc; MLX JANG is Apple Silicon. Use GGUF package. |
| Apple Silicon unified memory, 16 GB | Unsupported exact AWQ on Apple Silicon; vLLM-Metal not evidenced for this AWQ. Use MLX or GGUF. | Apple Silicon 16 GB unified: llama.cpp GGUF route; 1.56 GiB weights plus cache. | Apple Silicon 16 GB unified: MLX JANG route; 2.5 GiB weights plus KV cache. |
| CPU system RAM, 16 GB | CPU 16 GB RAM: exact AWQ route unsupported; vLLM requires supported GPU. Use GGUF. | CPU 16 GB RAM: llama.cpp GGUF route; 1.56 GiB weights plus context cache. | CPU 16 GB RAM: unsupported exact MLX JANG; Apple Silicon only. Use GGUF. |
| Machine | MiniCPM5 2B GPTQ (AWQ GEMM 4-bit) | MiniCPM5 2B Heretic Abliterated Q4_K_M GGUF | MiniCPM5 2B CRACK JANG_8M MLX |
|---|---|---|---|
| NVIDIA CUDA GPU, 8 GB VRAM | 8192 tokens starting; AWQ KV cache grows. Publisher max 131072; reduce if 8 GB VRAM tight. | 8192 tokens starting; GGUF KV cache grows with context. Publisher max 131072. | Unsupported on NVIDIA; use GGUF or AWQ package. |
| AMD ROCm GPU, 8 GB VRAM | Unsupported exact AWQ on AMD ROCm; use GGUF package. | 8192 tokens starting; GGUF KV cache grows. Publisher max 131072. | Unsupported on AMD; use GGUF package. |
| Intel Arc GPU, 8 GB VRAM | Unsupported exact AWQ on Intel Arc; use GGUF package. | 8192 tokens starting; GGUF KV cache grows. Publisher max 131072. | Unsupported on Intel Arc; use GGUF package. |
| Apple Silicon unified memory, 16 GB | Unsupported exact AWQ on Apple Silicon; use MLX or GGUF package. | 8192 tokens starting; GGUF KV cache grows. Publisher max 131072. | 8192 tokens starting; MLX KV cache grows. Publisher max 131072. |
| CPU system RAM, 16 GB | Unsupported exact AWQ on CPU; vLLM requires supported GPU. Use GGUF. | 8192 tokens starting; GGUF KV cache grows in system RAM. Publisher max 131072. | Unsupported on CPU; use GGUF package. |
A metadata plan for running MiniCPM5 2B locally for a small, reviewable code task. The exact publisher GPTQ repository is a 4-bit AWQ GEMM checkpoint intended for vLLM on Linux NVIDIA. An uncensored portable GGUF package provides llama.cpp routes across Windows, macOS, and Linux. An Apple Silicon MLX package provides a macOS route for Apple Silicon machines.
Select the weights you want. The instructions, downloads and settings change with your choice.
Save your machine in My Hardware to get an automatic starting choice. You can always choose any package yourself.
Estimate based on 2,099,615,880 bytes of root safetensors. Requires a runtime that can load AWQ GEMM checkpoints; vLLM is the source-backed exact route.
Choose where you want to run it. The instructions below update to match your selection.
Source-backed route: the publisher repacked the GPTQ layout into AWQ GEMM because vLLM's AutoGPTQ path rejects this exact configuration. Use vLLM's AWQ loading path.
Use a Linux machine with an NVIDIA GPU of compute capability 7.5 or newer and 8 GB or more VRAM. Install Python 3.12, the NVIDIA driver, and the CUDA toolkit. Keep the model repository and runtime separated from system Python.
Python 3.12 downloads NVIDIA CUDA downloadssudo apt update
sudo apt install -y python3.12 python3.12-venv python3.12-dev git curl jq
nvidia-smiCreate a dedicated working folder for this exact AWQ package. Create and activate a Python 3.12 virtual environment inside it so vLLM and its dependencies do not alter the system Python.
mkdir -p "$HOME/MiniCPM5-2B-vLLM"
cd "$HOME/MiniCPM5-2B-vLLM"
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheelInstall the official vLLM GPU wheel for NVIDIA CUDA. vLLM does not support Windows natively; the official Linux route uses uv with the PyTorch CUDA extra index. The exact AWQ GEMM checkpoint loads through vLLM's AWQ path, not the AutoGPTQ path. The cu129 index is used because vLLM's latest GPU documentation installs CUDA-enabled vLLM from the PyTorch cu129 wheel index.
vLLM GPU installation guideuv pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
python -c "import vllm; print(vllm.__version__)"Download the exact publisher repository openbmb/MiniCPM5-2B-GPTQ. The config declares a 4-bit AWQ GEMM layout repacked from an asymmetric GPTQ checkpoint. Use the Hugging Face CLI or huggingface_hub to keep the download in the working folder.
$HOME/MiniCPM5-2B-vLLM/models/openbmb/MiniCPM5-2B-GPTQ/mkdir -p models/openbmb
cd models/openbmb
git clone https://huggingface.co/openbmb/MiniCPM5-2B-GPTQ
cd MiniCPM5-2B-GPTQ
git lfs pull
cd "$HOME/MiniCPM5-2B-vLLM"Start vLLM with the exact local path to the AWQ GEMM checkpoint. Use --quantization awq so vLLM uses its AWQ loading path instead of the AutoGPTQ path that rejects this exact (4, False) layout. Set a starting context of 8192 tokens for an 8 GB VRAM GPU. Keep this server running in the first terminal.
vLLM OpenAI-compatible server guidesource .venv/bin/activate
vllm serve "$HOME/MiniCPM5-2B-vLLM/models/openbmb/MiniCPM5-2B-GPTQ" --served-model-name MiniCPM5-2B-GPTQ --quantization awq --dtype float16 --max-model-len 8192 --gpu-memory-utilization 0.90 --port 8000In a second terminal, verify that the server exposes the exact served model ID. This confirms the AWQ checkpoint loaded and is ready to serve OpenAI-compatible requests.
source .venv/bin/activate
curl -s http://127.0.0.1:8000/v1/models | jq .Send a deterministic prompt whose correct answer is known. Confirm the first local model response before configuring a coding client. If the answer is wrong, stop and inspect the server logs from the first terminal.
curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"MiniCPM5-2B-GPTQ","messages":[{"role":"user","content":"What is 12 multiplied by 7? Reply with only the number."}],"max_tokens":16,"temperature":0.0}' | jq -r '.choices[0].message.content'Install the official Visual Studio Code editor and the official Cline extension. These provide the coding client that connects to the local OpenAI-compatible endpoint.
Visual Studio Code download Cline extension Cline OpenAI-compatible provider configurationcode --install-extension saoudrizwan.claude-devOpen VS Code, open the Cline panel, and configure it to use an OpenAI-compatible provider. Set the Base URL to http://127.0.0.1:8000/v1 and the Model ID to MiniCPM5-2B-GPTQ. Set the context window to 8192 to match the server's --max-model-len. Keep approval controls enabled so that file changes and terminal commands require explicit confirmation.
Create a new empty project folder and open it in VS Code. Ask Cline to make one small, concrete code change, such as creating a Python script that reads a CSV of numbers and prints the mean. Review the proposed diff before accepting it.
mkdir -p "$HOME/MiniCPM5-2B-vLLM/disposable-project"
cd "$HOME/MiniCPM5-2B-vLLM/disposable-project"
printf '%s\n' 'value' '2' '4' '6' > numbers.csv
code "$HOME/MiniCPM5-2B-vLLM/disposable-project"Review the diff Cline proposes, then accept or edit it. Run the resulting script and check its output against numbers.csv. The publisher maximum position embedding is 131072 tokens, but the active runtime context is the --max-model-len value. If the GPU runs out of memory, stop the server and restart with --max-model-len 4096; the KV cache memory use grows with the active context length.
python mean.py
cat mean.pyThe selected package serves through a local OpenAI-compatible endpoint; a coding client can complete a small, reviewable code task. Tool calls should return OpenAI-compatible tool_calls when the runtime applies the MiniCPM5 XML-style parser. The reader inspects the diff before accepting generated code.
Package selection and platform routes are source-backed. Verify exact runtime flags and model revision against the linked primary sources before deployment.
The publisher config notes that vLLM's AutoGPTQ path may raise because it lacks a matching (4, False) entry; the package is repacked as AWQ GEMM for this reason. Confirm the runtime is loading it as AWQ, not legacy GPTQ, and use the exact publisher revision.
The model card recommends temperature=1.0, top_p=0.95, min_p=0.0. If repetition persists, add repetition_penalty=1.05. Do not silently switch to greedy decoding.
MiniCPM5 emits XML-style function calls. SGLang documents a dedicated MiniCPM5 tool-call parser. If using vLLM or llama.cpp, verify whether the server/client supports the supplied chat template and tool parsing; otherwise tool calls may appear as text.
Choose the smaller GGUF Q4_K_M package or reduce the runtime context length. The exact AWQ package and MLX bundle have larger weight footprints.
Original instructions, model files, and compatibility notes behind this setup.
Source-backed package and platform-route plan for private local chat with Qwen3.5 · 9B. Preserves the official Ollama 9B package, a smaller 4B family fallback, and a clearly labeled community uncensored GGUF option.
Run Qwen3.8 27B with Ollama, connect it to Cline in VS Code, and make one reviewable code change in a tiny local project.
Generate images from text prompts in ComfyUI with the fast distilled or more adjustable base 4B model. Save your results and reuse the visual workflow.