Run a compressed private reasoning assistant
Use Bonsai 2’s dedicated low-bit runtime, download its PQ2_0 package, and verify a local response without accidentally using stock llama.cpp.
Before you begin
- Difficulty
- Intermediate
- Software
- PrismML llama.cpp fork, Hugging Face CLI
- Hardware
- The PQ2_0 package is about 7.2 GB. Prism ML publishes measured RTX 5090 and RTX 4090 speeds; other machines need a specific runtime check.
Sources and files
Publisher quickstart and measurements Known-good binaries and setupChoose a model for this task
The steps below use the recommended model. Alternatives have their own package and command; open their model pages before switching.
The workflow
Choose the publisher runtime
Use the pinned binary linked from Prism ML’s Bonsai demo or build PrismML-Eng/llama.cpp. Stock llama.cpp cannot run the PQ2_0 or PTQ1_0 weights correctly.
Set up the binary
Download the archive for your operating system from PrismML-Eng/llama.cpp releases and extract it so llama-cli is under bin. On macOS and Linux the publisher also documents building its fork from source.
Download the PQ2_0 file
Install the Hugging Face CLI and download the named GGUF into your working directory.
hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .Run the publisher example
From that working directory use the publisher’s fork binary. This starts with 32K context and a short test prompt; reduce context if memory is tight.
./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 -fa on -c 32768 --temp 1.0 --top-p 0.95 --top-k 20 -p "Explain quantum computing in simple terms." -n 256Confirm local reasoning works
A successful run loads the GGUF and prints an answer to the prompt. If the runtime says the quantization type is unknown, stop and install Prism ML’s fork.
Adjust for your task
Replace the test prompt with your own reasoning or coding question. Use the publisher’s demo scripts for server mode, tool calling, or vision; those modes need different flags and sometimes extra files.
When it doesn’t go to plan
The wrong llama.cpp build is likely installed. Use PrismML-Eng’s fork or its pinned release; the publisher says stock binaries do not support these packs.
Check the exact GGUF filename and current directory, lower the context setting, and use the publisher’s hardware-specific setup before trying CPU offload.
The model behind this workflow
Bonsai 2 · 27BWill it run on your machine?
Save your machine to see a personalized rating and its reasoning.
Add my hardwareKeep exploring
Set up a private AI chat assistant
Install a local runtime, run Qwen3.5 9B, confirm responses, and know when to choose the smaller 4B package.
Build a private local coding assistant
Connect an open coding-capable model in Ollama to Cline, run a small repository task, and review the result locally.
Generate images on your own GPU
Use the official FLUX.2 Klein 4B ComfyUI template with exact model files, a first prompt, and an output check.