Workflows/Set up private chat with Bonsai 2

Text / Intermediate

Set up private chat with Bonsai 2

Run the 27B Bonsai 2 model in a local browser chat using PrismML’s own runtime. Choose your operating system, copy the setup commands, and confirm your first response.

Verified sourceSource checked 9/23/2026

LOCAL

Before you begin

Difficulty: Intermediate
Software: Git, PrismML Bonsai Demo
Hardware: The default PQ2_0 package is 7.21 GB of model weights. Leave additional RAM or VRAM for the runtime and conversation context; start at 8K context on a memory-limited machine.

View the setup source

Sources and files

PrismML official setup and launch commands Bonsai 2 model card, formats, and sizes PrismML environment variables

LOCAL CHAT / MAC · WINDOWS · LINUXPUBLISHER-SOURCED SETUP

Your local chat setup, step by step.

A focused path to your first private chat. PrismML’s setup downloads its compatible runtime and the default PQ2_0 weights; your prompts go to a server on your own machine.

Choose a package Run the setup Check the result

BEFORE YOU BEGIN

Choose your package

The quick-start steps below use the recommended package. Check the exact size before downloading.

Recommended for this guide

PQ2_07.21 GB model weights

The publisher’s default download. Use this for the steps below; it trades some disk space for faster prompt processing.

Smaller alternative

PTQ1_05.95 GB model weights

A tighter weight package for memory-limited systems. It needs a separate package selection; follow the publisher’s format guide before switching.

THE WORKFLOW

Set up on your machine

Pick your operating system and follow the commands in order.

Get the developer command-line tools

If Git is not installed, run this once and complete the macOS installer. You can skip this step if git --version already works.

xcode-select --install

Get the official Bonsai setup

Clone PrismML’s maintained demo and enter its folder in Terminal.

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo

Install the runtime and model

The publisher setup selects a compatible build and downloads Bonsai 2 PQ2_0. This command skips the optional Open WebUI and code interpreter to keep the first run focused.

BONSAI_OPENWEBUI=0 BONSAI_CODE_INTERPRETER=0 ./setup.sh

Start your private chat server

Begin at 8K context to leave memory headroom. Keep Terminal open while you chat.

BONSAI_CTX=8192 ./scripts/start_llama_server.sh

SUCCESS CHECK

Send your first message

Open the local address in your browser and ask a short question. A response in the chat window confirms that the downloaded model and compatible server are working together.

http://localhost:8080Opens on your machine

Bonsai 2 uses PrismML’s compatible runtime. Stock Ollama and stock llama.cpp do not run these Bonsai 2 ternary packages correctly. Package size is not total memory use; longer context and other applications need headroom.

Back to setup steps

When it doesn’t go to plan

The server says invalid ggml type or gives unusable output

Run the launcher installed by PrismML Bonsai Demo. Bonsai 2 requires PrismML’s compatible runtime, not stock Ollama or stock llama.cpp.

The machine slows down or runs out of memory

Close other GPU-heavy applications and lower BONSAI_CTX to 4096. The 7.21 GB weight file is only part of the total memory requirement.

Windows uses the CPU rather than the NVIDIA GPU

Confirm the NVIDIA driver and nvidia-smi work, then rerun the publisher setup to select the appropriate compatible build.

The model behind this workflow

Bonsai 2 · 27B

MY HARDWARE

Will it run on your machine?

Save your machine to see a personalized rating and its reasoning.

Add my hardware

Keep exploring

Text

Beginner

Set up a private AI chat assistant

Install a local runtime, run Qwen3.5 9B, confirm responses, and know when to choose the smaller 4B package.

Build a private local coding assistant

Connect an open coding-capable model in Ollama to Cline, run a small repository task, and review the result locally.

SOFTWAREOllama + Cline

Generate images on your own GPU

Use the official FLUX.2 Klein 4B ComfyUI template with exact model files, a first prompt, and an output check.

SOFTWAREComfyUI

LOCALRENTED GPU

Verified source