Workflows/Set up private chat with openjev

Text / Intermediate

Set up private chat with openjev

Run OpenJev locally on Linux with vLLM, send a typed decision request to its decision API, and confirm the model returns a choice, a yes/no probability, or a score.

EstimatedSource checked 9/25/2026

LOCALRENTED GPU

Before you begin

Difficulty: Intermediate
Software: vLLM, Linux, Python
Hardware: This setup uses vLLM. The exact checkpoint format, context, and runtime overhead determine its memory need; a weight-file size alone is not a GPU recommendation.

TEXT / VLLMSOURCE-LINKED SETUP

Set up private chat with openjev, step by step.

Run OpenJev locally on Linux with vLLM, send a typed decision request to its decision API, and confirm the model returns a choice, a yes/no probability, or a score.

Choose a package Run its commands Check the result

BEFORE YOU BEGIN

The documented package

This setup uses the package documented for this task. Review its source and supported platforms before starting.

The publisher has not provided a complete memory target for this task. Check its hardware guidance before starting.

CURRENT SETUP

openjev · official

This setup uses vLLM. The exact checkpoint format, context, and runtime overhead determine its memory need; a weight-file size alone is not a GPU recommendation.

View weight source

THE WORKFLOW

Set up openjev on your machine

Pick your operating system. Every command below is for the selected package and runtime.

Install vLLM and confirm an 80 GB GPU

Prepare a Linux host with an 80 GB GPU. Follow the official vLLM installation instructions for that platform to install vLLM and its dependencies so the vllm command is available in your shell.

Download the model repository

Clone or download openjev/openjev from the Hugging Face Hub into your working directory so the model files are available locally as ./openjev. Populate the openjev directory with the repository's files, including openjev/helper/shim.py.

Start the vLLM server

Run the publisher-provided serve command. It starts the server on 127.0.0.1:8000, exposes the model under the served name qwen, enables prefix caching with a 16384 token limit, uses up to 90% of GPU memory, limits each prompt to one image, trusts remote code, allows 256 sequences and 64 logprobs, selects the Triton gdn-prefill backend, and applies fp8 quantization.

vllm serve ./openjev --host 127.0.0.1 --served-model-name qwen --port 8000  --enable-prefix-caching --max-model-len 16384 --gpu-memory-utilization 0.90  --limit-mm-per-prompt '{"image":1}' --trust-remote-code --max-num-seqs 256  --max-logprobs 64 --gdn-prefill-backend triton --quantization fp8

Start the decision API in front of vLLM

In a second terminal, run the shim that turns the OpenAI-compatible vLLM server into the typed decision API on 127.0.0.1:3000. It needs to reach the vLLM server over HTTP. Note: the VLLM environment variable must contain http://, while the TOKENIZER, READOUT_, SHIM_, and --host/--port arguments are set for the readout configuration shown. Run this command from the directory that contains the openjev folder so the relative TOKENIZER path resolves.

VLLM=http://localhost:8000/v1 TOKENIZER=./openjev \
READOUT_T=0.85 READOUT_NOUL_T=1.829074 READOUT_NOUL_BIAS=0 \
READOUT_TARGETED=1 READOUT_INSTR_STYLE=pyrepr SHIM_STAGGER=1 \
python openjev/helper/shim.py --host 127.0.0.1 --port 3000

Send a first typed decision request

Use the document's example request to produce a model response. It asks three typed decisions about one customer message and specifies the labels for the choice question.

curl -s http://localhost:3000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "openjev",
  "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": null, "shipping": null, "technical": null}},
    "angry":   {"type": "noul",   "instructions": "Is the customer angry?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?",
                "criteria": ["can wait", "this week", "today", "right now"]}
  }
}'

SUCCESS CHECK · openjev

Check the first local reply

Ask a short question with a known answer. Confirm the selected local model responds and verify the answer yourself before using it for private work.

This is a source-linked setup, not a YouRunAI hardware test. Confirm your exact runtime version, package, and output before relying on it.

Back to setup steps

When it doesn’t go to plan

The shim cannot reach the vLLM server.

Set the VLLM environment variable to http://localhost:8000/v1 and keep the vLLM server running on port 8000.

The shim cannot find the tokenizer.

Run the shim from the working directory that contains the openjev folder, so TOKENIZER=./openjev resolves.

vLLM fails to load the model.

Confirm the openjev/openjev repository files are present in ./openjev and that vLLM startup includes --trust-remote-code, --quantization fp8, and --gdn-prefill-backend triton exactly as shown.

REFERENCE LIBRARY

Sources and files

Original instructions, model files, and compatibility notes behind this setup.

2 SOURCES

01Publisher model card and weightshuggingface.co 02Publisher run instructionshuggingface.co

The model behind this workflow

openjev

MY HARDWARE

Will it run on your machine?

Save your machine to see a personalized rating and its reasoning.

Add my hardware

Keep exploring

Text

Beginner

Set up a private AI chat assistant

Install a local runtime, run Qwen3.5 9B, confirm responses, and know when to choose the smaller 4B package.

Build a private local coding assistant

Connect an open coding-capable model in Ollama to Cline, run a small repository task, and review the result locally.

SOFTWAREOllama + Cline

Generate images on your own GPU

Use the official FLUX.2 Klein 4B ComfyUI template with exact model files, a first prompt, and an output check.

SOFTWAREComfyUI

LOCALRENTED GPU

Verified source