Serve a private coding model on rented GPUs
Use Mistral Small 4 with its publisher-recommended two-GPU vLLM path, connect a coding client, and validate the private endpoint.
Before you begin
- Difficulty
- Advanced
- Software
- vLLM, Python
- Hardware
- The publisher shows tensor parallelism across two GPUs. Verify the selected checkpoint, GPU memory, context, and concurrency on the exact rental before paying for a long session.
Sources and files
Mistral Small 4 serving and client examples Official vLLM server documentationChoose a model for this task
The steps below use the recommended model. Alternatives have their own package and command; open their model pages before switching.
The workflow
Choose a private multi-GPU rental
Find a Linux machine with at least two suitable GPUs and enough aggregate memory for the chosen Mistral checkpoint. Check the provider’s actual GPU configuration before renting. Keep network access restricted to your own client.
Install the current vLLM package
Use a fresh Python environment and install the publisher-recommended inference server. The model card notes that mistral_common 1.11.0 or newer should be installed.
uv pip install -U vllmVerify the Mistral package version
Check the installed mistral_common version against the publisher’s minimum before downloading the large checkpoint.
python -c "import mistral_common; print(mistral_common.__version__)"Start the model server
Use the publisher’s two-GPU vLLM example as the starting point. Reduce the maximum context, batch size, or sequence count if the exact rental cannot load it. Keep the port private.
vllm serve mistralai/Mistral-Small-4-119B-2603 --max-model-len 262144 --tensor-parallel-size 2 --attention-backend FLASH_ATTN_MLA --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral --max_num_batched_tokens 16384 --max_num_seqs 128 --gpu_memory_utilization 0.8Connect a coding client
Point a trusted OpenAI-compatible client at the private vLLM endpoint on port 8000. The publisher provides a client script that reads its system prompt and sends a test request; use that example before enabling tools.
Validate the setup
Ask the model to explain a short function, then run a bounded coding task in a disposable repository. Confirm the response came from the private vLLM endpoint and review any proposed file edits before accepting them.
When it doesn’t go to plan
Confirm both GPUs are available, reduce max-model-len and concurrent sequences, or choose the publisher’s smaller NVFP4 checkpoint with its own documented settings.
Confirm the private endpoint and port, check the server log for a fully loaded model, and keep the publisher’s Mistral tool-call and reasoning parser flags.
The model behind this workflow
Mistral Small 4 · 119BWill it run on your machine?
Save your machine to see a personalized rating and its reasoning.
Add my hardwareKeep exploring
Set up a private AI chat assistant
Install a local runtime, run Qwen3.5 9B, confirm responses, and know when to choose the smaller 4B package.
Build a private local coding assistant
Connect an open coding-capable model in Ollama to Cline, run a small repository task, and review the result locally.
Generate images on your own GPU
Use the official FLUX.2 Klein 4B ComfyUI template with exact model files, a first prompt, and an output check.