Compare the same search candidates before and after reranking. Use a supplied example or upload the results from your private document search, then export the improved order.
Conservative starting points for this small CPU-first app. GPU memory is not required; the listed memory is system/unified RAM.
| Machine | Qwen3 Reranker 0.6B |
|---|---|
| NVIDIA PC · 8 GB RAM | Runs on CPU; 16 GB RAM gives more room for other apps |
| AMD PC · 8 GB RAM | Runs on CPU; no ROCm or Vulkan dependency |
| Intel PC · 8 GB RAM | Runs on CPU; an Arc GPU is not required |
| Apple Silicon · 8 GB unified memory | Runs on CPU; 16 GB unified memory gives more headroom |
| CPU only · 8 GB RAM | Suitable for the supplied small collection; larger batches use more memory |
These are memory estimates for short passages and one local user, not speed measurements. The guide uses CPU on every OS for a consistent first run.
| Machine | Qwen3 Reranker 0.6B |
|---|---|
| CPU / Apple Silicon · 8–16 GB RAM | Start at 2,048 tokens per query + passage; batch size 1 |
Long candidates are truncated at the tokenizer limit. Use the short overlapping passages from the search workflow so the answer stays inside the scored input.
Compare the same search candidates before and after reranking. Use a supplied example or upload the results from your private document search, then export the improved order.
This setup uses the package documented for this task. Review its source and supported platforms before starting.
Save your machine in My Hardware to get an automatic starting choice. You can always choose any package yourself.
Start at 2,048 tokens per pair and batch 1. More tokens or a larger batch raises RAM use.
Choose where you want to run it. The instructions below update to match your selection.
Native CPU inference. The browser is the same on each OS; project paths and Python executables differ.
Install uv from its official instructions, then open a new PowerShell window. The commands below create an isolated Python 3.11 environment; there is no need to change your system Python.
uv installationpowershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"Use a new folder named yourunai-rerank in your home directory. These commands run in PowerShell. This starter deliberately uses CPU inference: its small model works with NVIDIA, AMD, Intel, Apple or no GPU. No CUDA installation is needed. Leave this window in the project folder for the remaining commands.
uv virtual environments PyTorch installation choicesNew-Item -ItemType Directory -Force -Path "$env:USERPROFILE\yourunai-rerank" | Out-Null
Set-Location "$env:USERPROFILE\yourunai-rerank"
uv venv --python 3.11
uv pip install --python .\.venv\Scripts\python.exe torch --index-url https://download.pytorch.org/whl/cpu
uv pip install --python .\.venv\Scripts\python.exe sentence-transformers==6.1.0 gradio==6.28.0Right-click the download link below and choose Save Link As. Save the .py file directly in the project folder created above; keep its .py extension, not .txt. This complete YouRunAI starter creates the browser interface and output folders. You can inspect it in a text editor. Model weights download separately when the runtime needs them.
Download rerank-results.pyRun this in PowerShell, leave it open, and open http://127.0.0.1:7861 after the weights finish downloading. Port 7861 lets this app run beside the document-search example.
.\.venv\Scripts\python.exe rerank-results.pyLeave the upload empty and click Rerank results. The sample asks how to return an item, with shipping, returns and support passages in an intentionally unhelpful initial order. Compare the Before and After tables; the return policy should move above unrelated opening hours.
In the document-search app, run a query and download candidates.json. Upload that file here and click Rerank results. The file must contain a query string and 1–20 candidates, each with text and an optional file name. Reranking can reorder those candidates but cannot recover a relevant document the first search never found.
Read the top passage and confirm it answers the query. Check its Before rank against its new position; good candidates may already have been first. Download reranked.json and find the same file in outputs/reranked.json. Scores are relative relevance logits, not percentages. Keep the query with the ranked passages if you pass them to a local assistant.
Press Ctrl+C in the running PowerShell window to stop the app. Your files stay in yourunai-rerank. Next time run the commands below, then reopen http://127.0.0.1:7861. Dependencies and downloaded weights are reused. Keep this address local; there is no public sharing link.
Set-Location "$env:USERPROFILE\yourunai-rerank"
.\.venv\Scripts\python.exe rerank-results.pyThe after table and saved JSON contain the original filenames/passages, their old ranks, and new relevance scores. You can compare the order rather than trusting an unexplained score.
Keep the Python process open and wait until it prints the local URL. If another app uses that port, stop that app or run these examples one at a time.
The first run downloads the publisher model to the Hugging Face cache. Watch the Terminal/PowerShell progress and allow the download to finish; later starts reuse it.
Run the installation command from this guide again in the project folder. Launch with the .venv Python path shown here so you use the same environment.
Close memory-heavy apps and other models. Keep the supplied short input limits and batch size. The small-model CPU route needs system RAM, not GPU VRAM.
Use candidates.json from the search guide, not vectors.npy or passages.json. It needs a nonempty query and 1–20 candidate objects, each containing a nonempty text field.
Improve the initial retrieval query or include more candidate passages first. The reranker only reorders the candidate set you gave it.
Original instructions, model files, and compatibility notes behind this setup.
Source-backed package and platform-route plan for private local chat with Qwen3.5 · 9B. Preserves the official Ollama 9B package, a smaller 4B family fallback, and a clearly labeled community uncensored GGUF option.
Run Qwen3.8 27B with Ollama, connect it to Cline in VS Code, and make one reviewable code change in a tiny local project.
Generate images from text prompts in ComfyUI with the fast distilled or more adjustable base 4B model. Save your results and reuse the visual workflow.