⚙️

Local LLM Inference Engine Selector v2

Weighted scoring for Ollama, vLLM, SGLang, TensorRT-LLM, MLX, ExLlamaV2, KTransformers, and more. Pick by GPU, model size, quantization, batch size, and priority.

Scenario presets

Select a preset to auto-tune weights, batch size, and recommendation.

Your setup

Weight priorities (0–5)

Last updated: 2026-08-07. See notes.

Top pick

Recommended engine

—

Score: —

VRAM needed

—

Est. tok/s

—

Select your setup to see a recommendation.

VRAM math

Model weights: —

KV cache: —

Total estimate: —

GPU VRAM: —

Engine rankings

Ollama

Score: —

One-command local prototyping and personal agents.

VRAM

—

Tok/s

—

—
OpenAI APIStructured outputGGUF

llama.cpp

Score: —

Maximum hardware compatibility and edge deployment.

VRAM

—

Tok/s

—

—
OpenAI APISpeculative decodingMulti-GPUGGUF

vLLM

Score: —

High-throughput multi-user serving.

VRAM

—

Tok/s

—

—
OpenAI APIContinuous batchingStructured outputSpeculative decodingMulti-GPUGGUFSafetensorsFP8AWQGPTQ

SGLang

Score: —

Structured generation, batch programming, and fast multi-turn.

VRAM

—

Tok/s

—

—
OpenAI APIContinuous batchingStructured outputSpeculative decodingMulti-GPUSafetensorsFP8AWQGPTQ

TensorRT-LLM

Score: —

Maximum NVIDIA throughput with pre-compiled engines.

VRAM

—

Tok/s

—

—
OpenAI APIContinuous batchingStructured outputMulti-GPUFP8AWQGPTQ

MLX (Apple Silicon)

Score: —

Native Apple Silicon performance with unified memory.

VRAM

—

Tok/s

—

—
Structured outputSafetensors

TabbyAPI

Score: —

Fast single-GPU EXL2/GPTQ serving with OpenAI-compatible API.

VRAM

—

Tok/s

—

—
OpenAI APIContinuous batchingStructured outputSpeculative decodingEXL2GPTQ

ExLlamaV2

Score: —

Maximum 4-bit throughput on a single NVIDIA GPU.

VRAM

—

Tok/s

—

—
Continuous batchingSpeculative decodingMulti-GPUEXL2GPTQ

Hugging Face Transformers

Score: —

Reference implementation and newest models.

VRAM

—

Tok/s

—

—
Multi-GPUSafetensorsFP8AWQGPTQ

llamafile

Score: —

Portable single-file model distribution.

VRAM

—

Tok/s

—

—
GGUF

KoboldCpp

Score: —

Storytelling/roleplay with built-in web UI.

VRAM

—

Tok/s

—

—
OpenAI APISpeculative decodingGGUF

Text Generation Inference (TGI)

Score: —

Production serving of Hugging Face models with sharding.

VRAM

—

Tok/s

—

—
OpenAI APIContinuous batchingStructured outputMulti-GPUSafetensorsFP8AWQGPTQ

KTransformers

Score: —

Very large MoE and dense models with CPU+GPU offloading.

VRAM

—

Tok/s

—

—
OpenAI APIMulti-GPUGGUFSafetensors

ONNX Runtime

Score: —

Cross-platform deployment and optimized transformer models.

VRAM

—

Tok/s

—

—
Multi-GPUONNX

Feature matrix

Engine Formats Platforms OpenAI API Batching Multi-GPU Structured License
Ollama GGUF Macos, Linux, Windows ✅ Basic — ✅ MIT
llama.cpp GGUF Macos, Linux, Windows ✅ Basic ✅ — MIT
vLLM Safetensors, PyTorch, AWQ, GPTQ, GGUF Linux, Windows ✅ Continuous ✅ ✅ Apache 2.0
SGLang Safetensors, PyTorch, AWQ, GPTQ, FP8 Linux ✅ Continuous ✅ ✅ Apache 2.0
TensorRT-LLM TensorRT-LLM engines Linux, Windows ✅ Continuous ✅ ✅ Apache 2.0
MLX (Apple Silicon) Safetensors, MLX Macos — Basic — ✅ MIT
TabbyAPI EXL2, GPTQ Linux, Windows ✅ Continuous — ✅ AGPL
ExLlamaV2 EXL2, GPTQ Linux, Windows — Continuous ✅ — MIT
Hugging Face Transformers Safetensors, PyTorch Macos, Linux, Windows — Basic ✅ — Apache 2.0
llamafile GGUF Macos, Linux, Windows — Basic — — Apache 2.0
KoboldCpp GGUF Windows, Linux, Macos ✅ Basic — — AGPL
Text Generation Inference (TGI) Safetensors, PyTorch Linux ✅ Continuous ✅ ✅ Apache 2.0/Hugging Face
KTransformers GGUF, Safetensors Linux, Windows ✅ Basic ✅ — MIT
ONNX Runtime ONNX Macos, Linux, Windows — Basic ✅ — MIT

Frequently asked questions

What is the fastest local inference engine for a single RTX 4090?▼

For 4-bit quantized dense models, TabbyAPI / ExLlamaV2 or TensorRT-LLM are usually fastest. For native FP16/BF16 models, vLLM and SGLang extract the highest throughput thanks to continuous batching.

Which engine should I pick for a production API?▼

Use vLLM or SGLang for open-model serving with continuous batching and OpenAI-compatible endpoints. If you only serve NVIDIA and can pre-compile engines, TensorRT-LLM can squeeze out extra throughput. TGI is also a proven production option.

Can I run Mixtral 8x22B on a 24 GB consumer GPU?▼

Not fully loaded in FP16. Use KTransformers or llama.cpp with CPU/NVMe offloading, or quantize to Q4 GGUF. Expect lower tok/s because expert layers move through system memory.

What should I use on an Apple Silicon Mac?▼

MLX is purpose-built for M-series chips and gives the best tokens-per-watt. Ollama and llama.cpp are also excellent if you already have GGUF models.

Do I need continuous batching?▼

Only if you serve multiple concurrent users. Single-user chat or local agents rarely benefit; one of the simpler GGUF engines is often easier to set up.

Are the tok/s numbers exact?▼

No. The calculator uses directional multipliers based on memory bandwidth, quantization, and engine efficiency. Real numbers depend on drivers, prompt length, KV cache, and PCIe/NVLink topology.

Inference Engine Selector v1 → VRAM Calculator → Quantization Advisor → Open-Source LLM Selector →

Scores and tok/s estimates are directional, based on public benchmarks, release notes, and typical user reports. Real-world performance depends on driver version, quantization recipe, context length, batching, CPU overhead, and PCIe/NVLink topology.

🚀 Get AI automation insights daily

15:00 MST. One-click unsubscribe.

Subscribe