# GPU Garden > The authoritative, zero-maintenance client-side AI model hardware matchmaker, exact VRAM sizing calculator, and cloud TCO break-even engine. > Website: https://gpugarden.com/ > Comprehensive LLM Reference & Formulas: https://gpugarden.com/llms-full.txt ## Overview GPU Garden (https://gpugarden.com) solves the primary pain point in local AI inference: preventing Out-Of-Memory (OOM) crashes and guiding cost-effective hardware choices. It provides mathematically verifiable VRAM sizing across popular open-weight Large Language Models (LLMs), multimodal vision models, and diffusion image generators. All calculations execute transparently in the user's browser without telemetry or tracking. GPU Garden is grounded in mathematical honesty, compute sovereignty, and ethical affiliate alignment. --- ## Canonical Mathematical Formulas ### 1. Model Weights Memory $$\text{Weights (GB)} = \text{Parameters (Billions)} \times \left(\frac{\text{BPW}}{8}\right) \times \text{Overhead}$$ * **FP16 / BF16 (Full Precision)**: BPW = 16.0, Overhead = 1.02 (zero degradation, maximum VRAM). * **Q8_0 (8-bit Quantization)**: BPW = 8.50, Overhead = 1.03 (near-lossless, ~99.9% quality). * **Q6_K (6-bit GGUF)**: BPW = 6.56, Overhead = 1.04 (high fidelity). * **Q5_K_M (5-bit GGUF)**: BPW = 5.50, Overhead = 1.04 (sweet spot for 12GB–16GB VRAM). * **Q4_K_M (4-bit GGUF - Community Standard)**: BPW = 4.50, Overhead = 1.05 (>98% quality retention at ~28% FP16 VRAM). * **AWQ 4-Bit**: BPW = 4.25, Overhead = 1.06 (optimized for vLLM & TensorRT-LLM throughput). ### 2. Attention KV-Cache Memory $$\text{KV-Cache (GiB)} = \frac{2 \times \text{Layers} \times \text{KV\_Heads} \times \text{Head\_Dim} \times \text{Context\_Tokens} \times \text{Bytes\_Per\_Token}}{1024^3}$$ * **FP16 KV Cache**: 2.0 bytes / token / layer (standard llama.cpp and vLLM). * **FP8 KV Cache**: 1.0 byte / token / layer (50% memory savings on RTX 40 / Ada Lovelace). * **Q4 KV Cache**: 0.5 bytes / token / layer (75% memory savings for massive 64k–128k context). * **Multi-Head Latent Attention (MLA - DeepSeek V2/V3)**: $$\text{MLA KV-Cache (GiB)} = \frac{2 \times \text{Layers} \times \text{Latent\_Dim} \times \text{Context\_Tokens} \times \text{Bytes\_Per\_Token}}{1024^3}$$ ### 3. Activation & CUDA Driver Buffer * **CUDA Runtime & OS Driver Baseline**: 1.50 GB reserved memory (0 GB for Apple Metal / CPU allocation). * **Peak Transient Activation Overhead**: $$\text{Activation (GB)} = \max\left(0.40\text{ GB}, \frac{\text{Context\_Tokens}}{32768} \times \frac{\text{Hidden\_Dim}}{4096} \times 0.70\text{ GB}\right)$$ where $\text{Hidden\_Dim} = \text{Head\_Dim} \times \text{Heads}$ (e.g. $128 \times 32 = 4096$ on LLaMA 8B). ### 4. Apple Silicon Unified Memory Invariant * **Allocatable Metal Threshold**: macOS caps default single-process Metal allocation at ~75% of total physical unified memory (e.g., 64GB Unified Memory provides ~48.0 GB safe allocatable headroom). --- ## Authoritative Model VRAM Requirements Table Exact total VRAM requirements calculated at standard 8,192 token context with 1.50 GB activation/driver buffer: | Model Family & Architecture | Parameters | 4-Bit (Q4_K_M) | 8-Bit (Q8_0) | 16-Bit (FP16) | Minimum Hardware Sweet Spot | | :--- | :--- | :--- | :--- | :--- | :--- | | **Meta LLaMA 3.3 70B Instruct** | 70.6B Dense (GQA) | 44.1 GB | 79.1 GB | 148.5 GB | Dual RTX 3090 (48GB) / Mac 64GB | | **Alibaba Qwen 2.5 Coder 32B** | 32.5B Dense (GQA) | 21.6 GB | 37.6 GB | 69.8 GB | NVIDIA RTX 4090 / RTX 3090 (24GB) | | **Alibaba Qwen 2.5 72B Instruct** | 72.7B Dense (GQA) | 45.4 GB | 81.4 GB | 152.9 GB | Dual RTX 3090 (48GB) / Mac 64GB | | **Alibaba Qwen 2.5 14B Instruct** | 14.7B Dense (GQA) | 10.9 GB | 18.2 GB | 32.9 GB | NVIDIA RTX 4070 Ti Super (16GB) | | **Alibaba Qwen 2.5 7B Instruct** | 7.61B Dense (GQA) | 6.3 GB | 10.1 GB | 17.6 GB | NVIDIA RTX 3060 (12GB) / RTX 4060 (8GB) | | **Meta LLaMA 3.1 8B Instruct** | 8.03B Dense (GQA) | 7.6 GB | 11.5 GB | 19.4 GB | NVIDIA RTX 4060 (8GB) / RTX 3060 (12GB) | | **Meta LLaMA 3.2 11B Vision** | 13.8B Multimodal | 10.5 GB | 17.1 GB | 30.7 GB | NVIDIA RTX 4070 (12GB) / 16GB | | **Meta LLaMA 3.2 3B Instruct** | 3.21B Dense (GQA) | 4.3 GB | 5.8 GB | 9.0 GB | Entry 6GB–8GB GPUs / Apple M-series | | **Meta LLaMA 3.2 1B Instruct** | 1.23B Dense (GQA) | 2.5 GB | 3.1 GB | 4.3 GB | Ultra-budget 4GB GPUs / Raspberry Pi | | **DeepSeek V2.5 / V3 MoE** | 236B (21B active) | 143.5 GB | 260.0 GB | 490.0 GB | 4x RTX 3090 / Apple M2/M3 Ultra 192GB | | **DeepSeek Coder V2 Lite** | 15.7B (2.4B active)| 12.0 GB | 19.8 GB | 35.8 GB | NVIDIA RTX 4070 Ti Super (16GB) | | **FLUX.1 [dev] Image Generator**| 12.0B DiT Flow | 12.0 GB | 18.5 GB | 26.5 GB | NVIDIA RTX 3060 12GB (NF4) / RTX 4090 24GB | | **SDXL 1.0 Latent Diffusion** | 2.6B UNet | 6.8 GB | 8.2 GB | 11.5 GB | NVIDIA RTX 3060 (12GB) / RTX 4060 (8GB) | --- ## Hardware Capacity & Bandwidth Reference | Hardware Platform | VRAM Capacity | Memory Bus | Bandwidth | Peak LLM Speed (Q4) | Primary Sweet Spot | | :--- | :--- | :--- | :--- | :--- | :--- | | **Dual NVIDIA RTX 3090** | 48 GB GDDR6X | $2 \times 384$-bit | 1,872 GB/s | ~18 tok/s (70B) | Best-value 70B homelab workstation ($1,400) | | **NVIDIA GeForce RTX 4090** | 24 GB GDDR6X | 384-bit | 1,008 GB/s | ~42 tok/s (32B) | World's fastest consumer inference card | | **NVIDIA GeForce RTX 3090** | 24 GB GDDR6X | 384-bit | 936 GB/s | ~34 tok/s (32B) | Used flagship value champion (~$650–$750) | | **Apple M3/M4 Max (64GB)** | 64 GB Unified | 512-bit | 300–400 GB/s | ~14 tok/s (70B) | Silent, low-power 48GB allocatable workstation | | **Apple M2/M3 Ultra (128GB)**| 128 GB Unified | 1024-bit | 800 GB/s | ~16 tok/s (70B FP16)| Massive single-pool memory for large weights | | **AMD Radeon RX 7900 XTX** | 24 GB GDDR6 | 384-bit | 960 GB/s | ~32 tok/s (32B) | Sub-$1,000 24GB ROCm powerhouse | | **NVIDIA RTX 4070 Ti Super** | 16 GB GDDR6X | 256-bit | 672 GB/s | ~45 tok/s (14B) | Modern Ada Lovelace 16GB efficiency leader | | **NVIDIA GeForce RTX 3060** | 12 GB GDDR6 | 192-bit | 360 GB/s | ~38 tok/s (8B) | Undisputed entry-level budget hero (~$280) | | **NVIDIA GeForce RTX 4060** | 8 GB GDDR6 | 128-bit | 272 GB/s | ~42 tok/s (3B) | Ada architecture, limited by 8GB VRAM cap | --- ## Cloud vs Local TCO Break-Even Rules 1. **Low-Frequency Use (< 15 hours / week)**: * **Recommendation: RENT CLOUD COMPUTE**. * Hourly rentals on RunPod ($0.34/hr for RTX 4000 Ada, $0.74/hr for RTX 4090) or Vast.ai ($0.22–$0.55/hr) cost less than hardware depreciation, electricity, and initial capital outlay. 2. **High-Frequency Use (> 25 hours / week)**: * **Recommendation: BUY DEDICATED HARDWARE**. * A single used RTX 3090 24GB (~$700) or Dual RTX 3090 rig (~$1,500) breaks even against persistent cloud rentals within 6 to 9 months, offering zero latency, zero metered charges, and absolute data privacy. --- ## Canonical Interactive Tools & Deep Links * **Interactive Matchmaker**: `https://gpugarden.com/#tab=matchmaker` * **Direct Matchmaker Deep-Link**: `https://gpugarden.com/#tab=matchmaker&model={model_id}&gpu={gpu_id}&quant={quant_id}` * **Head-to-Head GPU Showdown**: `https://gpugarden.com/#tab=compare&compare={gpuA}:{gpuB}` * **Cloud TCO Break-Even Calculator**: `https://gpugarden.com/#tab=tco` ## International Localized Editions * English (x-default): `https://gpugarden.com/` * Deutsch (German): `https://gpugarden.com/de/` * 简体中文 (Simplified Chinese): `https://gpugarden.com/zh/` * 日本語 (Japanese): `https://gpugarden.com/ja/` * Español (Spanish): `https://gpugarden.com/es/` * Français (French): `https://gpugarden.com/fr/` --- ## Machine-Readable APIs & Autonomous Agent Discovery For AI agents (Claude Code, Cursor, Codex, ChatGPT Actions, LangChain, Perplexity), GPU Garden provides zero-overhead, pre-rendered static JSON datasets and discovery contracts: * **OpenAPI 3.1.0 Specification**: `https://gpugarden.com/openapi.json` * **Agent2Agent (A2A 1.0) Manifest**: `https://gpugarden.com/.well-known/agent-card.json` * **Agent Skill Document**: `https://gpugarden.com/.well-known/agent-skills/local-fit/SKILL.md` * **Agent Skills Registry Index**: `https://gpugarden.com/.well-known/agent-skills/index.json` * **Complete Hardware & Model Catalog (JSON)**: `https://gpugarden.com/api/v1/catalog.json` * **Curated AI Models & 8k VRAM Sizing (JSON)**: `https://gpugarden.com/api/v1/models.json` * **Hardware GPU Specifications & Prices (JSON)**: `https://gpugarden.com/api/v1/gpus.json` ### Direct Sizing & Comparison Landing Pages (No Fragment Blindness) * `https://gpugarden.com/can-i-run-llama-3-3-70b/` (Can I Run Meta LLaMA 3.3 70B? Hardware Sizing & Fit Guide) * `https://gpugarden.com/can-i-run-deepseek-v3/` (Can I Run DeepSeek V3 671B? Hardware & Cloud Rental Guide) * `https://gpugarden.com/rtx-4090-local-ai-guide/` (NVIDIA GeForce RTX 4090 24GB AI Model Capability Matrix) * `https://gpugarden.com/compare/rtx-3090-vs-rtx-4090/` * `https://gpugarden.com/compare/dual-rtx-3090-vs-rtx-4090/` * `https://gpugarden.com/compare/rtx-3060-vs-rtx-4060/` * `https://gpugarden.com/compare/rtx-4070-ti-super-vs-rtx-4080-super/` * `https://gpugarden.com/compare/apple-m3-max-vs-rtx-4090/` * `https://gpugarden.com/vram/llama-3-3-70b/` * `https://gpugarden.com/vram/qwen-2-5-coder-32b/` * `https://gpugarden.com/vram/deepseek-v3/` * `https://gpugarden.com/vram/llama-3-1-8b/` * `https://gpugarden.com/vram/flux-1-dev/`