Running a large language model on your own machine, with no cloud API in the loop — "Local LLM" — has become practical over the past few years. It's the product of two currents converging: better open-weight models, and model-shrinking via quantization techniques.

OllamaOllama
Hugging FaceHugging Face

Image: Ollama / Hugging Face logos, Wikimedia Commons

The Rapid Rise of Open-Weight Models

The open-weight landscape as of 2026 keeps reshuffling on a timescale of months. DeepSeek pushes high efficiency across its Flash and Pro variants; Qwen has climbed from "a serious contender" to being rated "top-tier at graduate-level reasoning," on the strength of how broad its hardware and model-size options are. Meta's Llama 4 delivers a context window as long as 10 million tokens for an open model. More recently, Z.ai's GLM-5.2 made a big jump in coding-agent performance and was integrated into agent frameworks within days of release, and Kimi K2.7 Code HighSpeed claims a 6x speedup on multimodal coding reasoning — the pace of change is extremely fast.

Benchmark results mark an important shift too. On SWE-bench-style coding evaluation suites, the gap between the top open-weight models and the leading commercial models has narrowed to single-digit percentage points, and some argue that gap has effectively closed for everyday engineering work.

The Foundation: The Attention Mechanism

Every major large language model today — including the ones run through Ollama — is built on the transformer architecture, and its core is self-attention. Given query Q, key K, and value V matrices derived from the input sequence, it computes scaled dot-product attention as follows.

\text{Attention}(Q, K, V) = \text{softmax}\!\left( \frac{QK^{\top}}{\sqrt{d_k}} \right) V

QK^\top represents how related (how similar) tokens are to each other, and dividing by \sqrt{d_k} (the square root of the key dimension) prevents the dot products from growing so large that softmax's gradient vanishes. Repeating this operation across a model's full parameter count and layer depth is what transformer inference actually is — and how far the bulk of those parameters (the weight matrices) can be shrunk is exactly the role quantization plays, covered next.

The Basic Form of Linear Quantization

The idea underlying GGUF, AWQ, and GPTQ alike is linear quantization: converting a floating-point weight x into a lower-bit integer q.

q = \text{round}\left( \frac{x}{s} \right) + z

s is the scale (the real-number width per step) and z is the zero-point (where, in the integer representation, real-number zero actually lands) — both calibration parameters, and how each method decides these two values, and at what granularity (whole model, per-layer, per-weight), is what drives the performance differences between them.

The Maturing of Quantization: GGUF, AWQ, GPTQ

Alongside better model performance, quantization — shrinking model size — has become just as essential. Quantization methods like GGUF, AWQ, and GPTQ have managed to cut model size by roughly 70% while keeping the accuracy loss under 2%, which is how a 32-billion-parameter-class model now fits into 16 GB of memory. Local inference on typical consumer hardware now reaches 70–85% of the quality of a top-tier model, at zero additional cost per request.

The three formats each have their own strengths. GGUF (formerly GGML) is the native format for llama.cpp and its ecosystem (Ollama, LM Studio, and the like), strong at hybrid CPU+GPU inference. GPTQ excels at raw inference speed on GPU-only setups, and AWQ is generally regarded as the strongest option for accuracy-per-size at 4-bit quantization. For general local-development use, GGUF via Ollama is often recommended as the default choice.

The Technical Differences Between the Three Quantization Methods

GGUF, AWQ, and GPTQ are all quantization methods that shrink model weights from 16-bit/32-bit floating point down to around 4 bits, but they get there through technically different means.

GPTQ treats quantization as an optimization problem. It approximately uses second-order information from the loss function (the Hessian) to judge which weights' rounding errors matter most to the output, then, as each weight gets quantized in turn, distributes the resulting error as compensation across the not-yet-quantized weights in the same row. It's a method focused on GPU inference performance, and quantizing a 7B-parameter-class model takes roughly 2–4 hours on a single A100 GPU.

AWQ, in contrast, focuses not on the quantization step itself but on figuring out "which weights actually matter." It runs a calibration phase that observes the model's actual activations, identifies the "salient" weights that strongly affect output quality, and aggressively quantizes everything else while keeping those weights at high precision. It needs fewer calibration samples than GPTQ (roughly 128–512, versus 2,048+ for GPTQ often), which makes it substantially faster — around 10–30 minutes for a 7B model.

GGUF (llama.cpp's native format) uses a scheme called "K-quants," assigning a different bit depth per layer — 6 bits for a high-importance layer like attention, 4 bits for a feed-forward layer, and so on — achieving higher quality per bit than naive uniform 4-bit quantization. A representative setting, Q4_K_M, is said to retain about 92% of the original model's quality.

These technical differences map directly onto which use case fits which method. AWQ for GPU-only, high-speed inference; GPTQ when a mature GPU ecosystem and a large library of pre-quantized models matter most; GGUF (via Ollama, etc.) when ease of use in a hybrid CPU/GPU environment is the priority.

llama.cpp: How the Engine Behind Local Inference Works

Behind many local LLM runtimes, Ollama included, sits llama.cpp — an inference engine written in C/C++ — and GGML, the tensor library underneath it. GGML is a lightweight, low-dependency tensor computation library roughly analogous to PyTorch or TensorFlow, representing a model's entire computation as a "computation graph." Some tensors hold actual data (like weights), while others merely represent the result of an operation between other tensors, holding no value at all until the computation is actually executed. This computation graph can be run directly on CPU, or translated into instructions for an accelerator — CUDA for NVIDIA GPUs, Metal for Apple hardware — and that portability, running the same model file across a wide range of hardware configurations, is the technical foundation underpinning the GGUF format's popularity.

The Numbers Behind the Growth

How fast this has spread shows up in concrete numbers. Ollama's monthly download count grew from 100,000 in Q1 2023 to 52 million in Q1 2026 — a 520x increase in three years. The number of GGUF-formatted models on Hugging Face, formatted for local inference, grew from 200 to 135,000 over the same period. llama.cpp, the project underpinning all of this, has passed 73,000 stars on GitHub.

The Model Release Landscape in Late 2026

Tracking Ollama's official blog, one big release has followed another just in the second half of 2026. On August 10, Meta Superintelligence Labs released its first open model, a 30-billion-parameter multimodal model called "Muse Glimmer," under an Apache 2.0 license; the very next day, August 11, NVIDIA announced "Nemotron 3.5 Lightning," a 30B model designed for multi-step agentic tasks. On June 29, an update landed making Gemma 4 up to 90% faster on Apple Silicon's MLX runtime, continuing the push on execution speed for coding-agent use cases. Ollama itself announced an $88M funding round on July 9 and reported reaching 8.9 million developer users. The open-weight model ecosystem is moving past simply "releasing models" and taking on the shape of genuine commercial infrastructure.

The Cutting Edge of Quantization: NVFP4 as a New Option

Following GGUF, AWQ, and GPTQ, research on NVFP4 — a 4-bit floating-point format designed for NVIDIA's Blackwell architecture — has grown rapidly through 2026 as a new quantization format. A June 2026 study reported that a block size of 16 offers the best accuracy/storage tradeoff, and ScaleSweep (May 2026), which optimizes initial block-scale values via a sweep search, is one of several methods now reporting that even aggressive quantization can preserve more than 93% of full-precision performance. Post-processing techniques have also emerged, like H-Scale (August 2026), which refines per-group scale values using a second-order Hessian-based approximation with zero added inference-time overhead — a sign that quantization research itself is shifting from "how much can weights be cut" toward "how to recover accuracy after cutting them."

Compression of the KV cache (the memory region holding intermediate representations of past tokens during generation, which balloons with longer context) is also progressing in parallel. SemKV (August 2026), which dynamically assigns one of two precision levels per token based on an importance score, reports achieving a 6.0x storage reduction while avoiding the "quality cliff" phenomenon where quality degrades abruptly — and 7.9x when paired with an optimized quantizer. For local inference over long contexts, KV-cache compression is becoming just as much a practical focus as weight quantization itself.

Actual Use on the home-server

This project's home-server also installs Ollama natively on the host, giving it direct GPU access, with Open WebUI connected as the front end (see "How Local AI and Private Search Work" for details). Running a general-purpose conversational model alongside a less-filtered one for other use cases only became viable, with no added cloud billing, because quantization techniques like the above matured to the point they have.

References

#LLM #Ollama #Quantization