Benchmark local LLMs: first response, generation rate and memory

Tokens per second alone cannot explain the wait after sending a request. Measure first answer content, generation rate, completion time and memory separately. This article supplies a measurement procedure and client; no real-model benchmark or hardware ranking was performed.

Define timing boundaries

Diagram 1 · Use the button to switch views

TTFT normally means time to the first token, but an HTTP fragment is not necessarily one token. This script reports TTFC, time to the first nonempty response fragment. It separately records first thinking content when returned. This client metric includes queueing, loading, prompt processing and transport.

Generation rate instead uses the server's generated-token count and generation duration. Ollama durations are nanoseconds; the Generate API definitions were checked on 2026-09-07.

This is not throughput over the whole request. Different tokenizers also mean token/s alone is not a fair comparison of Japanese answer volume or quality.

Fix the conditions

Record model digest, quantization, Ollama version, CPU/GPU, RAM/VRAM, power limits, concurrency and prompt. Preserve returned counts because identical text can tokenize differently. Separate the first request with an unloaded model from subsequent resident-model requests.

Repeated prompts may use caches. Treat resident/same-prompt and resident/new-input tests separately. Record execution order because thermal conditions and background work can bias comparisons.

Read the stream

Place llm_benchmark.py beside prompt.txt containing your comparison text. It uses only Python's standard library. Replace YOUR_MODEL with an installed model:

python3 llm_benchmark.py --model YOUR_MODEL --prompt-file prompt.txt --runs 5 > runs.jsonl

The default endpoint is 127.0.0.1:11434/api/generate. Requests use temperature=0, seed=42, num_predict=128, num_ctx=4096 and keep_alive=5m. These are experiment settings, not a guarantee of determinism or suitability for every model. Check output count and done_reason for early termination or different thinking behavior.

Validate outputs and failures

Each JSON line includes ttfc_s, first_thinking_s, client_total_s, generation_tokens_s and server counts/durations. No answer fragment produces null TTFC; zero generation duration produces null throughput. Interrupted streams and API errors are status=error, never zero-second successes.

Parsing, timing fields and failure handling were tested with synthetic streaming responses, not actual Ollama generation. Start with five trials to validate the procedure, then increase repetitions with explicit warm-up rules. Report success count, median and range; do not present a small-sample p95 as precise.

Measure memory separately

The script does not measure memory. The running-model API exposes residency information including size_vram, but not whole-system usage or transient peaks. Sample OS/GPU metrics consistently during idle and generation, stating the observation interval.

Process RAM, page cache and GPU allocations are different quantities; adding them can double-count. Check boundaries particularly on unified-memory systems. A placement change such as partial CPU processing should not be described purely as a difference in model capability.

Build a useful comparison table

Include model/quantization, input/output tokens, residency/cache conditions, successes, median TTFC, median generation rate, total time and memory metric/sample interval. Evaluate answer correctness separately. For energy, connect the same workload to Wh using server power measurement.