Benchmark local LLMs: first response, generation rate and memory

Tokens per second alone cannot explain the wait after sending a request. Measure first answer content, generation rate, completion time and memory separately. This article supplies a measurement procedure and client; no real-model benchmark or hardware ranking was performed.

Try synthetic stream timing in your browser

Use the browser Lab to compare event arrival times without installing Python or a model. It replays synthetic events through the unchanged client's timing rules; it does not call Ollama or measure hardware. Open the timing Lab.

Change the first-arrival delay, thinking duration (only in the thinking-then-answer scenario), and answer-fragment interval. The timeline separates thinking, answer, done, error and empty-fragment arrivals. Dots at the same time overlap; the event table retains every event. Save A, change a condition, then run B. The delayed-arrivals preset changes client timing while keeping the server counters fixed.

The teaching fixture has three answer fragments, a final event one interval later, and 50 ms of client cleanup after the stream. Thinking adds one event before the answers. The no-answer and final-answer cases keep the same synthetic final-event schedule. Successful fixtures supply 20 generated tokens and 2 seconds of server generation time (10 token/s); these are stipulated synthetic inputs, not a physical model of generation or a claim that three fragments contain 20 tokens. The zero-duration scenario substitutes zero and makes the rate unavailable. Arrival controls do not predict cold starts, caching or real network effects.

TTFC starts at the first nonempty answer, even when it occurs in the final event. Thinking is separate. No answer produces an unavailable TTFC; interrupted streams and API errors mark the stream failed and leave all success metrics unavailable. A dash means unavailable, never zero. Server duration is shown separately because it does not establish where generation occurred on the client timeline. The browser's own compute time is not an inference benchmark.

Inspect the browser adapter and its unchanged client copy. Together they also support optional local reproduction of this synthetic exercise. The real-endpoint procedure below is a separate, optional experiment requiring your own model. No memory, answer quality, exact token latency or hardware throughput is measured by the browser Lab.

Define timing boundaries

Diagram 1 · Use the button to switch views

TTFT normally means time to the first token, but an HTTP fragment is not necessarily one token. This script reports TTFC, time to the first nonempty response fragment. It separately records first thinking content when returned. This client metric includes queueing, loading, prompt processing and transport.

Generation rate instead uses the server's generated-token count and generation duration. Ollama durations are nanoseconds; the Generate API definitions were checked on 2026-09-07.

This is not throughput over the whole request. Different tokenizers also mean token/s alone is not a fair comparison of Japanese answer volume or quality.

Fix the conditions

Record model digest, quantization, Ollama version, CPU/GPU, RAM/VRAM, power limits, concurrency and prompt. Preserve returned counts because identical text can tokenize differently. Separate the first request with an unloaded model from subsequent resident-model requests.

Repeated prompts may use caches. Treat resident/same-prompt and resident/new-input tests separately. Record execution order because thermal conditions and background work can bias comparisons.

Optional: measure a real endpoint

Place llm_benchmark.py beside prompt.txt containing your comparison text. It uses only Python's standard library. Replace YOUR_MODEL with an installed model:

python3 llm_benchmark.py --model YOUR_MODEL --prompt-file prompt.txt --runs 5 > runs.jsonl

The default endpoint is 127.0.0.1:11434/api/generate. Requests use temperature=0, seed=42, num_predict=128, num_ctx=4096 and keep_alive=5m. These are experiment settings, not a guarantee of determinism or suitability for every model. Check output count and done_reason for early termination or different thinking behavior.

Validate outputs and failures

Each JSON line includes ttfc_s, first_thinking_s, client_total_s, generation_tokens_s and server counts/durations. No answer fragment produces null TTFC; zero generation duration produces null throughput. Interrupted streams and API errors are status=error, never zero-second successes.

Parsing, timing fields and failure handling were tested with synthetic streaming responses, not actual Ollama generation. Start with five trials to validate the procedure, then increase repetitions with explicit warm-up rules. Report success count, median and range; do not present a small-sample p95 as precise.

Measure memory separately

The script does not measure memory. The running-model API exposes residency information including size_vram, but not whole-system usage or transient peaks. Sample OS/GPU metrics consistently during idle and generation, stating the observation interval.

Process RAM, page cache and GPU allocations are different quantities; adding them can double-count. Check boundaries particularly on unified-memory systems. A placement change such as partial CPU processing should not be described purely as a difference in model capability.

Build a useful comparison table

Include model/quantization, input/output tokens, residency/cache conditions, successes, median TTFC, median generation rate, total time and memory metric/sample interval. Evaluate answer correctness separately. For energy, connect the same workload to Wh using server power measurement.