FastInfer — an inference performance laboratory

Install

Probe the machine before you believe a number.

Python 3.11 or newer. torch and onnxruntime are optional — whatever is missing is reported as an unsupported backend, with the import error, rather than crashing the run.

Install into a virtualenv

Editable, with the development extras that carry the test suite.

python3.11 -m venv .venv          # 3.11+; 3.14 also works
.venv/bin/pip install -e ".[dev]"

Ask what this machine supports

Every precision is probed by executing it. Nothing is inferred from a version number.

fastinfer probe
Run a benchmark

Each run writes runs/<run_id>/run.json and a Markdown rendering of it. The JSON is the artefact.

fastinfer benchmark configs/cpu_baseline.yaml

Inspect, compare, or serve

Six commands in total. No dashboard, and nothing to log into.

fastinfer inspect  runs/<run_id>               # summarize a run
fastinfer compare  runs/<run_a> runs/<run_b>   # diff two runs
fastinfer batch-server --max-batch-size 16 --max-wait-ms 10

What it is

A harness that measures inference, and a scheduler worth measuring.

Most inference “benchmarks” are timing scripts: a loop, a stopwatch, a number in a README. They omit the things that decide whether the number means anything — whether the device finished the work before the clock stopped, whether the sample was large enough to support a p99, whether the faster configuration still computes the right answer.

FastInfer is built the other way round. Its engineering core is an asynchronous dynamic batching scheduler and a set of standardized backend adapters that let PyTorch eager, torch.compile and ONNX Runtime be compared on identical inputs under identical timing rules. Everything else — the environment probe, the percentile gating, the correctness checker — exists to stop the report saying more than the run supports.

It ships as a CLI with six commands, JSON and Markdown reports, and a test suite of 317 tests. There is no dashboard, and it does not claim to make anything faster.

Architecture

Eight modules, one boundary.

Inputs and outputs cross backend boundaries as NumPy arrays. That single rule is what lets the correctness checker diff a torch run against an ONNX Runtime run without either knowing the other exists — and what makes a new runtime a subclass rather than a rewrite.

backends/

Adapters for PyTorch eager, torch.compile and ONNX Runtime behind one interface, plus capability probing that executes each mode rather than reading a version string.

batching/

The dynamic batching scheduler and the HTTP server that exposes it. Queueing and compute latency are tracked separately.

workloads/

Reference models — an MLP, a small CNN, a decoder transformer — sized so CI can run them, with deterministic weights from a seed.

metrics/

Monotonic timing, warmup-excluding collection, and nearest-rank percentiles that are withheld when the sample cannot support them.

hardware/

CPU, accelerator, RAM, OS and library capture. On unified-memory hardware it records memory_model: unified and emits no VRAM figure.

correctness/

Max absolute error, mean absolute error and task agreement at per-precision tolerances, plus token-for-token KV-cache validation.

reporting/

The run schema, JSON and Markdown writers, and run comparison that refuses to rank an invalid configuration as a winner.

cli.py

benchmark · compare · inspect · batch-server · probe · list

The scheduler

Requests arrive alone. Accelerators are efficient in groups.

A batch is dispatched when it fills the row budget, or when the oldest request in it has waited max_wait_ms. Keying the deadline to the oldest member is what bounds the tail: keyed to the newest arrival, a steady trickle would starve the first request forever.

Compatibility

Only compatible requests share a batch — by default every input’s non-batch shape and dtype. Generative workloads bucket by padded sequence length instead, because exact-length keys make batching impossible under real traffic.

Compute runs in a thread

The model is a blocking call. On the event loop it would stall intake for its whole duration, so the queue could never grow and batching would collapse to batch-size-one under exactly the load it exists to help.

Two clocks, not one

Queueing and compute time are recorded per request. A slow p99 means something different depending on which half it came from, and the two have opposite fixes.

Fairness

Dispatch runs oldest-first across buckets. Insertion order lets a bucket that always keeps a remainder hold its slot indefinitely — a stream of short prompts would permanently starve the long-prompt bucket.

What happens when things go wrong
Situation	Behaviour
Model raises	Every future in the batch receives the exception; the batch is recorded with its error; the scheduler keeps running.
Cancelled before dispatch	Dropped and counted. It never reaches the model.
Cancelled mid-flight	Compute cannot be recalled — it completes, the result is discarded, and it is counted rather than hidden.
Queue at capacity	`QueueFullError` immediately — backpressure, surfaced as HTTP 503. Not an unbounded queue.
Shutdown	Drains queued work, or fails it explicitly. No request is left awaiting a future nobody will complete.

Measured · Apple M4 Pro

The same scheduler, opposite verdicts.

Identical code and identical experiment design — Poisson arrivals replayed against an unbatched control — on two different workloads. The control is the point: a throughput number with nothing to be a gain over means nothing.

CPU · dense MLP · not saturated

1.00×

Throughput unchanged, at 11.5× the median latency. The model served a request in 0.36 ms, so at 1,200 req/s the hardware was never the bottleneck. Every batch was dispatched by the timer, never by filling — the scheduler had nothing to recover and its wait was pure added delay.

Batched throughput: 1,159.7 req/s
Unbatched control: 1,164.9 req/s
Median latency: 7.60 ms vs 0.66 ms
Mean batch size: 10.53

MPS · small CNN · saturated

6.35×

Throughput and latency improved together — median latency fell to 0.03× — because the unbatched control saturated at 869 req/s against 6,000 offered and its queue collapsed into a 1.4-second p99. On this device batch-1 yields 978 items/s and batch-128 yields 36,973.

Batched throughput: 5,520.9 req/s
Unbatched control: 869.2 req/s
Median latency: 21.59 ms vs 751.94 ms
Control queue p95: 1,398.74 ms

The correctness gate

The fastest configuration was the one that failed.

An optimization that changes the answer is not an optimization. Every configuration is compared against a baseline before its numbers mean anything.

Small CNN · throughput in items/s · higher is better
Configuration	b=128	Gate
onnxruntime · CoreML · fp32	92,468	FAIL
torch_compile · mps · fp32	40,555	pass
torch_eager · mps · fp32	35,432	pass
onnxruntime · cpu · fp32	9,468	pass
torch_eager · cpu · fp32	539	pass

Why it failed

Asking the CoreML provider for fp32 does not get you fp32 arithmetic — it computes in reduced precision internally. Mean absolute error was 2.98e-05 against a tolerance of 1e-05. Every argmax still matched, so the configuration may well be fine for a classifier — but it is not the fp32 the table claims.

The report prints those rows as FAIL, fastinfer benchmark exits non-zero, and fastinfer compare refuses to call the configuration a winner — it reports the speed for reference and marks the verdict invalid. The tolerance was not widened.

Implemented · verified · unavailable

Three different claims, listed separately.

“Verified” means the mode was executed on the machine above — not that it is expected to work. Nothing here is inferred from a version number.

Capability	Implemented	Verified here	Note
PyTorch eager · fp32 / fp16 / bf16	yes	yes	CPU and MPS
`torch.compile`	yes	yes	Slower than eager on these small models
ONNX Runtime · CPU provider	yes	yes	fp32, fp16, INT8
ONNX Runtime · CoreML provider	yes	fails gate	Reduced precision internally
INT8 · torchao / ORT quantizer	yes	yes	Converted-layer count is verified, not assumed
INT4 · torchao	yes	unavailable	torchao installed, but its low-bit kernel package is not available for this platform — reported as a skip with the real `ImportError`
CUDA devices	plumbed	no hardware	Probes report unsupported with the reason; no CUDA numbers are published
KV-cache generation	yes	yes	torch backends; 6.37× on CPU, unreliable on MPS at this size
Generation on ONNX Runtime	by design, no	n/a	Cached decoding needs a different exported graph; the backend declines rather than mislabelling an uncached run

Methodology

The rules the numbers had to survive.

Monotonic clock

perf_counter_ns, with a device synchronize before the clock stops — otherwise an async backend reports the cost of queueing work, not doing it.

Warmup excluded

Recorded, then excluded from every statistic. Kept because the gap to steady state is itself a result; excluded because averaging it in describes nothing.

Repeats

Independent blocks, reported separately. If blocks of the same work disagree by more than their own spread, the machine was busy and the run is not trustworthy.

Percentiles gated

Nearest-rank, and only reported when the sample supports them — p95 needs 20 samples, p99 needs 100. Below that the value is null with a recorded reason, never quietly the maximum.

Limits

What these numbers are not.

Every figure on this page describes one machine, one OS, one set of library builds, recorded in full inside the run records. They do not generalize to other hardware, other model sizes or other batch regimes, and they are not backend-versus-backend verdicts.

A concrete illustration from the same run: PyTorch at batch 1 takes a single-threaded path and completes in 0.174 ms; at batch 4 — four times the work — it takes 1.690 ms, a 9.7× jump, then gets faster at batch 16 (1.354 ms). The three measurement repeats agreed to within 8%, so it is a real threshold on this machine and this build, not contention. Carried anywhere else, it is not evidence.

FastInfer does not accelerate inference. Whether batching helps depends on the model, the hardware and the arrival rate — which is why it measures both regimes against a control instead of quoting a number.