LLM Inference GPU Benchmark: H100 vs A100 vs L40 in 2026

Why memory bandwidth is the bottleneck

During autoregressive decoding, each generated token requires reading the entire model's weights and the KV cache from GPU memory — once per token. The arithmetic intensity (FLOPs per byte read) is very low, so the GPU's compute units spend most of their time waiting for memory. This makes LLM inference a memory-bandwidth-bound workload: tokens per second scales roughly linearly with HBM bandwidth, not with peak TFLOPs.

This is why two GPUs with similar FLOPs can have very different LLM inference speeds, and why a GPU with lower peak compute but higher bandwidth can win. It is also why KV cache quantization and weight quantization (FP8, INT4) matter so much — they reduce the bytes that need to be read per token, directly translating bandwidth into throughput.

Specs at a glance

Note the A100's lack of FP8 — it is an Ampere-generation card and predates the FP8 Tensor Cores introduced in Hopper. This is a real disadvantage in 2026, when FP8 quantization is table stakes for cost-efficient serving.

Throughput: tokens per second (Llama-3-70B, FP16)

For a 70B model in FP16 — a memory-bandwidth-bound decode workload — throughput scales with HBM bandwidth. The H100's 3.35 TB/s versus the A100's 2.0 TB/s translates to roughly 1.5-1.7x higher tokens/sec per GPU. The L40S, despite decent FP16 TFLOPs, is crippled by its 864 GB/s bandwidth — it manages roughly one-third the A100's throughput on this workload.

The L40S is effectively disqualified for 70B-in-FP16 work because 48 GB cannot hold the ~140 GB of weights even at tensor parallelism 2. For 70B, you either use 80 GB cards (H100/A100) or quantize to FP8/INT4 to fit the L40S.

Cost per million tokens

Throughput alone is misleading — what matters is throughput per dollar. Using the mid-2026 cloud rates and the throughput numbers above, here is the effective cost per million output tokens. This is the number that should drive GPU selection.

The FP8 factor

FP8 quantization is the biggest single lever in this comparison, and it changes which GPU wins. FP8 halves the weight memory and doubles the effective Tensor Core throughput, but only Hopper (H100) and Ada (L40S) support it natively. The A100 — still common in legacy fleets — is locked out.

For a 70B model, FP8 means the weights drop from ~140 GB to ~70 GB, fitting on a single 80 GB H100 instead of two. That alone halves the cost-per-token and is the reason most 2026 production deployments of 70B models run FP8 on H100. If you are on A100s and cannot migrate, INT4/FP8 KV cache quantization and weight-only INT4 are your fallbacks, but they are less accurate and more complex than native FP8.

Multi-GPU: NVLink vs PCIe

For models that span multiple GPUs (70B+ in FP16, or large-context batches), the inter-GPU link matters. The H100 and A100 have NVLink (900 GB/s and 600 GB/s respectively), which is essential for tensor parallelism to not become a bottleneck. The L40S has PCIe only (~64 GB/s), making multi-GPU tensor parallelism painful — a 70B model split across two L40S loses significant throughput to PCIe transfers.

This is why the L40S shines for single-GPU workloads (7B-13B models) but falls off for multi-GPU tensor-parallel serving. If your model fits on one card, the L40S's NVLink absence is irrelevant; if it doesn't, the H100's NVLink fabric is essential.

Decision matrix by workload

The utilization caveat

Cost-per-token numbers assume high GPU utilization. In practice, if your traffic is bursty and your GPUs sit idle 50% of the time, the cheaper-per-hour GPU (A100, L40S) wins — you are paying for idle time either way, so minimizing hourly cost matters more than maximizing peak throughput. Conversely, if you serve steady high traffic and can keep GPUs at 85%+ utilization, the H100's throughput advantage compounds.

This is also why serverless / spot pricing changes the calculus: if you can rent H100s on spot at A100 prices during off-peak, the H100 is always better. Use a cost optimization strategy that routes traffic to spot capacity when available.

FAQ

Is the H100 worth it over the A100 for LLM inference?

For decode-heavy workloads, the H100's 3.35 TB/s bandwidth gives roughly 1.5-1.7x higher tokens/sec than the A100's 2.0 TB/s. At high utilization this wins on cost-per-token; at low utilization the A100's lower hourly rate is cheaper. The H100's FP8 support is the decisive factor for 70B models — it halves memory, letting one card do what two A100s do.

Is the L40 good for LLM inference?

For 7B-13B models, yes — the L40S is the cost-per-token leader because its low hourly rate outweighs its lower bandwidth at small model sizes. It struggles on 70B+ models (48 GB VRAM, 864 GB/s bandwidth, PCIe-only multi-GPU). Pick it for small-model cost optimization, not for large-model throughput.

How much VRAM do I need for LLM inference?

Model weights need ~2 bytes/parameter in FP16 (7B ≈ 14 GB, 70B ≈ 140 GB), plus KV cache that scales with concurrency and context length. For 7B with healthy concurrency, 24-48 GB suffices. For 70B FP16, you need 2-4×80 GB GPUs or FP8/INT4 quantization. Use our KV Cache Calculator to size precisely.

Does FP8 change the GPU choice?

Yes. FP8 (H100, L40S — not A100) halves weight memory and doubles Tensor Core throughput. It extends the L40S's useful range to 13B FP8 models and makes the H100 dramatically more efficient for 70B. The A100's lack of native FP8 is a genuine disadvantage in 2026.

Related deep dives

• vLLM vs SGLang vs TGI — which inference engine extracts the most from each GPU
• TensorRT-LLM: When to Use It — the H100's compiled-kernel advantage
• KV Cache Quantization — FP8/INT4 cache techniques that extend GPU life
• KV Cache Calculator — size your VRAM needs before choosing a GPU
• LLM Cost Optimization Playbook — the full cost-reduction stack beyond GPU choice