GPU Rent HubGPU for LLM inference

Choosing a GPU for LLM inference.

GPU Rent Hub rents dedicated GPU servers for LLM inference from $250 a month for an NVIDIA L4 to $2,290 a month for an H200 SXM, paid in Bitcoin, Ethereum, USDT, USDC, Monero, Solana or Litecoin, with no identity verification.

Two numbers pick the card: the VRAM the weights and KV cache need, and the memory bandwidth that sets tokens per second.

Updated 2026-09-01

75 GBA 70B at FP8, 8k context, batch 1

3.35 TB/sH100 SXM memory bandwidth

$2.35H100 SXM per GPU-hour on a month

20 TBOutbound included per instance

01 Sizing

How much VRAM does an LLM need for inference?

Two figures decide the card here. The derivation belongs to the VRAM guide; the term that moves is the KV cache, and it moves with your traffic rather than with the model.

The two figures this page argues from

A 70B model in FP8 is 75 GB at 8,192 tokens and batch 1, and 2.7 GB of that is KV cache. Everything else, the weights-plus-cache-plus-overhead formula, the bytes per parameter at each precision and a row for every model size, is derived term by term in the GPU VRAM guide and is not repeated here. Those two figures do the work below: 75 GB is why an 80 GB card serves a 70B with 5 GB spare, and the 2.7 GB cache is the term carried forward into the 120B row of the table.

The cache term takes over at long context, where it scales with context length times batch size, not parameter count. At 128k and 32 concurrent requests it can exceed the weights. That is why the H200 SXM with 141 GB costs more than an H100 SXM running the same silicon.

Model size	BF16, 2 B/param	FP8, 1 B/param	4-bit, 0.5 B/param	Cheapest NVIDIA card that fits, FP8
7B–8BLlama 8B, Mistral 7B, Qwen 7B	17 GB	10 GB	7 GB	L4, 24 GB
13B–14B40 layers, MHA, large cache	35 GB	22 GB	15 GB	L4, 24 GB, tight
32B–34B	72 GB	38 GB	21 GB	RTX 6000 Ada, 48 GB
70B	145 GB	75 GB	40 GB	H100 PCIe, 80 GB
120Bsame formula, no guide row	245 GB	125 GB	65 GB	H200 SXM, 141 GB
405B	816 GB	411 GB	209 GB	8× MI300X, 1,536 GB

Weights plus a KV cache at 8,192 tokens and batch 1 plus 2 GB of overhead, quoted from the GPU VRAM guide along with the cheapest card that holds each FP8 figure. The last column is NVIDIA only, because the alternative is a different software stack: on the 120B row the 192 GB MI300X holds the same 125 GB for $1,590 a month against the H200 SXM's $2,290, at the price of ROCm rather than CUDA and Dallas rather than three sites. The 120B row is the same formula carried one step further, taking the 70B row's 2.7 GB cache, because no guide row covers that size. The 405B row is the exception to the column, because no single card of either brand holds it and the cheapest node that does is the 8× MI300X. A 70B at FP8 leaves 5 GB spare on an 80 GB card at 8k and stops fitting at 32k, where the cache alone is 10.7 GB.

02 Bandwidth

Why does memory bandwidth matter more than TFLOPS for LLM inference?

Decoding one token reads every weight once. The card waits on memory, not on maths.

At a batch of one, decoding streams the whole weight set out of VRAM for every token and does about two floating-point operations per parameter, which at BF16 is one operation per byte loaded. Keeping an H100 SXM's tensor cores busy takes roughly 300, taking its 1,979 sparse FP16 TFLOPS as about 990 dense against 3.35 TB/s of bandwidth. The card is memory bound by a wide margin, so the ceiling on tokens per second is bandwidth divided by the bytes of weights read per token.

An 8B model in BF16 is 16 GB. On an RTX 4090 at 1.01 TB/s the ceiling is 63 tokens a second; on an H100 SXM at 3.35 TB/s it is 209. Those are spec-sheet ceilings and real decoding lands below them, by a fraction we will not quote because we cannot source one. The ranking holds. Prefill is the opposite case, compute bound, which is where FP8 tensor cores earn their keep.

Ceiling, tokens per second = memory bandwidth ÷ (parameters × bytes per parameter). Batching raises throughput per card without raising single-request speed, because the weights are read once for the whole batch.

Two facts from the table that read badly for us. An A100 SXM at 2.04 TB/s has more bandwidth than an H100 PCIe, and an RTX 3090 at 936 GB/s more than an L40S. Neither is the better inference card. Section 03 says why.

GPU	VRAM	Bandwidth	Ceiling, 8B in BF16	Monthly	Per GPU-hour
B200Blackwell, HBM3e, 2-GPU minimum	180 GB	8 TB/s	500 t/s	$3,890/mo	$5.40
MI300XCDNA 3, HBM3	192 GB	5.3 TB/s	331 t/s	$1,590/mo	$2.21
H200 SXMHopper, HBM3e	141 GB	4.8 TB/s	300 t/s	$2,290/mo	$3.18
H100 SXMHopper, HBM3	80 GB	3.35 TB/s	209 t/s	$1,690/mo	$2.35
A100 SXMAmpere, no FP8	80 GB	2.04 TB/s	128 t/s	$990/mo	$1.38
H100 PCIeHopper, HBM2e	80 GB	2 TB/s	125 t/s	$1,490/mo	$2.07
RTX 5090Blackwell, GDDR7	32 GB	1.79 TB/s	112 t/s	$540/mo	$0.75
RTX 4090Ada, GDDR6X	24 GB	1.01 TB/s	63 t/s	$390/mo	$0.54
RTX 6000 AdaAda, GDDR6 ECC	48 GB	960 GB/s	60 t/s	$460/mo	$0.64
RTX 3090Ampere, no FP8	24 GB	936 GB/s	59 t/s	$230/mo	$0.32
L40SAda, GDDR6 ECC	48 GB	864 GB/s	54 t/s	$520/mo	$0.72
A40Ampere, no FP8	48 GB	696 GB/s	44 t/s	$240/mo	$0.33
L4Ada, 72 W	24 GB	300 GB/s	19 t/s	$250/mo	$0.35

The ceiling column is bandwidth divided by 16 GB of BF16 weights: an upper bound from the spec sheet, not a measured result. Per GPU-hour is monthly divided by 720. The B200 price is per card and the card is sold on an 8-way HGX baseboard, so the smallest B200 order is two cards, $7,780 a month before the two-card discount. All fifteen cards on the GPU catalogue.

03 Precision

Does FP8 matter, and why is the A100 a worse inference card than its price suggests?

FP8 halves the weights. That halves the bytes read per token and doubles the model that fits.

Hopper, Ada Lovelace and Blackwell have FP8 tensor cores: the B200, H200, H100 SXM, H100 PCIe, L40S, RTX 6000 Ada, RTX 5090, RTX 4090 and L4. Ampere does not. The A100, A40, RTX A6000, RTX A5000 and RTX 3090 are FP8 not supported, and that is silicon, not a driver gap.

So a 70B model in FP8 is 75 GB and runs on one H100 SXM at $1,690 a month, $2.35 per GPU-hour. On Ampere it runs in BF16 at 145 GB, meaning two A100 SXM cards: $1,980 at list, $1,881 a month with the 5% two-card discount, which is $2.61 an hour for the endpoint against $2.35 for the single H100 SXM, with tensor-parallel traffic on every token. The cheaper card builds the more expensive endpoint.

Ampere is still worth renting

For fine-tuning, for 4-bit serving, and where 80 GB under $1,000 a month is the point. It is a poor default for a production token endpoint, and the $990 price tag hides that.

The MI300X is the exception worth knowing. 192 GB of HBM3 at 5.3 TB/s for $1,590 a month fits a 70B model in BF16 on one card, 145 GB with 47 GB spare, no quantisation and no sharding. It boots our ROCm 6.4 template with a PyTorch ROCm build.

04 Picks

What is the best GPU for LLM inference at each model size?

Sorted by model size. One card per band, chosen on the arithmetic above.

Serving this	Pick	VRAM	Weekly	Monthly	Per GPU-hour
7B–8B, always on, low QPS72 W, single slot, FP8	L4	24 GB	$70	$250/mo	$0.35
7B–14B, latency matters1.01 TB/s, 265 in stock	RTX 4090	24 GB	$110	$390/mo	$0.54
14B–24B in FP8fastest memory under $600	RTX 5090	32 GB	$155	$540/mo	$0.75
24B–34B in FP8ECC, 8-way nodes, RT cores	L40S	48 GB	$148	$520/mo	$0.72
32B in BF16, no quantisationHopper at 350 W	H100 PCIe	80 GB	$420	$1,490/mo	$2.07
70B in FP8, one cardthe production default	H100 SXM	80 GB	$475	$1,690/mo	$2.35
70B in BF16, one cardmost HBM per card	MI300X	192 GB	$450	$1,590/mo	$2.21
100B–120B, or 70B at 128kKV cache headroom	H200 SXM	141 GB	$640	$2,290/mo	$3.18
405B in FP8, one node8-way, NVSwitch 1.8 TB/s	8× B200	1,440 GB	$7,850	$28,010/mo	$4.86

Per GPU-hour is monthly divided by 720, and for the node by 8 cards and 720 hours. Multi-GPU discounts run 5% at two cards, 8% at four, 10% at eight, so an 8× H100 SXM node is $12,170 a month, $2.11 per GPU-hour. In stock: 63 H100 SXM, 100 RTX 5090, 74 L40S, 265 RTX 4090. These are the cards we would run the job on; the VRAM guide's lookup names the cheapest card that merely holds it, so a row here is sometimes the more expensive card and the reason is in the row. The 405B row is the clearest case: the guide gives the cheapest node that holds a 405B in FP8 as the 8× MI300X at $11,450, and this table takes the 8× B200 for its 1.8 TB/s fabric.

Prices are identical in DFW-1 Dallas, IAD-1 Ashburn and PDX-1 Hillsboro, so choose the region on latency. The MI300X is Dallas only. Of the eight B200, six are in DFW-1 and two in PDX-1, with none in IAD-1.

05 The instance

What do I get on a GPU Rent Hub LLM inference server?

Bare metal, root, the serving stack preinstalled, nothing listening until you enable it.

vLLM 0.9 template

Ubuntu 24.04 with vLLM 0.9 and an OpenAI-compatible server unit installed. The unit ships disabled. You enable it and decide what may reach the port.

Nothing is exposed by default.

Ollama template

For a model answering in two minutes rather than a tuned server. Ollama is bound to localhost and you expose it yourself, usually over WireGuard.

Same rule: localhost until you move it.

20 TB outbound included

At roughly 100 bytes of JSON and SSE framing per streamed token, that is about 200 billion tokens a month. A text endpoint will not reach it.

Then $4/TB. Inbound unmetered.

100G port, 10G guaranteed

Shared 100G uplink with a 10 Gbps floor, one IPv4 and an IPv6 /64. A dedicated 100G port is $220 a month if you stream audio or images.

Private VLAN between instances, $25/mo.

Always-on inference is the clearest case for a term: busy at unpredictable hours, intolerant of cold starts, holding weights a metered platform would pull again on every scale-up. The card is yours for 720 hours whether traffic arrives or not, and cannot be preempted. Crossover arithmetic on monthly versus hourly GPU rental, deployment detail in the documentation. Fund a balance in BTC, ETH, USDT, USDC, XMR, SOL or LTC: how crypto payment works, and what no-KYC GPU hosting means.

BTCETHUSDTUSDCXMRSOLLTC

06 When not to

When should you not rent a dedicated GPU for LLM inference?

Three cases where a dedicated card is the wrong tool, and we would rather say so.

Under roughly ten million tokens a month, use a hosted token API. A 7B model on an L4 costs $250 a month, which is $25 per million tokens at that volume, and commodity per-token rates for a comparable open model sit below that.

Second, bursty traffic. An endpoint answering a hundred requests on Tuesday and nothing until the next week keeps the card busy for minutes and rents it for 720 hours. Use an hourly provider. Our shortest term is seven days and there is no meter.

Third, a smaller card doing the job. Quantise a 34B model to 4-bit and it fits in 21 GB, which an RTX 3090 at $230 a month serves at 936 GB/s with 3 GB spare, faster memory than the cheaper RTX A5000 the guide's lookup names for the same cell. Do not rent an H100 SXM because the model has a large number in its name.

Rent from us if

The endpoint is always on, per-token billing has become your largest line, the weights and prompts must stay on hardware nobody else touches, or you want to pay in crypto without an identity check.

Do not rent from us if

You need a card for an afternoon, traffic is unpredictable and low, or you have not yet measured whether the model fits in 4-bit on a card a quarter of the price.

Training is a different sizing problem, dominated by optimiser states: see GPU for fine-tuning.

07 Questions

GPU for LLM inference, asked plainly.

What is the best GPU for LLM inference?

It depends on model size. For 7B to 14B an RTX 4090 at $390 a month is the value pick. For 24B to 34B in FP8, an L40S with 48 GB at $520. For 70B in FP8, one H100 SXM at $1,690, which is $2.35 per GPU-hour. For 100B and up, an H200 SXM at $2,290.

What does a 70B endpoint need beyond the weights?

Cache, and it scales with concurrency rather than with the model. The weights and a single-stream figure are in the GPU VRAM guide: 75 GB in FP8 at 8,192 tokens and batch 1. Each further concurrent request at that context adds 2.7 GB, so eight in flight is about 22 GB on top and thirty-two is about 85 GB. That is the difference between an 80 GB card serving one stream with 5 GB spare and an H200 at 141 GB or an MI300X at 192 GB serving an endpoint with real concurrency. Quantising the cache to FP8 halves every one of those figures.

Can I rent a GPU for vLLM?

Yes. Any instance boots the vLLM 0.9 template on Ubuntu 24.04, which includes an OpenAI-compatible server unit that ships disabled until you enable it. You have root on bare metal, so you can replace it with your own build. The MI300X uses the ROCm 6.4 template with a PyTorch ROCm build.

Why is the A100 not a good LLM inference card?

Ampere has no FP8 tensor cores, so a 70B model runs in BF16 at 145 GB. That needs two A100 SXM cards, $1,980 at list and $1,881 a month once the 5% two-card discount applies, against one H100 SXM at $1,690 serving the same model in FP8 at 3.35 TB/s. The A100 is still a good fine-tuning and 4-bit serving card.

How many tokens per second will a GPU produce?

The upper bound is memory bandwidth divided by the bytes of weights read per token. An 8B model in BF16 is 16 GB, so an RTX 4090 at 1.01 TB/s tops out near 63 tokens a second and an H100 SXM at 3.35 TB/s near 209. Real stacks land under the ceiling, and how far under depends on your kernels, batch shape and scheduler, which is why we publish the bound and not a fraction of it.

Is a monthly rental right for an always-on LLM endpoint?

Usually. A flat term costs the same whether traffic arrives or not, so an endpoint answering at unpredictable hours pays no idle premium, has no cold start and never queues for capacity. The crossover follows from the prices: $1,690 for an H100 SXM month against an illustrative $2.49 to $3.49 meter breaks even between roughly 480 and 680 used hours of a 720-hour month, so below that an hourly provider is cheaper. The full working is on our monthly versus hourly page.

Do you include enough bandwidth for a token API?

Yes. Each instance includes 20 TB of outbound a month, which at roughly 100 bytes of JSON and SSE framing per streamed token is about 200 billion tokens. Inbound is unmetered, so pulling weights costs nothing. Beyond 20 TB it is $4 per TB, and a dedicated 100G port is $220 a month.

Do I need to give ID to rent an LLM inference server?

No. An account is a username and a password with no email address required, and you fund a balance in Bitcoin, Ethereum, USDT, USDC, Monero, Solana or Litecoin. GPU Rent Hub Compute LLC has run this way since 15 March 2021, with 6,300 GPUs across DFW-1 Dallas, IAD-1 Ashburn and PDX-1 Hillsboro.

Deeper sizing cases in the GPU VRAM guide, every card on the GPU catalogue, term discounts on the pricing page.

Put the endpoint on hardware nobody else touches.

Deploy vLLM on a dedicated card in DFW-1, IAD-1 or PDX-1 in about ninety seconds. Minimum deposit $20 equivalent, no email, no ID.

Create an account →