This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable.
https://github.com/xbill9/gemma4-dev
The whole exercise cost under three dollars, and that is the part worth keeping. Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts. It also bought five wrong claims, each caught by measuring instead of reasoning. On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.
What is this project trying to Do?
Three rigs in this monorepo serve google/gemma-4-E2B-it on an AWS G5g instance. One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers. The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.
For months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison.
Prerequisites
- An AWS account with G-family quota in
us-east-1. Eachg5g.2xlargeis 8 vCPU, so 16 vCPU of spot quota runs two at once. - A subnet, a security group opening TCP 8000, and an instance profile carrying
AmazonSSMManagedInstanceCoreplus read on the Hugging Face token secret. - A Hugging Face token in Secrets Manager. It is fetched at boot into a root-only
EnvironmentFileand never appears in user data. -
boto3and the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.
AWS EC2 G5g
| Instance |
g5g.2xlarge — 8 vCPU, 16 GiB host |
| Host CPU | AWS Graviton2, aarch64 |
| GPU | 1x NVIDIA T4G, Turing, SM 7.5 |
| GPU memory | 15,360 MiB per nvidia-smi; AWS lists 16,384 nominal |
G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.
Gemma 4 E2B
google/gemma-4-E2B-it is the reference instruction-tuned release. It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.
The dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.
The Three Runtimes
| runtime | engine | how it serves |
|---|---|---|
| vLLM | v0.27.2rc0, built from source for sm_75
|
continuous batching, paged KV, prefix caching |
| JAX | this project's own port | hand-written KV ring with a bucket ladder |
| PyTorch |
AutoModelForCausalLM + transformers |
past_key_values, one request at a time |
Turing has no bfloat16 datapath, so all three run float16. It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.
Check the Quotas
check_g5g_quotas
| Quota | vCPUs |
| Running On-Demand G and VT instances (vCPU) | 16 |
| All G and VT Spot Instance Requests (vCPU) | 16 |
`g5g.2xlarge` needs 8 vCPUs.
That is the constraint behind every launch below: two rigs in parallel, and no more.
The Sweep Could Not See vLLM
The sweep script read its throughput figure straight out of the response body:
"decode_tps": usage.get("decode_tokens_per_second", 0.0),