PART-2: Enable Multimodal (Image-to-Text) on Gemma 4 26B-A4B on AWS Trainium2
Building on Part 1's text-only deployment, we now turn on Gemma 4's vision path — feeding the model an image plus a prompt and getting a written answer — on the same single Trainium2 chip.
PART-2: Enable Multimodal (Image-to-Text) on Gemma 4 26B-A4B on AWS Trainium2
Building on Part 1's text-only deployment, we now turn on Gemma 4's vision path — feeding the model an image plus a prompt and getting a written answer — on the same single Trainium2 chip.
Introduction
In Part 1 we ported Google's sparse Mixture-of-Experts Gemma 4 26B-A4B to AWS Trainium2, compiled it at TP=4 on a single
trn2.3xlarge , and benchmarked text inference. But Gemma 4 is natively multimodal — it can also read images. In this part we enable that image-to-text capability: hand the model a picture (a document, a card, a scene) and a question, and it answers in natural language.The interesting part is how multimodal runs on Trainium efficiently. The model has two halves — a vision encoder that turns pixels into embeddings, and the language decoder that reasons over those embeddings plus your text. We keep the vision encoder on the host CPU (no Neuron compile) and run only the language decoder on the NeuronCores. That split keeps the accelerator focused on the expensive part and makes the vision path a drop-in.
By the end, you will have:
- The multimodal Gemma 4 26B-A4B (
NeuronGemma4ForConditionalGeneration) compiled at TP=4 on one Trainium2 chip
- A working image-to-text pipeline: image + prompt → answer, at ~50–58 tok/s (2K context) and ~41 tok/s (4K)
- A validation of what the model does well (reading text fields, describing scenes, resisting hallucination) and an honest limit (fine-grained chart grounding)
Note: This is Part 2 — the multimodal path. It assumes the Part 1 environment (instance, Neuron SDK, storage tiers, weights) is already in place — including Part 1's hardening:
/etc/fstab entries (with nofail ) so the EBS and FSx mounts survive reboots, an encrypted EBS gp3 durable tier, and a least-privilege IAM instance role.
How multimodal Gemma 4 works on Trainium
Gemma 4's image path is a soft-token injection design:
- An image is preprocessed and run through a SigLIP vision encoder (a 27-layer Vision Transformer, hidden size 1152) on the CPU.
- A projector maps the vision features into the language model's embedding space —
text_config.hidden_size = 2816for the 26B-A4B variant.
- The result is ~256 "soft tokens" per image. These are dropped into the input sequence at the positions of a special image placeholder token (
image_token_id = 258880, bracketed byboi = 255999/eoi = 258882).
- The language decoder — compiled to a Neuron NEFF — consumes
input_idsplus the vision embeddings and generates the answer. The fusion (scattering vision embeddings into the right sequence positions) happens inside the compiled graph.
INPUT image (JPEG/PNG) text prompt | | v v preprocess_image() tokenizer (raw tokenizers.Tokenizer) SigLIP ViT (27 layers, CPU) input_ids with <boi> <image>*N <eoi> + projector -> [1, ~256, 2816] | | | +----------------+-----------------+ v build vision_embeddings [B, S, 2816] + vision_mask [B, S, 1] (CPU) v ┌──────────────────────────────────────────────────────────┐ │ NeuronGemma4ForConditionalGeneration (Neuron NEFF, TP=4) │ │ forward(input_ids, position_ids, sampling_params, │ │ vision_embeddings, vision_mask) │ │ -> on-device scatter (soft-token fusion) -> decode │ └──────────────────────────────────────────────────────────┘ v answer textWhy vision on the CPU? The SigLIP encoder is small relative to the 26B language model, and running it on the host keeps it a plain PyTorch module — no separate NEFF to compile, no NeuronCore budget spent on it. Only the language decoder, which is the heavy compute, is compiled and sharded across the 4 logical cores.
The multimodal port: what changes from Part 1
Part 1 used the text-only contrib port ( PR #172 ), class
NeuronGemma4ForCausalLM . The multimodal path uses a different contrib port ( PR #109 ) under contrib/models/gemma-4/ , which ships the full vision+language implementation for all Gemma 4 variants.Aspect | Part 1 (text) | Part 2 (multimodal) |
|---|---|---|
Contrib port | PR #172 ( gemma-4-26b-a4b-it ) | PR #109 ( gemma-4 ) |
Model class | NeuronGemma4ForCausalLM | NeuronGemma4ForConditionalGeneration |
Vision encoder | n/a | SigLIP ViT on CPU (ships in the port) |
Compile entry point | smoke_compile.py (env-var driven) | Python API: model.compile(path) |
Extra forward inputs | input_ids only | + vision_embeddings , vision_mask |
Required flag | attn_kernel_enabled=False | attn_kernel_enabled=False (same) |
Note: use the multimodal class.
NeuronGemma4ForCausalLM is text-only — its compiled graph has no vision input slots, so feeding it images fails. The image path is NeuronGemma4ForConditionalGeneration .
Prerequisites
Everything from Part 1, plus:
- The PR #109 contrib code checked out (
contrib/models/gemma-4/).
- The same
google/gemma-4-26b-a4b-itweights already on the fast/datatier (no new download — Part 1's weights are reused).
- A test image to try (any legible JPEG/PNG works; see the note on legibility later).
Step 1: Check out the multimodal port (PR #109)
Keep the Part 1 (PR #172) tree intact by using a separate git worktree for PR #109:
cd ~/neuronx-distributed-inferencegit fetch origin pull/109/head:pr109git worktree add ~/nxdi-pr109 pr109 # keeps the PR #172 working tree untouchedls ~/nxdi-pr109/contrib/models/gemma-4/src/modeling_gemma4.py # the multimodal portThe
contrib/models/gemma-4/ directory contains the model ( src/modeling_gemma4.py ), the CPU vision encoder ( src/gemma4_vision_encoder.py ), and ready-made test scripts ( test/integration/verify_image.py ).
Step 2: Compile the multimodal decoder (TP=4)
Compilation is a one-time Python API call. The key settings mirror Part 1 (TP=4,
attn_kernel_enabled=False ) with the multimodal class:# compile_gemma4_vlm.py — run from contrib/models/gemma-4 with PYTHONPATH=srcimport torchfrom neuronx_distributed_inference.models.config import NeuronConfigfrom src.modeling_gemma4 import NeuronGemma4ForConditionalGeneration, Gemma4InferenceConfig
MODEL_PATH = "/data/model_weights/gemma-4-26b-a4b-it"COMPILED_PATH = "/data/compiled_gemma4_vlm_tp4"
neuron_config = NeuronConfig( tp_degree=4, # TP=2 OOMs on this box (NCC_EVRF009); TP=4 required batch_size=1, seq_len=2048, # validated at 2048 (and 4096); 256 image tokens + prompt + output max_context_length=2048, torch_dtype=torch.bfloat16, save_sharded_checkpoint=True, attn_kernel_enabled=False, # MANDATORY: Gemma 4 head_dim 256/512 unsupported by the kernel)
config = Gemma4InferenceConfig.from_pretrained(MODEL_PATH, neuron_config=neuron_config)model = NeuronGemma4ForConditionalGeneration(MODEL_PATH, config)model.compile(COMPILED_PATH)print(f"Compiled to {COMPILED_PATH}")Run it:
cd ~/nxdi-pr109/contrib/models/gemma-4source /opt/aws_neuronx_venv_pytorch_2_9_nxd_inference/bin/activateexport PYTHONPATH=src:$PYTHONPATH NEURON_RT_NUM_CORES=4 NEURON_RT_EXEC_TIMEOUT=3600python compile_gemma4_vlm.py 2>&1 | tee /data/compile_vlm.logIt's a single-pass, full-MoE compile at TP=4. On this box the two graph buckets take roughly ~2.5 min (token-generation) and ~22 min (context-encoding at 2K); a 4K build is ~30 min for the context bucket. When it finishes, back the NEFF up to the durable EBS tier (see the storage note below).
Note: the vision encoder is not compiled — it runs on the CPU. Only this language decoder becomes a NEFF.
Step 3: Run image-to-text inference
Option A — the port's own verify_image.py (fastest smoke test)
cd ~/nxdi-pr109/contrib/models/gemma-4export PYTHONPATH=src:$PYTHONPATH NEURON_RT_NUM_CORES=4
python test/integration/verify_image.py \ --model_path /data/model_weights/gemma-4-26b-a4b-it \ --compiled_path /data/compiled_gemma4_vlm_tp4 \ --tp_degree 4 --seq_len 2048The first run compiles (if not already) then prints a description of the image. Later runs reuse the NEFF and finish in ~2–3 minutes (mostly model load).
Option B — your own image + prompt
The pipeline is: preprocess the image and run the CPU vision encoder, build the
vision_embeddings / vision_mask tensors, load the compiled decoder, then greedy-decode. The core of it:import json, os, torchfrom neuronx_distributed_inference.models.config import NeuronConfigfrom neuronx_distributed_inference.models.model_wrapper import prepare_sampling_paramsfrom src.modeling_gemma4 import NeuronGemma4ForConditionalGeneration, Gemma4InferenceConfigfrom gemma4_vision_encoder import Gemma4VisionConfig, preprocess_image, load_vision_encoderfrom tokenizers import Tokenizer
MODEL_PATH, COMPILED_PATH = "/data/model_weights/gemma-4-26b-a4b-it", "/data/compiled_gemma4_vlm_tp4"IMG_PATH, PROMPT, SEQ_LEN = "/fsx/test_images/loyalty_card.jpg", "How many points are on this card and when do they expire?", 2048
cfg = json.load(open(os.path.join(MODEL_PATH, "config.json")))ITID, BOI, EOI = cfg["image_token_id"], cfg["boi_token_id"], cfg["eoi_token_id"] # 258880 / 255999 / 258882TH = cfg["text_config"]["hidden_size"] # 2816eos = cfg.get("eos_token_id", [1, 106])
# --- Vision on CPU (no NEFF) ---vcfg = Gemma4VisionConfig.from_dict(cfg["vision_config"])venc, vemb = load_vision_encoder(MODEL_PATH, vcfg, TH)pv, pos_ids, _ = preprocess_image(IMG_PATH)with torch.no_grad(): img_emb = vemb(venc(pv.to(torch.bfloat16), pos_ids).unsqueeze(0)) # [1, ~256, 2816]
# --- Build the sequence: <bos><start_of_turn>user <boi><image>*N<eoi> PROMPT ... model ---tok = Tokenizer.from_file(os.path.join(MODEL_PATH, "tokenizer.json"))N = img_emb.shape[1]ids = ([2, 105] + tok.encode("user\n").ids + [BOI] + [ITID]*N + [EOI] + tok.encode("\n" + PROMPT).ids + [106, 107, 105] + tok.encode("model\n").ids)input_ids = torch.tensor([ids], dtype=torch.int32)[:, :SEQ_LEN]padded = torch.nn.functional.pad(input_ids, (0, SEQ_LEN - input_ids.shape[1]), value=0)
# --- vision_embeddings [B,S,2816] + vision_mask [B,S,1] ---B, S = padded.shapepositions = (padded[0] == ITID).nonzero(as_tuple=True)[0]; n = len(positions)ve = torch.zeros(B, S, TH, dtype=torch.bfloat16); ve[0, :n] = img_emb.reshape(-1, TH)[:n].to(torch.bfloat16)vm = torch.full((B, S, 1), S - 1, dtype=torch.int32); vm[0, :n, 0] = positions.to(torch.int32)
# --- Load compiled decoder and greedy-decode ---nc = NeuronConfig(tp_degree=4, batch_size=1, seq_len=SEQ_LEN, max_context_length=SEQ_LEN, torch_dtype=torch.bfloat16, save_sharded_checkpoint=True, attn_kernel_enabled=False)model = NeuronGemma4ForConditionalGeneration(MODEL_PATH, Gemma4InferenceConfig.from_pretrained(MODEL_PATH, neuron_config=nc))model.load(COMPILED_PATH)sp = prepare_sampling_params(1)
sl, gen, nt = input_ids.shape[1], input_ids.clone(), Nonefor step in range(200): if step == 0: mi = input_ids.clone(); mi[mi == ITID] = 0 out = model(mi, position_ids=torch.arange(sl, dtype=torch.int32).unsqueeze(0), sampling_params=sp, vision_embeddings=ve, vision_mask=vm) else: out = model(nt, position_ids=torch.tensor([[sl + step - 1]], dtype=torch.int32), sampling_params=sp, vision_embeddings=torch.zeros(B, 1, TH, dtype=torch.bfloat16), vision_mask=torch.zeros(B, 1, 1, dtype=torch.int32)) lg = out.logits if hasattr(out, "logits") else (out[0] if isinstance(out, tuple) else out) nt = torch.argmax(lg[:, -1, :], dim=-1, keepdim=True).to(torch.int32) gen = torch.cat([gen, nt], dim=-1) if nt.item() in eos: breakprint(tok.decode(gen[0, sl:].tolist()))Feeding a loyalty-card image and the prompt above, the model answers:
There are 3450 points shown on the loyalty card, and they expire in March 2027.Note:
vision_embeddings and vision_mask are valid forward inputs on the multimodal class — there is no separate "fuse" step and no inputs_embeds recompile; the soft-token fusion happens inside the compiled graph. Each decode step after the first passes 1-token dummy vision tensors. The model may prefix answers with a reasoning thought channel, which you can strip.
Step 4: What it does well — and its limits
We ran a small validation battery against the compiled model (each answered in seconds once loaded, ~50–58 tok/s at 2K context):
Test image | Prompt | Result |
|---|---|---|
Loyalty card | Points and expiry? | Correct — "3450 points … expire March 2027" |
Invoice | Total due and due date? | Correct — "$128.50 … 2027-01-15" (ignored distractor fields) |
Colored shapes (no text) | What shapes and colors? | Correct — red circle, blue square, green triangle, with correct positions |
Loyalty card | Member's phone number? | Correct — answered "no phone number is shown" (resisted hallucinating) |
Bar chart | Which quarter is highest? | Incorrect — read the digits but mis-bound them to bars |
The takeaways:
- Strong at reading discrete text fields (documents, cards, invoices), describing scenes, and — importantly for reliability — saying when something isn't present instead of inventing it.
- Weak at fine-grained chart/plot grounding: it can OCR the numbers but mis-associates them with the right bars and gives inconsistent "which is highest" answers. Don't rely on it for exact chart/scoreboard/analytics extraction.
- Legibility matters: because an image becomes ~256 soft tokens, small text can be lost. Prominent, high-contrast text reads reliably; tiny fine print may not.
Context length vs throughput
We compiled the multimodal decoder at both 2K and 4K. Doubling the window compiled cleanly and produced correct output; the cost is throughput and a longer compile:
Compiled context | Decode throughput | Context-encoding compile |
|---|---|---|
2,048 | ~50–58 tok/s | ~22 min |
4,096 | ~41 tok/s | ~30 min |
Each build is a self-contained NEFF (~51 GB with sharded weights), so keep the one(s) you need for your prompt lengths.
Storage: the durable tier pays off again
Part 1's tiered storage design carries straight over. The multimodal NEFF is large (~51 GB per context length, because
save_sharded_checkpoint=True stores the presharded weights alongside the graph), and it lives on the ephemeral /data tier — so an instance stop would wipe a ~25-minute compile. We mirror it to the durable Amazon EBS gp3 tier:rsync -a /data/compiled_gemma4_vlm_tp4/ /mnt/ebs/compiled_gemma4_vlm_tp4/The recovery script from Part 1 already restores every
compiled_gemma4_* directory, so both the text and multimodal NEFFs come back automatically after an ephemeral wipe — no recompile.
Capabilities and limitations
Capability | Status |
|---|---|
Image-to-text (describe / read a single image) | Validated on trn2.3xlarge, TP=4 |
Document / card / invoice field extraction | Accurate |
Scene, object, and color description | Accurate (incl. relative positions) |
Hallucination resistance ("not shown") | Good |
Fine-grained chart / graph value grounding | Unreliable |
Video / audio | Not enabled — encoders exist in the port but are not used here |
Context length | 2K and 4K validated on this box |
Conclusion
Building on Part 1's text deployment, we enabled Gemma 4's multimodal image-to-text path on the same single Trainium2 chip — CPU-side SigLIP vision encoder, soft-token fusion, and the
NeuronGemma4ForConditionalGeneration decoder compiled to a Neuron NEFF at TP=4. The result reads documents and scenes accurately at ~50–58 tok/s, with an honest caveat around precise chart grounding, and it reuses the same durable storage strategy so the (larger) multimodal artifacts survive an instance stop.Together, the two parts show a full path: port an open sparse-MoE model to Trainium, run it cost-efficiently for text, and extend it to vision — all on a single, entry-level Trainium2 instance.
Need guidance on AWS Trainium adoption? Contact your AWS account team. To learn more:
Disclaimer: This article showcases how you can run an open-source multimodal model like Gemma 4 26B-A4B on AWS Trainium-powered Amazon EC2 instances. We used the model available on HuggingFace and did not fine-tune or customize it for a production application. Results and benchmarks were measured on a single trn2.3xlarge and will vary with instance type, configuration, image content, and workload. Any opinions in this article are those of the author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article