Compare
Llama 4 Behemoth vs Gemini 2.5 Ultra
Meta vs Google DeepMind. Specs, benchmarks, and real per-task cost — all in one page.
Verdict
Gemini 2.5 Ultra leads on LMSYS ELO (1385 vs 1342). Llama 4 Behemoth is ~2.7× cheaper on a 3:1 input:output blend. Llama 4 Behemoth is open-weights, the other is proprietary.
Meta
Llama 4 BehemothMeta’s open-weights flagship. 405B params, fully open license, runs on every major inference provider.
Best open-weights modelRuns anywhereTransparent training
Google DeepMind
Gemini 2.5 Ultra2M-token context, native video understanding, and Google’s deepest multimodal stack. The long-context king.
2M context windowNative videoBest multimodal reasoning
Pricing
Input / 1M$2.50$7.00
Output / 1M$8.00$21.00
Context256K2.0M
Max output16K64K
LicenseOpen weightsProprietary
Released2025-12-102026-02-05
Benchmarks
LMSYS ELO1342.01385.0
MMLU Pro87.190.4
HumanEval89.390.2
SWE-Bench52.458.3
MATH81.592.0
GPQA56.2—
MMMU—82.1
IFEval88.089.7
Per-task cost
Summarize a 1-hour meeting transcript$0.041$0.115
Review a 500-line pull request$0.036$0.098
Answer a customer support ticket$0.014$0.038
Extract structured data from a resume$0.021$0.056
Debug a stack trace with context$0.031$0.084
Per-call cost using published token counts for each task. Real-world prompts vary.