How to Run Meta Muse Glimmer-30B Locally on a 24GB GPU

Photo by Christian Wiediger on Unsplash

A single RTX 5090 can now host a 30-billion-parameter agent model that talks to your filesystem, calls tools, and never sends a token to a cloud API. This walkthrough covers a working Meta Muse Glimmer local GPU setup using the Q4_K_M GGUF build, the DFlash speculative-decoding drafter, and either llama.cpp or Ollama as the runner. If you already have 24GB of VRAM, you can be answering prompts in under an hour.

The model shipped August 10, 2026 under Apache 2.0, and the tooling has moved unusually fast because Meta seeded day-zero support with the llama.cpp maintainers. That is why this tutorial exists now rather than in three months.

What Muse Glimmer Actually Is Before You Download It

Muse Glimmer-30B is a ~29.6B dense transformer paired with a 1.8B ViT-G/14 perception encoder, released by Meta Superintelligence Lab under Apache 2.0 with no commercial restrictions. According to AI Weekly's launch coverage, Meta ships two ExecuTorch PTE 4-bit variants targeting 32GB and 24GB VRAM with claimed 0.2% and 1.0% accuracy degradation respectively against the full-precision model.

The 4-bit build is the entire reason this fits on consumer silicon. As TechMyMoney reported, full-precision BF16 weights require more than 55GB of VRAM, which is well beyond anything short of an H100 or a stacked workstation. The 4-bit quantized language model keeps that footprint under 20GB.

One clarification the vendor materials mostly gloss over: 20GB is the weights file, not the working set. Independent analysis from Kingy AI breaks it out at roughly 16.76GB for the K-Quant weights, 1.40GB for the vision projector, and 1.63GB for the DFlash drafter, before the KV cache, runtime buffers, and OS headroom. Plan for the full 24GB of a 5090 or 7900 XTX to be spoken for once a 131K context is loaded. This is the trap that catches most first-time installs.

The Minimum Muse Glimmer 30B Tutorial Setup That Will Actually Boot

Before anything else, confirm the hardware target. To run this AI agent locally on 24GB VRAM you need three files, not one:

• The Q4_K_M (or Unsloth UD-Q4_K_XL) GGUF weights, roughly 16.76GB
• The BF16 vision projector mmproj file, roughly 1.40GB, if you want image input
• The DFlash speculative-decoding drafter, roughly 1.63GB, if you want the 3.1x speedup

Pure text agents can skip the mmproj and reclaim 1.4GB. Skip the DFlash drafter and you keep the memory but lose most of the throughput advantage. TestingCatalog notes that the Q4_K_M GGUF quantization and DFlash drafter together are specifically what enable the 24GB fit at usable speeds.

Build llama.cpp with a working GPU backend

On NVIDIA, compile with CUDA. On AMD, the official AMD launch guide recommends the Vulkan backend, and reports up to 53 tokens per second on a single Radeon AI PRO R9700 with DFlash enabled. The Unsloth documentation shows the canonical clone-and-build:

git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j \
  --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

Swap -DGGML_CUDA=ON for -DGGML_VULKAN=ON on Radeon or -DGGML_METAL=ON on Apple Silicon.

Pull the weights from Hugging Face

The Unsloth Muse Glimmer documentation publishes a dynamic quant repository that llama.cpp can fetch directly. The hf download pattern grabs only what you need:

hf download unsloth/Muse-Glimmer-30B-GGUF \
  --local-dir unsloth/Muse-Glimmer-30B-GGUF \
  --include "*mmproj-BF16*" \
  --include "*UD-Q4_K_XL*"

Drop the mmproj include if you do not need vision. Add --include "*dflash*" to grab the drafter file for speculative decoding.

Get Muse Glimmer Answering on the First Prompt

With the binaries built and the files on disk, one command produces a working CLI session:

./llama.cpp/llama-cli \
  -hf unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL \
  --temp 1.0 --top-p 0.95 --top-k 64

Those sampling values are the ones Unsloth ships in its reference. Muse Glimmer was trained with a wider top-k than most 30B models, and dropping to --top-k 40 (a common default from earlier Llama recipes) tends to make tool-calling turns rigid.

If you plan to serve the model over HTTP, use llama-server instead:

./llama.cpp/llama-server \
  --model Muse-Glimmer-30B-UD-Q4_K_XL.gguf \
  --mmproj mmproj-BF16.gguf \
  --port 8001

That exposes an OpenAI-compatible endpoint on port 8001 that any existing agent framework, from LangChain to a homegrown Spring AI harness, can point at.

The DFlash flag that quadruples throughput

Speculative decoding is where the numbers get interesting. According to AI Weekly, the DFlash drafter lifts an RTX 5090 from 74.9 to 233.4 tokens per second, a 3.1x speedup. AMD's own testing hit up to 53 tokens per second on the Radeon AI PRO R9700 with --spec-draft-n-max=4 under the Vulkan backend, per the AMD engineering blog. Enable the drafter with the model's speculative flags and set --spec-draft-n-max=4 as a safe starting point. Going higher trades verification cost against acceptance rate, and 4 is the sweet spot Meta calibrated the drafter for.

If you would rather use Ollama

A full Muse Glimmer llama.cpp Ollama setup follows the same three-file pattern, with Ollama handling the runner. As dev.to's launch analysis documents, Ollama, LM Studio, and Unsloth all shipped support for local runs at launch, with llama.cpp, MLX, and ExecuTorch covering the edge deployment paths. Pull the model with ollama pull muse-glimmer:30b-q4_k_m, then run ollama run muse-glimmer:30b-q4_k_m. The DFlash drafter is bundled into Ollama's manifest, so you do not toggle it manually, though you lose the fine-grained --spec-draft-n-max control.

The Day-One Failure Mode Nobody Warns You About

Most first installs fail at the tool-call parser, not the model download. Muse Glimmer emits function calls in a schema that requires the muse_glimmer tool-call and reasoning parsers. On vLLM the invocation is vllm serve meta-models/Muse-Glimmer-30B --enable-auto-tool-choice --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer. Miss those flags and tool turns render as unstructured text that your agent framework silently discards, which looks exactly like a model that cannot use tools.

On llama.cpp the equivalent is a runtime that understands the same schema, or a thin adapter you write once and reuse. If your agent loop looks broken at the third or fourth tool call, this is almost always why.

The other trap is context. Muse Glimmer advertises 131K tokens, but a full 131K KV cache at 4-bit will not fit alongside the weights on a 24GB card. Cap --ctx-size 32768 for a 24GB build, or 65536 if you drop the vision projector. Full-context work needs the 32GB variant.

Wiring Muse Glimmer Into a Real Agent Loop

The model was trained as an open-weight agent model for 2026 workflows, not as a chat companion. TechMyMoney's writeup notes that Muse Glimmer is distilled from Meta's closed flagship Muse Spark through logit distillation, trained specifically to mimic the agent task performance of the larger teacher model. The vendor-published benchmarks reflect that intent: MCP Atlas 75.5, DeepSearch QA 74.6, SWE-Bench Pro 51.2, AIME 2026 94.7, GPQA Diamond 83.5, per the Hugging Face model card figures. No independent evaluation is attached to those numbers yet.

Treat those scores with skepticism until third-party runs land. Meta's own comparison table has Qwen3.6-27B winning several practical agent and multimodal tests, per Kingy AI's analysis of the launch materials. Where Muse Glimmer does look genuinely ahead in Meta's data is agent scaffolding: TestingCatalog cites MCP Atlas at 75.5 versus 54.2 for Gemma4-31B and 62.5 for Qwen3.6-27B, all on Meta's own bench.

Select reasoning strength through the system prompt. The model supports selectable reasoning strengths, more than 100 languages, and agent scaffolds such as OpenClaw, according to TestingCatalog. A short system prompt at reasoning: high roughly doubles latency but is the setting most agent tasks want.

For a hybrid deployment, keep the local model on repetitive high-frequency work and route hard reasoning to a hosted frontier model. That pattern is easier to justify commercially because Apache 2.0 puts no licensing friction on embedding the local model inside a paid product, as the dev.to analysis points out.

What This Release Actually Signals

Mark Zuckerberg framed the launch in political terms rather than technical ones. "Rather than centralizing superintelligence, we should distribute it widely and give every person the ability to direct it," he said in remarks reported by TechMyMoney. Zuckerberg also confirmed on X that weights for Muse Spark 1.2, the closed flagship, are planned for release in the near future, per TestingCatalog.

Read against the rest of the market, Muse Glimmer is not the strongest model you can talk to. Per Engadget's coverage cited in TechMyMoney's report, Meta's broader Muse Spark flagship still trails rivals from OpenAI and Anthropic. What Glimmer offers instead is openness and local control at a size that fits on hardware people already own. For anyone building on the Open-Weight AI side of the fence, that trade is the point.

If you want the same tooling story on other new hardware, the Intel Core Ultra Series 3 Panther Lake SKU guide covers the CPU side, and the recent Nvidia RTX GPU price increase is worth reading before you buy a 5090 specifically for this. Anyone planning to let an agent like this touch real files should also work through locking down AI agent sandbox permissions first. Muse Glimmer is trained to recover from failed tool calls, which makes it useful and also makes it persistent.

Next step: point your existing agent harness at http://localhost:8001/v1 and start replacing hosted calls one tool at a time.