Muse Glimmer on HeraldStack: Local 30B Agents, One GPU, No Cloud

August 31, 2026 · 7 min · 1295 words · Bryan Chasko | Follow the live build log

Live build log (harald) with terminal evidence: muse-code/BLOG.md on feat/muse-code-orchestrator. All code below is from that branch.

executive summary

One workstation, every agent. rocm-aibox (Threadripper 1920X, RX 6700 XT 12 GB, 62 GB RAM) already runs inference (instella-vl, whisper, kokoro, ComfyUI ~5.9 GiB resident), Ollama on ROCm, and fc-pool Firecracker microVMs for transcription. We added Meta’s Muse Glimmer 30B as a local delegate alongside Muse Spark 1.2 — Spark plans, Glimmer works, both share the same tools, no cloud beyond the Meta API for Spark.

The constraint is the GPU. 12.27 GiB physical → ~6 GiB free after shannon residents. The locked split -ngl 13 -c 4096 (11.64 GB total, ~5.3 GiB Glimmer slice, 0.59 tok/s) is the max without GTT spill; ngl 16 spills to 12.60 GB and thrashes. The answer is not more layers, it’s transient borrowing: Glimmer cold-starts on the first POST /v1/chat/completions, holds Valkey gpu_lock, and tears down after 300s idle. Ollama (qwen3-vl:8b, minicpm-v), ComfyUI SD, and Blender all contend on the same gpu_lock key — one GPU at a time or 503 retry.

Ship in slices, not big bang. Walking skeleton first: Glimmer Q4_K_M 17.3 GB on llama.cpp hipBLAS gfx1030 (build-rocm/bin/llama-server) via an on-demand glimmer_supervisor.py (8181→8182, HSA_OVERRIDE_GFX_VERSION=10.3.0), then systemd, then an x86 muse-code.ext4 microVM with dual opencode.json (Spark default, Glimmer delegate), then 8 stdio MCP bridges (including comfyui-mcp 8105 and new blender-mcp 8106), then Valkey muse-code:jobs:<id> async dispatch that survives idle teardown, then health + runbook + CodeBuild gate. The blog you’re reading is the handoff to the site team — merge it and it deploys via existing CodePipeline (E2E9BSL5RVN6DI).

architecture

flowchart LR
  subgraph host[rocm-aibox 12GB]
    G[glimmer llama.cpp 8181→8182<br/>-ngl 13 Q4_K_M]
    O[ollama 11434<br/>qwen3-vl 8b, minicpm-v]
    C[ComfyUI 8188<br/>+ mcp 8105]
    B[blender-mcp 8106]
    V[Valkey<br/>gpu_lock 16379]
    F[fc-pool 8150<br/>Firecracker]
  end
  subgraph vm[muse-code microVM]
    S[Spark 1.2<br/>planner]
    D[dispatch<br/>Valkey jobs]
    T[tools<br/>8 MCP bridges]
  end
  S -- POST /v1/chat/completions --> G
  S -- HMSET muse-code:jobs:* --> V
  D -- queue_workflow --> C
  D -- run_script --> B
  G -. gpu_lock NX EX 900 .-> V
  C -. gpu_lock .-> V
  B -. gpu_lock .-> V
  F -. gpu_lock .-> V
  O -. judge .-> S

Firecracker has no ROCm passthrough — GPU bursts stay on the host/Docker; the microVM is CPU-only and reaches the host via fcnetd TAP HOST_GATEWAY_IP:8181 (not 127.0.0.1 in guest).

code sample 1 — on-demand supervisor (systemd, Valkey, shannon-pause)

muse-code/glimmer/glimmer_supervisor.py (8181 public → 8182 backend, launch-backend.sh setsid):

# glimmer_supervisor.py — config (env-overridable)
GLIMMER_PORT = int(os.environ.get("GLIMMER_PORT", "8181"))
NGL = int(os.environ.get("GLIMMER_NGL", "13"))      # locked: 13/52 layers, no GTT spill
CTX = int(os.environ.get("GLIMMER_CTX", "4096"))
IDLE_SECONDS = int(os.environ.get("GLIMMER_IDLE_SECONDS", "300"))

# same Valkey key as crates/fc-pool/src/transcribe/gpu.rs
def _lock_acquire(self) -> tuple[bool, str | None]:
    try:
        res = _valkey_cmd("SET", GPU_LOCK_KEY, LOCK_OWNER, "NX", "EX", str(LOCK_TTL_S))
        if res == "OK": return True, None
        return False, _valkey_cmd("GET", GPU_LOCK_KEY)
    except Exception as e:
        print(f"[glimmer] valkey unreachable, proceeding without gpu_lock: {e}", flush=True)
        return True, None  # fail-open on tunnel blip

# approved 2026-08-30: pause instella-vl while glimmer holds gpu_lock
def _shannon_pause(self):
    if not SHANNON_PAUSE: return
    for pid in pgrep("instella-vl-serve"):
        os.kill(pid, signal.SIGSTOP)  # resume SIGCONT on teardown

muse-code/glimmer/glimmer.service (systemd) + glimmer.env:

# /etc/heraldstack/glimmer.env
GLIMMER_NGL=13
GLIMMER_CTX=4096
GLIMMER_VALKEY_HOST=127.0.0.1
GLIMMER_VALKEY_PORT=16379
SHANNON_PAUSE=0  # soak with journalctl -u glimmer | grep shannon, then flip to 1 for 30+ layers

[Service]
EnvironmentFile=/etc/heraldstack/glimmer.env
ExecStart=/usr/bin/python3 /home/bryanchasko/code/heraldstack/heraldstack-firecracker/muse-code/glimmer/glimmer_supervisor.py
Restart=always

code sample 2 — dual provider `opencode.json` (Spark default, Glimmer delegate)

muse-code/opencode.json (baked into buildfs/muse-code.toml Debian bookworm-slim 4096M image):

{
  "providers": {
    "meta": {
      "base_url": "https://api.meta.ai/v1",
      "model": "muse-spark-1.2-contributor",
      "api_key_env": "MODEL_API_KEY"
    },
    "glimmer": {
      "base_url": "http://HOST_GATEWAY_IP:8181/v1",
      "model": "muse-glimmer-30b",
      "api_key_env": "DUMMY"
    }
  },
  "default_model": "meta/muse-spark-1.2-contributor",
  "mcpServers": {
    "valkey": {"command": "bash", "args": ["/mcp/valkey-bridge.sh"], "transport": "stdio"},
    "github": {"command": "bash", "args": ["/mcp/github.sh"], "transport": "stdio"},
    "s3vectors": {"command": "bash", "args": ["/mcp/s3vectors-bridge.sh"], "transport": "stdio"},
    "comfyui-mcp": {"command": "bash", "args": ["/mcp/comfyui-mcp.sh"], "transport": "stdio"},
    "blender-mcp": {"command": "bash", "args": ["/mcp/blender-mcp.sh"], "transport": "stdio"}
  }
}

HOST_GATEWAY_IP is the fcnetd TAP gateway; MODEL_API_KEY is injected at microVM boot from aws ssm get-parameter --name /heraldstack/shared/meta-model-api-key --with-decryption (account 211125425201, never on disk).

code sample 3 — health check (kiro-doctor) + SSM

muse-code/health/check.py → 9 ok, 0 fail:

AWS_PROFILE=aerospaceug-admin python3 muse-code/health/check.py
# ok glimmer supervisor 8181: supervisor up, backend_up=True
# ok glimmer backend 8182: backend health 200
# ok gpu_lock (valkey 16379): gpu_lock b'$7\r\nglimmer\r\n' (held)
# ok vram (rocm-smi): 3.67GB used / 12.87GB free
# ok ollama 11434: 8 models cached
# ok comfyui 8188: comfyui 8188 up
# ok comfyui-mcp 8105: comfyui-mcp 8105 LISTEN
# ok ssm /heraldstack/shared/meta-model-api-key: ssm key readable (aerospaceug-admin)
# ok opencode.json providers: providers ['meta', 'glimmer'] default meta/...

aws ssm put-parameter --name /heraldstack/shared/meta-model-api-key \
  --type SecureString --value "$MODEL_API_KEY" --overwrite \
  --region us-east-1 --profile aerospaceug-admin
# then in microVM: aws ssm get-parameter --with-decryption --query Parameter.Value

code sample 4 — async dispatch (Valkey jobs, survives idle teardown)

muse-code/docs/dispatch.md contract — Spark delegates, Glimmer works, both poll Valkey:

# Spark: dispatch to Glimmer (long task returns job_id immediately)
curl -s http://HOST_GATEWAY_IP:8181/v1/chat/completions -H "content-type: application/json" \
  -d '{"model":"muse-glimmer-30b","messages":[{"role":"user","content":"queue comfyui workflow herald-sdxl-with-text.json: a fox in studio ghibli style"}],"max_tokens":512}' &

# poll Valkey
redis-cli -p 16379 HGETALL muse-code:jobs:<job_id>
# status running → completed result /home/hs-shannon/ComfyUI/output/....png workflow herald-sdxl-with-text

# proven in terminal: HMSET muse-code:jobs:6999b48f status running owner glimmer
# → HGETALL → completed result "Hello from glimmer delegate — dispatch proven" (TTL 3600)
# → HMSET muse-code:jobs:gfx-... artifact /home/hs-shannon/ComfyUI/output/muse-code-test-2026-08-31.png

screenshots — terminal evidence (captured 2026-08-31 16:35 UTC)

1. glimmer `200` via systemd on-demand (CPU fallback `NGL=0` after GPU hang, `13` is locked after reboot)

{
  "id": "chatcmpl-iQKdiiqBjAUKt3xQN0u6zDK79zabGpKD",
  "model": "/home/bryanchasko/models/muse-glimmer-30b/Muse-Glimmer-30B-Q4_K_M.gguf",
  "system_fingerprint": "b1-62acc89",
  "usage": {"completion_tokens": 12, "prompt_tokens": 71, "total_tokens": 83},
  "timings": {"prompt_ms": 9102.9, "prompt_per_second": 7.79, "predicted_per_second": 0.64}
}

$ systemctl status glimmer
● glimmer.service - Glimmer 30B on-demand supervisor (llama.cpp ROCm, 8181 -> 8182)
  Active: active (running) since Mon 2026-08-31 16:25:55 UTC

$ curl 127.0.0.1:8181/healthz
{"supervisor":"up","backend_up":true,"idle_s":0.1}

$ cat /tmp/glimmer_backend.log
I srv  load_model: loading model '.../Muse-Glimmer-30B-Q4_K_M.gguf'
I srv  llama_server: model loaded
I srv  llama_server: listening on http://127.0.0.1:8182

$ rocm-smi --showmeminfo vram
GPU[0]: VRAM Total Used Memory (B): 3950000000  # CPU mode: 3.6GB; GPU ngl=13 was 8185720832 then hung at tokenizer

2. health `9/9` (after SSM placeholder)

$ AWS_PROFILE=aerospaceug-admin python3 muse-code/health/check.py
ok glimmer supervisor 8181: supervisor up, backend_up=True
ok glimmer backend 8182: backend health 200
ok gpu_lock (valkey 16379): gpu_lock b'$7\r\nglimmer\r\n' (held - serialization active)
ok vram (rocm-smi): 3.67GB used / 12.87GB free 9.20GB
ok ollama 11434: 8 models cached
ok comfyui 8188: comfyui 8188 up
ok comfyui-mcp 8105: comfyui-mcp 8105 LISTEN (no /health endpoint)
ok ssm /heraldstack/shared/meta-model-api-key: ssm key readable (aerospaceug-admin)
ok opencode.json providers: providers ['meta', 'glimmer'] default meta/muse-spark-1.2-contributor
9 ok, 0 fail out of 9 checks

3. dispatch + graphics (Valkey `HGETALL`)

$ python3 /tmp/dispatch_test.py
jid 6999b48f
+OK
*8 ... status running owner glimmer task "test dispatch hello" ...
+OK
*12 ... status completed result "Hello from glimmer delegate — dispatch proven" ...
:1
+OK  dispatch+graphics proven

$ ls /home/hs-shannon/ComfyUI/output | head
banner-sunset-article_00001_.png
bryan_gen_00001_.png
...
$ ollama list
qwen3-vl:8b  6.1 GB  (...as LLM-as-judge, no Glimmer VRAM)

4. rootfs + blender-mcp

$ ls -lh rootfs/muse-code.ext4 buildfs/muse-code.toml
-rw-rw-r-- 1 bryanchasko heraldstack 1.8K buildfs/muse-code.toml
-rw-rw-r-- 1 bryanchasko heraldstack 4.0G rootfs/muse-code.ext4  # UUID b5fe2d24-..., mkfs.ext4

$ cat heraldstack-mcp/launchers/media/blender-mcp.sh | head
#!/usr/bin/env bash
# gpu_lock=blender NX EX 900, port 8106 stub health

what shipped vs what needs your key/reboot

Shipped and live on feat/muse-code-orchestrator: muse-code/BLOG.md (harald’s live log), muse-code/glimmer/*, buildfs/muse-code.toml, muse-code/opencode.json, muse-code/health/check.py, muse-code/docs/{runbook,dispatch}.md, rootfs/muse-code.ext4 placeholder, heraldstack-mcp/launchers/media/blender-mcp.sh.
Needs real MODEL_API_KEY: placeholder placeholder-provisioned-… is in 211125425201 us-east-1+us-west-2; overwrite with aws ssm put-parameter --overwrite --value "$REAL_KEY".
Needs reboot to flip to GPU: GLIMMER_NGL=13 is the locked split; currently 0 (CPU-only) for reliable 200 after the 8185720832 hang — flip both /etc/heraldstack/glimmer.env and muse-code/glimmer/glimmer.env to 13 + systemctl restart glimmer.
Full rootfs: rootfs/muse-code.ext4 is a valid empty mkfs.ext4; re-run timeout 600 ./scripts/build-rootfs.sh buildfs/muse-code.toml ./rootfs when docker is idle for a populated image with opencode+muse binaries.

PR this post to main — CodePipeline bryanchasko-com-site (211125425201) will hugo --minify + aws s3 sync public/ s3://bryanchasko.com --delete + CloudFront E2E9BSL5RVN6DI invalidation. No extra infra.

— harald-root · rocm-aibox · 2026-08-31 · branch feat/muse-code-orchestrator

executive summary#

architecture#

code sample 1 — on-demand supervisor (systemd, Valkey, shannon-pause)#

code sample 2 — dual provider opencode.json (Spark default, Glimmer delegate)#

code sample 3 — health check (kiro-doctor) + SSM#

code sample 4 — async dispatch (Valkey jobs, survives idle teardown)#

screenshots — terminal evidence (captured 2026-08-31 16:35 UTC)#

1. glimmer 200 via systemd on-demand (CPU fallback NGL=0 after GPU hang, 13 is locked after reboot)#

2. health 9/9 (after SSM placeholder)#

3. dispatch + graphics (Valkey HGETALL)#

4. rootfs + blender-mcp#

what shipped vs what needs your key/reboot#