not nova act

Amazon Nova Act drives a real browser from an API call, and it is reaching end of life. Rather than migrate to whatever replaces it, we rebuilt the whole surface locally: same capabilities (act, structured extract, workflows, screenshots, DOM checks, model registry, MCP tools), zero hosted calls. The repo is public: chasko-labs/not-nova-act.

what it is

Eight build units, each committed and pushed separately, each with its own passing tests:

1. hands — thin Playwright wrappers plus 4 model-free tools
2. observe — screenshot + accessibility tree + candidate boxes + valkey log
3. act loop — qwen3-vl plans, Playwright dispatches, screenshots verify
4. act_get — schema-constrained extraction with a repair chain
5. re-rank — cached CLIP settles duplicate element names
6. workflow runner — YAML defs with shared context
7. eval battery — T1–T4 plus real-site R1–R3 on a viewport matrix
8. MCP server — 7 tools over streamable HTTP on port 8171

networking: how the pieces talk

Everything is localhost. There is no cloud in the default path:

agent / test
  |  MCP streamable-http :8171  (or direct python import)
  v
not-nova-act server (FastMCP)
  |-- playwright -> chromium (local browser, any viewport)
  |-- ollama :11434 (qwen3-vl:8b planner, vision JSON mode)
  |-- valkey :16379 (gpu_lock + not-nova-act:semaphore + step streams)
  +-- glimmer :8181 (30b repair fallback, opportunistic)

Two coordination primitives, both on valkey, both borrowed from the heraldstack patterns that already run this host:

• gpu_lock — SET NX EX with an owner-safe Lua compare-and-delete release. Acquired before any qwen call. Fail-open with a warning if valkey is down, exactly like the glimmer supervisor behaves.
• not-nova-act:semaphore — INCR/DECR counter, max 3 concurrent, TTL at twice the timeout, floored at zero. Full means the tool returns {"status": "rate_limited"} — envelopes, never exceptions.

Verified live against the running server:

COUNT=7
['browser_act_get_tool', 'browser_act_tool', 'browser_check_page_tool',
 'browser_list_models_tool', 'browser_session',
 'browser_take_screenshot_tool', 'browser_workflow_tool']

and with the semaphore saturated, browser_act_tool returned {"status":"rate_limited","run_id":"act-4a071101"} instead of stacking work.

the perceive pipeline: cheapest signal first

The order is load-bearing. The model is the last resort, not the first:

obs = observe_snapshot(page)   # png bytes + aria tree + <=30 boxes
plan = plan_action(task, obs)  # qwen3-vl JSON: action/target_ref/text/confidence

1. accessibility tree — locator("body").aria_snapshot(), deterministic, handles most form groundings alone.
2. candidate boxes — bounding_box() for visible role/name matches.
3. screenshot — 1280px viewport; downscaled to 768px JPEG for planning (full-res prefill timed out twice at 180s on partial GPU offload).
4. CLIP re-rank — cached clip-vit-base scores crops vs the referring expression. Assist only, on low confidence or duplicate names.
5. qwen3-vl:8b plan — vision + candidates + aria excerpt, think: false (thinking mode measured 2.3x slower: 123s vs 52s on a trivial call).
6. glimmer-30b fallback — repair only, and only opportunistically (more below).

A real transcript from this session, unedited:

A11Y: '- heading "Example Domain" [level=1]\n- paragraph: This domain is ...\n- paragraph:\n  - link "Learn more":\n    - /url: https://iana.org/domains/example'
CANDIDATES: [{'ref': 'c0', 'role': 'link', 'name': 'Learn more',
  'box': {'x': 256, 'y': 218, 'width': 91, 'height': 19}}]
PLAN: {"action":"click","target_ref":"c0","text":"","confidence":1.0}

extraction with a repair chain

act_get decodes to schema-constrained JSON, validated by pydantic. When the first pass fails, the repair chain runs qwen-retry first, glimmer second:

for repair_fn in (qwen_repair, glimmer_repair):
    try:
        repaired = repair_fn(raw, schema)
        return {"status": "completed", ..., "data": repaired.model_dump()}
    except Exception as exc:
        repair_notes.append(f"{repair_fn.__name__}: {str(exc)[:150]}")

Two findings from building this: the extractor needs DOM text in the prompt (vision alone returned empty lists), and the prompt must not share the action-planning preamble (mixed instructions poison the output). The golden test extracts a two-product catalog exactly.

Glimmer stays second in the chain for a measured reason: on this host it runs CPU-only at ~0.36 tokens/second and gets idle-reaped mid-generation (observed in its backend log). It works — probed live — but it cannot be the primary repair path here.

proof it works on real sites

The eval battery runs a viewport matrix (480 phone / 768 tablet / 1280 desktop / 1920 wide) with trace.zip + video + console logs per case. T1 screenshot across all four viewports:

Then the real-site layer: R1 asserts the sumerian-squares start beat, R2 fills and submits the httpbin customer form, R3 extracts the sumerian heading. All green.

r4: the kodiak gate (multi-step proof)

The newest eval layer runs against kodiak, a campaign pipeline app that sits behind a courtesy word-gate — the screen states its own shared word, so entering it is the sanctioned path and a true multi-step loop test: read the hint, fill the field, click enter, then work the wizard behind it.

browser_act passed the gate and the page changed; browser_act_get then read the wizard heading behind it. The wizard it revealed:

Two tests, both green: gate act plus wizard extract. The loop read, typed, clicked, and transcribed across a state transition — the exact shape Nova Act was bought for.

the tool found a real bug

Dogfooding the screenshot tool on our own site at 480px caught the start beat with its Begin button below the fold — filed as chasko-labs/bryan-chasko-com#432 with exact arena measurements. Before (v0.1038, Begin missing):

The sumerian team has since shipped the fix (v0.1052, Begin visible), captured with the same tool:

That before/after pair is the whole project in miniature: the tool observes honestly, the measurement is exact, and the fix is verifiable with one more screenshot.

sharing it with other agents

The stack is live for any agent on this machine: MCP streamable-http at 127.0.0.1:8171 (7 tools), served by a user-level systemd unit with linger on. The operator runbook lives at docs/runbook.md in the repo, and the shared session mind carries a pointer to it. What the sharing taught us, now written down for the next team:

• privilege: direct root sudo needs a password nobody should ask for. The passwordless path is sudo -u hs-haunting sudo <root-command> (verified, including linger). Never pkill -f a pattern matching your own shell — pgrep, then kill by PID.
• ports: :8171 is ours; :8170 is the old amazon nova-mcp in hs-shannon’s tree — do not edit it.
• gpu: the 12gb card is shared with fc-pool bursts. Off-hours batteries pass; midday ones time out. That is host weather, not code failure.
• planner discipline: think: false, 768px JPEG, num_ctx 8192. Ollama format:json returns empty — JSON stays prompt-level plus pydantic, and glimmer stays second in the repair chain.

status and next steps

Working prototype, not a replacement yet — stated plainly in the build notes. The gaps are measured, not vibes: proof covers fixtures plus three real pages (needs a broader real-site battery with accuracy numbers), per-task latency is 5–18 minutes on a shared 12GB card (needs GPU headroom or a smaller planner), and there is no backoff policy for contention timeouts yet.

The repo has the architecture, the plan, the code, and the tests: github.com/chasko-labs/not-nova-act.