not nova act

Amazon Nova Act drives a real browser from an API call, and it is reaching end of life. Rather than migrate to whatever replaces it, we rebuilt the whole surface locally: same capabilities (act, structured extract, workflows, screenshots, DOM checks, model registry, MCP tools), zero hosted calls. The repo is public: chasko-labs/not-nova-act.

what it is

Eight build units, each committed and pushed separately, each with its own passing tests:

1. hands — thin Playwright wrappers plus 4 model-free tools
2. observe — screenshot + accessibility tree + candidate boxes + valkey log
3. act loop — qwen3-vl plans, Playwright dispatches, screenshots verify
4. act_get — schema-constrained extraction with a repair chain
5. re-rank — cached CLIP settles duplicate element names
6. workflow runner — YAML defs with shared context
7. eval battery — T1–T4 plus real-site R1–R3 on a viewport matrix
8. MCP server — 9 tools over streamable HTTP on port 8171

networking: how the pieces talk

Everything is localhost. There is no cloud in the default path:

agent / test
  |  MCP streamable-http :8171  (or direct python import)
  v
not-nova-act server (FastMCP)
  |-- playwright -> chromium (local browser, any viewport)
  |-- ollama :11434 (qwen3-vl:8b planner, vision JSON mode)
  |-- valkey :16379 (gpu_lock + not-nova-act:semaphore + step streams)
  +-- glimmer :8181 (30b repair fallback, opportunistic)

Two coordination primitives, both on valkey, both borrowed from the heraldstack patterns that already run this host:

• gpu_lock — SET NX EX with an owner-safe Lua compare-and-delete release. Acquired before any qwen call. Fail-open with a warning if valkey is down, exactly like the glimmer supervisor behaves.
• not-nova-act:semaphore — INCR/DECR counter, max 3 concurrent, TTL at twice the timeout, floored at zero. Full means the tool returns {"status": "rate_limited"} — envelopes, never exceptions.

Verified live against the running server:

COUNT=9
['browser_act_get_tool', 'browser_act_tool', 'browser_assert_visual_tool',
 'browser_check_page_tool', 'browser_compress_shot_tool',
 'browser_list_models_tool', 'browser_session',
 'browser_take_screenshot_tool', 'browser_workflow_tool']

The two newest tools landed in 54a30af (jpeg compression plus local-model visual assert): browser_assert_visual_tool returns a local-model true/false verdict on a visual claim, and browser_compress_shot_tool is the JPEG sibling of the screenshot tool for vision contexts that want smaller payloads.

And with the semaphore saturated, browser_act_tool returned {"status":"rate_limited","run_id":"act-4a071101"} instead of stacking work.

the perceive pipeline: cheapest signal first

The order is load-bearing. The model is the last resort, not the first:

obs = observe_snapshot(page)   # png bytes + aria tree + <=30 boxes
plan = plan_action(task, obs)  # qwen3-vl JSON: action/target_ref/text/confidence

1. accessibility tree — locator("body").aria_snapshot(), deterministic, handles most form groundings alone.
2. candidate boxes — bounding_box() for visible role/name matches.
3. screenshot — default 1280x800, but the viewport is now a parameter (see below), not a fixed assumption; downscaled to 768px JPEG for planning (full-res prefill timed out twice at 180s on partial GPU offload).
4. CLIP re-rank — cached clip-vit-base scores crops vs the referring expression. Assist only, on low confidence or duplicate names.
5. qwen3-vl:8b plan — vision + candidates + aria excerpt, think: false (thinking mode measured 2.3x slower: 123s vs 52s on a trivial call).
6. glimmer-30b fallback — repair only, and only opportunistically (more below).

A real transcript from this session, unedited:

A11Y: '- heading "Example Domain" [level=1]\n- paragraph: This domain is ...\n- paragraph:\n  - link "Learn more":\n    - /url: https://iana.org/domains/example'
CANDIDATES: [{'ref': 'c0', 'role': 'link', 'name': 'Learn more',
  'box': {'x': 256, 'y': 218, 'width': 91, 'height': 19}}]
PLAN: {"action":"click","target_ref":"c0","text":"","confidence":1.0}

extraction with a repair chain

act_get decodes to schema-constrained JSON, validated by pydantic. When the first pass fails, the repair chain runs qwen-retry first, glimmer second:

for repair_fn in (qwen_repair, glimmer_repair):
    try:
        repaired = repair_fn(raw, schema)
        return {"status": "completed", ..., "data": repaired.model_dump()}
    except Exception as exc:
        repair_notes.append(f"{repair_fn.__name__}: {str(exc)[:150]}")

Two findings from building this: the extractor needs DOM text in the prompt (vision alone returned empty lists), and the prompt must not share the action-planning preamble (mixed instructions poison the output). The golden test extracts a two-product catalog exactly.

Glimmer stays second in the chain for a measured reason: on this host it runs CPU-only at ~0.36 tokens/second and gets idle-reaped mid-generation (observed in its backend log). It works — probed live — but it cannot be the primary repair path here.

proof it works on real sites

The eval battery runs a viewport matrix (480 phone / 768 tablet / 1280 desktop / 1920 wide) with trace.zip + video + console logs per case. T1 screenshot across all four viewports:

Then the real-site layer: R1 asserts the sumerian-squares start beat, R2 fills and submits the httpbin customer form, R3 extracts the sumerian heading. All green.

r4: the kodiak gate (multi-step proof)

The newest eval layer runs against kodiak, a campaign pipeline app that sits behind a courtesy word-gate — the screen states its own shared word, so entering it is the sanctioned path and a true multi-step loop test: read the hint, fill the field, click enter, then work the wizard behind it.

browser_act passed the gate and the page changed; browser_act_get then read the wizard heading behind it. The wizard it revealed:

Two tests, both green: gate act plus wizard extract. The loop read, typed, clicked, and transcribed across a state transition — the exact shape Nova Act was bought for.

rendering honestly: viewport, webgl, and real mobile

Three capabilities merged this session closed the gap between “screenshot” and “screenshot that reflects what a real device sees.”

arbitrary viewport (PR #1, 1c59767). browser_take_screenshot_tool and browser_check_page_tool now accept a viewport {width, height}, threaded straight through to Playwright new_page. Before this, everything rendered at the default 1280x800 regardless of what you asked for — the viewport arg was simply not wired.

webgl actually paints headless (PR #2, e571b73). A bare chromium.launch() left the canvas unpainted — a gl.readPixels center-sample came back [0,0,0,0], transparent nothing. The fix is a shared launch-args constant, the SwiftShader-via-ANGLE flags:

CHROMIUM_GL_ARGS = [
    "--use-gl=angle",
    "--use-angle=swiftshader",
    "--enable-unsafe-swiftshader",
    "--ignore-gpu-blocklist",
    "--enable-webgl",
]

applied at both launch sites — p.chromium.launch(args=CHROMIUM_GL_ARGS). SwiftShader ships inside Playwright’s own chromium (libvk_swiftshader.so), so this is self-contained: WebGL renders to the captured framebuffer with no GPU driver dependency. The proof is the sumerian-squares alien host, rendered large and face-forward via ?square=alien at 1280 — a frame that was blank before the fix:

true mobile device emulation (PR #3, bd4b045). An opt-in mobile param enables CDP is_mobile + device_scale_factor + has_touch via new_context:

browser.new_context(
    viewport={"width": w, "height": h},
    is_mobile=True,
    device_scale_factor=2,
    has_touch=True,
)

This was a genuine gotcha. Without is_mobile, a 375 viewport only resized the window — device-width still resolved to the ~1280 screen width, so @media (max-width: 480px) rules never fired and width=device-width layouts evaluated at desktop metrics. It produced a false “off-canvas” reading until it was found and fixed. With device metrics on, device-width tracks the emulated width and the mobile media queries match. The game at a true 375 device viewport, device_scale_factor 2 (so the PNG is 750x1624):

the tool found a real bug

Dogfooding the screenshot tool on our own site at 480px caught the start beat with its Begin button below the fold — filed as chasko-labs/bryan-chasko-com#432 with exact arena measurements. Before (v0.1038, Begin missing):

The sumerian team has since shipped the fix (v0.1052, Begin visible), captured with the same tool:

That before/after pair is the whole project in miniature: the tool observes honestly, the measurement is exact, and the fix is verifiable with one more screenshot.

Since then the same tool has driven fixes beyond that single beat. It caught a tablet-768 breakpoint hole — the arena was a flex row splitting two children side-by-side at 768 portrait, fixed with flex-direction: column and shipped v0.1056 — and it drove an asset-perf investigation (#285): the geometry probes plus a readiness-flag capture recipe (?square=<hostId>, then poll window.__ssSquareReady before shooting) proved the host GLBs are 98.5% uncompressed textures, not a Draco problem. Layout bugs and asset-perf bugs, both at real mobile device metrics.

sharing it with other agents

The stack is live for any agent on this machine: MCP streamable-http at 127.0.0.1:8171 (9 tools), served by a user-level systemd unit with linger on. The operator runbook lives at docs/runbook.md in the repo, and the shared session mind carries a pointer to it. What the sharing taught us, now written down for the next team:

• privilege: direct root sudo needs a password nobody should ask for. The passwordless path is sudo -u hs-haunting sudo <root-command> (verified, including linger). Never pkill -f a pattern matching your own shell — pgrep, then kill by PID.
• ports: :8171 is ours; :8170 is the old amazon nova-mcp in hs-shannon’s tree — do not edit it.
• gpu: the 12gb card is shared with fc-pool bursts. Off-hours batteries pass; midday ones time out. That is host weather, not code failure.
• planner discipline: think: false, 768px JPEG, num_ctx 8192. Ollama format:json returns empty — JSON stays prompt-level plus pydantic, and glimmer stays second in the repair chain.

a different repo wired the judge in as an oracle

The first real downstream consumer arrived from outside this repo. The creative-automation-pipeline (a Kodiak Cakes image-generation tool) does not test browsers at all — it generates branded images through Amazon Bedrock Stability control-structure, restyling a seed photo into a brand theme. Its problem was a different shape than anything above: an intermittent failure where the model hallucinates garbled fake signage into the scene. One earlier render invented a round sign reading “ROSITAR” in nonsense letters, which is why the team calls it the ROSITAR gap.

Nothing new landed in not-nova-act for this. The pipeline reached in and consumed the tool that was already here — assert_visual, the local-model visual verdict — as a pluggable judge in a generate-and-judge loop. The local qwen VQA that decides “is the Begin button visible” now decides “is there any text or lettering in this render,” on AI-generated images instead of browser screenshots. Same tool, a caller in a different repo, exactly the reuse the runbook was written for.

The pipeline built a parameter-sweep harness that renders a controlled sweep of one diffusion parameter and judges each output. The judge is an injectable contract, not hardcoded — the default shells assert_visual in the not-nova-act checkout, so the judgment step costs zero AWS calls and runs against local qwen3-vl:8b:

# the judge is a pluggable contract, not hardcoded:
Judge = Callable[[Path, str], tuple[bool, str]]   # (image, question) -> (verdict, detail)

# default: not-nova-act's local qwen VQA, zero AWS cost
def default_not_nova_act_judge(image_path, question):
    # shells `uv run` in the not-nova-act checkout -> assert_visual(image, question)
    # returns (verdict_bool, detail_str) from qwen3-vl:8b
    ...

# same harness, any visual question, any diffusion param:
sweep_param(param="seed_value", values=[101..110],
            assertions={"has_text": "Is there any text or lettering in this image?"},
            judge=default_not_nova_act_judge)

The judge did real epistemic work twice. First it killed a theory. The prevailing belief was that the hallucinated text was a control_strength problem — turn the Bedrock parameter down and it goes away. The harness swept control_strength across 0.7 / 0.55 / 0.4 / 0.3 and the judge called every render text-clean, including 0.7. If control_strength were the lever, 0.7 would reproduce the bug and 0.3 would change it; neither happened. A plausible hypothesis became a measured wrong answer.

Then it measured the real rate. Swapping the sweep to the seed across 10 renders of a sign-heavy prompt, 2 of 10 came back with garbled text — a ~20% hallucination rate, quantified without a human eyeballing a single frame. The qwen judge flagged seed 102 with the exact strings it read off the render: “SUUTLATIN BEAUR”, “KOUTAK”, “1925 2027”, “BALL STILEE”, “POUSTERS”. The ROSITAR class, caught automatically:

A local vision model, wired in as a cheap injectable oracle, replaced human eyeballing in a generate-and-judge loop and did the epistemic work: it falsified a wrong lever, then put a number on the defect. That is the tool being used the way the sharing section intends — by a caller who never touches a browser.

status and next steps

Working prototype, not a replacement yet — stated plainly in the build notes. The gaps are measured, not vibes: proof covers fixtures plus three real pages (needs a broader real-site battery with accuracy numbers), per-task latency is 5–18 minutes on a shared 12GB card (needs GPU headroom or a smaller planner), and there is no backoff policy for contention timeouts yet.

One of those measured gaps was the tool’s own: the mobile-emulation hole (viewport-resize without device metrics) produced a false “off-canvas” diagnosis before it was found and closed with is_mobile. The limitation was measured, not hand-waved, and the fix is now proven with the 375 device screenshot above — which is exactly the ethos the project is built on.

The repo has the architecture, the plan, the code, and the tests: github.com/chasko-labs/not-nova-act.