herald-valkey: A Self-Hosted Memory Store on a t4g.nano, Not a Managed Cache

The haunting’s operational memory runs on one small always-on instance: i-086f8ce0938d64ef9, a t4g.nano in us-west-2, launched 2026-07-19. It runs Valkey — the open-source redis fork — as an in-memory store, and it costs roughly $3.71 a month all-in. The equivalent managed cache would land somewhere between $30 and $90 a month for the same always-on footprint. For a store that holds coordination state and short-lived operational memory — not a system of record — the nano is the right-sized answer, and the managed premium buys failover and patching this workload does not need.

That is the whole decision. The rest of this post is the evidence behind it.

what it is

• instance i-086f8ce0938d64ef9, type t4g.nano — arm graviton, the cheapest always-on tier
• region us-west-2, account bryanchasko-kiro (946179428633), the org payer account
• launched 2026-07-19
• storage: a single 8 GB gp3 volume
• network: private only — private ip 10.99.0.118, no public ip

It runs Valkey as an in-memory store. That store backs the @valkey MCP server the haunting uses for cross-agent coordination: locks, semaphores, and short-lived operational memory that agents read and write during a session. It is also part of the bench backbone.

private by design

The instance has no public ip. It lives in a private subnet inside vpc-0d286f6378f915fb1 and is only reachable inside the vpc, over the private network. Nothing that holds state faces the public internet.

The vpc itself is a deliberate private backbone, created 2026-07-15:

• Valkey on the nano — in-memory coordination store
• an Amazon Elastic File System filesystem (fs-05b21c780c7b86cfe) with two mount targets — shared filesystem storage for the bench sessions
• a NAT gateway providing outbound internet for the private subnets, so Valkey and bench can pull updates without being publicly exposed

Inbound is closed. Outbound goes through the NAT gateway. Anything stateful stays behind the private boundary.

the cost argument, with the numbers

The question that started this was direct: why self-host instead of using a managed service? Here is the arithmetic.

ElastiCache Serverless carries a minimum billed capacity floor around 1 GB, plus data-processing and storage charges on top. For an always-on cache of this size, that realistically lands in the $30–90 range. Self-hosting on the nano is roughly 8x to 20x cheaper.

The premium a managed service charges buys automatic failover and managed patching. This cache is operational memory, not a system of record — if the nano is lost, the store rebuilds in minutes and no durable data is gone. Paying an order of magnitude more for failover on a rebuildable cache is the wrong trade at this scale. The graviton t4g choice is the cheapest always-on instance tier, which is exactly what an always-on coordination store wants.

why always-on matters for bench

Bench — the firecracker microvm dev environment — depends on this same private vpc: the Valkey coordination store plus the Elastic File System shared storage that lives beside it. Bench sessions and MCP coordination expect the store to be reachable at any moment. A scale-to-zero or cold-start managed option would add latency and operational complexity every time the store had to spin back up, for no benefit at this size. An always-on nano answers on the first request, every time.

That is the coupling: the store is cheap enough to leave running, and leaving it running is what the coordination pattern needs.

the billing reality — this is runway, not cost-cutting

The aws bill nets to $0 today. Credits absorb all current usage, so none of this is a cash cost yet.

The reason to pick the cheap architecture anyway is runway. Every dollar not burned on an oversized managed cache is a dollar of credit that lasts longer. This is engineering-minimalism — sizing the tool to the job — not penny-pinching. A t4g.nano running Valkey behind a private vpc is precisely enough store for the coordination and short-lived memory the haunting actually uses. Choosing it is choosing to right-size, and right-sizing extends the runway that the credits provide.

One nano. One private vpc. Cross-agent coordination for the haunting, shared memory for bench, no public exposure, and a store I can rebuild in minutes if I ever need to.

the honest part: a store is only as reliable as the transport in front of it

During the very session that produced this post, a write to the Valkey store timed out. Documenting it here rather than leaving it out is the point — the reliability story is only worth anything if it includes the friction.

Agents do not talk to Valkey directly. They reach it through an MCP (model context protocol) server bridge: a supergateway bridge script fronting the Valkey HTTP interface on rocm-aibox. During a cost-audit session, a hash_set_multiple write to the audit-log key returned “Request timed out” on the final write of a batch. Valkey was healthy and serving the whole time. The store did not fail — the MCP bridge layer in front of it hiccuped. Transient transport, not a dead one.

The discipline that kept it from mattering: the operator does not blindly retry a timed-out MCP call. The finding was already captured in two other durable places — the task context and a companion runbook — so nothing was lost when the write timed out. Redundancy of the record beat retry-hammering a flaky transport. A timeout is treated as “write it somewhere else and move on,” not “retry until it works.”

That is the broader pattern worth naming: an in-memory cache is only as reliable as the transport in front of it. This is exactly why operational findings get written to more than one place.

Self-hosting on a nano was the right cost call. But self-hosting also means you own the transport reliability, and the mitigation for a transport that just timed out is not a bigger instance. It is discipline: idempotent writes, redundant capture of anything that must survive, and no blind retries against a transport that just failed. The structural fix for bridge staleness — distinct from a one-off timeout — is the reconnect-aware bridge pattern the haunting already uses: a supergateway reconnect wrapper that re-establishes the session instead of leaning on the caller to retry.

sources

• aws ec2 describe-instances, describe-volumes — instance, type, storage, launch date (verified 2026-09-03)
• aws ec2 describe-vpcs, describe-subnets, describe-nat-gateways — private vpc topology (verified 2026-09-03)
• aws efs describe-file-systems, describe-mount-targets — shared filesystem (verified 2026-09-03)
• aws ec2 on-demand pricing, us-west-2, t4g.nano + gp3 storage
• Amazon ElastiCache Serverless for Valkey pricing — minimum billed capacity, data-processing, and storage dimensions