Exactly-once actions for AI agents

Let agents act on their own. Never twice.

Blackout sits between your AI agents and the real world.

  • 1Every action happens once, even when the agent crashes mid-job.
  • 2"Done" is decided by code, not the agent's word.
  • 3When it can't tell, a human decides, with the evidence.
Install in Claude Code or any MCP agent
$ npx -y blackout-mcp init
$ claude mcp add blackout -- \
    npx -y blackout-mcp --config "$PWD/blackout.mcp.json"
Agents powered by NVIDIA Nemotron 3.5 Lightning Managed Inference
open source · built at the Recursive Agents Hackathon, SF
Three promises

What Blackout guarantees, and how we know.

1

Every action happens once.

Before an agent sends, pays or deploys, Blackout writes the action down with an ID. After a crash or restart it asks your system "did this already happen?" and skips it if so.

18 → 0 duplicate emails across 10 restarted agents · 10/10 vs 0/10 clean runs after a real crash
Runtime + MCP gateway, today · where your system can answer "did operation X happen?"
Without Blackout

A crash or restart repeats the action. Double refunds, double charges, the same email twice. We measured it: a restarted agent sent 18 duplicate emails across 10 runs, and after a real crash it re-sent in 10 of 10 runs.

2

"Done" is decided by code, not the agent.

The definition of done is written and frozen before the work starts. When the agent says it's finished, plain code checks the real system against it. A confident agent can't talk its way past a failing check.

0 false "done" claims accepted · without it, the agent reported DONE in 6 of 10 runs while customers had duplicates
Full runtime, today · coming to the gateway next
Without Blackout

The agent grades its own homework. The health check is green while every job fails, and the agent reports success anyway. We measured it: the agent said DONE in 6 of 10 runs while customers had duplicates.

3

When it can't tell, a human decides.

If Blackout can't confirm what happened, it never retries blindly. It stops and hands a person the evidence: what was attempted, what the checks found.

Every duplicate in our tests was caught and escalated, never reported as success
Runtime + MCP gateway, today
Without Blackout

It guesses, and the guess is usually "try again". An unknown outcome becomes a second charge or a second message, and you hear about it from the customer. In one of our runs the agent re-sent the whole batch, 7 extra messages.

10/10jobs finished clean after a real mid-delivery crash, with Blackout. 0/10 for the same agent without it.
18 → 0duplicate emails across 10 restarted MCP agents, once routed through the Blackout gateway.
0false "done" claims accepted by Blackout's verifier, in every measured run. Without Blackout, the agent itself reported DONE in 6 runs while customers had duplicates.
½¢AI inference per mission on Crusoe, measured from real usage. Cheap enough to verify everything.

All live Nemotron 3.5 Lightning runs on Oct 17, n = 10 per arm. Every run, caveat and raw file: results/RESULTS.md.

Watch the idea in 30 seconds

One message. One crash. What happens next.

Switch between the tabs to see the same moment with Blackout, without it, and through the MCP gateway.

1/7 Ask

The agent decides to message customer 5.

Built on Crusoe + NVIDIA Nemotron

Every judgment runs on Nemotron, through Crusoe.

Every judgment in Blackout, from writing the definition of done to repairing the service to deciding what to do after a crash, runs on Nemotron 3.5 Lightning through Crusoe Managed Inference, at about half a cent per mission. Crusoe's own telemetry independently confirms our numbers. That price is what makes "verify everything" the default, not a luxury.

We picked the model by measuring, not by name. On the same crash mission, Nemotron 3.5 Lightning finished in the fewest turns, Nemotron 3 Super also finished at about six times the cost, and Nemotron 3 Nano did not make progress. Lightning drives all 40 measured missions in the results table. When capacity was tight during the event, Blackout rerouted to Nemotron 3 Super and kept going. In the recorded race above, every call in both lanes ran on Lightning.

ModelNemotron 3.5 Lightning
30B A3B, both agents, native tool calling
InferenceCrusoe Managed Inference
OpenAI-compatible endpoint, zero glue code
Cost / mission≈ $0.005
from API usage counts × $0.05 / $0.20 per 1M tokens
Live missions60+
40 in the paired ablation, all reported in the repo
Observed40 req/min · 717 ms
first token, from Crusoe's own metrics API during our runs
FallbackNemotron 3 Super
measured to complete the mission; used only on 429s
The problem

Your AI agent just refunded a customer twice.

This isn't an exotic failure. It's what happens to any worker that is killed between doing something and writing down that it did it. (An illustration. The live demo below uses a simulated notification sender.)

  1. 1:59:58
    The agent issues refund #4417. The payment goes through.
  2. 1:59:58
    The agent's process dies a moment later, before it records "refund sent". A deploy, an out-of-memory kill, a timeout.
  3. 2:00:30
    A fresh agent takes over. Its records say refund #4417 is still pending.
  4. 2:00:31
    It sends the refund again. Then reports "all done", because as far as it can tell, it is.

They double-act

A restarted agent can't tell "didn't happen" from "happened, but nobody wrote it down". Retrying feels safe. It isn't.

They say "done" when it isn't

Health checks stay green while real work fails. Agents grade their own homework, and they grade generously.

Nobody can walk away

Without a clear definition of finished and a budget, agents loop, drift and burn money overnight.

Who it's for

Teams letting agents act on their own.

Builders

AI and platform engineers

Worry: "If a worker dies mid-run, will my agent redo things it already did?"

Blackout gives: a drop-in MCP gateway around the tools that touch the world. Keep your agent, model and prompts.

Operators

Support, fintech and growth ops

Worry: "Will customers get double refunds, double charges, double emails?"

Blackout gives: each action happens once, checked against your system's own records; anything it can't confirm goes to a human.

On call

SRE and infrastructure teams

Worry: "The auto-fix bot said it was resolved. Was it?"

Blackout gives: "done" decided by real checks, and a clear escalation with evidence when it can't finish.

Oversight

Risk and compliance

Worry: "Can we show what the agent did, and why it was accepted?"

Blackout gives: a frozen definition of done, receipts for every action, and an auditable proof-of-done record per run.

Use cases

Anywhere an agent does something you can't take back.

Sending, paying, creating, deploying. And anywhere "the agent said it's done" isn't good enough.

Today works now with npx blackout-mcp in front of an MCP tool · Adapter needs a small integration: a way to ask that system "did operation X happen?" · Full runtime uses the full runtime with a frozen definition of done and a code verifier. Tested so far on a simulated notification sender, an MCP mailer, and Claude Code.

Developers using coding agents

Claude Code, Cursor, Hermes and friends
  • Posting to Slack, GitHub or email Today

    A retried or restarted session can't post the same release note, PR comment or reply twice.

  • Creating tickets, issues and invites Today

    No duplicate Jira tickets, GitHub issues or calendar invites when the agent loops.

  • Long sessions that crash or hit limits Today

    The next session picks up the job without redoing what already went out.

Companies running agents unattended

Support, ops and growth
  • Customer messaging Today Adapter

    Order updates, reminders and sequences: each customer gets each message once, across crashes and redeploys.

  • Refunds and credits Adapter

    Support agents can't refund the same customer twice.

  • CRM and sales agents Today Adapter

    No double-sent sequences or duplicate contacts.

Finance and payments

Where a duplicate is a finance incident
  • Payouts, invoicing, billing runs Adapter Full runtime

    Every account settled once, and "done" only when totals reconcile in code.

Platform, SRE and infrastructure

Bots that touch production
  • Auto-remediation Full runtime

    No double rollbacks or restarts, and "fixed" is decided by real checks, not /health 200.

  • Provisioning Adapter

    No duplicate VMs, buckets or DNS records, and no surprise bills, when the agent retries.

Data teams

Moving data with agents
  • Backfills, migrations, syncs Adapter Full runtime

    Every record moved once; checks must pass before anyone says done.

Agent platforms and framework builders

Shipping agents to your users
  • Exactly-once for every connected tool Today

    Put Blackout in front of the tools your users connect. Each tool author doesn't have to rebuild it.

Compliance, risk and audit

Proving what an agent did
  • An auditable record per run Full runtime

    A frozen definition of done, an ID for every action, receipts, and a verdict from code instead of a chat log.

How it works · architecture

The model proposes. Code decides.

Nemotron does the thinking: what "done" means, how to fix the service, what to try next. Everything that has to be guaranteed, like IDs, receipts, fencing and the final verdict, is plain code with no AI in it.

Nemotron on Crusoe a model judgment { } Code, no AI a guarantee
  1. 1
    Snapshot the job{ } Code, no AI

    Before anything changes, the controller reads the service: which 10 jobs are in the batch and which customers already got a message.

  2. 2
    Propose the definition of doneNemotron on Crusoe

    The coordinator agent picks checks from a fixed menu (every job delivered, no duplicates, earlier outputs untouched, a fresh test job works…) and states its assumptions. It can't write checker code.

  3. 3
    Freeze the contract{ } Code, no AI

    Blackout adds the five checks it may never drop, rejects checks that are false from the start, hashes the result and stores it in the journal. Nobody can lower the bar later.

  4. 4
    Diagnose and repairNemotron on Crusoe

    A worker agent investigates with 12 tools (logs, workspace files, config patch, restart, test job, receipts, process jobs) and chooses every call itself.

  5. 5
    Write it down, then act{ } Code, no AI

    Every tool call that changes the world gets a stable operation ID, saved to SQLite before the request goes out. The ID travels with the request, so the service's receipt carries it.

  6. 6
    Crash and fence{ } Code, no AI

    We SIGKILL the worker right after a send. The controller starts a new worker generation; anything the dead one still tries to write is rejected.

  7. 7
    Reconcile from receipts{ } Code, no AI

    Before the new worker's model sees anything, Blackout asks the service about each unfinished operation ID. Happened: mark done. Never arrived: safe to retry with the same ID. Can't tell: escalate.

  8. 8
    Verify{ } Code, no AI

    The verifier reads the service's real database and checks it against the frozen contract. The agent's own "done" counts for nothing.

  9. 9
    Decide what's nextNemotron on Crusoe

    The coordinator reads the verifier's evidence and chooses continue, replan or escalate. It can't declare success, and the controller forces a replan when there's no progress.

  10. 10
    Verified, or a human decides{ } Code, no AI

    The mission ends VERIFIED only on a passing verdict. Otherwise it stops with an escalation report: what was checked, what failed, which operations are unknown.

The MCP gateway (npx blackout-mcp) is steps 5 and 7 packaged for any MCP agent. It has no model of its own: the agent brings its own, Nemotron, Claude or anything else.

See it work

Same AI agent. Same broken service. Same crash. One has Blackout.

A simulated notification sender breaks at 2am with 10 customer messages queued. Two copies of the same Nemotron agent fix it, and we kill each one mid-delivery. Press play: this is a recorded live run, replayed at 2× speed.

Everything in the replay came from real processes: Nemotron agents on Crusoe, a real SIGKILL, a verifier reading the service's database. Run it live: npm run web from the repo.

Use it today · MCP gateway

Put Blackout between your agent and its tools.

The Blackout MCP gateway works with any agent that calls tools over MCP. Your agent doesn't change: point it at the gateway instead of the tool server, and mark which tools have side effects.

# Try it in Claude Code (bundled demo mailer, no setup)
$ npx -y blackout-mcp init
$ claude mcp add blackout -- npx -y blackout-mcp --config "$PWD/blackout.mcp.json"
# then ask Claude to send the same email twice: the second comes back "duplicate suppressed"
// blackout.mcp.json, in front of your own MCP server
{
  "mission": "order-shipped-backlog",
  "downstream": { "command": "node", "args": ["your-mcp-server.js"] },
  "sideEffects": {
    "send_email": { "idempotencyArg": "idempotency_key",
                    "receipt": { "tool": "get_receipt", "arg": "idempotency_key" } }
  }
}

$ npx -y blackout-mcp --config blackout.mcp.json   # a stdio MCP server; add it to any agent
The insight

Agents already try to be careful. That's not enough.

Given an optional idempotency key, Nemotron made one up on its own. But after a restart it made up a different one, so the server saw a brand-new request and sent the email again. You can't prompt your way out of this: the key has to survive the agent.

Agent alonesession 1: ana@example.com→restart: order_shipped_ananew key, sent again ✗
Through Blackoutsession 1: bo-58f1…→restart: bo-58f1…same op, suppressed ✓

Blackout derives the key from the call itself and journals it before the call runs. Measured: 6 of 10 direct runs duplicated (18 emails); 0 of 10 through the gateway.

Repeats are suppressed

Every side effect gets a stable ID and is journaled before it runs. A retry or a restarted agent gets the original result back, not a second email.

Crashes are reconciled

If the gateway dies mid-call, it asks your system's receipt tool what happened before allowing a retry. If it can't tell, it refuses and escalates.

Measured live, n = 10

Same Nemotron agent, same task, restarted mid-job, told "don't send duplicates". Through the gateway: 0 of 10 runs duplicated. Direct: 6 of 10 did, 18 duplicate emails, and said DONE anyway. It invented idempotency keys, but the restarted agent invented different ones.

The gateway handles exactly-once side effects. "Code decides done" (frozen contract plus verifier) is the full Blackout runtime shown in the race above. Works with anything that can answer "did operation X happen?" by an ID you choose: Stripe idempotency keys, SQS FIFO dedup IDs, a Postgres outbox.

Where it goes

From a hackathon protocol to the reliability layer for agents.

Today

Runtime + MCP gateway

Journal, frozen contracts, receipt reconciliation, code verifier, budgets and escalation. A drop-in MCP gateway for any agent. 60+ live missions, 25 tests.

Next

Done-checks in the gateway

Bring frozen contracts and the verifier to the gateway, plus an SDK for frameworks that don't use MCP.

Then

Real adapters

Postgres outbox, Stripe test mode, email and SMS providers with idempotency keys, Kubernetes.

Later

Agent fleets on Crusoe

A self-hosted gateway for agent fleets on Crusoe Managed Kubernetes, with Nemotron on Crusoe drafting done-checks and summarising escalations. Code still makes the final call.

Building agents that send, pay or provision?

Blackout is open source. Star it, try the race, or open an issue to tell us what your agents touch. That decides which adapters we build first.