Metering AI model traffic is easy to demo and hard to trust when it comes down to $$$$. This post is a look under the hood of nexus-proxy, which AllCode’s Nexus governance solution uses. Nexus Proxy is a usage-metering platform for AI model traffic, and the engineering decisions that let it turn raw model calls into priced, auditable, reconcilable revenue.
What it is
nexus-proxy meters every AI model request and turns it into priced, auditable usage. It ships as two independent deliverables:
- The Proxy: a LiteLLM-based Python gateway. Developers point Claude Code, Codex, the OpenAI/Anthropic SDKs, or plain curl at it. It serves three dialects (Anthropic Messages, OpenAI Chat Completions, OpenAI Responses), authenticates the caller, forwards to Amazon Bedrock, and emits a usage event for every request.
- The Usage API: a Go service that ingests those events, prices them against an effective-dated price book, reconciles against Bedrock’s own CloudWatch numbers, and closes them into immutable customer invoices.
Both pieces install standalone on AWS via a Terraform module or a CloudFormation template, or run anywhere through a docker-compose bundle. The two are deployed as container images plus infrastructure-as-code, so a customer can adopt either piece without the other.
When the end user deploys AllCode Nexus, they have the option to use either The Proxy or our desktop applications for Claude Code, Claude Co-Work, or Codex, which leverage Sidecar to collect the telemetry data.
Requirements-driven, ID-traceable
The whole system is built on a shared ID spine (REQ, NFR, ADR, ENT, API, TC, SLO, TASK), where every requirement links to the architectural decision that shaped it, the entity that stores it, the API that exposes it, and the test that proves it. A traceability checker enforces that every ID resolves both ways and that implementation phases respect their dependencies. The requirements are the source of truth; the code is downstream of them.
That discipline is what makes the “billing-grade” claim defensible: nothing in the ledger is implemented without a decision record explaining why, and nothing ships without a test tracing back to a requirement.
Architecture at a glance
The request path is deliberately asynchronous past the point of the model response, so metering never sits in the caller’s latency budget. The proxy’s job ends when it has forwarded the response and durably handed off an event. Pricing, rollups, reconciliation, and invoicing are the Usage API’s problem, and they run on their own clocks.
Developer tooling
(Claude Code / Codex / SDK / curl)
│
▼
┌────────────────────────────────────────────────┐
│ Proxy: auth + model guard + dialect routing │
└────────────────────────────────────────────────┘
│
▼
┌────────────┴─────────────────────┐
│ │
▼ ▼
┌──────────────────┐ ┌────────────────────────┐
│ Amazon Bedrock │ │ Emit usage-event.v1 │
└──────────────────┘ └────────────────────────┘
model response │
└─▶ back to Developer tooling ▼
┌────────────────────────────────────────────┐
│ Durable handoff │
│ (S3 spool + SQS/DLQ + Forwarder Lambda) │
└────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Usage API (Go) │
│ /ingest -> Pricing engine -> Rollups │
└────────────────────────────────────────────┘
│
▼
┌───────────────┴───────────────────┐
│ │
▼ ▼
┌────────────────────┐ ┌────────────────────────┐
│ Postgres ledger │ │ Bedrock reconciliation │ ◀┄┄ heartbeat pull ┄┄ Bedrock CloudWatch usage
└────────────────────┘ └────────────────────────┘
│
▼
┌────────────────────────┐
│ Month-close + invoices │
└────────────────────────┘
└─▶ Postgres ledger
Everything above “Emit usage-event.v1” is synchronous and in the hot path; everything below it runs asynchronously and can retry, spill, and reconcile without ever touching the caller.
Getting the token accounting right
The hardest correctness problem was also the least glamorous: counting tokens. LiteLLM hands the callback three different usage shapes depending on the route, and they disagree on whether the input count already includes cached tokens. Anthropic’s raw shape excludes cache tokens from input_tokens; the Chat and Responses shapes fold them back in. Bill the raw number blindly and you either double-count or undercount cache reads on every single request.
Rather than guess, we pinned the exact behavior of our LiteLLM release and encoded it as a table of “shapes.” Each shape declares which key is the input, whether that input is inclusive of cached tokens, and where the cache-read and cache-write counts may sit. The four billable classes then come out clean and shape-independent:
@dataclass(frozen=True)
class TokenCounts:
"""The four billable classes; `input` never includes a cached token.
`cache_write_1h` is the 1-hour part of `cache_write`, a subset and never a fifth total.
"""
input: int
output: int
cache_read: int
cache_write: int
cache_write_1h: int = 0
def split_usage(usage: Any) -> TokenCounts:
"""Separate cached prompt tokens from billed input, whichever way the usage was shaped."""
return read_usage(usage).tokensTwo subtleties are worth calling out because they are exactly the kind of thing that quietly corrupts a ledger:
cache_write_1his a subset, not a fifth class. It is the 1-hour-TTL portion of the cache write, and it is only trusted when its 5-minute and 1-hour parts add up to the whole write (LiteLLM’s own invariant). If they do not add up, we do not invent a number.- Anomalies are logged as numbers, never guessed. When a usage shape cannot carry a breakdown, we emit a
cache_ttl_split_missinganomaly that a log-metric filter can count, rather than papering over the gap.
The usage event: small, safe, idempotent
Every request produces one usage-event.v1. Prompt bodies never travel inline. They are gzipped, spooled to S3, and referenced by a claim-check (body.ref plus a sha256), so the event stays small and the sensitive payload stays out of the queue:
{
"schemaVersion": 1,
"eventId": "01a0c448-d325-7b2c-94a7-d1f6082e5c31",
"ts": "2026-09-21T14:05:02.117Z",
"userExternalId": "u-8a41c2de",
"teamExternalId": "platform-eng",
"authMethod": "key",
"platform": "claude-code",
"apiDialect": "anthropic_messages",
"requestedModel": "claude-sonnet-4-5",
"bedrockModelId": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",
"awsRegion": "us-east-1",
"inputTokens": 5021,
"outputTokens": 1873,
"cacheReadTokens": 0,
"cacheWriteTokens": 0,
"status": 200,
"stream": true,
"latencyMs": 18744,
"ttftMs": 612,
"bodyStatus": "pending",
"body": {
"ref": "spool/2026/09/21/01a0c448-...-5c31.json.gz",
"sha256": "3ce93215be4e138fbe1b05a17276e8582c3642ab0071219aa1d174bf923a5722",
"bytesCompressed": 6104
}
}Ingest is idempotent on (org, eventId), so a redelivered SQS message never double-bills. The event also carries timing (latencyMs, ttftMs) and streaming state, so operational dashboards and the ledger read from the same source of truth.
Auth without long-lived secrets on the developer’s machine
From a developer’s seat, the whole system is a base URL and a token. Callers sign in with their existing AWS identity, and a small helper mints a short-lived token from a presigned STS GetCallerIdentity call. There is no static API key to leak from a laptop:
export NEXUS_PROXY_URL="https://ai-proxy.example.com"
export NEXUS_AWS_AUDIENCE="<installation id>"
export NEXUS_AWS_REGION="us-east-1"
aws sso login --profile dev && export AWS_PROFILE=dev
# Any OpenAI-compatible tool (base URL WITH /v1)
curl -sS "$NEXUS_PROXY_URL/v1/chat/completions" \
-H "Authorization: Bearer $(nexus-aws-token)" \
-H 'content-type: application/json' \
-d '{"model":"claude-haiku-4-5","max_tokens":20,
"messages":[{"role":"user","content":"Say hi"}]}'For Claude Code, the same helper plugs into apiKeyHelper so tokens refresh automatically. The proxy accepts AWS-minted tokens, minted nxk_ keys, and OIDC JWTs, and it validates OIDC issuers in the canonical form the API stores them in (trailing-slash and host-case normalized) so a subtle string mismatch can never silently reject a valid token.
What “billing-grade” actually meant
Metering is easy to demo. Trusting it with money is where the engineering lives:
- Deterministic pricing. Costs are integer micro-USD, priced per model per UTC day against an effective-dated price book seeded from a verified AWS Bedrock list-price extract. Day-grain pricing is apportioned exactly across invoice lines, so the pennies always reconcile and there is no floating-point drift.
- Reconciliation against Bedrock. A heartbeat Lambda pulls Bedrock’s own CloudWatch usage, and the ledger cross-checks itself against it, flagging drift instead of quietly diverging.
- Completeness and a month-close. Events can arrive late, so the ledger tracks per-hour completeness. A close step freezes prices and writes immutable customer invoices. An independent reference oracle in the test suite recomputes revenue and expense straight from raw events and asserts the invoices equal the ledger.
- Spill and overflow handling. When the outbound queue is under pressure, overflow events spill to S3 and drain back to SQS, so a burst never drops a billable event.
- Fail-closed auth. During an outage the proxy returns 503, never a 401 that a client could misread as “you are not allowed.”
An agentic installer with a real threat model
Installing into a customer’s AWS account is treated as an adversarial boundary. Customer role assumption uses per-phase session policies (read-only discovery is genuinely read-only), the credential broker holds credentials in memory only and never logs SDK wire traffic, and every change moves through a typed proposal with a verification step and an approval window. Upgrades look up the deployed secret ARN rather than trusting client-supplied input. The installer is exercised by happy-path, lifecycle, red-team, and credential-canary suites.
Verification we can point to
The test suite is part of the deliverable, not an afterthought:
- End-to-end suites for billing, spill/overflow, Bedrock reconciliation, prompt logging, CUR expense ingestion, and AWS sign-in, all running against a fake Bedrock and LocalStack in docker-compose.
- An isolation and permission matrix that exercises every spec route against every role, so a new endpoint cannot ship without an explicit authorization answer.
- An independent revenue-and-expense oracle that reconciles invoices against raw events.
- Load and chaos harnesses (proxy overhead, API outage replay).
- A merged coverage gate that fails the build under 90%.
- A nightly cloud job that actually installs the full stack into a sandbox account through the agentic installer, then tears it down.
Closing
The theme running through all of this is that correctness with money is a systems property, not a feature. It comes from pinning the exact behavior of your dependencies, refusing to guess when a number is missing, keeping metering off the caller’s latency path, making every side effect idempotent, and proving the ledger against an independent oracle. Those are the parts that do not show up in a demo, and they are the parts that let you put a number on an invoice and defend it.
nexus-proxy is one piece of the broader AllCode Nexus platform, alongside Nexus Factory, our autonomous, PR-generating agent pipeline.