A behind-the-scenes look at one week of changes to Nexus Factory, AllCode’s autonomous software factory. This covers both the work our human engineers did and the changes the factory shipped to itself.
The week at a glance
This past week split cleanly into three themes:
- A new public metrics endpoint so anyone can see live pipeline activity.
- A serious infrastructure cleanup around AWS Bedrock AgentCore that stopped a runaway cost loop and hardened our deploys.
- The factory writing its own docs: several changes were planned, coded, reviewed, and auto-merged by Nexus Factory itself, with no human in the loop.
Thirteen commits landed on main between October 20 and October 23. Here’s what mattered.
1. A public pipeline-stats API
Commit: e5287602 · feat(factory-web): add unauthenticated GET /api/stats pipeline metrics endpoint
We added a new unauthenticated GET /api/stats endpoint to the web service (factory-web). It exposes live pipeline metrics, the same kind of numbers you see tick up on the dashboard at nexusfactory.allcode.com, without requiring a login.
The change was small and self-contained: a new stats.rs handler (~247 lines), wired into the handler module and the crate’s router. Because it’s unauthenticated and read-only, it’s a clean building block for status badges, external dashboards, and “is the factory busy right now?” checks.
What makes this one notable isn’t the size, it’s the author. This feature was built by the factory itself, then auto-merged into main.
2. Taming AWS Bedrock AgentCore (the expensive part)
The biggest engineering story of the week was infrastructure, and it’s a good illustration of how a small cloud misconfiguration can quietly cost real money.
Commit: df638e96 · Fix ECS secrets ARN mismatch, cut prod AgentCore runtime off its capacity provider, add AgentCore version sweeper
Three distinct fixes shipped together here:
Cutting the runtime off its capacity provider
AgentCore “capacity providers” keep a warm fleet of EC2 instances plus gp3 scratch volumes running 24/7, regardless of traffic. Two of ours were costing roughly $365/day combined, for weeks, with essentially zero real usage behind them.
The catch we learned the hard way: a capacity provider can’t be resized or edited after creation, and a runtime’s binding to one is immutable. The only way out is to create a new runtime with no capacity provider (cold-start only, no warm pool), repoint it, verify with a real invoke, then tear down the old one. That’s exactly what this commit did for the prod runtime.
An AgentCore version sweeper
Every update to an AgentCore runtime creates a new version, and nothing deletes the old ones automatically. Those stale versions pile up and even block capacity-provider deletion. We added two cleanup paths:
- The provisioner Lambda now self-prunes old versions after each create/update.
- A new, scheduled version sweeper (shipped disabled and in dry-run mode by default) does the same account-wide, important because runtimes sometimes get created outside of our infrastructure code.
An IAM secrets gotcha
We also fixed a subtle, deterministic failure: ECS tasks were being denied access to their secrets. The root cause was using Secret.fromSecretNameV2 instead of Secret.fromSecretCompleteArn. The former produces a wildcard IAM pattern for grants but a bare partial ARN for the task’s valueFrom reference, and those two never match as literal strings, so GetSecretValue is denied at every task launch no matter how correct the IAM looks. Switching to full ARNs fixed it.
3. Writing the rules down: CDK-only infrastructure
Commits: e8c5385c, 7d25ba81, 30d25d59, 0e41df43
Alongside the fixes, we hardened the process so these problems don’t recur. The headline is a new, firm policy captured in AGENTS.md:
All infrastructure changes go through CDK code and
cdk deploy. No manual AWS CLI or console mutations as the actual fix.
Read-only investigation (describe-*, list-*, get-*) is always fine. But changing a resource by hand creates invisible drift that nobody, not a teammate, not the next automated session, not future-you, can diff, review, or safely redo. Nearly every broken thing in the account traced back to a hand-made resource with no code describing what should exist.
We also:
- Documented the AgentCore capacity-provider gotchas above so the next engineer doesn’t rediscover them at 2am.
- Cross-referenced a separate self-promotion loop finding (where the reactor was calling AgentCore control-plane APIs roughly once a minute even with an empty work queue).
- Fixed stale
deploy.shdocs (the script is now disabled; releases go through CDK). - Set the prod GitHub OAuth client ID in CDK context.
The through-line: if CDK can’t express something natively, use a CDK custom resource backed by a Lambda, not a one-off CLI call.
4. The factory shipped to itself
Four of this week’s merges came from Nexus Factory’s own autonomous pipeline, planned, coded, reviewed, and auto-merged with no human hands:
d7727312: add a marker line to the top of the README9fc5fa01: add a ticket line below the README header9140dd2e: add a test line near the top of the READMEe5287602: the/api/statsendpoint described above
The README edits are deliberately small, they’re end-to-end exercises of the full pipeline on prod, validating that the factory can take a requirement, open a branch (factory/run/...), make the change, and auto-merge it cleanly. The /api/stats feature shows the same loop producing real, non-trivial application code.
Takeaways
- Warm pools cost money even when idle. A flat daily EBS/gp3 charge with no traffic correlation is a red flag, check your AgentCore capacity providers first.
- Immutable cloud resources need a create-new-then-swap playbook, not an “edit in place” assumption.
- IAM that looks correct can still fail deterministically, partial vs. complete ARNs matter for ECS secret references.
- Codify your infrastructure, and codify your rules. The cleanup fixes were valuable; writing down why they happened is what prevents the next incident.
- The factory is increasingly shipping itself, from docs to real API endpoints.
Built by AllCode. Watch live pipeline progress at nexusfactory.allcode.com.
Recent Comments