Welcome to session 51! This one was mostly theory. Bala walked through how he thinks about software factories, then showed the planning tool his team uses, with a mock project standing in for real client work.
Bala is the founder of Applied AI and ZORP. He has been building and using this setup for about six months, and runs roughly half his own work through it.
Session overview
A software factory takes a requirement in and ships working software out, with people stepping in only where a decision is needed. The term spread after Simon Willison wrote about a "dark factory" earlier this year: a factory that runs with the lights off because no human is on the floor.
AI has already made writing code fast. What it has not fixed is everything around the code. When Bala's team started building products with agents, the cycle got longer, not shorter. Agents built something other than what was meant, test suites passed without proving anything, and new features broke old ones. Some of those problems reached customers.
Bala's conclusion is that planning is the bottleneck. If every question is answered before any code is written, and the work is cut into the smallest tasks possible with their own test cases, the build itself can run without anyone watching. His team spent the first four months of building their tool on planning alone.
Key takeaways
- The four levels of AI: Assistant (you ask, it answers), agent (it finishes a task while you watch), coworker (it takes the work, asks when stuck, you review) and autonomous (it plans, builds and ships with little help). Most teams sit at coworker and want to reach autonomous.
- The SDLC turns from a line into a loop: AI compressed development the most and review partly. Research and spec take as long as they did, and every failed review sends work back through the loop.
- Three ways agent-built software breaks: Drift from what you meant, tests that pass without testing the logic, and regressions in features that used to work.
- Three anchors: You no longer review every line of code. Defining the job and every edge case is still a human's job. There must be a clear way to judge success, mostly at the functional level.
- Plan until no question is open: Every assumption the planner makes is listed and needs a human's approval. The planner cannot close a question you raised.
- Use a second model to review: Bala plans with Claude and reviews with Codex. He has seen Codex catch issues that a Claude sub-agent missed.
- Independent test cases: The tests for each task are written before the code and run by a different agent, so passing tests mean something.
- Requirements now take longer than the build: For any feature, finishing the plan takes more time than writing the code.
- Put specs in GitHub: When specs live next to the code, the next agent can see why the code is the way it is.
Topics covered
What a software factory is
Simon Willison's post, the dark factory idea, and why "no human in the loop" is a goal Bala doesn't think teams should reach for a long time.
The four levels of AI and the SDLC
The assistant, agent, coworker and autonomous levels, mapped to chat tools, Cursor, Claude Code and agents like OpenClaw. Then Anthropic's AI-native SDLC, and where the time goes once development is cheap.
Where agent-built software breaks
Drift, fake test passes and silent regressions, with a small expense management tool as the example. Multiple entities, non-rupee invoices and foreign remittance filings were edge cases nobody wrote down, so each one turned up as a failure later.
Why reading a terminal doesn't scale
A real update from an agent: five phases, a decision to make and a question at the end. Reading text like that every few minutes, across two or three projects, is what Bala wanted to stop doing.
Better planning is the trick
Plan until no question is open, cut the smallest tasks you can, and set up independent test cases. Then the two flows: a Feature goes through intake, plan, review, tasks and approval, while a Fix skips plan and review and keeps the gates.
A look at the planner
Intake with an evaluator that checks the requirement says what should change, who it affects and what done looks like. A plan written like a PRD, where the AI adds comments and blockers and you answer them. A Codex review that sends the plan back until it is clean. Tasks with their own evaluation criteria, and a final evaluator before anything goes to development.
Execution is a loop too
On a VM with two pipelines, each requirement takes a pipeline. Pick the next task, create a worktree, write the code, review by another agent, run the linter, run CI, open the PR, then move to the next task. Human review happens at the PR.
Q&A highlights
- Where do evals fit, and do you write them at the spec stage?
- How do you stop an LLM from picking a solution that doesn't suit the use case?
- Why build your own tool instead of using Linear or Jira with agents?
- Does the reviewer have to be a different model, or can it be another session of the same one?
- How do you know a spec is done, and how do you handle the volume of review comments?
- Model makers are dropping plan mode. How much should you plan?
- How much of the spec does AI write, and why do some specs come out bad?
- What is the moat when anyone can copy software?
- Are PRs still reviewed by a person?
- Why a portal instead of a skill, and where should someone starting out begin?
Here's the entire recording of the session.