A system that builds software on its own: a five-week list of mistakes

At a testing conference in 2018, I apparently tweeted "AI is coming, of course, but when?" A few months before that, I had told someone on LinkedIn that "QA won't die until there's an AI that can do exploratory testing." Eight years later, AI has made its way into our daily lives. So I got curious: could I build a software company on my own with these agents today? I tried it on my own computer, and what came out was a simulation of a company that runs autonomously. There are no human employees besides me; everyone else is an agent.

This post covers the first five weeks of that system. But it isn't a success story. It's more of a list of mistakes and optimizations, because I spent the first two weeks writing the system that would write the product instead of writing the product, and it took me a week to notice.

5 weeks · 15 subagents · 1,100+ merged PRs · 9 runners, 3 machines

By the way, I wrote this post myself 🙂 I only had AI make small edits and pull out the numbers and charts. Of course, it could have written the whole thing on its own, but this was one part I wanted to do myself. (This English version is a translation of the Turkish original.)

Why I started

At my last job, we were writing almost all of our code with AI tools. But on projects with tight deadlines where you need to ship releases fast, you don't get much time for R&D. You still use the system as a fast assistant, and there's no chance to push its limits. Now I wanted to push those limits.

The goal was simple: to see what AI can really do and to build a setup that ships products on its own. I had already been building individual projects with Claude Code before this. My first project was a tool I needed myself but couldn't find a practical app for. But on every project I repeated the same loop: open a session, explain what you want, check it, fix it. In other words, what I had wasn't an agent; it was an intern who types very fast.

What I had in mind was different. I'd just give the idea, the product manager would write the product document, the business developer would come up with ideas to grow the product, the architect would produce the design, the developer would write the code, the reviewer would review it, and QA would test it. And as CEO, I'd make the decisions. I've spent 15 years working in and leading test, software and platform teams. I already knew this structure by heart, so I figured I could teach it to the agents too.

Setup

I made the first commit on September 1, 2026. It contained a 1,885-word constitution (CLAUDE.md), 14 agent definitions and a starter prompt. Each agent's role, the model it would use and its decision rights were written down. Who writes architecture decisions, who approves code, who decides on a release: it was all in one table.

The system's current roster: me as CEO, the team lead who hands out work, and 15 subagents. acting-ceo is one of them, but it doesn't wait for work from the team lead; it runs on its own every 30 minutes. The blue boxes are the agents that compile code and run tests.

The tech side was settled from the start: Go on the backend, Next.js on the web, Kotlin and Swift on mobile, PostgreSQL for the database. I left out Java, which I love at fanboy levels. If I wasn't going to write the code myself, I didn't think it was worth it given its resource usage. We might try Rust later, and for products that ship on two platforms at once we might go with Flutter. I use Google Cloud Run for hosting, Neon for PostgreSQL and Firebase for web hosting. All of them are free up to a certain number of requests.

Then there's work tracking. Every product has its own repo and its own GitHub board. Agents open issues, write documents like ADRs and RFCs, open branches, open PRs and approve each other's PRs with a separate bot account. Very professional on paper.

1- We wrote seven quality gates, and none of them worked

In the first two days, the agents wrote seven quality gates: a branch-name check, migration protection, linting, fast tests, a status summary at session start, a final check before declaring work done, and a merge gate. They wrote 508 test cases for them, and all of them were green. On top of that, they went through six review rounds.

0 / 7 gates actually worked. Yet all 508 test cases were green, and all of them had been through six review rounds.

Then, at the first retro, it turned out that the config line that switches these gates on had never existed. In other words, none of the seven gates had run even once in two days. It went into the metrics table as "Gates live: 0/7".

Honestly, coming from test automation, this one bothered me the most, because it's a classic mistake. The tests verified the gates themselves, not whether they were actually switched on. The retro summed it up nicely: "absence reads as success". Just because something doesn't stop you doesn't mean it works. The next day, a script arrived that reads the live config to check whether the gates are really on. A "how many gates are live" line was added to the daily standup, too.

2- We produced process instead of product

Once the gates started working, the opposite happened. By September 7, 50 infrastructure PRs had been merged, and only 28 on the product side. The merge gate alone had grown to 13,251 lines with 635 test cases. And what was it doing? It was rewriting GitHub's branch protection feature from scratch in bash, because the free plan doesn't offer it for private repos. I had 13 thousand lines of bash written just to avoid paying GitHub. Oh boy.

This gate was blocking between 2 and 7 legitimate pieces of work every day. The funniest part: it had also blocked the retro that described this very problem.

13,251 lines of bash and 635 test cases. We had written our own version of the branch protection feature from GitHub's paid plan.

Then there were the review rounds. When I looked on September 3, every PR was going through 3 to 5 rounds. And most rounds were fixing wording, not behavior. Security approval had become mandatory on 17 of 28 product PRs, because the agents read the "money" rule as money inside the application. On a finance-related project, of course, every PR touches money. What I actually meant was the money this system spends. Across all 17, those security approvals never once caught a useful blocker.

I noticed this while reading the standups. Every day, the "what was done yesterday" section was all gates, tests and checks. The product was always "tomorrow". On September 6, I wrote this to the agents, word for word (translated): "be more permissive when doing security checks and review checks, some things can be fixed after the MVP, don't forget that I'll be the only one using this app locally for a long time."

The next day there was a big cleanup. The merge gate was deleted entirely, 11.7 thousand lines went in a single day, and the number of gates dropped from seven to four. Review went down to a single round, and the reviewer started writing only blocking findings. Infrastructure PRs started getting merged without waiting for a second approval. A new article was added to the constitution as well: a code freeze. The system's own rules can now change only when there's an active product blocker or when I make a direct decision.

You can see the result in the chart. In the first week, infrastructure work outnumbers product work; after that, product pulls far ahead.

Merged PRs per week. Orange is company infrastructure, blue is product repos. The last column is a 3-day half week.

3- Every agent was reading the entire constitution

On day one, the constitution was 1,885 words. Every retro added a few articles to it, and by September 8 it had grown to 2,972 words. The problem was that this file was read from scratch every time an agent ran. If eight agents were running, it was read eight times. The product manager, for example, didn't need to know the rule about the iPhone simulator, but it read it anyway.

On September 11, I split the constitution in two. The rules everyone needs on every task stayed in the main file; the rest moved to a handbook that's read only when needed. The word count dropped to 1,119.

We rewrote the agent definitions with the same logic. Before, each agent's file was a long instruction text describing every detail of its job. Now each one is a short role card: what it owns, when it's called, what it decides, what it doesn't do, and what it writes when handing over its work. The details moved to skill files that are read only when needed.

Two more rules came the same day. First, when the team lead hands out work, it no longer passes along the entire history; it gives only the expected outcome, the files the agent will touch and when the work counts as done. Second, agents write at most eight lines when handing over their work.

The constitution's word count after each change. The drop on September 11 is the move to the handbook.

Look at the right side of the chart and you can already see what happened: the number climbed back up to 1,993. Adding rules is easy; removing them is hard. Companies made of people have the same pattern. The only difference is that there, removing a rule isn't hard, it's close to impossible :)

4- We were spending tokens on reading, not writing

This was my most expensive lesson. In mid-September, the 5-hour limit started running out in under 2 hours, and my weekly usage limit in 3-4 days. After that, you either sit and wait for the rest of the week, or hand over your credit card and watch your credits melt away fast.

Not much changed in the process during the second half of September, because I was on vacation for part of that period and didn't have much changed. When I got back, I had the agents analyze their own conversation logs. Between September 13 and 27, there were 34.6 thousand model calls, and the picture was surprising. 78% of the spend went to the model re-reading the previous conversation from the start every time. The part the model actually wrote was only 4%. When conversations got very long, more than half of the spend went there, because auto-compact was set to kick in far too late.

Breakdown of token spend between September 13 and 27. The slices overlap and don't add up.

The details are even funnier. In one session, the team lead checked "is there a new message?" 867 times, and the answer was "no" all 867 times. The linting hook injected lint output 1,700 times while the agent was still writing the file. And the agents writing code carried a task from start to finish, tests and review included, in one long conversation. A single task took between 300 and 1,000 turns to finish.

On September 27, we made these fixes:

1. Auto-compact kicks in much earlier.
2. The developer agent's work ends when it opens the PR. The team lead waits for CI, and findings that come back from review go to a brand-new agent that starts from scratch.
3. A cap was put on how many turns each agent can take.
4. Repeatedly checking "is there a message?" for nothing was blocked.
5. Test and tool output is read as a summary, not at length.

The team lead still runs on the most capable model; I didn't change that. We cut the cost not through the model but through the length and lifetime of the conversations. The result surprised even me. The limit used to run out in 3-4 days; now, even though we do much more work, it lasts 6-6.5 days. I finished the last week before this post without fully using up the limit, at 99%. And the 5-hour limit never runs out anymore.

Same plan, same model. The difference is the length of the conversations.

5- All the agents on a single 32 GB machine

All the agents run on my Windows machine at home with 32 GB of DDR5 RAM. The same machine also has four GitHub runners that run CI jobs. In total, there are actually nine runners across three machines. That includes a 2019 MacBook Pro with 32 GB of DDR4 RAM and a 2011 MacBook Pro with only 16 GB of DDR3 RAM that I installed Linux on. But I haven't yet built a setup that can start and run agents on the other machines; I'm working on that right now.

The agents are on one machine; CI jobs are spread across three.

At first, I set a single rule: "at most five agents at the same time". Then I realized that agents like the reviewer, the architect and the product manager use almost no resources on the computer, because their work runs on the model's side. On September 28, I split the cap in two: at most five agents that compile code and run tests (three while the Android emulator is open), plus at most three others. Light CI jobs like lint moved to the 2011 MacBook.

6- Mistakes should have been prevented before review, not caught in it

On the product the system worked on most, the entire MVP shipped and 514 PRs were merged. But something was bothering me: after every task, review and QA turned up a lot of bugs. On September 29, I asked the agents to dig into this.

Here's the picture: 100 of the last 300 PRs got "changes requested" at least once. About 45% of the blocking findings weren't even about code. The code had changed, but a design document, an ADR or a code comment still described the old behavior. On the QA side, there were 76 real product bugs. And testing had been done neither with real data nor on a small phone screen.

The September 29 analysis. Orange squares are PRs that came back at least once.

The real irony is elsewhere. There was a retro agent and a retro guide, but not a single product retro had been held for this application. Five recurring bugs turned up, meaning the same bug had been fixed again in a different place. After years of explaining this to teams, I had forgotten to explain it to the agents.

Before this, we also had issue inflation. The number of issues opened per day went from 4 to 31 in eight days. Opening an issue takes a minute, but closing it takes a separate workspace, a branch, a PR, a review and a CI run. Since September 14, findings that aren't bugs don't get their own issue. Each one goes in as a single line in that task's one "follow-ups" issue.

7- Every decision was coming to me

In the first two days, the agents opened 15 decision questions for me. Most of them were technical details I didn't need to think about.

The queue shrank over time but never went away. On October 5, I added an "acting CEO" (acting-ceo) agent. Spend, production deploys and the constitution stayed with me; it decides the rest. It can't change its own definition, the constitution or the settings; otherwise it could take the powers I kept for itself as well. And since October 2, notifications come to Telegram. The Claude mobile app gets notifications too, but not quite the way I want. Thanks to Telegram, I no longer need to sit at the computer.

Still, acting-ceo doesn't work exactly the way I want. Agents in different sessions can't talk to each other directly, or at least I haven't managed to make that happen yet. So when a decision is needed, agents open a "decision" issue on GitHub, and acting-ceo checks the open issues every 30 minutes and decides. In other words, it's a polling setup.

10 / 10 decision issues opened after acting-ceo arrived were all closed the same day. 75 decision issues have been opened in total, and none are open right now.

8- The one thing the agent couldn't see

The last story is small, but it's one of the ones that bothered me. My main goal is for the agents to move everything forward and for me to be involved as little as possible. Every task I have to do by hand bothers me, and I look for a way around it.

I had an extension built for my new tab page in Chrome; one part of the page also shows my Untappd and Letterboxd stats. On October 6, the Untappd card started showing "Couldn't read". Cloudflare was rejecting the extension's request and inserting its "Just a moment..." page.

The architect wrote a good ADR, went through the documentation and came up with a lot of options. Its first proposal was to run the Cloudflare challenge inside the extension on a hidden page (an iframe). But at one point it said: no agent can see whether the browser's Cloudflare clearance cookie is sent with the request or not. That cookie can't be read from code. Only I could tell, by opening DevTools in my own Chrome and looking.

I opened it and looked. The cookie wasn't being sent. So the iframe proposal could never have worked, and Cloudflare's own documentation said so too. The solution turned out to be simpler: when I visit Untappd, the extension captures the numbers on the page, and the card shows the latest value along with when it was taken. When it can't read them, it now says why.

The agent reads documentation better than I do and comes up with options faster than I do. And when it couldn't take the last step that touches the real world, it said so plainly. That turned out to be one of the things agents can't do yet: use DevTools inside an application like Chrome. As far as I know, they can't, though I didn't push very hard. Maybe they could do it through DevTools' own API; I haven't tried that yet either. For now, this one stays on my list of things I do by hand.

The problems I noticed in the first five weeks and the fixes that replaced them.

How it works today

Before any work starts, it's sorted into one of three routes. For a small bug, the product manager, architect and designer aren't called; a developer and a reviewer are enough. For a change that affects the architecture, the architect steps in and writes an ADR or RFC if needed. And QA doesn't test every PR; it tests a finished feature on the real system.

The three work routes and the roles each one requires.

Since September 1, 1,109 PRs have been merged across 11 repos, 833 of them in the last three weeks.

What I'd do if I started over

1. I'd still set up the process first, because I believe process should always come first. But I'd put more thought into it while setting it up and have different models review it too. While I was building this system, some models suggested the exact opposite: ship the product first, add process as the need arises. Most companies would actually find that more agile. But I lean more toward the philosophy that the structure should be built from the very beginning.
2. I'd believe a check works not because the agent that wrote it said "it works", but when a different agent has verified it by looking at the live config.
3. I'd measure usage from day one. What really eats the limit isn't the code being written; it's the conversations that get re-read from the start every time as they grow longer.
4. I'd make retros mandatory for the agents. Without retros, you end up fixing the same bug again in a different task.
5. From day one, I'd make it a rule that documentation is updated in the same PR as the code. Nearly half of the blocking findings came from code changing while the documentation stayed as it was.
6. I'd have QA test with real data and on a small phone screen from the very start. The first tests skipped both.
7. I wouldn't have a separate issue opened for every finding. Opening an issue takes a minute; closing it takes a PR, a review and a CI run.
8. Before adding a new rule, I'd check whether the existing rules already cover it, and I'd review the constitution at regular intervals. The constitution we cut down to 1,119 words climbed back to 1,993 within a few weeks.
9. I'd write down every task I do by hand on a list from day one. That list is the list of things the system can't do on its own yet, and it's my next target.
10. I'd write down on day one which decisions should come to me. For now there are three: spend, release and the constitution. The rest is an agent's job. Eventually, my goal is to hand those three over to the system too.

In short, I think the agents learned to write code a long time ago. I was the one who had to teach them how to work together. And my goal is still the same: they run the work from start to finish, and I make only the decisions that are truly mine. After five weeks, I'm much closer to that point than I was on day one, but every task I do by hand is still an obstacle to overcome.

The future of software engineering

In the teams I worked with, we'd sometimes debate the future of software engineering, and we rarely agreed. Most of the people I worked with believed the profession wouldn't end. In their view, AI tools would now write the code, but software engineers would still make the architecture decisions and design the systems. Engineering would even become more important than before because of that.

Where we are today, I think the answer to that question has largely become clear. In the system I describe in this post, an agent makes the architecture decisions and writes the ADRs too. While I shipped more than 1,100 PRs alone in five weeks, what slowed me down the most wasn't writing code. It was deciding what to build and verifying that what was built actually works.

I'm not the only one seeing this. Last year, Andrew Ng described how one of his teams, for the first time in his life, asked for more product managers than engineers: two PMs to one engineer. According to a forecast Gartner published in July, the share of companies working with small software teams is 15% today and will be 60% in 2029. At Shopify, teams have to show why a job can't be done with AI before asking for a new hire. According to a Stanford study using payroll data, employment of software developers aged 22-25 is 20% below its late-2022 peak. In short, the number of developers per product owner has already started to fall at many companies in the industry, Turkey included.

I think that in the coming period we'll see software companies working with teams made up of one PO and one developer. That's why we're heading into a period where the two roles converge: developers will need to work more with a product mindset, and POs more with a development mindset. As writing code gets cheaper, what becomes valuable is knowing what's worth writing and being able to prove that something actually works. And my most expensive mistake in this system was here, not in the code: none of the seven checks that were said to "work" actually worked.

The question I don't know the answer to is this: if junior positions shrink, where will tomorrow's seniors come from? Today, I can check the agents' work with the experience I gained over 15 years. If the places to gain that experience shrink, finding people to oversee the agents will get harder too.

But whether that will really be needed is another question. Today, I still have to check the agents' work. A year from now, I believe the models will have improved enough to do this oversight on their own as well. Then I think teams of one PO and one developer will give way to one-person scrum teams. Take the simulation a bit further and you arrive at one-person tribes, and from there at one-person companies. It may still sound far off today. But looking at how far things came between October 2025 and October 2026, it may not be as far off as we expect.