TL;DR
- AI executes a flawed plan correctly. It doesn't catch outdated assumptions in the plan itself.
- Five real incidents drive this piece: a dead tracing module, a 10,000-line refactor across 14 services, an 11-hour billing loop, seven orphaned instruction files, and a one-line CI break that took down 18 workflows.
- The fix isn't better code review. It's reviewing the premise before scoping, splitting one kind of risk per pull request, treating a green test run with suspicion, checking that the AI assistant actually loads its own rules, and measuring blast radius instead of diff size.
Our architecture doc carried the same item for months.
One of our services had a proper distributed-tracing module. The other thirteen didn't. Promote it. Standardize the fleet.
It's the kind of task you hand to an AI assistant without hesitating. Mechanical. Well scoped. The target code is already written. There is no product decision buried in it.
The module had never run.
Both of its backends sat behind optional imports, and neither dependency was installed anywhere in the repository. Every call site had been quietly receiving a do-nothing object since the day the module was written. Nobody noticed, because a tracing module that does nothing looks exactly like one that works; right up until you go looking for the traces.
Promoting it would have added two heavy dependencies to thirteen services in exchange for nothing.
We deleted it instead. 168 lines of module, plus 210 lines of tests that had been faithfully verifying the do-nothing path.
An AI assistant would have done that migration flawlessly. The migration was the problem.
1. AI code review: Review the premise, not the diff
When we finally picked up the work, the plan contained several false assumptions that changed the scope.
The module named for promotion was dead code.
A comment in the code explaining why consolidation was impossible was false. The constraint it described no longer existed.
A standard code review would not catch those faulty assumptions. A reviewer reads a diff and asks whether the change is correct. We needed to ask something different: were the plan's assumptions still true?
AI reduces implementation cost but leaves the cost of deciding whether the work is worthwhile unchanged. In an old codebase, evaluating the proposed work can require more effort than implementing it, and cheaper implementation can cause that evaluation to be skipped. This is the core discipline behind hiring and managing an AI-native engineering team, where the assistant's output is only as sound as the human judgment scoping it.
Takeaway: Before scoping AI-assisted refactor work, verify the ticket's assumptions against the current codebase rather than the document that described it.
2. AI refactoring: Keep one kind of risk per pull request
We set out to remove roughly ten thousand lines of duplicated scaffolding across fourteen background services. Years of copying had caused their logging, configuration, database, and queue code to drift apart.
We didn't do it in one pass. We retired one layer at a time, and every layer shipped as two pull requests.
The message-queue layer followed this structure.
| What ships | What changes | How you review it |
| PR 1 | The shared module, plus the three services that already behaved that way | Reviewers confirm that the shared module preserves behavior. |
| PR 2 | The remaining ten services, flipped onto the new contract | Reviewers check one repeated behavior change across ten services |
Smaller pull requests let each review focus on one kind of risk.
When extraction and adoption are combined, the reviewer must distinguish structural moves from behavior changes line by line. AI-generated volume makes that distinction harder to review.
The same split held on every layer: extract in one change, adopt in a later one.
Takeaway: Split extraction and adoption into separate pull requests. Existing tests should verify that the first pull request preserves behavior. The second should move call sites to the new contract through one defined behavior change at a time.
3. AI coding assistant risk: green is not a verification strategy
Our dashboards failed to detect two production and delivery failures.
A background worker entered a retry loop and spent about eleven hours repeating the same paid model call for the same records. Every call returned a 200 response, so the dashboards stayed green and the bill is what exposed the problem.
Our pre-commit hook checked for this configuration error, but web-editor commits bypassed local hooks. The check never ran on those commits.
A green test run provides evidence only when CI enforces checks that can detect these failures.
CI must enforce coverage through the command it actually runs. Our plain test command collected no coverage data, while a separate measured command served as the gate.
Test steps must fail when they collect no tests. One step in our pipeline accepted the "no tests were collected" exit code instead, so an unloaded suite could report success.
Tests must exercise database logic against a real database rather than only a mock. A mocked query can test application behavior, but it does not execute the SQL against the target database.
We commit our review rulebook to the repository. It tells reviewers what to catch and what to ignore, and our automated reviewer reads the same file.
Takeaway: Confirm that the exact CI command enforces coverage and that test steps fail on an empty suite. Test database logic against a real database rather than only a mock.
4. Legacy code modernization: check that the AI assistant reads its own rules
Verify that the AI assistant loads the rules intended to govern its work.
Our assistant loads instructions from one specific directory, but we had placed seven instruction files in a directory it ignored. Together, those instructions had recorded 110 uses over their lifetimes, which showed that they had once loaded successfully. Our internal documentation still listed them as available.
The assistant reports no error when it fails to load an instruction. It therefore completed the affected tasks without the constraints those files defined.
Our instruction list had also exceeded its size limit, causing the assistant to truncate entries mid-word. Truncation can remove the final phrases that tell the assistant when an instruction applies. The assistant then retains the rule but no longer selects it.
The tracing module and the misplaced instructions both existed without doing their intended jobs or reporting an error. Both failures point to the same requirement laid out in what makes an engineer AI-native: someone has to own the configuration the assistant depends on, not just the code it produces.
Takeaway: Audit the directory from which the assistant loads instructions and move every rule file there. Then inspect the loaded descriptions for truncation.
5. AI legacy code modernization: Measure blast radius, not diff size
We safely rewrote ten services at once because each one received the same known change.
Blast radius measures risk more usefully than line count does.
AI can increase both line count and blast radius, so manage blast radius directly. That distinction between output volume and delivery risk is also what separates a delivery model from a partner model in AI-native engineering work.
None of these five habits are exotic. They are the ordinary discipline of a senior engineering team, applied deliberately to a tool that removes the friction that used to force that discipline by default. Howdy's partners get engineers who already work this way, so AI-assisted modernization ships without becoming the sixth incident on this list.
FAQs
What is the biggest risk of using AI for legacy code modernization?
The biggest risk is that AI executes a flawed plan correctly because the plan relies on an outdated assumption about the codebase.
Can code review catch these anti-patterns?
Standard code review checks whether a change implements its plan correctly. Reviewers must separately verify that the plan still reflects the current codebase.
How should large refactors be structured when AI is doing the implementation?
Split refactors into separate extraction and adoption steps. Existing tests should verify that extraction preserves behavior, and each adoption pull request should introduce one defined, repeatable behavior change.
