Imagine this.
A junior developer gets access to an AI coding agent. Two weeks later, the repository contains roughly 100,000 lines of code.
The application runs.
The interface looks polished.
Tests are green.
The demo goes well.
Then someone asks a few basic questions.
Where does a user request enter the system?
Which module makes the authorization decision?
Where is data validated before it reaches the database?
What happens if this function changes?
Which service owns this transaction?
What breaks if this dependency disappears?
The developer cannot answer.
The source code is visible. Nothing is encrypted. Nothing is hidden behind a proprietary binary.
And yet the system has become a black box.
This is one of the most important engineering problems created by AI-assisted software development.
The danger is not simply that AI may generate bad code.
The deeper problem is that AI can generate software faster than humans can understand it, verify it, and accept responsibility for it.
That changes what software teams need to measure.
The question is no longer simply:
Does the code work?
The more important question is:
Does anyone actually understand what we are putting into production?
What 100,000 Lines of Code Really Means
The first mistake would be to stare at the number.
100,000 lines sounds enormous.
It may be enormous.
Or it may be misleading.
Modern repositories contain far more than handwritten application logic.
Generated API clients may contribute thousands of lines. Database migrations accumulate over time. Test fixtures and snapshots can become huge. Lockfiles describe dependency trees. Vendored libraries may contain code written by completely different organizations.
Configuration files, infrastructure definitions, duplicated implementations, obsolete experiments and unreachable code may all be counted as part of the repository.
GitHub Linguist explicitly distinguishes generated and vendored files because raw repository statistics can otherwise give a distorted picture of what developers actually maintain.
A package-lock.json file can contain thousands of lines while representing dependency resolution rather than thousands of human engineering decisions.
At the same time, dismissing everything that looks like configuration would also be dangerous.
A five-line authorization policy can matter more than 5,000 lines of interface code.
A database migration can destroy production data.
A CI configuration can expose credentials.
An infrastructure rule can give a service far more access than it should have.
So the first question during an audit should not be:
Who is going to read all 100,000 lines?
It should be:
What exactly are these 100,000 lines?
How much of the repository represents unique, reachable production behavior?
Where are the critical control points?
What was generated automatically?
What is duplicated?
What is unused?
What is security-sensitive?
What can change data?
What can change permissions?
That classification immediately turns an intimidating pile of code into something closer to an engineering system.
Generation Is Not Understanding
AI coding tools have changed one part of software development dramatically: the speed at which implementation can be produced.
They have not automatically increased human comprehension at the same rate.
That distinction matters.
A 2026 IEEE Transactions on Software Engineering study involving 69 participants found that AI assistance increased implementation completeness while participants performed worse on questions about the code they had produced.
The reduction in measured understanding was 12.5 percent in that experimental setting.
The exact percentage should not be projected onto every engineering team.
The important result is the separation itself.
More code can be completed while less of the resulting implementation is understood by the person responsible for it.
That gives us four concepts that should never be treated as interchangeable.
Code generation means that an implementation exists.
Code comprehension means that someone can explain how important parts of that implementation work.
Technical ownership means that a named person or team understands a component well enough to operate, modify and repair it.
Engineering accountability means that an organization accepts responsibility for requirements, verification, risk decisions, deployment and remediation.
AI directly accelerates the first category.
It does not automatically provide the other three.
A developer can therefore be the person who committed the code without being ready to own that code in production.
The Speed Problem Is More Complicated Than It Looks
It is tempting to assume that AI always makes software development faster.
The evidence is more interesting.
In a 2025 randomized controlled trial, METR studied 16 experienced open-source developers working on 246 real issues in repositories they already knew.
When the developers were allowed to use the AI tools available at the time, they took 19 percent longer.
Before doing the work, they expected AI to make them approximately 24 percent faster.
Even afterward, they still believed AI had accelerated them.
That gap matters.
Developers can feel faster without actually completing the total engineering task faster.
By early 2026, METR believed newer tools were probably producing much larger gains for some developers, although selection effects made precise measurement difficult.
So the useful conclusion is not that AI speeds everyone up or slows everyone down.
The useful conclusion is that AI productivity is highly dependent on the developer, task, codebase, tool and quality standard.
Typing more code is not the same thing as finishing engineering work.
Finishing a feature is not the same thing as understanding the system.
Merging more pull requests is not the same thing as creating maintainable software.
Claude Code Makes This Question Bigger
Traditional autocomplete suggests code.
Agentic coding systems can operate across an entire repository.
Claude Code, for example, can inspect files, edit code, execute commands and perform multi-step work.
That means the unit of AI-generated change can be much larger than one function suggested inside an editor.
A human developer may ask for a feature.
The agent may inspect several modules, create new files, change configuration, install a dependency, modify a database query, add tests and update another component to make everything pass.
From the outside, the developer asked for one thing.
Inside the repository, dozens of engineering decisions may have been made.
This is not proof that Claude Code produces poor software.
It is simply a new governance problem.
The faster and more broadly an agent can modify a system, the more important it becomes to know who understands those modifications afterward.
The Working Demo Trap
A polished demo is psychologically powerful.
Click the button.
The screen changes.
Data appears.
Everyone smiles.
Ship it.
But a successful demonstration proves only a narrow thing:
For the demonstrated input, in the demonstrated environment, one visible path produced the expected visible result.
It does not prove that authorization works correctly for every role.
It does not prove tenant isolation.
It does not prove safe handling of malicious inputs.
It does not prove transaction integrity.
It does not prove database migrations are reversible.
It does not prove recovery after partial failure.
It does not prove secrets are stored correctly.
It does not prove third-party dependencies are safe.
It does not prove the system behaves correctly under concurrency.
It does not prove that another engineer can maintain it six months later.
This matters even more with AI because agents are very good at optimizing toward visible success signals.
If the acceptance signal is “make this test pass,” the system may satisfy the test without satisfying the real requirement.
If the acceptance signal is “make this screen work,” the generated implementation may optimize the demonstrated path while leaving hidden failure modes untouched.
A working interface is evidence.
It is not assurance.
When AI Writes the Code and the Tests
Tests can create another dangerous illusion.
Imagine this workflow:
The developer gives the AI a requirement.
The AI interprets it incorrectly.
The AI writes the implementation.
Then the same AI, with that implementation already in context, writes the tests.
The implementation and tests can now agree perfectly.
They can also both be wrong.
This is the software equivalent of writing the exam and grading your own answer using the same misunderstanding.
Recent research has begun examining this problem directly.
A 2026 empirical study found that tests generated after exposing the model to faulty generated code detected fewer faults than independently generated tests in that experimental setup.
Peer-reviewed ICSE 2026 research into generated database-system tests also found examples of flaky assumptions being introduced or copied from examples in model context.
This does not make AI-generated tests useless.
It means test quantity is a weak metric by itself.
A repository containing 5,000 tests may still provide poor assurance if those tests merely encode the same assumptions as the implementation.
The better question is:
What independent information do these tests contain?
Useful verification includes tests derived directly from requirements, negative tests, adversarial inputs, boundary conditions, independently determined expected results, integration tests, security checks and tests designed specifically to fail plausible incorrect implementations.
Coverage is useful.
Coverage is not truth.
The Five-Minute Architecture Test
There is a very simple test that can reveal whether a software project is drifting toward black-box territory.
Choose one critical feature and ask its owner to explain this flow:
Input
to processing
to authorization
to database
to output.
The explanation does not need to include every class or function.
But the engineer should be able to describe the major steps.
Where does untrusted input enter?
Where is it validated?
What identity is attached to the request?
Where is authorization checked?
Which business operation executes?
Which database tables are read or written?
Which external systems are called?
What happens if one step fails?
What gets returned?
What gets logged?
What information must never leave the system?
If nobody can explain that path, the organization has a much larger problem than missing documentation.
It cannot reliably reason about trust boundaries.
It cannot confidently predict blast radius.
It cannot determine where authorization is enforced.
It cannot know whether one customer's identifier might reach another customer's data.
It cannot know where a payment is actually committed.
It cannot know what happens if an external API succeeds but the local database operation fails.
It cannot know which data needs to be restored after a bad migration.
OWASP threat-modeling guidance treats system understanding, processes, data stores, flows and trust boundaries as prerequisites for meaningful security analysis.
You cannot protect a system you cannot describe.
Do Not Read 100,000 Lines From Top to Bottom
The natural reaction to a mysterious repository is often:
Fine. We will review the code.
Then someone imagines opening the first file and reading until line 100,000.
That is the wrong strategy.
Software risk is not evenly distributed.
Five authorization lines can matter more than thousands of UI components.
One destructive SQL migration can matter more than an entire generated client library.
A secrets configuration can matter more than a month of frontend work.
A dependency can introduce more risk than thousands of locally authored lines.
The review therefore needs to start with architecture and risk.
First classify the repository.
Then identify entry points.
Then map services and modules.
Then identify databases, external APIs and queues.
Then locate authentication and authorization.
Then trace the most important data flows.
Then identify destructive operations.
Then identify dependency and infrastructure boundaries.
Then perform deep source review where risk is concentrated.
The sequence should look like this:
Inventory.
Architecture.
Critical flows.
Trust boundaries.
Verification evidence.
Targeted source review.
Not:
Line 1.
Line 2.
Line 3.
Eventually line 100,000.
How to Take Control of an AI-Generated Codebase
A team inheriting a large AI-generated repository should temporarily stop treating feature generation as the priority.
The priority is control.
Start with the repository itself.
Produce an inventory of files by language, type, size and purpose.
Separate production code from tests, fixtures, snapshots, generated files, vendored code, migrations, documentation and configuration.
Identify generated code and document exactly how it is regenerated.
Find duplicated implementations.
Find unused modules.
Find unreachable code.
Find abandoned experiments.
Inventory dependencies, including transitive dependencies.
Record exact versions.
Generate a software bill of materials where appropriate.
Then rebuild the application from nothing.
Not from the original developer's laptop.
Not from an environment containing months of hidden configuration.
Start with a clean machine, container or CI environment.
Can the repository install?
Can it build?
Can it start?
Can a fresh database be created?
Can migrations run?
Can a new environment be deployed?
Can the release be rolled back?
If the software only works inside one developer's machine because nobody knows which invisible local state makes it function, the organization does not yet control the system.
Reconstruct the Architecture
Next, build an architecture package.
Do not allow the repository itself to remain the only explanation of the repository.
Create a simple system-context diagram.
Identify major modules.
Identify entry points.
Identify databases.
Identify queues and caches.
Identify third-party services.
Identify background jobs.
Identify authentication boundaries.
Identify authorization boundaries.
Identify deployment infrastructure.
Identify production-critical components.
Then assign a human owner to each one.
For the most important user journeys, draw the data flows.
Login.
Account creation.
Payment.
Password reset.
Administrative actions.
Data deletion.
Sensitive exports.
Any feature capable of changing money, identity, permissions or important data deserves explicit tracing.
Verify Requirements Independently
One subtle risk of AI-generated systems is circular validation.
The requirement is interpreted by the model.
The model implements its interpretation.
The model generates tests for its interpretation.
The same implementation is demonstrated back to the team.
Everyone sees consistency.
But consistency is not correctness.
Someone must compare the running system with the original business requirement independently of the generated code.
If the product requirement says a manager may access records belonging only to their region, testing should begin with that rule.
Do not begin by asking what the implementation appears to do.
Generate acceptance scenarios directly from the requirement.
Then attempt to break them.
Treat Security as Architecture, Not Cleanup
Security review should focus first on the areas where failures matter most.
Authentication.
Authorization.
Secrets.
Input validation.
Dependency integrity.
Database access.
File uploads.
External API calls.
Administrative functions.
Destructive operations.
Background jobs.
Infrastructure permissions.
CI/CD credentials.
Logging of sensitive information.
Multi-tenant boundaries.
Do not simply run a scanner and declare victory.
Static analysis is valuable.
Dependency scanning is valuable.
Secret scanning is valuable.
But scanners operate inside a model of the software.
Human reviewers still need to understand what the system is supposed to protect.
Make the Developer Change Something
One of the strongest tests of ownership is surprisingly simple.
Choose an important but controlled modification.
Before touching the code, ask the developer responsible for the component to explain:
Which modules will be affected?
Which data flows will change?
Which tests should fail before the implementation is updated?
Which tests should remain unchanged?
What could break?
Which database structures are involved?
Which security assumptions are involved?
What should be monitored after deployment?
Only then allow the modification.
This exercise separates familiarity with prompts from familiarity with the system.
A developer who can only say “I will ask the agent to change it” does not yet own that component.
That does not make the developer incompetent.
It means the organization has allowed implementation throughput to exceed knowledge transfer.
This Is Not Just a Junior Developer Problem
It would be convenient to blame the junior developer.
That would also miss much of the problem.
Organizations create incentives.
If management celebrates lines of code, AI can produce more lines.
If management celebrates pull-request counts, AI can produce more pull requests.
If teams are rewarded for visible features, agents can create visible features rapidly.
If deadlines value demos over architecture, demos will win.
If engineers are praised for speed but never asked to explain what they built, understanding becomes economically irrational.
The result is predictable.
Generate.
Merge.
Generate again.
Eventually the repository becomes too large for anyone to challenge comfortably.
The fact that AI made the growth possible does not mean AI created the management system around it.
Teams need metrics that reward outcomes rather than output volume.
Reproducibility.
Change failure rate.
Security findings.
Recovery capability.
Maintainability.
Architectural clarity.
Ownership.
Reliability.
Time required for another engineer to safely modify the system.
Those measures are less dramatic than “100,000 lines in two weeks.”
They are also much closer to what determines whether software survives contact with production.
Who Owns AI-Generated Code?
There is a useful engineering answer that is separate from legal questions about copyright.
You own a production component when your organization can explain it, operate it, verify it, modify it and accept responsibility for its consequences.
Typing every line personally is unnecessary.
Modern engineers already depend on compilers, frameworks, libraries, code generators and massive open-source ecosystems.
AI is another layer of abstraction.
The difference is that agentic AI can produce large amounts of project-specific logic very quickly.
That increases the importance of proving understanding after generation.
A useful rule is:
The person presenting the code should carry the burden of demonstrating control.
The reviewer should not be required to prove that 100,000 lines are bad.
The team presenting the system should demonstrate that it is understandable, reproducible, testable, sufficiently secure for its intended use and maintainable by accountable humans.
That reverses a dangerous assumption.
Instead of saying:
The application seems to work. Find something wrong with it.
The organization says:
Before production, show us why we should trust it.
Before Production Approval
For a large AI-generated system, I would require several things before approving production use.
The repository must be classified.
The build must work in a clean environment.
Deployment must be reproducible.
Critical dependencies must be known.
The architecture must be documented well enough for another engineer to navigate it.
Critical data flows must be traceable.
Authentication and authorization must be explicitly reviewed.
Business requirements must be tested independently of the implementation.
Tests must demonstrate more than coverage.
Database migrations and destructive operations must have recovery plans.
Secrets and infrastructure permissions must be reviewed.
Major unused and duplicated code must be explained or removed.
Critical components must have named human owners.
At least one controlled modification should demonstrate that those owners can predict consequences before asking an AI agent to make the change.
And the system must have an operational plan for monitoring, rollback and incident response.
None of these requirements are anti-AI.
They are normal engineering controls applied to a development process whose output speed has changed dramatically.
The Real Bottleneck Has Moved
For decades, writing code was one of the expensive parts of software development.
AI is reducing that cost.
But software was never valuable because someone managed to type it.
Software is valuable because it behaves correctly enough, securely enough and predictably enough to solve a real problem.
As code generation becomes cheaper, the scarce resource moves elsewhere.
Understanding becomes scarce.
Verification becomes scarce.
Judgment becomes scarce.
Architecture becomes scarce.
Accountability becomes scarce.
That is why the visible-code black box matters.
The source may be sitting directly in front of us.
We may be able to search every file.
We may even be able to ask another AI to explain it.
But a production system is not truly under control until accountable humans can reason about what it does, why it does it, what can go wrong and what will happen when they change it.
The future software organization may not win by generating the most code.
It may win by being able to generate rapidly without surrendering understanding.
AI can write faster than humans.
The engineering challenge is making sure responsibility does not disappear in the gap.
Sources
GitHub Linguist
https://github.com/github-linguist/linguist
IEEE Transactions on Software Engineering, More Code, Less Understanding?
https://ieeexplore.ieee.org/document/11458664
METR, Early 2025 AI Experienced Open-Source Developer Study
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
METR, 2026 Uplift Update
https://metr.org/blog/2026-02-24-uplift-update/
Anthropic Claude Code Documentation
https://docs.anthropic.com/en/docs/claude-code
NIST Secure Software Development Framework
https://csrc.nist.gov/pubs/sp/800/218/final
OWASP Threat Modeling Cheat Sheet
https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html
SLSA Software Supply Chain Security Framework
https://slsa.dev/
USENIX Security 2025, Package Hallucinations in Code Generation
https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen
METR Frontier Risk Report and MirrorCode
https://metr.org/blog/2026-05-19-frontier-risk-report/