AI Legacy Code Modernization Is Not a Translation Problem

The pitch for AI legacy code modernization fits on one slide. Point a model at the COBOL, get Java back. Point it at the Delphi, get C#. Forty years of accreted business logic, translated over a weekend, by something that never gets bored reading PERFORM VARYING.

The pitch answers the wrong question, very well.

Translating between languages has been a tractable engineering problem since the 1960s. Rule-based transpilers exist. They have existed for decades. No mainframe survived into 2026 because the transpiler was missing. It survived because at some point between 1988 and 2004 the last person who understood the settlement module retired, the design documents went into a filing cabinet in a building that has since been demolished, and the only remaining specification of what the system does is the system.

That is the actual problem. Comprehension first, then behaviour preservation. Translation is the bit in the middle that was never the constraint.

Which makes the real question about large language models here narrower than the marketing suggests. Models are unusually good at the comprehension half. They are unreliable at the behaviour-preservation half in a specific and dangerous way. The danger is not that they get it wrong. It is that they get it wrong quietly.

What the benchmarks say when nobody is selling

Factory AI published Legacy-Bench on 1 April 2026. It is the most useful thing I have read on this subject in two years, largely because it is unflattering.

Hundreds of tasks across six legacy language families: COBOL at 46%, Java 7 at 32%, then BASIC, C89, Fortran and x86 assembly at roughly 5% each. Twelve model-agent combinations were evaluated, including GPT-5.3-Codex, Gemini 3.1 Pro, Opus 4.6 and GLM-5. Pass rates landed between 16.9% and 42.5%. The same frontier models score above 70% on SWE-bench Verified and Terminal-Bench 2. Of 44 unsolved tasks, 31 were COBOL.

Then the number that should end most vendor conversations. In 97% of failures, the agent believed it had solved the task. Factory's worked example is an agent computing a health deduction as $275 where the correct figure was $125, with 24 of 29 tests still passing. Nothing in the loop raised a hand.

The translation literature agrees, less dramatically. A July 2026 paper introducing SEDCoT took 319 COBOL programs from IBM's CodeNet and translated them to C. Vanilla accuracy came in at 28.7% for Gemma-3-27b, 33.8% for Llama-3.3-70b, 34.5% for Qwen2.5-Coder-32b and 39.3% for GPT-4o-mini. Wrapping symbolic execution, generated test cases and delta debugging around those same models pushed GPT-4o-mini to 71.8%.

Read that lift carefully. It did not come from the model. It came from the verification machinery bolted around the model.

Domain adaptation helps too. COBOL-Coder, published April 2026, reports 49.33 Pass@1 on COBOLEval against GPT-4o's 16.4, and a 73.95% compile rate against 41.8%. Genuine progress. Also worth knowing that COBOLEval is 146 problems ported from HumanEval, which means self-contained puzzle functions.

Nobody's payroll runs on HumanEval.

A diff you cannot explain is a business rule you have not found yet.

Where AI legacy code modernization actually pays

Anthropic's 23 February 2026 post on COBOL is, to its credit, mostly about the right phase. The argument is that AI makes understanding legacy code cheaper than rewriting it, and that the exploration and discovery work which used to require armies of consultants can be automated. The claim of quarters instead of years is a marketing number. The framing underneath it is sound.

Four things work well enough that I would build process around them today.

Documentation reconstruction. Feed a model a program with its copybooks and it produces a readable account of control flow, inputs and outputs. It will be wrong about intent sometimes. A domain expert corrects that in twenty minutes, which is exactly the point.

Dependency mapping. Which JCL step calls which program, which CICS transaction touches which VSAM file, which copybook is shared across eleven modules. Tedious, mechanical, high recall from a model, checkable against a static analyser.

Characterization tests. Michael Feathers named these in 2004: tests that describe what the code currently does rather than what it should do. Models draft plausible ones quickly, and unlike a rewrite, a bad characterization test fails loudly the first time you run it against real traces.

Dead-path identification. Flagging branches that appear not to have executed in years is a hypothesis a model can propose and instrumentation can settle.

The asymmetry across all four is identical. Comprehension output is cheap to check, because a human who knows the domain reads it and says yes or no. Translation output is only checkable by running it against thirty years of production behaviour, and most organizations cannot produce thirty years of production behaviour on request.

The business rules are not in the code

Here is where the pitch breaks, and it breaks in the same place regardless of language or decade.

In Shenzhen the interesting legacy is not COBOL on a mainframe. It is the mid-2000s manufacturing layer: Delphi 7 and VB6 front ends over SQL Server 2000, written by two or three people who dispersed years ago, still running lines that ship real product. I have read a lot of this code. Comments in GB2312-encoded Chinese that a UTF-8 pipeline turns into mojibake, identifiers as unexpanded pinyin abbreviations, and a build that only works on one Windows XP virtual machine somebody keeps snapshotted.

Point a good model at that and it produces clean English documentation of the source. Useful. Also, on one system I looked at, largely beside the point, because the rules deciding which station a part went to next were not in the source at all. They were rows in a table that supervisors edited through a form. The code was a generic interpreter. The business logic was data.

I have seen the same shape in Chinese factory software, in European insurance systems and in a 1996 logistics stack. Configuration tables, rule engines, spreadsheets imported nightly, values written into a database at go-live and never touched since. The model reads the code, reconstructs a specification with total confidence, and the specification describes the interpreter rather than the rules.

There is a smaller version of this failure that costs more time than it should. When the encoding is mangled, models do not report mojibake. They infer meaning from it and write fluent, plausible nonsense.

Behaviour that includes the bugs

The second failure mode is subtler and worse.

A production system's observable behaviour includes its defects. A 1987 routine truncates where a modern developer would round half-up, and by 2003 three downstream reconciliation processes had been written to expect the truncated figure. A date parser treats one malformed input as 1 January instead of rejecting it, and forty thousand records now carry that assumption. A sort is unstable, and a report has depended on the accidental ordering for twenty-five years.

Translate that faithfully to spec and you have shipped a regression that passes every test you wrote and breaks the month-end close in week six.

Models are trained to produce correct code. Legacy work often needs code that is correct in precisely the way the old thing was incorrect. Those are different objectives, and only one of them is in the training signal. Bug-for-bug compatibility does not appear in any product announcement I have read this year.

Verification is the entire constraint

The best public evidence for LLM-assisted migration remains Google's FSE 2025 industry paper. Thirty-nine distinct migrations, three developers, twelve months, 595 code changes and 93,574 character edits. The model generated 74.45% of the changes and 69.46% of the edits. The developers estimated a 50% reduction in total time.

Impressive. Now read the conditions.

A six-stage validation pipeline ordered by increasing cost: response completed, change is non-trivial, AST parses and differs from the original, model self-assessment, build compiles, regression tests pass. Mandatory human review before anything landed. A monorepo, a hermetic build system and a test suite that actually runs on every target. Even then, 25.55% of changes needed manual work, and pre-existing test failures blocked otherwise correct ones.

That pipeline is not a footnote. It is the result. The model was allowed to be wrong frequently because being wrong was cheap to detect.

Most legacy estates have none of that. No characterization tests. A build nobody can reproduce. Production data that cannot leave the environment for regulatory reasons. In that setting a model does not remove the bottleneck, it feeds it. GitClear's January 2026 analysis of 623 million code changes since 2023 found duplicated blocks up 81%, error-masking constructs up 47%, two-week churn up 15% and refactoring collapsed to 3.8% of changed lines from 21% in 2022. Throughput rises. Review capacity does not.

One more wrinkle, in the other direction. A May 2026 study of 6,164 translations across GPT-4o, DeepSeek-Coder and Magicoder found that a meaningful share of reported translation failures were evaluation artefacts: compiler flags, library linking and runtime configuration rather than wrong logic. Our measurements of how bad translation is are themselves noisy. Be less confident both ways.

The strongest case against everything above

Take it seriously, because it is not weak.

First, Google shipped. 74.45% is not a rounding error and those migrations were real. Second, AWS Transform for mainframe reached general availability in May 2025 doing exactly the reverse-engineering step, turning COBOL, PL/I and JCL into structured business rules before any target code is written. Third, and most seriously, every benchmark figure quoted above is a snapshot. Legacy-Bench is from April 2026. COBOL-Coder tripled GPT-4o's COBOLEval Pass@1 in one paper, mostly through better data curation. The comprehension-good, translation-bad split I have drawn may describe the capability frontier in August 2026 rather than anything structural.

I think that last objection is probably correct. I still do not think it changes the plan.

The binding constraint is not model capability. It is the cost of establishing ground truth about behaviour, and that cost sits outside the model entirely. Trace capture, test harnesses, data access, regulatory approval to run production traffic through a shadow system. A model twice as good at translation halves none of it. It makes unverified output arrive sooner.

Thoughtworks named the three practical failure modes on 2 March 2026: hallucination, incompleteness and drift. Drift is the underrated one. Same prompt, same source, different answer on a different day. You cannot build a migration plan around a component whose output is not reproducible, and reproducibility is not something scaling has fixed.

IBM's Rob Thomas said in February that translation captures almost none of the actual complexity. He sells mainframes, so discount accordingly. He is also not wrong, and IBM's accompanying note that roughly 40% of COBOL does not run on mainframes is a reminder that even the size of this problem is contested. Industry surveys have put the world's COBOL at 800 billion lines and at 250 billion. Both cannot be right.

A sequence that survives contact

The order matters more than the tooling.

Pick one seam. The smallest component with a real boundary, ideally at the edge of the system rather than the settlement engine. Resist the instinct to start where the value is.

Have the model read it and draft the specification. Hand that draft to whoever still understands the domain and let them mark it up. Their corrections are the asset, not the draft.

Capture live traces. Real inputs and outputs from production, at volume, across a window that includes month-end and year-end. This is the step that gets skipped, and it is the step that decides the outcome.

Turn those traces into characterization tests. Before the rewrite, not after. Writing tests against the new implementation is faster, and the tests pass, and it is a lie that takes eight months to surface.

Rewrite the unit, model-assisted, however you like. This part is now the cheap part.

Shadow-run. Both implementations against live traffic, outputs diffed. Any mismatch sends the unit backwards. Never forward, never with a waiver, never because the difference looks cosmetic. A diff you cannot explain is a business rule you have not found yet.

Then repeat, keeping both systems live and shifting routing incrementally. Which means something durable has to hold routing and reconciliation state for months, and that orchestration layer is usually harder to get right than any individual rewrite.

What the good estates did in 2024

The organizations that modernize well between now and 2029 will not be the ones with access to the best models. Model access is a credit card. They will be the ones that spent the previous two years building the unglamorous half: production trace capture, a build that reproduces, a test harness that runs unattended, a written definition of correct that somebody signed their name to.

None of that appears on anyone's slide.

There is a tension here I have not resolved. If a model can reliably reconstruct the specification from the code, then the specification becomes the asset and the code becomes a build artefact. At which point rewriting COBOL into Java in 2026 starts to look like the same category error as rewriting it into Java in 2011, executed faster and at greater volume.

The 97% figure is the one I keep returning to. An agent that is wrong and flags it is a tool. An agent that is wrong and confident is a liability that scales linearly with how much you use it. Nobody is benchmarking the distance between those two states, and it is the only number that would tell you whether any of this is safe.

Tools referenced

Claude, reviewed here: Claude review.

Cursor, reviewed here: Cursor review.

GitHub Copilot, reviewed here: GitHub Copilot review.

Devin, reviewed here: Devin review.

Semgrep, reviewed here: Semgrep review.

Promptfoo, reviewed here: Promptfoo review.

Sources

Factory AI, Legacy-Bench: Can AI Agents Maintain the World's Most Critical Software? (1 April 2026): https://factory.ai/news/legacy-bench

SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging (arXiv, July 2026): https://arxiv.org/abs/2607.04092

COBOL-Coder: Domain-Adapted Large Language Models for COBOL Code Generation and Translation (arXiv, April 2026): https://arxiv.org/abs/2604.03986

Migrating Code At Scale With LLMs At Google (FSE 2025 Industry Track, arXiv): https://arxiv.org/abs/2504.09691

Anthropic, How AI helps break the cost barrier to COBOL modernization (23 February 2026): https://claude.com/blog/how-ai-helps-break-cost-barrier-cobol-modernization

Thoughtworks, Claude Code and COBOL modernization: What's the reality? (2 March 2026): https://www.thoughtworks.com/en-us/insights/articles/claude-code-cobol-modernization-reality

GitClear, The Maintainability Gap: AI Code Quality in 2026: https://www.gitclear.com/the_ai_code_quality_maintainability_gap

ITPro, IBM's response to Anthropic's COBOL modernization claims (24 February 2026): https://www.itpro.com/software/development/anthropic-says-claude-code-can-help-streamline-cost-prohibitive-cobol-modernization-but-ibm-says-its-not-that-simple-decades-of-hardware-software-integration-cannot-be-replicated-by-moving-code