Agents that migrate pipelines nobody fully documented
Legacy SAS and Teradata pipelines, moving to BigQuery. The translation itself is tedious but tractable; the hard part is that an LLM will hand you a confident answer whether or not it's the right one, and the pipelines have behavior that isn't written down anywhere. So the interesting design question isn't "can an agent write the SQL" — it's "what does the system do when the agent is wrong."
- 1SAS / Teradatalegacy source
- 1Planner agentLangGraph
- 2Migration workstreams ×5+concurrent
- 3Airflow / Dataprocorchestration
- BigQuerytarget
-
01
A plan that reads correctly and isn't. Model output is fluent by default, so fluency tells you nothing — a single long generation gives you one place to be wrong and no way to localize it.
Built the orchestration as discrete, individually checkable steps rather than one monolithic generation, so a bad step surfaces at the step instead of at the end of a three-week migration.
-
02
One workstream taking the others down. Five-plus migrations run at once; without isolation, one bad translation stalls a team that has nothing to do with it.
Designed the backend services (Python, FastAPI) for modularity and fault tolerance so workstreams fail independently — a stall is contained to one team, not the program.
-
03
Long transformations dying halfway and leaving partial state — the failure that's worse than no run at all, because now the warehouse looks fine and isn't.
Everything executes through Airflow and Dataproc, so retry and re-run are the ordinary path rather than a manual recovery someone performs from memory.