Super-intelligence, provenance, and the new work of the analyst

A recent Hacker News discussion about a Meta engineer’s discomfort shipping AI-generated code they no longer fully understood prompted a question we often ask ourselves: what happens when production accelerates faster than comprehension? (news.ycombinator.com)

Super-intelligence can create problems. Software researcher Margaret-Anne Storey describes two related problems associated with AI software production: cognitive debt, the erosion of shared understanding, and intent debt, the loss of recorded reasons, goals and constraints. Simon Willison described losing his mental model of projects after generating entire features with AI. (arxiv.org, simonwillison.net) At Building Insights Together, these are practical research questions. As agents take on more engineering, we are developing ways to preserve the judgment and evidence needed to challenge what they produce.

Our starting point is disciplined validation. Anthropic’s guidance on evaluating agents emphasizes explicit success criteria, checking actual outcomes, and combining automated tests with human judgment. (anthropic.com) Forge, BIT’s internal production system, requires a challenge code before code can be accepted, demands strong empirical validation before accepting a methodology, and runs nightly quality assurance to identify problems and opportunities for improvement. We are developing this into a repeatable research pattern: state the claim, identify evidence that could disprove it, challenge the implementation, and retain the evaluation. The distinction matters because a pipeline can execute perfectly while measuring the wrong population or treating a policy announcement as evidence of implementation.

Provenance makes improvement inspectable. Our methods have distinct editions, with records connecting source materials, transformations, analytical decisions and outputs. Changes retain their rationale so that successive editions can be compared. (building-insights.org) We connect proposed improvements with the failures that motivated them, the relevant data and implementation versions, and the evidence used to evaluate them. Self-improvement then becomes a testable claim: did error decrease against independent reference evidence? Did performance deteriorate for a particular place or building type? Did an apparent improvement come from changing the definition of the indicator? A future agent needs access to why a decision was made, including the limitations that remain.

This changes the emphasis of analyst work. AI agents take on more implementation, architectural iteration, testing and technical documentation. So instead we invest in broad skills across statistics, climate science, economics, social conditions and policy, with the technical literacy to interrogate the resulting systems. Compared with a conventional software development role, more of our analysts’ attention goes toward interpretation: what a measure means, how an institution works, whose circumstances a dataset represents, and where an assumption stops making sense. This is a technical practice with a strongly humanistic component—close reading, contextual understanding, comparison and judgment.

An analyst’s “smell” is often where an investigation starts. For instance, two agreeing datasets might share an upstream source. Or, lower household energy consumption might reflect under-heating. A policy target might have quietly become a claimed outcome. Recognizing these possibilities requires knowledge that crosses disciplinary boundaries. The next step is to investigate: a confirmed concern can become a regression test, a revised evaluation rubric, a methodological change or an explicit limitation. This is how BIT is advancing the practice of applied AI research. We are building systems that turn human judgment into durable, inspectable improvements, so that each investigation strengthens both the evidence and the way the next investigation is conducted.