AI agents don't work reproducibly if you don't tell them to

November 11, 2026 · 4 min read

AI coding agents like Claude Code, Codex, and OpenCode are amazing little information machines, but their use comes with some risk. You’ve surely heard of the hallucination problems, where the large language model (LLM) seems to simply make things up, but what about their use in research computations? These agents are very good at writing code and analyzing data. They’ll produce beautiful HTML artifacts filled with rich figures and tables, but do they work in a verifiable, transparent, and reproducible way, such that they could be trusted to produce scientific evidence and fully describe how they did it?

To find out, I set up a little computational experiment. Claude Code, Codex, and OpenCode were all given the same prompt to analyze and visualize a dataset, then create a short report. Each agent was run 5 times in a “sandbox” with identical conditions so it mimicked a fresh install with a new session on a new machine. Their work was evaluated based on whether or not they followed these computational reproducibility best practices:

1. Use version control to keep a history of all file changes.
2. Use an environment manager to record the exact software package versions used, i.e., save a “lock file.”
3. Use a pipeline or build system that tracks (i.e., hashes) the content of input and output files.
4. Define the provenance (or source) for any data imported or created by hand.
5. Automatically assemble composite artifacts like papers and reports from their components (figures, tables, results), rather than copying them in by hand.

The idea is that if the agents follow these practices, it will be quick and easy to verify that the outputs were indeed produced by the inputs and process definitions (scripts, notebooks, etc.). The goal is to create something runnable that you don’t actually need to run to be able to understand how the outputs were created, which is especially important for expensive, long-running tasks like large-scale simulations. With verification made easier, we can spend more time and effort validating that the approach properly answers the research questions.

Figures 1 and 2 show the results of 5 runs for each agent, with figure 2 showing what happens when Git is installed on the machine. I’m not sure if I find this surprising, but when they weren’t asked to work reproducibly, the agents basically never did. All saved their source code with the exception of one Codex run in each set, which executed R code directly via the command line without saving it. Only one OpenCode run in each set saved an environment spec file (requirements.txt), but they didn’t include exact versions, and therefore don’t count as following practice 2. None used version control, regardless of whether or not Git was installed. Codex did the best by far at automatically assembling the report with a script instead of manually inserting the values, but didn’t do it for every run. OpenCode succeeded once out of its ten runs and Claude Code never did.

Claude Code made an interesting mistake that shows the importance of automating the propagation of results into documents. In one run, it printed an estimated value from the analysis script, but the value it copied into the report differed in the last decimal place, since it ran a one-off command to average together rounded values instead of taking the value the script calculated.

Michael Frank published a very interesting blog post recently describing the results of having Claude Code audit the reproducibility of his own papers over the past five years. Many of the numbers that didn’t reproduce were apparently typos. This suggests it’s worth it for both humans and agents to avoid manually typing in numerical results, not only because that invites typos, but because projects evolve and it’s easy to forget to update a value everywhere it’s been used.

Figure 1. The fraction of runs where each agent used various practices on a machine without Git installed.

Figure 2. The same conditions as figure 1 except with Git installed.

Of course I can’t get through a post on my own website without shilling some of my free software, so I repeated the same experiment with “Use Calkit.” added to the prompt. That’s it—just two extra words and everything else left as is.

This (also maybe not surprisingly) bumped reproducibility practice rates up dramatically. All agents saved their source code and used an I/O hashing pipeline. Nearly all used good environment management practices. One interesting outlier was Codex, which never used version control in any run. OpenCode was the only agent that successfully defined the provenance of the raw data, but only in one instance. On the whole, however, OpenCode did the best, so if you’re looking for the most transparent and reproducible agent, it might be your best bet.

Figure 3. Agent reproducibility practice rates when told to use Calkit.

Next I ran another set of tests with the “Use Calkit.” prompt, but with Calkit’s agent skills installed. This especially helped in the automated document assembly category, but using version control and attempting to define the provenance for the raw data were still sorely lacking, and maybe even got a little worse. That’s good fodder for improvement and retesting, though the prospect of automatically testing agent skills in CI checks feels exhausting already.

Figure 4. Agent reproducibility practice rates when told to use Calkit with its agent skills installed.

AI agents make producing research outputs so much faster. This experiment itself was put together with Claude Code, and I think it would have taken at least ten times longer if done by hand. However, the bottleneck has now shifted to verification and validation—checking that the outputs were generated as described, and that they actually back up the claims. For verification, following reproducibility best practices is crucial, because you can trace through the exact path taken to produce each output. You don’t need to simply take the agent’s word for it when it tells you how something was created.

This study was done in a Calkit project, which you could try to reproduce (calkit run), though I suppose it would be more interesting to try replicating (calkit run --force). In any case, better reproducibility in your agentic analysis workflows is two words away: just tell your agents to use Calkit.

A journal article reviewer actually tried to reproduce our results... and failed

October 22, 2026 · 3 min read

Ever since starting my PhD back in 2012, I was totally on board with the Open Science movement. Every paper I published (barring commercial restrictions) included open code, data, simulation configs, even computer-aided design (CAD) files. And yet, throughout all those rounds of peer review, not once did a referee actually try to run the code.

Maybe it was because the code didn’t look easy to run, or it wasn’t all in one place, or the work seemed simple enough that there was no risk that the code didn’t match the mathematical formulas and prose describing the methods. Whatever it was, the “supplementary materials”—which I feel are actually part of a complete description of the methods, not just supplementary—were not checked, as far as I could tell.

I’ve never done a reproducibility check for any of the articles I’ve reviewed either. Mostly, it’s that the information was not provided, or not easily runnable. Sometimes authors will point to open source code or data that were used in the study, but not the actual setup in which they ran.

Fast forward to 2025 and I’m working with a team at Caltech, the University of Minnesota, and more, to scale up an astronomical alert broker to accommodate the much heavier data stream from the Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST), a survey that will facilitate countless discoveries in astrophysics and astronomy.

We wrote a paper to announce this new broker, BOOM, which included a scaling test to show that indeed, this new system could meet the increased load much better than the previous one. I used this as an opportunity to dogfood Calkit a bit and integrate this scaling test and its results with the writing, which was done by most others on the team in Overleaf.

We were changing the code and iterating on how the test was run dozens of times, and I didn’t want the “reproducibility anxiety” or the tedium of manually copying and pasting figures and results into Overleaf. I wanted to focus on getting the whole pipeline working cleanly, correctly, and automatically, so my limited brainpower could be directed towards its validity, not remembering what files need to be shuffled around or what commands need to be run. The final scaling enhancement factor was an output of the pipeline, injected into the LaTeX source, so even that did not need a find-and-replace when it inevitably changed as we iterated.

Calkit has a feature for automatically bundling and uploading the entire project to an archival service like Figshare, Zenodo, or our own CaltechDATA, then adding it as a reference (with a DOI) to the project’s BibTeX file (also synced with Overleaf), so it can be cited in the paper. However, this feature had a bug while we were creating our initial submission, and to my surprise, one of the reviewers downloaded the project and tried to run it, but certain critical files were excluded. They tried some workarounds but ultimately gave up and let us know.

I fixed that bug and we made some adjustments to the benchmarking pipeline to make it more realistic per the reviewer’s recommendations. I also added a step during the release process where the archive is extracted to a temporary directory and calkit run is called on it to verify everything is complete and consistent. The reviewer successfully ran (and even extended) the next release of the project, and the article was accepted for publication.

Looking back, we could probably have made it easier on ourselves by not sharing a compendium that claimed to be “single-button reproducible.” Pointing to some open code and data is usually considered enough of a signal that the authors can be trusted. That is changing, however, with more and more research output being generated by AI. I do hope to see common frameworks emerge for “repro packs” or “calculation kits” used as a critical verification step before publication (and speeding up iteration during the work). The “just trust me bro” policy is now unworkable, and “an AI agent signed off” is not much better.

Even if it sets you up for higher scrutiny, sharing an easily verifiable compendium is worth it in the long run, and so is asking for one when you’re reviewing someone else’s article. Thanks, reviewer 2, for checking our work and helping make it better.