A journal article reviewer actually tried to reproduce our results... and failed
Ever since starting my PhD back in 2012, I was totally on board with the Open Science movement. Every paper I published (barring commercial restrictions) included open code, data, simulation configs, even computer-aided design (CAD) files. And yet, throughout all those rounds of peer review, not once did a referee actually try to run the code.
Maybe it was because the code didn’t look easy to run, or it wasn’t all in one place, or the work seemed simple enough that there was no risk that the code didn’t match the mathematical formulas and prose describing the methods. Whatever it was, the “supplementary materials”—which I feel are actually part of a complete description of the methods, not just supplementary—were not checked, as far as I could tell.
I’ve never done a reproducibility check for any of the articles I’ve reviewed either. Mostly, it’s that the information was not provided, or not easily runnable. Sometimes authors will point to open source code or data that were used in the study, but not the actual setup in which they ran.
Fast forward to 2025 and I’m working with a team at Caltech, the University of Minnesota, and more, to scale up an astronomical alert broker to accommodate the much heavier data stream from the Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST), a survey that will facilitate countless discoveries in astrophysics and astronomy.
We wrote a paper to announce this new broker, BOOM, which included a scaling test to show that indeed, this new system could meet the increased load much better than the previous one. I used this as an opportunity to dogfood Calkit a bit and integrate this scaling test and its results with the writing, which was done by most others on the team in Overleaf.
We were changing the code and iterating on how the test was run dozens of times, and I didn’t want the “reproducibility anxiety” or the tedium of manually copying and pasting figures and results into Overleaf. I wanted to focus on getting the whole pipeline working cleanly, correctly, and automatically, so my limited brainpower could be directed towards its validity, not remembering what files need to be shuffled around or what commands need to be run. The final scaling enhancement factor was an output of the pipeline, injected into the LaTeX source, so even that did not need a find-and-replace when it inevitably changed as we iterated.
Calkit has a feature for automatically bundling and uploading the entire project to an archival service like Figshare, Zenodo, or our own CaltechDATA, then adding it as a reference (with a DOI) to the project’s BibTeX file (also synced with Overleaf), so it can be cited in the paper. However, this feature had a bug while we were creating our initial submission, and to my surprise, one of the reviewers downloaded the project and tried to run it, but certain critical files were excluded. They tried some workarounds but ultimately gave up and let us know.
I fixed that bug and we made some adjustments to the benchmarking pipeline
to make it more realistic per the reviewer’s recommendations.
I also added a step during the release process where the archive is extracted
to a temporary directory and calkit run is called on it to verify
everything is complete and consistent.
The reviewer successfully ran (and even extended) the next release
of the project, and the article was accepted for
publication.
Looking back, we could probably have made it easier on ourselves by not sharing a compendium that claimed to be “single-button reproducible.” Pointing to some open code and data is usually considered enough of a signal that the authors can be trusted. That is changing, however, with more and more research output being generated by AI. I do hope to see common frameworks emerge for “repro packs” or “calculation kits” used as a critical verification step before publication (and speeding up iteration during the work). The “just trust me bro” policy is now unworkable, and “an AI agent signed off” is not much better.
Even if it sets you up for higher scrutiny, sharing an easily verifiable compendium is worth it in the long run, and so is asking for one when you’re reviewing someone else’s article. Thanks, reviewer 2, for checking our work and helping make it better.