America publishes a lot of data about itself. Much of it is public in name only. The file exists. The download link works. That is roughly where the public part ends.
Concrete example. Earlier this year, as part of the DOGE transparency push, HHS published Medicaid’s payment records: every payment to every billing provider, month by month, 2018 through 2024. About 238 million rows. Sum the payment column and the file says Medicaid paid out 21.8 trillion dollars. The actual program spent about a quarter of that, which you can check against CMS’s own audited reports.
The file is real. The numbers need work.
what the file says · $21.8T
what Medicaid actually spent · about $5T
Doing that work has always meant a team. Someone to decode the files, someone to find where they lie, someone to cross-check the totals, someone to write down what can and cannot be trusted. If you just wanted to know what the data says, there was no good answer short of doing all of that yourself.
healthsets is an open-source experiment built on a bigger bet: AI models can now do that work themselves. Not as autocomplete for a data scientist. The whole job, in days instead of months, and done more carefully than schedules usually allow, because being careful stops being expensive.
You bring the questions. The model develops the domain expertise.
So I ran the experiment. I pointed Claude Fable, Anthropic’s model, at the Medicaid file with one question: who does Medicaid actually pay? (Fable, because nobody has given me Mythos access. If you work at Anthropic and want to fix that, my inbox is open.)
I picked the question and kept veto power over the boring. The rest was the model’s: it acquired the file, found the traps, designed the cleaning rules, and checked its own work against the outside world. Some of what that meant in practice:
- a text encoding trap that crashes standard tools halfway through a file
- a fake provider ID quietly carrying billions of dollars in a single year
- fifty states billing the same hour of care under different codes, which makes some states nearly invisible in national data
- reconciling the cleaned totals against CMS’s audited books, for all seven years
- sorting half a million billing IDs into the real organizations behind them
- a build rule that refuses to publish the report if any company it names is missing its citations
All of that happened inside a week, most of which I spent saying yes, no, and simpler.
the same task, as three states record it:
New YorkT1019
TexasS5125
CaliforniaZ9525
Z9525 is California's own invention. It appears in no national code book.
The answer to the question surprised me. The report opens by asking you to guess, so I will not spoil it here:
Everything on that page is computed in your browser from a published set of data tables, and every number can be re-derived from them. The tables, the row-level records, and the code are on GitHub, definitions included.
This is the first public piece of healthsets. The pipeline behind it is not ready for release yet. More datasets are coming, because Medicaid is one file in a universe of public data that deserves the same treatment.
If the bet is right, the expensive part of making public data public is now mostly a matter of asking.