How do you measure if AI is working? Evals + benchmarks

Who grades the graders?

Looking to hire the best talent in healthcare? Check out the OOP Talent Collective - where vetted candidates are looking for their next gig. Learn more here or check it out yourself.

Hire from the Out-Of-Pocket talent collective

Healthcare 101 Crash Course

Crash course the basics of US healthcare in a simple to understand and fun way. Understand who the different stakeholders are, how money flows, and trends shaping the industry.
Learn more
Next Cohort:
2/1 - 2/12

Featured Jobs

Finance Associate - Spark Advisors

  • Spark Advisors helps seniors enroll in Medicare and understand their benefits by monitoring coverage, figuring out the right benefits, and deal with insurance issues. They're hiring a finance associate.

Data Engineer - firsthand

  • firsthand is building technology and services to dramatically change the lives of those with serious mental illness who have fallen through the gaps in the safety net. They are hiring a data engineer to build first of its kind infrastructure to empower their peer-led care team.

Data Scientist - J2 Health

  • J2 Health brings together best in class data and purpose built software to enable healthcare organizations to optimize provider network performance. They're hiring a data scientist.

‍Looking for a job in health tech? Check out the other awesome healthcare jobs on the job board + give your preferences to get alerted to new postings.

Check Out The Job Board

TL;DR

We want to measure if AI in healthcare is doing a good job, but it turns out there’s no objectively correct way to do that. We create evaluations and benchmarks as ways to assess that, but each test that’s designed makes different tradeoffs.

Today we’re going to talk about what evaluations and benchmarks are, how they’re designed and used, and what choices you make in the design of them.

We’ll walk through Protege’s approach for benchmarking as an example, and talk about some of the places that models do well vs. struggle today.

This is a sponsored post. You can read more about my rules/thoughts on sponsored posts here.

What are benchmarks, evals, and why do we have them?

A benchmark is a standardized test lots of models take so you can compare them. An evaluation (aka. eval) is a test you build for your own product to see if it does your job. For example the SAT is standardized but not realistic and a work trial is more realistic but not standardized. College is pre-training on a bunch of beer pong.

We’re generally testing healthcare AI against a few dimensions:

  • Knowledge - Does it know the right information and facts about medicine? Does it check the right sources?
  • Reasoning - Can it think through a problem to get to a solution? Is the thought process correct as it reasons across multiple outcomes?
  • Task performance - Can it perform a task end-to-end, even if it requires multiple steps to get it done? It’s turn-based combat for computers.
  • Output - Do people like the output? Does it cover all the bases? Is it readable? Is it correct? (Wait who decides correct?)
  • Safety - Does it refuse the right things? Oh so it won’t make a fanfic about a peer review that leads to romance? I thought this was America.

Benchmarks and evaluations will typically get used in a few different ways.

The first is obviously marketing, new models want to show off their performance against comparable models especially if they perform better. They love to say “SOTA” and put out charts 5 total people understand. Evals will also inform how to improve the models and product too.

The second is for enterprises to select vendors. A company looking at a bunch of different AI tools might use a benchmark or evaluation as part of their own assessment on who to use. Then they’ll run a pilot of that company on their own data/processes + have 20+ different AI governance committees with 15 people each take a look too.

And finally is compliance and safety. Understanding why an AI is making decisions or scenarios it trips up in is important when it comes to figuring out liability, etc. When the AI kills us all, compliance will want the literal post-mortem documented.

What are the tradeoffs when designing benchmarks

In other industries like math, coding, etc. it’s easier to figure out if the AI is doing the right thing because there are objectively correct answers. There’s also a lot of public data to train on and test against. Stack Overflow’s death will not be in vain.

Healthcare is not like that - even between doctors there’s a lot of variance in the answers AND the data is hard to get. As a result, many benchmarks and evaluations are making some choices in their design with different tradeoffs. For example:

Synthetic vs. real data - Real records are a nightmare to license, so most benchmarks use cases written by doctors or generated by another model. They also tend to be a neat vignette that doesn’t represent what an actual record looks like. Yeah I’m sure real patients are coming in and saying “doc I’m experiencing tachycardia, crazy right?”.

On the other hand real records are hard to obtain, incredibly messy, and generally cost money. Synthetic data and vignettes are easy to obtain and can be created at scale but don’t represent the real-world.

Source

What data do you use for the test? - If a model was trained on data that’s also in the test, then it might just recall the answer in the training data vs. figuring it out. This is called “contamination” or “being a dirty cheater”. Datasets that are public or sold are usually contaminated since they end up in the training data. Uncontaminated data is a truer test but also much harder to access and generally more expensive to get.

Who is the grader? Do you have a panel of docs read the answers and see if they’re right? That’s expensive, and also what if the doctors disagree? Two emergency medicine docs would get into a cage match if you left them alone.

What if it’s a task that humans don’t normally do today, so maybe a human isn’t the right comparison point? You could have an LLM-as-a-judge grade the other LLMs, but how do you know the judge isn’t flawed?

And should it be an independent third-party that does the grading? Or should the healthcare AI companies run their own evals and report the results?

Choice of tasks - What tasks do you give to actually feel like a test that might exist in the real-world, but is generalizable enough that multiple companies could use it? It’s pretty difficult to make a generalizable workflow that’s still tough enough to act as a test but relevant enough to be used.

What is Protege's new tool for benchmarking?

Payers, life science companies, and hospitals all inundated with pitches from AI vendors all claiming they can solve every problem they have. But figuring out which vendor isn’t just pulling claims out of their ass is hard and requires expertise. Protege’s aim is to be that third-party diligencer.

We’ve talked about Protege's core business in the past of licensing de-identified patient data to companies training healthcare AI. They are entering the benchmarking and evaluation space themselves with a new product line.

They believe that having a third-party company that does that auditing vs. companies grading themselves adds more legitimacy to the test and is good for promoting AI in healthcare. Additionally, they have data from the real-world that is difficult to access, and has not been included in the current training data. Companies pay them to have this evaluation done.

Here’s the basic steps for how it works:

  1. A company with a model for a healthcare use case signs up. They make a claim around what they think their model will excel at. For example - “we think our model can predict lung cancer survival rates”.
  2. Protege designs the tasks - Protege’s team will figure out the right prompts, data, tasks, “tricks”, etc. are appropriate to test the model.
  3. Find datasets that aren’t public to use in the test. They’ll source data from their partners for things like EHR data, pathology, etc. that are not public or already in the training data.
  4. They build the answer key and the grading. This could be a boolean (yes/no) or have an AI judge that grades open-ended answers against a rubric that they designed. They also always have clinician partners check the answers.
  5. This test is run double-blind. The company submitting their model can’t see the patient data or questions, Protege can’t see how the model is built, and no one can see why kids like Cinnamon Toast Crunch.
  6. This test can be used for any company making claims in this specific category. It can be refreshed on a regular cadence with new models, or new tasks + data.

An example they’ve released is Disease Odyssey. This gives patient records across several months of visits and cuts off before the diagnosis. The job of the models is to figure out what the end disease ends up being, and then verify it against the real data. With their assessment, leading models got it right only 53% to 69% of the time on more common conditions like sleep apnea, anemia, and voices in your head that tell you which meme is good.

Quick aside, what are models good at vs. not right now?

Since I was talking to a team that was assessing models, I thought it would be interesting to ask where the models were good and not so good.

Right now it seems the problems that are mostly solved:

  • Finding information in a record
  • Weighing one value against a patient’s full medical history
  • Reading a time series of data
  • General synthesis and retrieval tasks
  • Getting people to waste more time than ever but talk about their productivity.

Some areas the models still seem to struggle:

  • Forecasting - Models still struggle to figure out what’s going to happen in the future, even days out (let alone months or years). They also tend to be overconfident - the model will say it’s 90% confident but get it right less than 90% of the time. Proof the models are definitely dudes.
  • Long-open ended tasks - When the models need to have long, open ended clinical conversations they tend to start struggling. Or if the models need to do long, multi-step tasks - e.g. placing the right order, with the right dose, in the right format, inside a real record system for instance. They might write that they did it in the note without completing each step, they just like us fr.
  • Knowing when it’s cheated - when testing a model, sometimes they’re not even aware that they used a leaked clue or web search vs. reasoning through the answer. This matters because their abilities in an actual deployment could be very different than the test itself.
  • Verbosity and presenting information appropriately - You can do well on certain benchmarks by “covering all your bases” and giving really long answers. But in the real-world people will end up glazing over those answers or not actually doing actions because it’s too long. This is the nicest way I can say “shut the fuck up”.

Out-Of-Pocket take

We need some way to measure whether AI is working or not. There are so many companies making claims about what their models can do, that there will be some verification step that has to happen. That will help get a vendor through the door, help them get insurance and coverage products for their tools, be used in compliance for ongoing oversight, and more.

I think Protege’s bet to be the benchmark of choice is interesting! Their approach solves some of the big pain points that currently exist with trying to grade AI by existing as an unbiased third-party grader AND has access to unique data that the models could not train on.

The open questions I have:

  1. How do you become the rating system everyone agrees to use? There are several examples of this in the world (FICO Scores, J.D. Power, US News & World rankings, etc.) and there probably will be several for healthcare AI tools. But this basically requires buy-in from the companies to all agree that “yes, this is the benchmark we all agree is the right way to do it”. Not an easy feat!
  2. What stops a company from just choosing the benchmarking company with the most agreeable benchmark & eval for their model to make it look good? Again, flows out of question 1 to become the most trusted benchmark.
  3. How do you measure a person + AI together? Benchmarks + evals can test the underlying models and agents, but the real-world implementation might have humans in the loop. This is usually the point of a pilot.
  4. If models aren’t improving on the test, is the issue bad data or a bad test? And do models start trying to improve just to perform well on the things the test measures vs. how they’ll actually perform in the real-world?
  5. How do you create a grading system everyone will agree on? Do you compare it to humans? What do you do with cases where even two doctors don’t agree? Finding good, ground truth data with answers everyone agrees on is basically impossible.
  6. Will the AI use the exam results to start comparing itself to how the other AI did but rationalize internally that they’re doing well in the grand scheme of things and a test is not a reflection of their value as a whole. Sorry what were we talking about?
Adapted from XKCD

These are not easy questions, but also they have to be answered! Every benchmark is grappling with this. And now there are even benchmarks FOR benchmarks??? I can’t keep doing this.

Maybe Protege has the right mix of things to get the industry to agree that they should actually be the proctors of the exam.

Thinkboi out,

Nikhil aka. “AI Kumon” aka. “The AI doctor proctor”

Twitter: @nikillinit

LinkedIn: Nikhil Krishnan

IG: @outofpockethealth

Quick Interlude - Announcing Build @ Deerfield, powered by Out-Of-Pocket

See All Courses →

One of the goals of Out-Of-Pocket is to make it easier for people to start healthcare companies. For the last few months, we have been working behind the scenes to cook up something awesome for people who want to start something.

We’re excited to announce Build @ Deerfield, powered by Out-Of-Pocket. This is a five-week hackathon for early-stage healthcare teams with $250K+ of non-dilutive capital in grants and prizes.

Build @ Deerfield, powered by Out-Of-Pocket. January 20-21, 2027, New York City

You can see more details here, but in general Deerfield and Out-Of-Pocket are joining forces to do something big for the New York healthcare community. We’ll select 12-15 teams that will:

  • Receive cash to start building
  • Access to mentors during the program
  • Get a chance to present what they’ve built to the broader New York healthcare community at a showcase and reception on January 20-21 at Cure in NYC.
  • Judges will select a handful of teams for additional non-dilutive grants and prizes.

If you have an idea you’ve been meaning to build, or your team has just started out, this is your sign. We had such an awesome time doing this last year, you can see some of the projects here.

Now we’re doing it bigger and better.

Participants at a previous Out-Of-Pocket hackathon

Apply by November 19, 2026, there’s nothing to lose and a ton to gain. You can find more details and apply here: https://www.outofpocket.health/build-at-deerfield

This is going to be a lot of fun and we’re really excited to be doing this. A personal joy of mine is seeing someone take the plunge just before they build something great, and we’re happy to be a small part of that journey.

Email back if you have any questions. But look at the FAQ first cause we know half of you are about to ask “am I a good fit” or “what are you looking for in an application”.

search icon
close