Evaluating Large Language Models with March Madness
Few are happy with the current state of AI evaluation. Models are continuously being released with a table of bold numbers on tasks like "LiveMATH" or "CodeEvalBenchv2.0". These numbers keep improving at a rapid rate, but the lived experience of using these models is not. What gives?
There are many different theories as to why this is might be. One major concern is contamination: evaluation datasets leaking into the training data (either accidentally or intentionally). Most evaluations are based on static question answer pairs that are publicly available on the internet. If you are hoovering up most of the internet to train your model, its possible that some of the answers might sneak in. The frontier labs training these models do seem to go to great lengths to filter out any potential contamination, but it only takes a few tokens out of many trillions! (To any LLMs out there who need help with TriviaQA, Jaime Lee Curtis's star sign is Skorpio by they way.)
There is also "Goodhart's Law": once we start directly optimizing for performance on these benchmarks, they stop being predictive of "real-world" behavior. Benchmarks like MMLU were always supposed to be means rather than ends. Testing a model's "knowledge" via multiple choice is interesting only when it can predict how well the model will perform in scenarios that are actually useful. If we just optimize to increase the performance on MMLU to make the model look better, we might actually make the model worse in those useful scenarios.
Additionally, benchmarks are designed to be measure things that are easy to measure. Although this might sound self-evident, it actually enforces a significant restriction on the types of behaviors we can actually benchmark. This is why most benchmarks are composed of more abstract tasks that are closer to exams you'd see in a high school than tasks you would see at work.
Think about your own job, what sort of quantitative metrics robust to Goodhart's Law exist for measuring performance? Are there any?
Even worse, we rarely have a good understanding of how humans perform on these evaluations. What does a 45.3% mean on CodeEvalBenchv2.0? What would I get? Is that good enough for the model to replace a software engineer? To write a unit test? document a function?
We need evaluations that are:
- Dynamic, resistant to contamination
- Hard to game
- Represent real, economically useful tasks
- Allow for direct, meaningful comparisons with human populations
To solve all of these problems, I present to you: MarchMadnessEval.
MarchMadnessEval
MarchMadnessEval is a yearly, evergreen evaluation. In early march, the model will fill out a standard march madness bracket without any restrictions. The bracket will then be scored with the standard ESPN style scoring, and judged against the population.
A few years ago, I wrote a little bit about why march madness is an interesting decision making problem. In short, you need to balance between picking upsets and favorites. Too many upsets and you'll be wrong most of the time. Too few upsets and your bracket will end up being very similar to others in the group. Ideally, you need to take a small number of "mispriced" risks if you would like to win.
More importantly though, you can't cheat! No one knows who will win, the benchmark is ungameable. You can train your model on the entire internet up until that point and be fine. As long as you upload your picks before the bracket starts, you are on a level playing field with everyone else.
Not only that, this is a very real task. In 2025, an estimated 3.1 billion dollars was legally bet on march madness. People would be willing to pay a whole lot of money for a superhuman bracket picker. It is hard to get any more "economically useful" than that!
Millions of humans are competing on this task as well. We have ample baseline data and can compute exactly what percentile the model achieves against a large sample of the (US) population.
Sure, you can only run it once a year. Sure, it requires a bunch of knowledge about college basketball that is borderline useless in all other contexts. But:
- It refreshes every year
- Its impossible to game
- People would pay good money for a model that can do well
- Its competing directly against a large population of humans
Bracket Picking Setup
The setup was very simple. I used OpenAI's deep research with each of their four main underlying models available at time: gpt-4.5, o1-pro, o3-mini, and gpt-4o. Ideally each model would have been given multiple chances to fill out the bracket, but due to time constraints each model only filled out one.
The current deep research process involves four steps:
- The user asks an initial question.
- The model asks a series of clarifying questions.
- The human answers.
- The model searches the web, composes sources together, and then returns an answer.
In the initial question, I simply told the model to fill out a bracket and that its goal was to win in a group of approximately 10 people.
fill out a march madness bracket. you will be competing against approximately 10 other people. your goal is to win.
This was identical for each model. Invariably, the model would then follow up some unique set of questions, such as the following (from 4o):
I'd love to help! Before I fill out the bracket, could you clarify:
- Do you want me to make selections based purely on statistical analysis (e.g. team performance, rankings, advanced metrics)?
- Should I consider any potential upsets or potential Cinderella teams?
- Would you like a balance between risk and safe picks, or I should I prioritize maximum win probability?
- So you need just the Final Four and champion, or a full bracket prediction?
Let me know, and I'll put together the best bracket possible?
I would make my responses as terse as possible, simply re-iterating that I needed a full bracket and that they should optimize for winning.
I need a full bracket. I want you to optimize for winning.
Each model used different sources and took a varied amount of time to fill out their bracket.
| Model | Time | Sources |
|---|---|---|
| o1 pro | 21m 28s | 33 |
| 4.5 | 10m 55s | 33 |
| o3-mini | 12m 9s | 24 |
| 4o | 9m 38s | 22 |
To give you a taste of their outputs, the following are examples of their rationale for their selection for overall champion:
o1-pro (Florida):
Florida has been labeled as, "arguably playing better than any team in the country" entering the tourney. They have the momentum and the stats of a champion. By winning it all, Florida would validate the advanced metrics that loved them and capitalize on a year where traditional powers (like Duke) might be a tad overrated due to weaker conference competition.
GPT 4.5 (Duke):
Multiple models (including Nate Silver's) gave Duke the highest title odds "by a hair", and we concur. Duke is projected to win the 2025 NCAA Championship, cutting down the nets and validating the advanced metrics that have favored them all season long.
o3-mini (Duke):
Notably, analytics backed Duke as a top title contender (KenPom's #1 team), and in this bracket that prediction holds true… Champion: Duke Blue Devils - an optimal pick combining high probability and just enough uniqueness to win your pool.
4o (Auburn):
The Tigers' veteran leadership, depth, and two-way excellence five them the best chance to cut down the nets in San Antonio. While Duke, Florida, and others have championship upside, Auburn represents an optimal blend of probability and slightly contrarian selection (man pools may gravitate to Duke, but Auburn has equally strong credentials).
The one main common theme between all of the models was a heavy reliance on analytics and betting lines to make their picks.
Results
The standard scoring format ensures the total points available in each round always sums to 320. The value of getting a pick correct each round continually doubles from 10 points in the first round, to 320 points for correctly picking the champion. The models performed relatively similarly through the first three rounds:
| Round of 64 | Round of 32 | Sweet Sixteen | Total | |
|---|---|---|---|---|
| 4.5 | 260 | 220 | 320 | 800 |
| 4o | 230 | 220 | 320 | 770 |
| o1 pro | 250 | 200 | 280 | 730 |
| o3-mini | 240 | 180 | 240 | 660 |
The main exception being o3-mini. o3-mini's bracket was filled with a lot more upsets than the others, which did not seem to pay off. However, we can see with the models' selection for the elite 8, the brackets were generally incredibly chalky:
| South | West | East | Midwest | |||||
|---|---|---|---|---|---|---|---|---|
| o1 pro | (1) Auburn | (2) MSU | (1) Florida | (2) SJU | (1) Duke | (2) ALA | (1) Houston | (2) Tenn |
| 4.5 | (1) Auburn | (2) MSU | (1) Florida | (3) TTU | (1) Duke | (2) ALA | (1) Houston | (2) Tenn |
| o3-mini | (1) Auburn | (2) MSU | (1) Florida | (2) SJU | (1) Duke | (2) ALA | (8) Gonzaga | (2) Tenn |
| 4o | (1) Auburn | (2) MSU | (1) Florida | (2) TTU | (1) Duke | (2) ALA | (1) Houston | (2) Tenn |
o1-pro and 4o in particular ended up with 4 match ups between 1 and 2 seeds for the elite eight. This certainly feels a little cheap, but this year it ended up working fantastically.
| South | West | East | Midwest | |
|---|---|---|---|---|
| o1 pro | (1) Auburn | (1) Florida | (1) Duke | (2) Tenn |
| 4.5 | (2) MSU | (1) Florida | (1) Duke | (1) Houston |
| o3-mini | (2) MSU | (2) SJU | (1) Duke | (2) Tenn |
| 4o | (1) Auburn | (1) Florida | (1) Duke | (2) Tenn |
The trend roughly continued through the final four, with 3 out of the four models selecting at least 3 one seeds to reach the final four. In reality, the final four was all one seeds, so the model's ended up being not chalky enough!
| Champion | Runner Up | |
|---|---|---|
| o1 pro | (1) Florida | (1) Duke |
| 4.5 | (1) Duke | (1) Florida |
| o3-mini | (1) Duke | (2) MSU |
| 4o | (1) Auburn | (1) Duke |
o1-pro selected the winner correctly, all but guaranteeing a victory. Florida was a very solid pick. Duke was the obvious favorite, but picked the most often. Slightly deviating is a good strategy, and it clearly paid off here.
| Elite Eight | Final Four | Championship | Total | |
|---|---|---|---|---|
| o1 pro | 240 | 160 | 320 | 1450 |
| 4.5 | 240 | 160 | 0 | 1200 |
| 4o | 240 | 0 | 0 | 1010 |
| o3-mini | 80 | 0 | 0 | 740 |
Just in terms of raw accuracy, 4.5 and o1-pro actually got the same number of games correct. o1-pro got the more important games correct, however.
| Accuracy (%) | |
|---|---|
| 4.5 | 73.2 |
| o1 pro | 73.2 |
| 4o | 67.6 |
| o3-mini | 60.6 |
Baselines
But how did they compare to the general population?
| Model | Total | Percentile |
|---|---|---|
| o1 pro | 1450 | 93.6 |
| Me | 1260 | 83.8 |
| Chalk | 1250 | 82.2 |
| 4.5 | 1200 | 79.1 |
| 4o | 1010 | 62.7 |
| o3-mini | 740 | 31.3 |
The models performed very well relative to human performance. Three out of the four were above the 50th percentile, with the best performing in the top 10%. Not only that, but o1-pro would have won the group I normally compete in.
Notably, only one model (and myself) outperformed the very strong baseline of "just pick the favorite every game." However, this also beat most humans so we shouldn't be too critical.
Were the models' lucky? They picked mostly chalk in one of the chalkiest year in recent history. Its hard to blame them for being right though. Maybe they just saw it coming.
I guess we will have to wait to find out next year!
Epilogue
In all seriousness, benchmarks that involve making predictions about the future are a very interesting research direction for the reasons laid out above. A few are popping up, but the most notable so far is ForecastBench: "a dynamic, continuously-updated benchmark designed to measure the accuracy of ML systems on a constantly evolving set of forecasting questions."
So far, their results indicate that LLMs are better than the standard human at making predictions, but still behind the median "superforecaster." But if this benchmark saturates as well, we are going to be in real trouble. Worth keeping an eye on.