Project summary
Insights from prediction markets can offer an alternative to the increasingly untrusted news companies and governments. Yet, the broader public is not tapping into the data coming out of prediction markets. We believe the gap stems from a lack of understanding of how good prediction markets are at forecasting the future. Metrics like brier scores or calibration charts are confusing to most, we seek to evaluate the accuracy of prediction markets and share the data in a way that is easily digestible to most.
What are this project's goals and how will you achieve them?
Calculate accuracy scores for closed prediction markets that can be tied to active predictions allowing anyone to quickly understand accuracy. There are two key components to our project:
1) Analyze closed prediction markets to estimate their accuracy (% of the time market predicts correctly) based on how many days away from the expiry date.
2) Group historical prediction markets into categories such as elections, economic metrics, war, and AI.
Once we establish a % accuracy score within a category we can take that % accuracy score based on days away from market expiry and apply it to active prediction markets. For example, “The US presidential election is X days away, election markets at X days away have a Y% chance of correctly predicting the winner.”
A basic one liner like the above will ascribe real value to the current prediction market and allow for more people to capture the benefits of these markets. Benefits of understanding what will happen in the world can help people decide if they should stay in a country that is at risk of war, go to vote, take on debt…
To establish accuracy scores by market categories, we will gather historical data from: Manifund, Kalshi, Polymarket, Predictit, and Metaculus. We will use python and sql to analyze the data producing reports that we will make open for anyone to see. At the conclusion of the project we will publish a paper explaining our methodology and our findings. We will continue to publish accuracy scores along with new active markets through Base Rate Times, a popular twitter account (https://twitter.com/base_rate_times). We believe news should rely more heavily on prediction markets and an accuracy score will increase that usage.
How will this funding be used?
The funding will go to data scientist wages to download and analyze historical data. Some prediction markets like Kalshi make this process very easy but others like Polymarket require a technical knowledge of blockchain smart contracts. Our team has above average python developer and database experience but will seek to hire experts to navigate some of the more complex prediction markets.
Who is on your team and what's your track record on similar projects?
Brooker Belcourt is the founder of a VC backed fintech company, Covey, that seeks to find and reward the best investment analysts. Brooker has built and scaled covey.io to thousands of users, the project is a virtual portfolio competition open to anyone, where the best rise to the top and are rewarded. On the data scientist side he has worked with big data to produce trading strategies that direct millions of dollars at hedge funds like Citadel and Coatue. You can see the public samples of his data science work at Covey published here: https://medium.com/@brookerbelcourt.
Marcel is the founder of The Base Rate Times, the first news site to prominently feature prediction markets in its coverage. On the data side, he has led analytics projects at multinational corporations.
What are the most likely causes and outcomes if this project fails? (premortem)
We will seek some help from the Manifund team to get the right historical data and then we will be able to get historical data for at least 2 other prediction markets. Where we could fail is if we are unable to access historical data from Polymarket and Predictit. Those two markets present challenges for our team. We are confident we can get three of the top markets. There is a strong potential upside to this project if we are able to convince Metaforecast to help us access the historical data, which they have gathered, we are actively building that relationship now and would benefit greatly from funding to help push that along.
What other funding are you or your project getting?
None
