How is the project going?
We collected data from Polymarket, Metaculus, and Manifold, by writing scripts to retrieve time series data from their APIs and putting together a dataset of comparable questions with matching resolution criteria (43 questions so far). We ran comparisons between the platforms by calculating time-weighted accuracy scores (log and Brier), restricted to the time intervals in common between each pair of platforms.
Currently, in general the scripts and collected results look good but we're still working on validating the data - there were a number of data bugs that we already found and fixed, so there's a pretty good chance there's still some bugs, and we aren't ready to report results yet.
How much money have you spent so far? Have you gotten more funding from other sources? Do you need more funding?
We did not get any additional funding, and no additional funding is needed.
How well has your project gone compared to where you expected it to be at this point? (Score from 1-10, 10 = Better than expected)
3/10 - We were able to collect a lot of good data a broad range of questions, including extras like counts of participants and predictions at each point in time, and have analysis in progress, but I was hoping to have at least basic results published at this point.
Are there any remaining ways you need help, besides more funding?
Nothing strictly needed, but one nice-to-have: If we had a better data source for Polymarket, that would be nice to get more accurate price data - what we currently have uses transaction prices, but if we had precise historical order-book mid prices that would be useful.
And having more questions that are in common between different platforms would of course be helpful as well. There will be a large batch of these at the end of the year, so I will plan to add them to the analysis then.
Any other thoughts or feedback?
I will have a block of time to work more on forecasting-related projects this fall, so I expect to be able to complete this analysis by then and possibly work on additional follow-up investigations.
Thanks to Michael Wheatley who worked with me on this project and did most of the data collection work.