@RyanKidd Thank you for the contribution and the kind words! I agree AI forecasting is very important and is therefore one of the primary topic areas I intend to be featured on the site. I also think that the most important questions in that area will be long-horizon and future accuracy may not be reflected by past performance, but I'm sure there is still plenty to learn.

@lucas I just shot you a DM on discord, but email works as well!

This seems like a technically interesting solution but extremely niche application. It will be difficult to get people to trust you with their anonymity for the most beneficial situations and for less privacy-critical situations many groups will default to the simpler solution of “text X person if you’re interested”. There might be some growth potential if the site can become a “platform” like Kickstarter or GoFundMe, but I’m not sure how that would work with your target applications.

What does success look like for you? Is there a path forward where this product exists but without a full-time support team?

Nice to see this project is still going, I haven’t heard much about it recently.

Some questions:

• Does https://brierfoxforecast.com/ represent the current state of development? Are there any other currently-active instances?

• What’s the advantage to running an independent instance? Why would a small group go through the trouble to run an instance instead of using an already-established server?

• Do you have any changes to market structure or feature set planned to differentiate you from competitors? Are you implementing the same AMM used by Manifold?

• If the application is intended to be run locally, have you tested the resource usage for a typical instance? Could it run from, say, a laptop or home computer? What size is the DigitalOcean server used for the demo instance?

An interesting set of ideas! Some questions:

• I believe Manifold is getting rid of the managram system as part of the prize point pivot, so unfortunately I don’t think managram subscriptions will be very useful.

• For an automated arbitrage bot I think it would probably be better to submit this separately from the comments section to avoid revealing your linked positions. If someone had that information they could take advantage of it by e.g. forcing you to trade into their limit orders. **If this was in your web app you could overlay the two target markets and keep a list of active arbitrages, or even arbitrage against a different site’s market.

• The secret market idea is interesting, but if I wanted to do that right now I’d just make one called “Secret Market” and communicate the true purpose of the market to whoever I wanted to trade with separately, through direct messages or a private google doc. What advantage would this extension have?

• The manifoldmaven account is neat! I would love to see something like that with a little more signal:noise. The latest 3 posts are all about the RFK news, and the same Trump twitter market is on there three times. There’s also a few that are only caused by transient spikes which were quickly reverted. I think there’s also some low-hanging fruit to finding the root cause of a swing, usually a news article posted recently in the market.

  • Additionally, I wanted to see what the “Get these alerts in your inbox” link would do, but it just takes me to a place to pay $20/mo. If I were to pay this would I get all of these alerts in my email inbox? SMS inbox? Would I be able to customize the swing percentage or timeframe? More information would be good!

  • Unfortunately I don’t think I would pay any amount of money for this information, since learning about news instantly is nice but has basically no value to me over learning about the news whenever I get around to reading it.

  • Reading further down I think the Predictor's Digest is much closer to what I would want out of this, a full newsletter linked to prediction markets. I’m unsure of the AI aspect but would be interested to see.

• I think loan infrastructure for Manifold would be very nice and has lots of potential, but with mana exchange going away I think loans and tips will be no more.

• For the personal prediction tracker, how would this compare to something like Fatebook? I have used Fatebook in the past and it has a very nice interface for this sort of thing.

This seems like an extremely beneficial service that is sorely needed by the community!

• Could you provide an example of content you have produced for an EA org in the past?

• Approximately how many additional projects would you be able to take on if this proposal is completely funded?

• What types of social science research are you expecting to fund with this?

• Why should we award this proposal rather than finding those researchers directly?

• What service do you use to find participants and ensure they are valid participants?

I am unfamiliar with both fields involved here (neuroimaging and cryptography) so I had to do some digging to understand this proposal. Here’s my understanding of the situation and some basic questions, let me know if I’m misunderstanding:

• The problem is that researchers aren’t sharing anonymized brain imaging data. This is already legal but rarely done in practice due to privacy concerns.

  • What types of advancements are lacking good brain scan data? What sorts of research is this data good for?

• The solution is to apply a special form of encryption to the brain scan data. When the data is encrypted this way it allows researchers to do some stuff with the data but not other stuff.

  • What exactly can you do with an encrypted scan vs a non-encrypted scan? If it’s already anonymized, what danger exists in a non-encrypted scan that’s removed when it’s encrypted?

• There are a lot of ways to do this special encryption. The first part of this project is to take some existing brain scans and try each way of encrypting them, hopefully finding one that works but is still easy to use for researchers.

  • What is your list of HE implementations? What criteria do they need to pass in order to be useful? What sort of performance impact do you expect from this intermediate processing step?

• The last part of this project is to take a live brain scan and encrypt it in real-time so it’s never unencrypted.

  • What benefit does this have? Shouldn’t original researcher should still have full access to the original scan?

Wow, I love what you’ve done so far. As someone who’s gone through the same process, your blog post on serverless ETL was very interesting!

• I think the new data platform screenshots are great. Being able to successfully match markets to trending news stories is nothing short of magical. If you can connect equivalent markets across platforms then I’d likely just start using your API instead of building my own!

• I think the RSS reader idea is nice, but I don’t see myself moving away from FreshRSS anytime soon. I am quite interested in a newsletter however!

What progress have you made since your last update?

Our biggest updates to Calibration City this month have been the addition of a few new tabs!

🤔 First up we have the introduction, a sorely-needed explainer for this whole “calibration” thing. This is mainly formatted as a dialog-based blog post where we explain the basics of quantified predictions, calibration, accuracy, prediction markets, and the site’s purpose. I hope it is able to bring the ideas behind prediction markets to a few new eyes, directly alongside the statistics to (potentially) back it up.

🥇 Up next we have the accuracy tab, with a set of graphs I teased last month. Ever wondered if longer markets are more accurate? What about markets with more traders? Now you can compare all markets across every supported platform against any attribute we measure - plus you can combine it with any of the other standard filters! Do sports markets on a specific platform get more accurate the closer to the end of the market? Are markets that resolve yes more accurate than those that resolve no? Experiment to your heart’s content!

🔍 Wait, what markets are in that bin of sports markets on that one platform that resolved no? Well to find out you can go search on that site… or you can use the new list page! List, sort, filter, and browse the markets to your heart’s content. (The API endpoint that powers this page can also be used to download all of the markets in my database, in case you’re interested in double-checking my math or rolling your own 😉)

👨‍🎓 Interested in learning more about the site? Head on over to the FAQ tab! It’ll answer your question… as long as your question is “where can I learn more about scoring prediction markets” or “give me more nitty-gritty implementation details”. I’ve been trying to include responses to actual frequently-asked questions on the relevant page with better wording or hover-text, so nothing has actually made it to this page yet 😅 Feel free to give suggestions for what I should add, though!

What are your next steps?

🏆 More scoring options! We can already calculate and show the Brier score for every market, but there are a lot more scoring methods! I plan to add logarithmic scores, spherical scores, and more.

🖇️ Even more platforms! After adding Polymarket (and then working with their dev team to get even more information) I paused adding new platforms to the site in order to get the user experience the way I wanted. Now, I think we’re in a great state and we can get moving on integrating even more data!

✅ Corpus of questions! The biggest issue with the site as it stands now is that you’re comparing apples to oranges - not all markets are comparably difficult! I don’t want to punish sites for catering their questions to their communities, and I don’t want to reward sites that might attempt to “game” statistics like calibration or accuracy. I want to build off of existing datasets, leverage tools to replicate questions across platforms, and build a large enough corpus of questions that users can confidently see how accurate different platforms were for key questions or in aggregate.

Is there anything others could help you with?

Absolutely: I’m not a technical writer, so please review what I’ve written! The introduction dialog is a quick summary of how I understand calibration and accuracy, but it could be wrong or misleading in important ways. That’s why I’ve set up this bounty for users to submit their feedback on the site, especially anything that was confusing, unintuitive, or incorrect.

I’m also interested if there is any prior art to grouping a corpus of questions like we will need to compare accuracy across platforms. Aside from doing it manually, we could leverage the markets that MirrorBot has replicated or use LLMs to determine when a market is “close enough” to be identical, but each of those have drawbacks. Since this is going to be a big feature with a high future maintenance burden, I would be happy to hear any ideas that may make it simpler!

As before, I’m always looking for ways to make the site better. If you have any ideas for things that would bring you to the site more often or features you would love to see, I’m all ears.

❤️ Thanks!

@Austin Thanks for the information!

I'm a bit confused about this proposal. Are you looking for retroactive funding as compensation for past site moderation? To my knowledge the community fund is not continuing so I'm not sure how you would expect to gain investors for future work. Are you claiming that you'll stop the vast majority of your moderation if this is not filled? I think some context would be helpful.

One more sneak peek... prototyping alternative Brier score visualizations here and here (answering the question "why does Polymarket's brier score drop so suddenly when you increase the minimum duration?")

Working on categories now... take a peek at the new filter (🤫) at https://beta.calibration.city/

And Polymarket has now been added! 🎉

What progress have you made since your last update?

🎉 Calibration City 2.0 is live! You can find it at https://calibration.city/

We've integrated 2 more platforms - Kalshi and Metaculus - and completely rewritten the backend to be hundreds of times faster! Now you can experiment to you heart's content without waiting ten seconds for each reload.

What are your next steps?

We've got plenty ahead of us, which you can follow along with here at the repo. A few things we're working on:

• 🖇️ More platforms! Three is good, but we want all of them. Polymarket is up next, and we have many more in our sights.

• 📶 Categories! We want to be able to filter down to a category, and see those types of markets across every platform.

• 📜 Market list! "What's that outlier?" Check the list of what markets are in each sample, and which are weighted the highest!

• ❓ Explainers! New users should have an easier way to understand forecasting, prediction markets, and calibration so the site can be more useful to them.

• 🔬 Brier analysis! Now that we have all of this information, calibration plots aren't the only thing we can build. We can plot the Brier scores per platform against time, number of traders, markets volume, and more.

Is there anything others could help you with?

Feedback, comments, and suggestions are always welcome. Tell me what you're using it for and what can be done to make it better!

@Austin Thanks, but I'd rather keep going with Themis and let my prospective investors utilize their money elsewhere rather than lock up their bids an additional few weeks.

@Sinclair If you're talking about the platform icon in the Themis repo I'll get that updated before I deploy anything. If you're talking about the brass foldies I'll use that as a starting point if I manage to get funded in the next 24 hours.

Project update: During the proposal stage I have started working on a rewrite of Calibration City, currently under the name Themis. You can see my progress here: https://github.com/wasabipesto/themis

If this proposal is not funded I plan to continue working on Themis and have high hopes for it. If the proposal is funded I may spend some time on it while also working on whatever project is selected.

@MarcusAbramovitch I don't plan to separate out any of these projects, but if this proposal is funded I will weigh my investors' advice on which to pursue much more heavily. Currently project #3 is ranked second-lowest on the market, so it would take a bit of convincing for me to pursue it over the others.