EA - [Cause Exploration Prizes] Training experts to be forecasters by Sam Abbott

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Cause Exploration Prizes] Training experts to be forecasters, published by Sam Abbott on August 26, 2022 on The Effective Altruism Forum. This essay was submitted to Open Philanthropy's Cause Exploration Prizes contest. If you're seeing this in summer 2022, we'll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag! Summary Improving decision-making through forecasting requires both knowledge of how to forecast, and domain expertise to develop the forecast question, provide guidance for sources of evidence and their synthesis, and help facilitate the use of the results of the forecasting process. We argue that one, currently neglected, strategy of making forecasting more useful is to focus on making domain experts better forecasters. The current dominant strategy to make forecasting useful relies on the use of aggregate forecasts from a pool of self-selecting forecasters, providing feedback and incentives, identifying high-performing forecasters from this pool, and focussing on forecasts from this subset. We refer to this approach as forecast consulting and suggest the model is similar to other forms of consultancy. Significant resources are being spent on training and identifying high-performing forecasters with some of this focus being to give these forecasters the time, space, and training needed to gain relevant domain knowledge. Currently little is done to encourage existing domain experts to become better forecasters themselves or to identify those with innate potential. We believe that encouraging and helping experts to become better forecasters has potential benefits that far exceed what is feasible through the current dominant strategy of employing generalist forecasters as consultants alone. Context on forecasting There is some consensus in the EA community that forecasting, as commonly practised in the community, is important and helpful. Forecasting is “big if true”: In principle, it is an excellent way to summarise existing knowledge. It can help guide and improve relevant and important decisions by making implicit intuitions and assessments explicit, and by revealing uncertainty and variation in opinions. Forecasting can be understood as a thought process, that can unmask blind spots, help make disagreements clear, and aid in synthesising evidence. For many, forecasts are also a form of “news source” that helps them interpret and contextualise current events. In the following, we focus on forecasting that is explicitly used to inform decision-making. In this context, forecasting can provide value if We can select a useful forecast target. The forecasters can forecast the target quantity meaningfully. This means both that it is possible to forecast the target, and that the forecasters have the required expertise to understand the target and make a forecast. The forecasts are “good” or “good enough” to inform the decision being made. If practised by decision-makers the exercise of forecasting, rather than the output of any given forecast, may be beneficial by helping to structure and formalise, synthesise evidence, calibrate expectations, and foster useful debate. Relevant people trust good forecasts and use them as part of their decision-making process. In the following, we argue that we can achieve our goal of improving decision-making through forecasting if more focus is given to increasing the use of forecasting by domain experts and improving their forecasting practice. Note: we’re using the term “expert” very broadly here. An expert could be an academic, but could also be a policy maker or someone with personal experience and deep knowledge of the subject. The current relationship between forecasting and domain expertise Today, the overlap between forecasters, domain experts...