My PhD dissertation was about bias in cost and ridership forecasts for transit projects. Before getting into any data analysis, I address the question of how we should even be evaluating forecasts in the first place. One response to evidence that forecasts for transit projects have generally proven to be overwhelmingly biased has been an argument that forecast accuracy is unimportant, or less important than other considerations. And it\u2019s true that accuracy isn\u2019t the only possible way to evaluate a forecast.<\/p>\n
A 1993 essay<\/a> on weather forecasting by Allan Murphy (which I came across by way of Nate Silver\u2019s book The Signal and the Noise<\/em>) defines forecast \u201cgoodness\u201d in terms of three characteristics:<\/p>\n (1) Consistency<\/em>:<\/strong> Is the published forecast consistent with the forecaster\u2019s best judgment? Does the forecaster actually believe her own forecast?<\/p>\n (2) Quality<\/em>:<\/strong> Does the forecast correspond with what actually occurred? Was it proven to be accurate?<\/p>\n (3) Value<\/em>:<\/strong> Is the forecast useful to forecast users? Does it help them to make the best decisions?<\/p>\n Individual forecasts might be good in one or more of these ways, without being good in all three. For example, a financial forecaster might try to defraud investors by intentionally inflating her firm\u2019s earnings forecast, but unexpected events occur later that end up making the inflated forecast accurate (thus, the forecast has good quality<\/em>, but poor consistency<\/em>). A weather forecaster might intentionally overstate the seriousness of a storm (poor consistency<\/em>) because she\u2019s knows that people would otherwise under-prepare. Although the storm turns out to be less serious than her published prediction (poor quality<\/em>), lives were saved because people were more prepared than they otherwise would have been (good value<\/em>). An economic forecast might be really rigorous and accurate, but so vague and presented with so much technical jargon that it\u2019s ultimately useless to the lay forecast user (good quality<\/em>; poor value<\/em>).<\/p>\n In evaluating all of a particular forecaster\u2019s forecasts over time (rather than just one individual forecast), Murphy proposes that the best way to maximize all three types of goodness is to maximize forecast quality<\/em>. Consistently accurate forecasts will increase the forecaster\u2019s confidence in good methodologies (improving consistency<\/em>), and the value<\/em>\u00a0of a set of forecasts increases with forecast credibility, which correlates with the accuracy of past forecasts.<\/p>\n Murphy goes on to argue that another reason to emphasize forecast quality<\/em>\u00a0over other types of goodness is that it\u2019s often the only of the three types of forecast goodness that can be observed at all, since we\u2019re generally no better at reading minds than we are at predicting the future. We don\u2019t know what\u2019s going on in the mind of the forecaster, so it\u2019s hard to really judge how well a published forecast corresponds to the forecaster\u2019s best judgment. Likewise, since we can\u2019t read the minds of forecast users, we can\u2019t really say how the forecast has influenced their decision-making. So we\u2019re left with accuracy.<\/p>\n I think these three types of goodness are a useful way to think about what we might mean when we say, \u201cThe Church is True.\u201d We hear and say this a lot in our church, and it strikes me that different people mean and understand different things by it. Applying Murphy\u2019s three types of goodness, we might be referring to:<\/p>\n (1) Consistency<\/em>:<\/strong> Current and past church leaders and teachers at various levels are sincere in their teachings and truth claims.<\/p>\n (2) Quality<\/em>:<\/strong> The teachings of the church [fn] accurately reflect reality.<\/p>\n (3) Value<\/em>:<\/strong> Decisions that follow church teachings can improve a person\u2019s life.<\/p>\n Thinking about which of these three types of goodness should get the greatest emphasis when we\u2019re talking about the truthfulness of the church is a little different than when we\u2019re evaluating forecasts. The quality <\/em>(accuracy) of most forecasts can be empirically observed, but most church teachings (the nature of God, for instance) don\u2019t lend themselves well to that kind of direct confirmation. We have the same problem with evaluating consistency<\/em> as we do in the case of forecast evaluation. No one but Joseph Smith (or President Monson) can really know how sincere he was (or is) in his belief in his prophetic calling.<\/p>\n Speaking generally about all people who encounter church teachings, we might run into the same problem with evaluating the value<\/em>\u00a0of church teachings that we do with the value<\/em> of a forecast. We don\u2019t necessarily know how people are using them. However, we can<\/em> evaluate the value of church teachings on a personal, individual level. For you personally, does the Church work the way you need it to? Are the doctrines helpful to you?<\/p>\n I think scripture verses like Matthew 7:16-20<\/a>, John 7:17<\/a>, and Alma 32:27<\/a> are an argument that \u2014similar to how a set of forecasts with good\u00a0quality<\/em> is most likely to also have good consistency<\/em> and good value<\/em>\u2014 a set of doctrines that you\u2019ve found to have good value<\/em> is likely to also have good consistency<\/em> and good quality<\/em>. This comes pretty close to how I think about my own testimony and experience with the church. Not that the accuracy of the church\u2019s claims or the sincerity of church leaders are unimportant, but that my own personal experiences with church doctrine and church participation are (first of all) also important, and (second) the closest I can come to any kind of evidence of the other two types of goodness.<\/p>\n [fn] There could be a related, but entirely separate discussion about what \u201cthe teachings of the church\u201d include or don\u2019t include. I\u2019m choosing not to get into that here.<\/p>\n","protected":false},"excerpt":{"rendered":" My PhD dissertation was about bias in cost and ridership forecasts for transit projects. Before getting into any data analysis, I address the question of how we should even be evaluating forecasts in the first place. One response to evidence that forecasts for transit projects have generally proven to be overwhelmingly biased has been an argument that forecast accuracy is unimportant, or less important than other considerations. And it\u2019s true that accuracy isn\u2019t the only possible way to evaluate a forecast. A 1993 essay on weather forecasting by Allan Murphy (which I came across by way of Nate Silver\u2019s book The Signal and the Noise) defines forecast \u201cgoodness\u201d in terms of three characteristics: (1) Consistency: Is the published forecast consistent with the forecaster\u2019s best judgment? Does the forecaster actually believe her own forecast? (2) Quality: Does the forecast correspond with what actually occurred? Was it proven to be accurate? (3) Value: Is the forecast useful to forecast users? Does it help them to make the best decisions? Individual forecasts might be good in one or more of these ways, without being good in all three. For example, a financial forecaster might try to defraud investors by intentionally inflating her firm\u2019s […]<\/p>\n","protected":false},"author":10392,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1058],"tags":[],"class_list":["post-36832","post","type-post","status-publish","format-standard","hentry","category-guest-bloggers"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/posts\/36832","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/users\/10392"}],"replies":[{"embeddable":true,"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/comments?post=36832"}],"version-history":[{"count":3,"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/posts\/36832\/revisions"}],"predecessor-version":[{"id":36838,"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/posts\/36832\/revisions\/36838"}],"wp:attachment":[{"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/media?parent=36832"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/categories?post=36832"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/timesandseasons.org\/index.php\/wp-json\/wp\/v2\/tags?post=36832"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}