The Guardian has printed an excerpt of economics Nobelist Daniel Kahneman's new book, Thinking, Fast and Slow. The story addresses the question of why investment bankers believe they are doing anything useful when statistics show they average worse than the market. Kahneman draws upon his own experience as a psychologist in the Israeli Army, predicting leadership:
We were willing to make that admission because, despite our definite impressions about individual candidates, we knew with certainty that our forecasts were largely useless. The evidence was overwhelming. Every few months we had a feedback session in which we learned how the cadets were doing at the officer training school and could compare our assessments against the opinions of commanders who had been monitoring them for some time. The story was always the same: our ability to predict performance at the school was negligible. Our forecasts were not much better than blind guesses.
I think the army story is much more interesting than what he describes about his later interactions with investment houses. Using methods developed by the British army, Kahneman's outfit attempted to evaluate leadership potential in candidate officers based on their interactions on a physical task that required teamwork. They believed their assessments to be reliable and based on real data, and yet they did very poorly compared to the officers who ultimately trained the men.
Predicting leadership seems much more relevant to our evolutionary history than predicting investment returns. In many ways, leadership is a much less predictable game than investment. Kahneman describes the interactions among men who may not have previously worked together. But in the absence of reputation and repeated interactions, some effective strategies for attaining and defending status as a leader simply won't work. More aggressive leadership strategies may work in the short term, while judgment and fairness become important among people who know each other well.
The usual argument for why we don't predict marginal probabilities well is that we haven't encountered them in nature. But here's a task it seems we ought to be pretty good at. And maybe we aren't. Worth further exploration.