On Monday's Econtalk, Taleb said that knowledge was harmful. Here's my take.

Formalisms are born in academia, where they establish boundaries within which solutions are sought. Those boundaries make it possible for ideas to engage, by directing people to compete on the same problems. Boundaries are necessary for focus, they make it easier to share and hybridize ideas. The real game (of life) is broken up into sub-games which academics play in for status and respect.

The sub-games of academia have independent  intellectual histories and traditions. Within each sub-game, there is enough competition that solutions are dependable and valid. However, to actually use an academic result in the real world, one has to make sure the sub-game itself is set up correctly. Sub-game incorrectness takes longer to catch - paradigms have inertia. Assumptions are often accepted by players to make the competition a good game with clear winners, not so much to reflect reality.

Alas, the academic quantification of risk in the finance sub-game has falsely enthralled us all, when in fact it is a weak shadow of real world uncertainty.

In decision theory, risk adversity is a terminal goal

In decision theory, risk adversity is represented by concavity in the utility function. Concave functions bend downwards, so a straight line drawn between any two points would lie below the function line, and averaging any two function values by any weight would have a smaller value than the function at the averaged position.

Px*f(x)+Py*f(y) < f(Px*x+Py*y)

For a given average outcome, the expected utility is higher if there are fewer possible outcomes. Maximizing expected utility then makes the agent prefer a smaller variance in outcomes. This is how decision theory generates risk adverse behavior.

Presented in this way, the desire to reduce variance is fundamental, an ends in itself, and not subject to justification.

Risk adversity seems to be instrumental in reality

When I consider a risky choice, however, I don't undergo anything like the process of utility maximization. If I am highly uncertain, I may

1. delay the decision and acquire more information
2. avoid the choice to avoid complicating the context of future decisions
3. avoid the choice because of the fear that uncertainty makes me susceptible to fraud, since other agents have freedom in the uncertain space.

These reasons all point to uncertainty as an indicator of incomplete information, and as a cue to spend more on information or a chance to avoid future information cost, something a priori excluded if you assume information costs to be zero, as does canonical economic theory.

Presented in this way, the desire to reduce uncertainty is instrumental, a means of bettering the expected outcome, not a goal in itself.

Conclusion

Risk adversity's relationship to the management of ignorance cannot be denied. Characterizing risk adversity as the maximization of concave utility functions misses this point. Exogeny is useful for compartmentalizing and communicating theories, but in making ignorance exogenous you restrict an agent from choosing to learn more or to avoid the cost of future learning. In the presence of this restriction, the model will not display a realistic reaction to risk.

Related

• http://en.wikipedia.org/wiki/Search_theory
  

Note: I use the words "value" and "goal" interchangeably in this post.

Terminal values are pursued for their own sake. Instrumental values are pursued as a means to attaining terminal values, or other instrumental values closer to the terminal values.

For example, if I had the ultimate goal of maximizing profit, I might break that up into the instrumental goals of increasing revenue and decreasing cost. Depending on the situation, those two instrumental goals can be broken up into even more specific instrumental sub-goals.

Instrumental sub-goals help to reduce repetition in thought. Conditions change in such a way as to change the actions demanded by an instrumental sub-goal, but not change the validity of the way in which the terminal goal is divided into sub-goals.

Irrationality is often caused by holding on to instrumental sub-goals even though they no longer serve your terminal goal. This is what Eliezer calls thought caching.