I had Cass R. Sunstein’s book, Infotopia: How Many Minds Produce Knowledge on my wish-list at Amazon for some time before finally purchasing it. I wish I had done so sooner: it sheds valuable light on key aspects of knowledge creation, decision-making and social software.
Sunstein’s raw material is simple: Condorcet’s Jury Theorem ; some studies of deliberative groups; Eric S. Raymond’s The Cathedral and the Bazaar ; and a few examples of powerful uses of technology (especially wikis and prediction markets). By combining these he creates a book dedicated to answering a few simple questions. Why do random groups tend to produce better decisions than carefully constructed discussions? Why does open source software and thinking tend to be more robust than that produced by closed processes?
The book starts by exploring and expanding on Condorcet’s Jury Theorem, which holds as follows (as described in Wikipedia).
The assumptions of the theorem are that a group wishes to reach a decision by majority vote. One of the two outcomes of the vote is correct, and each voter has an independent probability p of voting for the correct decision. The theorem asks how many voters we should include in the group. The result depends on whether p is greater than or less than 1/2:
- If p is greater than 1/2 (each voter is more likely than not to vote correctly), then adding more voters increases the probability that the majority decision is correct. In the limit, the probability that the majority votes correctly approaches 1 as the number of voters increases.
- On the other hand, if p is less than 1/2 (each voter is more likely than not to vote incorrectly), then adding more voters makes things worse: the optimal jury consists of a single voter.
In doing so, Sunstein puts a useful gloss on the raw mathematics of Condorcet.
[I]t would wrong to suggest that the best approach to hard questions is to ask a large number of people and to take the average answer. That approach is only likely to work under distinctive circumstances: those in which many or most people are more likely than not to be right… In all of these cases, there is reason to trust the people being asked, and hence the average answer is likely to be right.
But it would make no sense to make policy by asking everyone in the world whether the United States should sign the Kyoto Protocol, or whether genetic engineering poses serious risks… In these cases, there is a great risk that error and confusion will be replicated at the level of group averages. (pp42-43)
Crucially for Sunstein, the consulted group needs to have real (even if only fragmented) information about the facts at issue. The further we get from this position — either because the answer sought is not fact-based or because there is an increasing risk of systematic error or misinterpretation — the less likely it is that a Condorcet approach will work.
Sunstein then turns to the deliberative approach. Can groups (of experts or of generalists) reach reliably good conclusions by discussing the options available? Here he gives the game away in advance by the heading of Chapter 2: “The Surprising Failure of Deliberating Groups.” Sunstein’s case is that the process of deliberation actually works against the kinds of outcome that we intuitively think it should promote: accurate decision-making, open sharing of information, and improved quality of discussion, when compared to random groups.
- It cannot be shown that deliberating groups generally arrive at the truth…
- Much of the time, deliberating groups do quite poorly at aggregating the information that their members have…
- Deliberating groups sometimes outperform statistical groups, but sometimes the opposite is the case. (pp57-58)
These problems are, broadly speaking, a consequence of human psychology and group behaviours. In reaching these conclusions, Sunstein examines a wealth of research in those areas. I have a slight reservation about this aspect of the book. Here and elsewhere research studies and reports are mentioned, but I felt the absence of any context. Because this is not a scientific monograph, it is perhaps reasonable for there not to be a comprehensive analysis of all the research on the topics under discussion. However, it is not possible to judge from the material Sunstein uses how much has been selected to support his argument. I am willing to trust him, but I don’t feel particularly comfortable doing so — particularly in view of the fact that the book depends significantly on exposing and challenging received opinion.
Having laid the theoretical foundations, Sunstein moves on to practical matters. He focuses on two areas: prediction markets and social and open source software. These chapters are fascinating. In his treatment of prediction markets, Sunstein revels in showing how markets in information match Hayek’s assertions about markets generally. However, he also highlights the problems that can be caused when the purity of the market can affect the quality of information sharing. These markets are as subject to manipulation, bias and bubbles as traditional ones. Within certain parameters, though, Sunstein is confident.
Prediction markets remain in their early stages. … [I]n many domains, they perform extremely well — better than surveys, better than deliberating groups, better than experts. Their promise is most likely to be realized when knowledge is genuinely dispersed, when a wide range of people know relevant facts, and when their incentives lead them to reveal, through investments, what they know. (p144)
There is a curious detour in the middle of this chapter, in which Sunstein looks at the way Hayek’s views on markets relate to the persistence of information over time (rather than space, which is the focus of the rest of the book) and compares them to Edmund Burke’s assertion that long-standing traditions are generally to be preferred, being rooted in the judgments of particular people tested over time. Ultimately, Sunstein is less convinced by Hayek’s proximity to Burke on the question of meritorious persistence.
What Hayek and Burke appear to miss is that traditional practices are often the congealed product of earlier informational or reputational cascades. Often morality consists of inefficient or unjust practices defended by entrenched groups and factions. …
In many cases, traditions last not because they are excellent, but because influential people are averse to change and because of the sheer burdens of transition to a better state. It is for this reason that celebrations of traditions have always met an ambivalent reaction in free countries. (p125)
It was particularly striking to me that Sunstein invoked Burke at this stage, as he had earlier ignored the Englishman’s dictum on representative democracy in preference to the identical view expressed by Roger Sherman in the first US Congress:
I think, when the people have chosen a representative, it is his duty to meet others from the different parts of the Union, and consult, and agree with them on such acts as are for the general benefit of the whole community. If they were to be guided by instructions, there would be no use in deliberation.
Sunstein has previously relied on Sherman’s words in his works The Partial Constitution (1993: 22) and Republic.com (2002: 41), so we should give him some credit for recycling.
When Sunstein turns to examine technology, he treads much more familiar ground. His analysis owes much to Eric Raymond’s, topped up with lessons learned from Wikipedia, Intellipedia and similar efforts, and a critique of blogs as mechanisms for exaggerating apparent consensus. There is a long-standing tradition of approval of open source and similar processes coupled with denigration of proprietary and closed development. (I even contributed to the tradition myself in a critique of competing Internet standards processes.) Sunstein adds little here, beyond publicising the existing views and, much more importantly, linking them to his more abstract material. This really makes a difference, but I am still unsure why people deliberating together in groups do a poorer job of reaching the right conclusion than those contributing openly to a wiki, to open source software development or to an Internet standards process. What is different? Is it the quasi-anonymity offered by the latter processes? Or is it a consequence of politicisation (in the most general sense) of the deliberative process. Sunstein doesn’t even pose these questions, let alone offer an answer.
What he does do, in his final chapter, is to start to offer solutions for the flawed deliberative process. He recognises that, rightly or wrongly, this tradition is not going away. Instead he suggests ways in which the improved effectiveness of some of the other approaches in the book can be introduced into deliberating groups. Unfortunately, this chapter is tantalisingly short. I would have appreciated more here — Sunstein’s solutions are just too sketchy.
Infotopia provides a valuable and informative summary of some important concepts. Its measured and thoughtful tone stands out amongst works of a more evangelical nature that cover similar ground, and is therefore highly recommended. There are gaps, but perhaps in time those will be filled by others.
Mark Gould... 