Abstract:

Simple aggregation of individual judgements has shown impressive success in a variety of tasks, ranging from political forecasting to the location of a disappeared submarine. On the other hand, such aggregation can be contaminated by group-think and may ignore the expert knowledge of a specialized minority. Prelec et al. (2017) propose the “Surprisingly Popular Algorithm” (SPA) as a remedy. In addition to letting respondents choose between one of two options, the SPA asks respondents to predict the answers of others and subsequently picks the answer which is more commonly chosen than on average predicted. Empirically, the SPA yields better results than majority voting in six different areas of general knowledge and estimation tasks. Generalizing the underlying model of Prelec et al. (2017), I show that when the respondent pool is small, the SPA may sometimes perform worse than majority voting and that we can in theory improve upon both methods by asking respondents directly for the appropriate threshold for each answer.
I test both SPA and this newly derived “threshold method” in an experiment. Compatible with the idea that respondents act as approximate Bayesians, I find that both SPA and threshold method outperform majority voting and that average responses approximate correct conditional expectations well. Thus, asking respondents for assessments more complex than their individual judgement can improve the performance of simple aggregation. However, individual responses to the threshold question are markedly similar to those to the prediction question and as a result the threshold method does not outperform the SPA.