From labor to trader: Opinion elicitation via online crowds as a market

We often care about people's degrees of belief about certain events: e.g. causality between an action and the outcomes, odds distribution among the outcome of a horse race and so on. It is well recognized that the best form to elicit opinion from human is probability distribution instead of simple voting, because the form of distribution retains the delicate information that an opinion expresses. In the past, opinion elicitation has relied on experts, who are expensive and not always available. More recently, crowdsourcing has gained prominence as an inexpensive way to get a great deal of human input. However, traditional crowdsourcing has primarily focused on issuing very simple (e.g. binary decision) tasks to the crowd. In this paper, we study how to use crowds for Opinion Elicitation. There are three major challenges to eliciting opinion information in the form of probability distributions: how to measure the quality of distribution; how to aggregate the distributions; and, how to strategically implement such a system. To address these challenges, we design and implement COPE Crowd-powered OPinion Elicitation market. COPE models crowdsourced work as a trading market, where the "workers" behave like "traders" to maximize their profit by presenting their opinion. Among the innovative features in this system, we design COPE updating to combine the multiple elicited distributions following a Bayesian scheme. Also to provide more flexibility while running COPE, we propose a series of efficient algorithms and a slope based strategy to manage the ending condition of COPE. We then demonstrate the implementation of COPE and report experimental results running on real commercial platform to demonstrate the practical value of this system.