Authored by Aaron Shaw, John Horton, Daniel Chen

Download PDF

Summary

This paper presents the results of an experiment comparing the effects of social, financial, and hybrid incentive schemes on worker performance in an online labor market (Amazon's Mechanical Turk). Overall, workers performed better than chance, but performance varied considerably depending on task difficulty. With regard to the incentives, we find that asking workers to prospectively think about the responses of their peers -- when combined with financial incentives -- produced more accurate performance than a control condition. In other words, workers performed best when their payment was tied to their ability to anticipate the answers their fellow workers would give to the same questions. Notably, workers in India and workers with lower web-browsing skills performed worse than their peers irrespective of the treatment condition. We attribute these variations to a combination of the behavioral and institutional factors.

Acknowledgments

This project benefitted from the participation of a number of people at the Berkman Center. In particular, Professor Yochai Benkler provided early guidance and support throughout. Jason Callina and Anita Patel designed the Not-a-Number web application that handled the data collection for the study. Eszter Hargittai provided several survey questions on Web skills and permission to use them. The authors also thank the Ford Foundation, the Ewing Marion Kauffman Foundation, and the Law Lab for support.

Last updated March 21, 2011

Join the Community

Follow us on Twitter, sign up for our mailing lists, contact us, explore our courses, or check out current job openings.

Collaborators