Design And Implementation Of A Crowdsourcing Annotation System

The new era of Big Data has brought the whole Machine Learning community, espe-cially for the researchers of Deep Learning; opportunities as well as challenges. More and more massive data means that we need more manpower and resources for data annotation. Instead of full-time employees, Crowdsourcing assigns the annotation tasks to a large group of Internet users. This will greatly reduce the annotation cost, and its scalable feature also makes the annotation of massive data possible. Crowdsourcing for its unique advantages at-tracts a great deal of interest in the research community. Researchers from around the world designed a lot of Crowdsourcing platforms. However, existing Crowdsourcing platforms are, either targeting a specific application scenarios, or a closed platform which is difficult to extend.To solve this problem, we designed and implemented a generic Crowdsourcing system, which could act as a annotation platform and also be used by researcher for implementing and experimenting all kinds of Crowdsourcing related algorithms. For system design, the modular design improves the system’s generality, making our system being able to support a variety of Crowdsourcing tasks. Focusing on the "Image Classification" task, we designed a unique annotation interaction interface, and conducted a series of experiments to verify the advantage of our design in term of reducing annotation time and cost. Furthermore, in order to solve the problem of the Crowdsourcing annotation quality, we implement a number of algorithms to control and improve the quality of Crowdsourcing annotation.