Reduction Of Unknown Unknowns In Optimization Of Deep Learning Image Caption Models With Crowdsourcing

With the approaching of Artificial Intelligence,machine is taking the place of more and more labor duplications in fields such as Facial Recognition,Automatic Speech Conversion and so on.Deep Neural Networks and Deep Learning Algorithms make good use of the complex inner model structure for automatic combination of low-order features into high dimensional characteristics,which easily acquire not only much better outcomes,but also the problems of difficult in parameter adjustment and fine-tuning,low interpretation in high dimensional traits.On the other hand,the optimization of deep learning models also depends a lot on the quality of training data.Unknown Unknowns(UUs)is a kind of serious problem caused by distribution bias of training data,which is hard to find with the help of evaluation metrics of models.It is easy to be neglected in the process of optimization and may leads to unforeseen identification errors and be obstacles for these models to be used in reality.Considering that UUs are difficult to be found in tons of data,Human-In-The-Loop plays an important role in the work flow of deep learning model optimizations,for that human are always much sensitive on images and audio data comparing to machines.With the assistances of Crowdsourcing,which is one of the mainstream method for targeted data collection and cleansing.Those original data combining with human decision making and annotations relying on their commonsense and domain knowledge is then used for training and tuning of initial models.Present ways of UUs identification mainly depends on asking crowdsourcing workers to label all the training datasets,or data fragments divided by feature engineering.However,those decisions on data division are always difficult to make and there may always be lots of redundant annotations which calls for extra cost.Also,deep learning models,such as image caption generation and machine translation,tend to use great amount of data in each iterations.The cost of annotation acquiring rises quickly with the number of targeted data.How to collect as much data with high quality as possible becomes an important problem to be solved.Our work focus on this series of problems,aiming at solving the task of eliminating UUs found over deep leaning models in the field of image captions with guidance from crowdsourcing for true labels.Quality control and other related aspects are also investigated in detail.Our main works are as follows:(1)We propose a complete work flow for training data adjustment which not only reduce the number of UUs,but also improve the performance of the model on generated captions.This work flow include three main steps: collect synonyms of certain key words of UUs,use these words to acquire similar images of targeted scene,ask crowdsourcing works for help on image filtering and labeling.The external data in format is then sent as additional training data for UUs elimination and model optimizations.We prove that this work flow is capable of reducing UUs and can improve the accuracy of wrongly captioned images with target UUs by over 10%.Other valuation metrics are not influenced by these additional data,some of them even rose slightly.(2)Considering the cost and quality problem during crowdsourcing,we blend "Gamification" into crowdsourcing tasks for task wrapping,so as to implicitly collect our targeted data from workers with no rewards' cost.Experiments show that annotation tasks which calls for only common sense can be done with no rewards at all.Setting of certain parameters in the gamified tasks may even promote the working efficiency of "human computation".