Research On Incentive Mechanism In Mobile Crowdsensing Based On Reinforcement Learning

Mobile crowdsensing is a newly-emerging sensing paradigm in recent years.It uses smart devices carried by ordinary users to collect data and completes large-scale and complex sensing tasks through their collaboration.Compared with the traditional sensing paradigm,mobile crowdsensing has many advantages such as universality,in-telligence,low cost,etc.Thus,it has broad application prospects and important research significance.Data quality and user incentives are two important issues in mobile crowd-sensing.High data quality can improve the total revenue of sensing system,and a rea-sonable incentive mechanism can attract more users to participate in mobile crowdsens-ing.Most of the current research on the incentive mechanisms in mobile crowdsensing assumes that user's sensing quality is known in advance.However,the sensing quality is often unknown in practice,which makes it difficult for the system platform to select users who can provide high-quality data.Therefore,it is essential to study the incentive mechanism design for mobile crowdsensing under the case where users' sensing quality is unknown a priori.Based on the above background,we combine quality-learning into incentive mech-anism design,and study the reinforcement learning based incentive mechanism in mo-bile crowdsensing.On the one hand,we use reinforcement learning to learn users'sensing quality.On the other hand,we design reasonable incentive mechanism to at-tract users to participate in mobile crowdsensing.For different application scenarios,we propose two different incentive mechanisms based on reinforcement learning,and verify the significant performance through substantial theoretical analysis and experi-mental simulations.Our major contributions are summarized as follows:1.We propose a multi-armed bandit mechanism based on UCB(i.e,Upper Confi-dence Bound)and auction.We adopt reverse auction to model the whole data collection process and adopt the multi-armed bandit in reinforcement learning to model the unknown user selection problem.We design a UCB-based user se-lection strategy and adopt the critical value payment method to calculate remu-neration.In addition,in order to incentivize users to provide high-quality data,we correlate users' remuneration with their sensing quality.Through substantial theoretical analysis and experimental simulations,we prove that our mechanism satisfies truthfulness and individual rationality in each round,and analyze the computational efficiency and regret bound of the mechanism.2.Considering large-scale data collection and user's task coverage,we propose an exploration-separated combinatorial multi-armed bandit mechanism.We model the user selection process as a combinatorial multi-armed bandit problem and se-lect multiple users in each round.In addition,in order to ensure truthfulness in the whole process,we separate the learning process as the exploration and ex-ploitation phases and then design different user selection strategies and payment calculation methods for the two phases.Through rigorous theoretical analysis and experimental simulations on real-world data traces,we verify the truthfulness,in-dividual rationality and computational efficiency of the mechanism.Moreover,we analyze the worst regret bound and derive an approximately optimal budget allocation for the exploration and exploitation phases.