Multi-objective Bid Optimization For E-commerce Sponsored Search Based On Multi-agent Cooperative Evolutionary Strategy

Online advertising is a marketing strategy utilizing the Internet as a medium to help advertisers attract target consumers and conversions via Real-Time Biding(RTB).Ecommercial sponsored search is a mainstream form of online advertising,where consumers enter user requests and advertisers bid for traffic in real time to achieve subsequent conversion behaviors..In the highly dynamic marketing environment of e-Commerce platforms(such as Taobao,Amazon,Jing Dong),there are typically millions of advertisers,billions of Page-Views and hundreds of billions of auctions.How to optimize the real-time bidding problem in such a large-scale e-commerce environment and meet the diversified target demands of advertisers has become a more challenging problem.Bid optimization for online advertising from single advertiser's perspective has been thoroughly investigated in both academic research and industrial practice.However,existing work typically assume competitors do not change their bids,i.e.,the wining price is fixed,leading to poor performance of the derived solution.Although a few studies use multi-agent reinforcement learning to set up a cooperative game,they still suffer the following drawbacks:(1)They fail to avoid collusion solutions where all the advertisers involved in an auction collude to bid an extremely low price on purpose.(2)Previous works cannot well handle the underlying complex bidding environment,leading to poor model convergence.This problem could be amplified when handling multiple objectives of advertisers which are practical demands but not considered by previous work.In this paper,we propose a novel multi-objective cooperative bid optimization formulation called Multi-Agent Cooperative Evolutionary Strategy,(MCES).MCES sets up a carefully designed multi-objective optimization framework where different objectives of advertisers are incorporated.A global objective to maximize the overall profit of all advertisements is added in order to encourage better cooperation and also to protect self-bidding advertisers.To avoid collusion,we also introduce an extra platform revenue constraint.We analyze the optimal functional form of the bidding formula theoretically and design a policy network accordingly to generate auction-level bids.Then we design an efficient multi-agent evolutionary strategy for model optimization.Evolutionary strategy does not need to model the underlying environment explicitly and is more suitable for bid optimization.Offline experiments and online A/B tests conducted on the Taobao platform indicate both single advertiser's objective and global profit have been significantly improved compared to state-of-art methods.And the model has been fully deployed to Taobao's E-commerce platform to serve millions of advertisers in real-time bidding optimization,and achieve an effect improvement of more than 5% for platform turnover and advertisers' multiple objectes.