An Automated Negotiation Model Based On Market-driven Agent And Learning Mechanism

With the development of computer technology and network technology, and the wildlyuse of the Internet in particular, agent automated negotiation technology has gained increasinginterest internationally. This study focused on the agent automated negotiation protocol andstrategy in multilateral multi-issue negotiation of E-commerce environment. E-commerceagent uses network negotiation to reach an agreement in order to transactions of goods,services and other resources.Multilateral multi-issue negotiation is a very complex issue of agent automatednegotiation, with the development and popularization of Internet, electronic transactions areno longer restricted by region, the commodity resources, business services, trading partners,logistics, distributions and other trading resources are extremely rich, and the tradingopportunities are increased, so these makes the electronic marketplace shows more opennessand volatility. The key to E-commerce consultation is that the buyers and sellers how to takefull advantage of the rapidly changing electronics market supply and demand information toestablish a feasible and effective negotiation strategy, to balance supply and demand, and tomaximize individual interests. Market-driven agent model is an adaptive negotiation strategyfor changing market trading environment. In negotiation process, considering tradingopportunities, competitive pressures, the consultation time limitation and the enthusiasm ofthe consultation results, agent describes the market consultative environment, decides thenegotiation strategy, carries out the concession reasonably in the dynamic process ofelectronic transactions, and reduces the transaction differences to reach an agreement.In order to protect their own interests and securities, the individual generally encrypts theindividual preferences, the reserve price of the consultation, the effective evaluation function,and the negotiation strategies. In addition, the negotiation environment is complex and varied,the knowledge of agent is incomplete, ambiguous and even contradictory, and the agent oftenrestricts by the negotiation time, so it is difficult to pre-determine a set of effective negotiationstrategy. In multi-agent system, agent has the ability to adapt, to learn and to change their ownbehaviors, so the important things to successful negotiation in complex E-commercenegotiation environment are that agent studies from the previous negotiation experience, takesthe initiative to adapt the environment changes, and adjusts the negotiation strategyautomatically.This study describes the current negotiation state using the Market-driven Agent (MDA)model, and uses a variety of learning methods, including Bayesian learning, reinforcement learning, genetic algorithms, and artificial neural network to optimize its negotiation strategyvia the Agent's negotiating experience. Consequently, we propose five automated negotiationstrategies based on Market-driven Agent and learning mechanism:(1)We build a multi-strategy fuzzy inference negotiation model bases on the offersatisfactory degree of the negotiating parties.With the constant changing of the open electronic market environment, the intention of anegotiation and the satisfaction evaluation of the negotiation results will change over time.Therefore, the correct prediction of the negotiation result based on the negotiatingenvironment before the start of the consultation will guide the whole negotiation process andwill become the basis of the negotiation strategy selection at different stages. The consultationprocess is divided into pre, early, medium and late period of the negotiation by theauto-negotiation model which bases on the bid satisfaction of the negotiating parties. Thepre-negotiation is the preparation for prediction of the consulation target before the start of theneogotiation. The consultation agent combines the obtained market information, uses fuzzydecision making to judge the bargaining power and the third-party environmental factors,forecasts the expected goal of the consultation, and divids the whole negotiation bid area intodifferent bid satisfaction fuzzy sets according to the goal. In the process of negotiationinteraction, based on the different bid satisfactions, agent determines the different stages ofthe negotiation progress, decides to take the corresponding combined negotiation strategy, anduses fuzzy inference to generate a new round of the overall concession value. This negotiationstrategy applies multi-objective fuzzy decision-making and multi-strategy fuzzy inference,guided by the explicit overall goal, and improves the success rate and the overall benefits ofthe consultation.For multi-issue negotiation, we develop an annealing algorithm based on the similarcriteria, which is trade-off between the multi-issue to get the optimal multi-issue bid which isbeneficial to the two negotiators. Furthermore, in order to improve the algorithm efficiency,we also use a Bayesian method to learn the preference weights of opponent for varied issues.(2)We propose an automated negotiation model bases on MDA which can meet themultilateral negotiation environment in the E-commerce transactions.MDA evaluates the negotiating information to adapt the open and changing marketenvironment from four market factors: trading opportunity; competitive pressures; timepressure and eagerness to complete a deal. Therefore, we establish a multi-agent system tomanage the process of multilateral negotiation, and develop the agent automated negotiationstrategy which computes its new round's total concession value according to the weightedstatistical for the four market factors.(3)An automated negotiation strategy bases on distributed reinforcement learning hasbeen introduced. The MDA makes prudent compromises by reacting to the four changing market factors.However, it is a complex problem to combine the change of these factors to choose anappropriate strategy because these factors are influenced by each other, and this is not suitableto combine them linearly. So we develop an adaptive market-driven agent (AMDA) usingimproved WoLF PHC multi-agent reinforcement learning algorithm to improve theadaptability of MDA in the multilateral negotiation process. As a result, it can adaptivelyacquire the appropriate negotiation strategy by using its own experience. Comparing AMDAswith MDAs through extensive simulations, the results show that AMDAs can adaptivelyimprove their strategies and consequently outperform MDAs with the knowledge gatheredabout past negotiations.(4)In order to deal with continous state, we evolve the automated negotiation strategybases on fuzzy neural network.The WoLF PHC perceives the current state based on evaluation functions about themarket factors and stores them in the look-up Q_table, so it can only deal with discrete statesand actions, but the discrete granularity of WoLF PHC method is hard to decide, we employfuzzy neural network to learn and optimize the strategy. The input of the fuzzy neural networkare the continuous state and candidate action, the output is the Q value of the pair ofstate-action, and the network connection parameters are adjusted by temporal-difference errorand back propagation(BP) method. The state variable is calculated from the four factorsevaluated by MDA model, the action that the agent acts on the environment is the predictivevalue of the concession ratio under current state, and the value of this action is obtained withmaximum Q output of fuzzy neural network of a candidate action set in the current state andgenerated through a random Gaussian noise.(5)We propse the negotiation strategy bases on Actor-Critic reinforcement learning.The actual implementation of action from the model based on fuzzy neural networkabove is taken from the candidate action set, however, the implemented processing is not trulyfor continues actions. To solve this problem, we optimize the negotiation strategy by usingActor-Critic reinforcement learning method, we build two neural networks based onactuator-Actor and evaluator-Critic, respectively, the Critic network parameters adjusted bytemporal-difference error and BP algorithm, and the Actor parameters are optimized bygenetic algorithm. This learning method implements the fuzzy processing of continues stateand action, which is the beneficial exploration for the fuzzy reinforcement learning, and wetry to apply this method to optimize the compromise strategy in the automated negotiation.We achieve the above sequence learning-based automated negotiation strategies, thelatter one is to solve the drawback of the previous, and we verify the efficiency andeffectiveness of the corresponding method by a large number of experiments.