Research On Organizational Structure Discovery In Hierarchical Complex Business Systems

With the rapid development of big data,cloud computing,and artificial intelligence technologies,the scale of Internet data center(IDC)has also exploded,and the operation and maintenance management that Safeguards the safe operation of data center is crucial.Early manual operation and maintenance is difficult to maintain in the era of rapid business growth and high labor costs.Automated operation and maintenance replaces people to perform simple and repetitive tasks and improves operation and maintenance efficiency.However,with the complexity and variety of business services,the automated operation and maintenance based on rule guidance is insufficient.The system complexity is constantly upgrading,and the system architecture is constantly evolving.So the existing technology makes it difficult for the operation and maintenance department to accurately and quickly control system architecture and system changes,and they cannot automatically sense the data failure problem in the configuration management database(CMDB)caused by the actual system change.However,the data center generates a large amount of server log data all the time.If we can effectively analyze the organizational structure of the business system based on these data and realize the automation of the business system architecture,we can change the shortage of traditional manual operation and maintenance and improve operation and maintenance efficiency and timeliness of system architecture diagram information.This thesis analyzes the server log data collected by the data center.The servers driven by the service establish TCP connections for communication.Therefore,the TCP network can reflect the logical relationship of the business functions between the servers.Generally,servers in a business system provide functional services in the form of clusters,and different server clusters have a sequential order in the business process flow.Therefore,the organizational structure of a complex business system is a hierarchical network structure.So,this thesis decomposes the problem of organizational structure discovery in hierarchical complex business systems into two sub-problems:server cluster discovery and server cluster hierarchy discovery,and proposes a cluster discovery and hierarchical discovery method based on server log data collected by the big data platform(CD-HD).Based on the server log data collected by the big data platform,the CD-HD method first uses the server attribute and the category label information of the known partial server to train iterative classifier to predict whether the servers belong to the same category based on the server attributes,and the classifier is used to predict all server-to-server relationship without category tag information;then,on the server TCP network,calculate the similarity of the server communication behavior pattern,and combine the classifier's prediction on the relationship between servers to construct a server similarity network.On this similarity network,server cluster discovery is performed by community discovery.Finally,the functional level of the server cluster in the business system is determined according to the business flow,thereby realizing the discovery of the organizational structure of the complex business system.Experiments are carried out on the real server log dataset from a large data center.The experimental results show that the proposed method can effectively solve the pain point that the organizational structure of complex business system in the operation and maintenance field can only rely on hand-drawn and realize the business system architecture automated combing.