Antnest: A Distributed Computing System Supporting Multiple Computational Modals

With the development of computer and the Internet, the growth of the informationhas reached an unprecedented speed, the increasing accumulation of massive data makespeople ushered in the "big data" times. How to extract the useful information from thesehuge data correctly and efficiently and make a proper decision becomes a challenging task.Facing this problem, people tend to consider using a distributed computing for processing.However, for most of the information enterprise, always have many business withdifferent needs, they applicable to different distributed computing platform, which led to awide variety of distributed computing platform, to learn and maintain theseplatforms often requires a very high cost, this bring huge economic pressure to newInternet enterprise. Therefore, Design a unified distributed computing platform becomesthe urgent needs of each enterprise.Antnest catagorizes three computation models for the enterprise data processing, andgives a comparison for them. We propose a computation model to unify the three computation models after founding they are equivalent. The proposed computation model mainly employs a same data source interface and data processing interface. Antnest composite thecharacteristics of batch distributed computing, streaming distributed computing andautomatic scheduling task framework, provide good support to three basicenterprise business——Large-scale static data processing business,online real-time business, background timing business, to reduce theenterprise overhead caused by the use and maintenance of a variety of platforms.Antnest provides users with a unified task create mode, users can use the same rules todefine data sources and calculation unit configuration file according to their actualbusiness needs, and processing logic calculation unit in the corresponding task interfaceand then be able to upload tasks to running in the platform. Antnest shields the differencebetween the tasks of different computational model. In addition, theunderlying implementation mechanism, such as fault tolerance, messagecommunication, task partitioning and scheduling is completely transparent for the user. The tests show that Antnest can support different types of tasks well, to complete thebasic operations of task, and can support different databases.