Benchmarking Infrastructure-As-A-Service Cloud Systems Automatically With Extensibility

Cloud computing has become one of the most popular topics in the information technology ?eld. Among the major models of cloud computing services, Infrastructure-as-a-Service cloud system aims at providing low-level resources. Its convenience and ?exibility attract many individual users and small companies to migrate their services and products to the public or private clouds. The emergency of evaluating cloud systems increases with the growing demand and rapid development of cloud computing. However, it is usually quite di?cult to evaluate a cloud due to the complexity of the cloud system. Performance testers often face with the embarrassment of dealing with a lot of complex manual operations to interact with the clouds. A good cloud service benchmark tool becomes valuable for both the cloud customers and the cloud providers.The paper introduces a cloud performance evaluation framework which achieves both broad cloud API support and good workload extensibility, to measure the overall capability and scalability of Infrastructure-as-a-Service cloud systems in an automatic way. A CloudAPI model is provided as a high level generalization for all basic cloud resource management operations. The abstraction of workloads is also carefully designed to ensure the extensibility of the workload set. The framework separates the logic of controlling tests and running workloads, and provides a set of Coordinator interfaces to format the work?ow of workload execution. Two default modes of simulating performance scenarios for tests involving multiple workloads, the static parallel scenario and the dynamic incremental scenario, are added to the framework. To better understand and analyze the results of performance testing, the paper also proposes a methodology to build the performance model for different aspects of the cloud system, as well as its scalability.The CloudAPIs supporting Amazon Elastic Compute Cloud and OpenStack are implemented, and a set of workloads for assessing different aspects of a cloud system is added to the framework. They are used to carry out a series of experiments on the Amazon cloud and two clouds deployed with OpenStack. The results of the experiments are applied to the scalability model to evaluate the elasticity and performance of the cloud systems under test. Models of different scenarios give similar results. The scalability and the performance per instance are in polynomial relation.The experiments show the robustness and compatibility of the framework,which makes remarkable guarantee that it can be leveraged in practice for researchers and testers to perform their further study.