Research On Memory Resource Sharing In Virtual Computing Environment

The growth, autonomy and diversity characteristics of Internet resources have brought many challenges on resource sharing. Internet-based Virtual Computing Environment (iVCE) is a novel network computing approach based on Internet resource autonomisation. With the key mechanisms of on-demand aggregation and autonomic collaboration, iVCE focuses on multiplicate resources sharing and collaborative working over the open network infrastructure.Memory resource sharing is an important but special problem in iVCE. On one hand, traditional network memory technology can work well for local memory resource sharing within the cluster and improve the performance of memory-intensive or I/O-intensive applicatons, while it can not fit the characteristics of resources in loosely-coupled network computing environment; on the other hand, many works have gotten excellent solutions to share the resources such as data, storage and so on, while the memory resource usually has different characteristics. In this thesis, we will extend traditional network memory approach to the common network infrastructure based loosely-coupled computing environment using the idea of iVCE, and deeply study on the autonomisation method, on-demand aggregation and autonomic collaboration mechanisms of memory resources.Traditional network memory approaches are usually designed for tightly-coupled cluster system, and will be restricted by limited resource. In this thesis, we introduce the idea of memory resource sharing into the loosely-coupled network computing environment, and propose the concept of memory resource sharing system upon iVCE named iVCE/M, which can usually be applied into the applications with mass small-granularity and random disk access. We firstly propose the basic principle of single-service, single-identity and exponential-probing. Upon the idea of resource autonomisation in iVCE, we also describe the five types of nodes in iVCE/M, their transition relationship, and the two types of services among the nodes. In addition, we design the basic behavior strategies of the nodes with consideration of the growing, autonomy and diversity characteristics for wide area distributed resource. The strategies are compared with the similar parts in iVCE. Through the real meteorologic application trace based simulation, we prove that iVCE/M can improve the performance of memory intensive or I/O intensive applications efficiently in the network with good performance, such as a campus network.Due to the special characteristics of memory resource, it can hardly borrow the traditional centralized, unstructured decentralized and structured decentralized resource information management systems. In order to implement the iVCE idea of retrieving resource information dynamically with actual requirements in a distributed way, we propose a network latency based resource clustering mechanism considering the characteristics of memory resource sharing. It can partition the nodes in iVCE/M into several groups, and guarantee with high probability that the memory resource sharing should only be applied within the group. The resource information management problem can be downgraded efficiently. We propose the basic model of memory resource clustering and the corresponding distributed greedy algorithm, and then analyze their shortcomings. After that, with the force field and potential energy theoretic in physics, we propose the force field-potential energy model, which includes various factors such as the diverse and dynamic network delay and disk lantency. We also design the corresponding force field-potential energy clustering algorithm with the definition of collision set and the probability estimate of network latency. With the simulation on real network topologies, the primary target such as false positive, false negative of the two models and algorithms, and the memory resource capacity in the group are also evaluated.The performance of iVCE mainly depends on the cost of network communication during the memory page accessing. We consider the approach of hiding the network communication cost in this thesis. Prefetching has been widely used in computer systems. However, the effect of prefetching, which depends on the prediction of the data accessing pattern, is restricted with the resource limitation. Based on the resource autonomisation in iVCE, we propose a push-based iVCE/M prefetching mechanism. The memory node should not only respond the operation of memory page accessing of the user node, but also automatically analyze the pattern of the user node during their accessing, predict the sequential operation, and push the potential needed memory pages to the user node actively. We also propose the I/O pattern prediction algorithm based on the sequential pattern mining method in data mining area. This algorithm can perform better analysis with smaller granularity than others. The time and space cost and the correctness are also analyzed and proved. We study the impact of several important parameters in our prefetching mechanism through real trace based simulation.The performance of iVCE/M can also be optimized from time and space considerations. In time dimension, the network communication efficiency among the memory nodes and the user node can be improved through parallel transmission of several memory nodes; in space dimension, there may be multiple memory nodes storing much data with the same content, which should be merged for more memory resources in order to contain larger work set of applications in iVCE/M. In this thesis, we analyze the traditional data stripe and I/O paralleling methods from the time aspect, then model the network transmission process, and propose the conception of time correlation. After that, the idea of adaptive parallel transmission with memory page interchanging is also described. From the space aspect, we propose the idea of merging redundancy memory pages for larger work set of applications. The memory page redundancy is detected and merged using Bloom Filter method for the sake of the distribution of the memory nodes. The low cost on time and space make this mechanism rather practical. We also collect the trace of three I/O intensive applications under different configurations, and evaluate the approaches using the trace based simulation.