Pydra: Improving Software Reuse In A Distributed Computing Framework

Distributed computing frameworks have proved their effectiveness in reducing the development complexity of parallel programming. Atypical framework of this kind provides a distributed runtime environment and a set of restricted interfaces with which developers can write programs. In essence, frameworks make it easier to write paralle programs by means of software reuse. Despite of this fact, existing frameworks support reuse only at the levels of code and libraries, but not components. By allowing one to reuse code in a coarse-grained fashion, component-based reuse (CBR) enables flexible development and is particularly a good solution to complex application scenarios. CBR is also the foundation of advanced development displines such as software product line engineering. In order to implement CBR, a distributed framework has to provide infrastructural support for component management and component communication, which is, however, absent in existing systems. Lack of CBR support limits the capability of a framework and possibly leads to low productivity.This paper presents Pydra, a distributed computing framework designed to aid code reuse in parallel programming. With its task-oriented architecture, Pydra manages user-submitted tasks as components, and provides infrastructural support for components to communicate with each other, thus achieving CBR. At runtime, Pydra co-ordinates the interaction among components, isolates their execution contexts, and uses sandboxes to withstand malicious code. Pydra also supports several programming paradigms found in other computing frameworks, such as data parallelism and MapReduce. Our current prototype runs on a cluster of computers interconnected by high-speed LAN. It adopts a master/slave model, in which users submit their tasks to the master, and the master schedules tasks to run on available slave nodes with a fair scheduling algorithm.Experiments suggest that Pydra has promising performance and good scalability. Our experience also proves that it is helpful for writing highly-reusable programs, reducing development effort, and ultimately improving the productivity of parallel programming.