Design And Implementation Of A Message Queue Based Distributed Image Processing System

With the big data era, distributed systems show a new promising future, and as the number of images grows fast on the internet, new demands arise both in engineering and research areas. Nowadays, Hadoop has grown to be a successful project, and distributed image processing solutions based on Hadoop emerge from some labs. Since image processing is a computation intensive application, it brings some new challenges. Hadoop based solutions have some deficiencies in this application, so we designed an improved system. Since traditional distributed image processing systems based upon Remote Procedure Call and NFS, they have inherent defects in communication and storage, we will not do comparison with them.We’v designed a message queue based distributed image processing system. With elaborately designed message queues for communication between each component, we implemented a simplified and low coupling computation framework. In processing of a large number of images, our framework reduces the processing and transferring overhead of the intermediate results between map and reduce tasks in hadoop based solutions, also reduces the coupling between each component of our system, which benefits scalibity of our system.We also implemented a task dispatching mechanism for distributed image processing, in which we dispatch tasks for a working server based on the completeness of former tasks. With FastDFS for image storage, we can break down the batch task into single image, which makes it easy for task dispatching and recovery.Our system is implemented with C/C++, and provids interface for native language implemented image processing libraries, like OpenCV and ImageMagick. It’s more convenient for user to program, and more efficient to process images.At last, we did some experiements comparing with a Hadoop based system, and the result shows that our system is more efficient in processing of a large number of images.