Research On Visual Measurement And Obstacle Avoidance Control Of Apple Harvesting Robot

The apple harvesting robot is a kind of intelligent mechanized harvesting system with perceptual recognition ability, and can be used to accomplish apple picking and other tasks automatically. The basic function of apple harvesting robot is to reduce labor intensity and productive cost, improve operation efficiency and product quality, and guarantee fruit picking on time. To this end, the research on the related technique for apple harvesting robot is of great practical significance. To the best of the author’s knowledge, the study on harvesting robot is relatively late in China. Since the non-structural picking environment and complicated picking object, the picking automation level is still low. Moreover, with the increment of population aging and the agricultural production cost, and the decrement of labour force and the ability of market competition, the automation on fruit picking becomes an urgent problem. This paper will investigate the visual measurement and obstacle avoidance control problems for apple harvesting robot. The research work is supported in part by National High Technology Research and Development Program of China (Grant NO.2006AA10Z254)—"Research on the key technologies of apple harvesting robot" and in part by Research Fund for the Doctoral Program of Higher Education of China (Grant NO.20093227120013)—"Research on the motion performance optimization and obstacle avoidance control strategy of fruit picking manipulator ". The main content can be listed as follows:1. According to opening design rules, the apple harvesting robot was built with visual system, sensory perceptual system, AC servo system and control system. The visual and sensory perceptual systems were used to perceive and recognize the picking target and environment. The AC servo and control systems were used to drive mechanism to pick fruits safely. These works would provide hardware support and the theoretical base for the following visual measurement and obstacle avoidance control.2. The used acquisition methods of static and dynamic apple images were introduced. The acquired apple images were performed characteristic analysis and classified by difference of light acceptance and growth state. We serve up the statistics on the fruits, leaves, branches and sky in images based on the different color feature under RGB, XYZ, HIS and I1I2I3color spaces. The color difference among fruits, leaves, branches and sky was found from the above statistical data, which provided the basis and lay foundation for the following image segmentation based on color feature.3. The static fruit recognition method in natural scene was researched. At first, the OTSU dynamic threshold segmentation method with I2color characteristic in I1I2I3color space was chosen by comparing the apple image segmentation methods based on the color feature in different color spaces. Next, the image perfection and noise removal were carried out for the above segmentation image, and the apple fruit contour model can be established. The apple fruits were recognized by edge detection and the improved RHT transformation method, in which the phenomenon of rough edges and broken edges were solved by edge thinning and edge connection, and apple fruits overlapping and severely shaded by the branches and leaves were performed the separation and restoration operations respectively before they were recognized. Finally, the apple fruits recognition test was done, and the test result showed that the recognition rate of the proposed method in this study was100%for apple fruits in separate and non-shaded and overlapping shaded states, while for apple fruits shaded by branches and leaves, the recognition rate was higher than85%; the average recognition time is0.44s for non-shaded apples,0.72s for overlapping apples, and0.77s for badly shaded apples, which can meet the requirements of real-time picking. In addition, the fast tracing recognition method of target fruit for apple harvesting robot was also discussed. Firstly, the picking target fruit was determined by the principle of the nearest to image center. The target fruit in the following images were traced and recognized with the improved fast mean-residual normalized product correlation template matching algorithm while the image process area was narrowed frame-by-frame continuously by the correlated information between the acquired images. Finally, the matching recognition tests based on the different gray value, brightness and contrast and the recognition time measurement tests with the new and old methods were done, which verified the availability of the designed method.4. Since the fruit state needs to be discriminate for the sake of selecting the different harvesting methods before it is fast picked, the fruit state discrimination method was developed. Firstly, the acquired apple images were segmented and the picking target fruit was determined based on the principle of the nearest to image center from the first segmented image. Secondly, the inter-frame difference method was applied for the two segmented fruit images, and then the target fruit state was got using the connection number discrimination for the difference image and the centroid coordinates matching discrimination for the oscillating fruit image. Lastly, the test results showed that the proposed algorithm was feasible and effective for most cases in nature environment, and the discrimination time was less than0.2s. The fast mean-residual normalized product correlation template matching algorithm was improved to be the property of resistance to rotation, which was used to recognize the dynamic images. The validation test showed that the improved algorithm had the rotation invariance in the wide range of [-55°60°]and could recognize the oscillatory fruit accurately. In addition, because of the update of the template, the improved algorithm could meet the requirement. Under the above case that the picking target fruit was detected to the oscillatory fruit, a kind of fast harvesting method in oscillation condition for the harvesting robot was also researched in order to solve the problem of fruit oscillation influence on picking efficiency. At first, the oscillatory fruit in the acquired images were recognized and its2D centroid coordinates were extracted, and then the fruit oscillation period was calculated by FFT algorithm. When obtaining the depth distance of the oscillating fruit, the forward speed of the translation joint for the harvesting robot was calculated. The harvesting robot started to harvest, and the oscillatory fruit was in balance position when it was gripped. Finally, it can be concluded by testing that the successful rate of harvesting was84%, and the proposed method was better than the past harvesting methods for picking the oscillating fruit, and could evidently improve the harvesting speed.5. In order to provide fairly perfect environment information for automatic navigation and obstacle avoidance of apple harvesting robot, the obstacle perception and recognition of apple harvesting robot was studied. Firstly, the work flow of obstacle perception and recognition system for apple harvesting robot was analyzed. Secondly, the obstacle recognition of apple harvesting robot was especially researched. The most effective multiple segmentation method of trunk and branch obstacles was selected after discussing and comparing, and then the morphological closed operation and hole filling operation were performed. The main characteristics of trunk and branch obstacles, i.e., the axle wire was extracted by the skeletonizing method and removing subbranch method based on different templates. Finally, the trunk obstacles and the branch obstacles were located respectively based on binocular vision and monocular vision. The test results showed that the detecting rate of trunk and branch obstacles was95%, the recognition average time was less than0.4s, the measurement error based on binocular vision was within20mm in the pale of500-1500mm, and the measurement error based on monocular vision was within15mm in the pale of250-400mm.6. The obstacle avoidance control of apple harvesting robot was definitely divided into two parts, i.e., the target searching obstacle avoidance and the target picking obstacle avoidance. To search target fruit safely and prevent equipment damaging, the target searching obstacle avoidance paths were designed by combining the obstacle information perceived by the collision sensor on the small arm of apple harvesting robot. For the target picking obstacle avoidance, the kinematics model of apple harvesting robot was firstly established and simulated. The obstacles in the picking environment were equivalent to point-shaped, round-shaped and line-shaped obstacles, and then they were built model by the boundary feature and done the rasterization in C-spce, which maked robot to control the mechanical joints directly. At last, the smoothing direction-first path planning algorithm based on tree structure was presented, which accomplished the loacal obstacle avoidance path planning of apple harvesting robot in C-space. The test showed that this algorithm could avoid obstacles successfully when picking target fruit.