Study On Detection And Analysis In Fruit Phenotype Based On Imaging Techniques

China today has the world’s largest production of melon, which has various varieties. Among them, netted muskmelon(Cucumis melo L. var. reticulatus) is recognized as upscale melon because of its good quality and beautiful netting. Fruit quality is influenced by genetic heritage and growing environment. In addition, fruit phenotype parameters such as size, netting and epidermis color directly reflect genetic characteristics and environmental response. Non-destructive measurements of fruit phenotype is now an urgent demand on the breeding and production practices of netted muskmelon.Traditional measurements for fruit phenotype parameters mainly rely on manual measurements, which have disadvantages, such as time-consuming, labor-intensive, complicated operation and subjective measurement accuracy. As the development of phenomics, machine vision has been increasingly employed in plant phenotype quantization measurement and plays an increasingly important role. In the use of imaging techniques for plant phenotype measurement, development of algorithms for different crops in different growing environment is critical for measurement and analysis of plant phenotype. In addition, using the phenotype information extracted by imaging techniques, combined with the genetic information and environmental information, to build models to characterize plant traits can be useful for breeding and precise production management.Fruit phenotype parameters detection and analysis algorithm during the growing in greenhouse are key issues for netted muskmelon production. Based on imaging techniques, image processing, and mathematical modeling, this study aimed to(1) non-destructively detect fruit phenotype parameters and analysis,(2) estimate fruit internal quality based on detected phenotype, and(3) simulate fruit surface temperature based on environmental parameters, by using image fusion segmentation, active shape model, multifractal dimension, lacunarity, active source and color ellipsoid methods. We carried out fruit phenotype detection and analysis, estimated nutritional quality of fruit based fruit phenotype and simulated epidermis temperature of fruit surface based on environment parameters. Main contents and results are as follows:(1) Fruit locating and edge detection using RGB-D imaging and active shape model algorithmFor locating is rather complicated in greenhouse due to several factors such as illumination changing, fruit displacement and noise influence. In this study, we developed fruit locating algorithm based on the integration of RGB image and depth image. This method combined H component image and depth image, and used weighted average method for image pixels fusion. To ensure precise positioning and improve noise immunity, first threshold detection method based on the histogram was used to achieve an accurate fruit detection. Comparing with a single image, the fused image had higher accuracy and robustness on fruit locating. For the images collected at the frontlight in ideal circumstances, accuracy of fruit recognition was 90%.To obtain accurate edge of located fruit, we used edge detection segmentation algorithm based on active shape model. We obtained accurate edge and repeat accuracy of diameters as ±2.51 mm and ±3.97 mm respectively with mean standard deviation was 4.17 mm. The obtained results provide reference information for accurately and non-destructively monitoring fruit morphological information and quality.(2) Detection and analysis of netted muskmelon netting based on multifractal analysis and lacunarity analysisThe netted muskmelon epidermis netting is one of the most important phenotypic traits, because it directly reflects fruit’s growth condition and genetic characterization. Traditional measures of netted muskmelon netting, including netting description(non-netted, sparsely netted, partially netted and completely netted), netting coverage rate, and wrinkled epidermis or unwrinkled epidermis, are commonly applied in breeding and cultivation processes. These measurements were confirmed to be sufficient for some studies. However, they are less well fitted for quantifying changes of netting distribution and the last two methods are mainly through visual inspection. For the case of inconsistent from traditional assessment methods and being unable to characterize difference in the distribution of netting, we focused on the benefits of multifractal and lacunarity analysis for quantifying the netted muskmelon epidermis netting based on detected fruit images. We used the multifractal analysis and lacunarity analysis on three cultivars(Wanglu, Feicui and Luhoutian) and four different development stages. Their efficiencies were confirmed by comparing the classical texture features(co-occurrence matrices, Gabor filters and the wavelet transform) in supervised classification processes(Ada Boost and support vector machine classifiers). Using the images from growth monitoring system, some image processing-mathematical morphology operations were used before analysis. The epidermis netting showed fractal properties. Comparisons among cultivars showed that the extracted generalized dimensions of netting were significantly different while their coverage rate was less different. The generalized dimensions D0, D1, D2 and the lacunarity parameter b could be applied to discriminate netting from different development stages. Based on multifractal analysis and lacunarity analysis, we presented an automated extraction tool of the netted muskmelon epidermis netting. These results demonstrate that multifractal dimension and lacunarity are valuable additions to traditional measures of epidermis netting. Features obtained by fractal, lacunarity, multifractal features contributed to new texture characterization and complementary for classical features(co-occurrence matrices, Gabor filters and the wavelet transform) used in fruit epidermis netting.(3) Color grading and quantification of netted muskmelon color under active light source based on color ellipsoidIn the natural environment, fruit color grading and quantification of netted muskmelon were mainly influenced by light. We used active light source to reduce the effects of light and applied seven grading method of fruit color. Quantification method of fruit color based on color ellipsoid was proposed. Tests showed that the rating accuracy was 90% under three different outdoor lighting conditions(5000K, 5900 K, 7900K). By using images obtained under active light source, L value, a value and b value were clustered to be ellipsoid. The direction and half shaft of the ellipsoid reflected the heterogeneity of epidermis color and compensated the shortage of using a single point of color value and standard deviation. It has been successfully used in different growth stages and different varieties of netted muskmelon.(4) Interior quality estimation of netted muskmelon based on fruit phenotype and BP neural networkIt has increasingly demand by growers on how to use phenotype information as interior quality guide to improve crop management in production. Based on machine vision imaging technology, phenotype characteristics(color features and texture features) and BP neural network were used to rapidly qualitatively and quantitatively estimate fruit internal quality. Netted muskmelon images were collected based on machine vision. A total of 57 melon sample images from three different growth stages were collected. Based on the melon sample images, phenotype features were characterized with L*a*b* color model, HSV color model and gray level co-occurrence matrix(GLCM). The epidermis netting was quantified by multifractal and lacunarity methods. These phenotype parameters were taken as input of BP neural network. In the quantitative prediction, we established predictive models for every interior quality parameter such as sugar contents(the real value of glucose, fructose, sucrose and total sugar content), total soluble solids content and vitamin C(VC) content. The results showed that sugar content of netted muskmelon had high correlations coefficient of prediction up to 0.90. In qualitative estimation, using BP neural network to predict the growth stage of melon by phenotype parameters, the correlation coefficient between predicted growth stage actual growth stage was 0.89. Training set of model contained 30 samples. The hidden nodes and training functions that are important parameters for the network were optimized. For 15 test samples, growth stage predictive value aggreed with the actual value. These results provide a good theoretical basis for predicting the quality of muskmelon and improve netted muskmelon production management.(5) Modeling fruit surface temperature dynamic using thermal imaging and weather dataThe exposure of fruit surfaces to direct sunlight during the summer months can result in sunburn damage, color and texture change. The objective of this part was to develop and validate a model for simulating fruit surface temperature(FST) dynamics based on energy balance and measured weather data, using thermal imaging technology. A series of weather data(air temperature, humidity, solar radiation, and wind speed) was recorded in time windows of 11:00h–18:00h for two months at 15 minutes intervals. To validate the model, the FSTs of “Fuji” apples were monitored using an infrared camera in a natural orchard environment. The FST dynamics were measured using a series of thermal images. For the apples that were completely exposed to the sun, the RMSE of the model for estimating FST was less than 2.0 °C. A sensitivity analysis of the emissivity of the apple surface and the conductance of the fruit surface to water vapor showed that accurate estimations of the apple surface emissivity were important for the model. The validation results showed that the model was capable of accurately describing the thermal performances of apples under different solar radiation intensities. Thus, this model could be used to more accurately estimate the FST relative to estimates that only consider the air temperature. In addition, this model provided useful information for sprinkling irrigation of sunburn protection.In conclusion, we studied fruit phenotype as the main objective, and focused on the dynamic detection of shape, analytical quantification of netting, grading and quantifying of color and model building. The theory and methods of active shape model algorithm, multifractal dimension, lacunarity, active source and color ellipsoid were proposed to increase accuracy and precision in non-destructive detection of fruit phenotype and advance the development of fruit phenotype genomics. It is also significant to establish muskmelon phenotypic expression of important genes, environmental control and precise management of production. The results provided theoretical basis and technical methods for physiological and ecological characterization of the change process of fruit development with fruit phenotype.