Leaf Recognition Guided Intrusive Indoor Plant Modeling

With the rapid development of machine learning,data-driven cognition and understanding object are the trends.Digitization of our physical world is needed by data-driven methods.Plants are one of the most common objects in nature.Hence 3D modeling of plants is always an important topic in computer graphics.3D laser scanners or depth sensors are able to obtain the 3D shape information of the object being scanned.As the plants have large self-occlusion,the obtained data loses the information of the occluded parts.For indoor plants,people are mainly concerned about the leaves modelling,but the inner leaves are highly occluded by the outer leaves.To obtain complete plant information,the outer leaves can be cut off gradually during the scanning process to make the interior of plants visible.This kind of plant modeling method,which destroys the whole structure of plants,is called the "intrusive modeling method".How to automatically choose leaves for cutting is an important part of the intrusive plant modeling method.To solve this problem,we propose a method to identify and select leaves by instance segmentation network.Given a plant picture observed from a certain viewpoint,the leaves that should be cut are detected and selected using instance segmentation network(such as Mask R-CNN).To reduce the work of manual labeling plant leaves in real plant pictures,we generate images with the leaf contour information by rendering 3D plant models.A parametric model is used to model a leaf.By adjusting the parameters,leaves with various shapes could be represented.Then the characteristics of a specific plant are summarized,together with the corresponding leaf model is used to generate a virtual plant model.We classify the operations of the modeling process and propose an automatic modeling system with a client-server structure.We use a leaf index image to connect the image region and the leaf index of the model.We gradually remove the detected leaves from a virtual plant and reconstruct the leaf model based on the data obtained from the virtual scanner.When the system stops,all the reconstructed leaf models are added back to the rest of the virtual plant.To verify the effectiveness of the system,we perform simulation tests on typical plants,such as Epipremnum aureum and Schefflera octophylla.The results show that the reconstructed plant models can completely recover the 3D information of the original plants.This system has good scalability and can be applied to the modeling of real plants.