Photometric Stereo Method Based On The Unstructured Neural Network

Calibrated photometric stereo predicts the surface normal from a set of single-view images acquired under various illumination conditions to achieve high-precision estimation of 3D information.However,because of the unstructured photometric stereo data with uncertain quantity and order,mainstream machine learning networks such as Convolutional Neural Networks and Recurrent Neural Networks have difficulty in handling the photometric stereo input.Therefore,how to effectively process the unstructured photometric stereo data through machine learning methods is still an opening problem.Moreover,challenging problems under real-world illumination condition such as sparse lighting and special illumination effects will also greatly affect the performance of the photometric stereo model.To solve these problems,this thesis proposes an unstructured neural network that efficiently processes the unstructured photometric stereo data with uncertain quantity and order,and specially designs it for those challenging problems under real-world illumination condition.The achievements and innovations of this thesis are as follows.1.This thesis proposes a convolution strategy specially designed for the unstructured input data,based on the function approximation theory.It fits a non-discrete function as the convolutional filter using the truncated Chebyshev polynomials,and adaptively calculates the potential topological structure of the input data,which providing new ideas for processing unstructured data through machine learning methods.The proposed method is implemented by matrix operation and can be efficiently applied to the photometric stereo problem.2.This thesis proposes a deep photometric stereo network based on our unstructured convolution strategy,which is designed to solve the challenging problems under real-world illumination condition.To solve the sparsity problem under a small number of illuminations,it proposes a flexible aggregation strategy based on the set structure to significantly improve the performance of photometric stereo model under sparse lightings.To deal with special illumination effects such as shadows and specular highlights,it proposes the structure-aware adaptive weighting strategy,which improves the overall performance of the model.To predict surface normals with fine geometric structure,this thesis designs a multi-scale and multi-branch network architecture to avoid oversmoothing in the spatial domain and retain rich image details.A large number of comparative experiments on the synthetic and real-world dataset prove that the proposed photometric stereo network can better process the unstructured photometric stereo input data,and is more robust to the real-world illumination condition.Experimental results show that our model achieves an average accuracy improvement of approximately 10%,and retains richer geometric details of the image.