Multi-view Stereo Algorithm Based On Homogeneous Direct Spatial Expansion

Multi-view stereo reconstruction,which recovers 3D structures from 2D images,has become a research hotspot of computer vision for many years.It has wide applications in such fields as industrial inspection,reverse engineering,protection and display of cultural relics and heritage,city planning and so on.Its application is expected to become broader and broader along with the popularity of smartphone and high resolution but low cost image sensors.A lot of multi-view stereo reconstruction algorithms have emerged over the last two decades.They could be roughly classified into volume based algorithms,depth based algorithms and feature expansion based algorithms.Yet a lot of improvements still need to be made to perfectly hand various complicated situations in reality,such as highly curved surfaces,thin structures,weak texture,changing illumination,occlusion,and so on,in order to achieve high reconstruction accuracy,completeness and high efficiency.The dissertation offered a novel multi-view stereo algorithm based on homogeneous direct spatial expansion(MVS-HDSE).It extended the frame of the feature expansion based algorithm by an additional step for initial value modification utilizing already grown neighbor points.It also made important improvements to other existing steps like sparse seed points extraction,expansion and outlier filtering.The main contributions of this dissertation are summarized as follows.At the sparse seed points extraction step,a new method based on DAISY descriptor was proposed.The traditional SFM method first detects SIFT feature points in images and then matches the points between pair of images to extract the seed points.,However,the accuracy and number of seed points might reduce greatly due to mismatch or match failure.The proposed method describes the feature point by high performance DAISY descriptor and searches the best match point along the epipolar lines in several images,instead of the traditional matching among limited feature points in different images.As a result the number of initial seed points increases by multiple of times along with the improvements on accuracy.In order to implement the above method successfully,a series of supporting measures have been adopted such as computation of the fundamental matrix by random sample consensus algorithm after carefully selecting of several neighbor images and successful matching of some feature points among the selected images;searching of corresponding feature points along epipolar lines;computation by SVD method the 3D positions of those feature points that have been successfully matched in all the selected images,clearing away the uncertainty of matching in only a pair of images;filtering of the low accurate 3D points based on reprojection errors;finally,estimation of the normal of extracted seed points by the conditional-double-surface-fitting method.Both the above seed points extraction and the subsequent homogeneous spatial expansion are conducted on multi-layer image pyramids,which further improves the efficiency and the rate of success of proposed algorithm.At the expansion step,a homogeneous spatial expansion strategy was proposed.In traditional feature expansion based algorithms,the assignment of next expanding point still relies on a reference image,i.e.,the initial positions and normal of expanding points are preset in a local coordinate system which changes with the changing of the reference image.In addition the traditional algorithms had to adopt a greedy or "best-first" strategy based on a priority queue to prevent more outliers grown out from bad or low accuracy seed points.The sequential nature of these expansion strategy greatly limited their efficiency.In contrast,the proposed algorithm grows simultaneously all the seed points with same priority,making it quite suitable for parallel computation.The initial positions of expanding points are directly preset in a fixed world coordinate system along the tangent planes of nearby seed points.It is not necessary to always update the record in all images which pixels have been successfully expanded,saving a lot of memory.The possible spreading of errors from low quality seed points are effectively depressed by means of latter outlier filtering and initial value modification,making the expansion of each point relatively independent.At the same time,for each expanding point,best images are selected as reference and auxiliary images before the optimization and the selected images would be updated immediately during the optimization if the normal of the expanding point changes by an amount larger than a threshold to guarantee the reconstruction accuracy,completeness and efficiency.On the contrary,the reference images were always fixed in traditional algorithms.In addition such parameters as window size and the level of image pyramid are adaptively adjusted during the optimization according to the texture,which is beneficial for improvements of reconstruction completeness.At the newly added initial value modification step,a conditional-double-surface-fitting method was proposed to fit the real object surface based on the already grown neighbor points.The fitted surface was then used to modify the initial values of expanding point.In theory,each point can be expanded independently via optimization regardless of whether other points have been reconstructed.In reality,the initial values of position and normal for optimization are provided by an adjacent seed.If the initial values were far away from the true values,the optimization process might end up at a local minimum with low accuracy or even fails.So the initial value modification is of vital importance for both efficiency and accuracy.To do this,first it is checked to see whether the neighbor points are dense enough and whether the expanding point is near the center of the region covered by the neighbor points.If the two conditions are satisfied the first surface fitting will be conducted.The second surface fitting follows after el;iminating of those points with big residual error,which might come much closer to the real object surface.Then the expanding point can be projected back to the fitted surface to modify its initial values.At fittering step,three adaptive filters according to smooth consistence,depth consistence and normal consistence were designed to filter the outliers.These filters are capable to distinguish true outliers from false outliers.For this purpose some conditions would be checked at first to determine whether the filtering should be conducted to deal with the ever changing factors such as surface curvature,point density,occlusion,etc.Once the filtering was find necessary then the related parameters would be adaptively adjusted.The proposed algorithm was tested on various kinds of scenes from DTU benchmarks,Middlebury benchmarks,VGG dataset and that captured by ourselves.All the reconstruction experiments have come up with good results,demonstrating that the proposed algorithm was robust to deal with all kinds of scenes.Compared with other multi-view reconstruction algorithms,the local defects in the reconstruction results generated by the proposed algorithm reduced significantly.The quantitative evaluation results also indicated that both the reconstruction accurcy and completeness of the proposed algorithm are among the top ones,obviously better than other feature expansion based algorithms.This good performance of proposed algorithm owes to the successful extension of the algorithm frame and effective improvements on all the steps within the frame.