| Computer vision tasks are increasingly favored by artificial intelligence.Semantic segmentation has potential applications in many fields,such as automatic driving and face recognition.Real-time 3D reconstruction is closely related to artificial intelligence and intelligent robots,such as scene perception,indoor navigation,etc.These visual tasks greatly promote the progress of our country intelligent and embody the very high value.Based on deep learning technology,this thesis carries out corresponding research on semantic segmentation and 3D reconstruction of computer vision tasks.Aiming at the problems such as poor generalization of semantic segmentation model in different data domains,not making full use of the assistance of depth information and poor real-time performance of semantic segmentation and 3D reconstruction,a new solution is proposed,and a large number of experiments are conducted to verify it.In the aspect of semantic segmentation,To improve the segmentation performance and solve the problem of poor generalization of the model in different data domains,we propose a method based on depth information for semantic segmentation in the context of unsupervised domain adaptation.It includes a Depth-aware Adaptation Frame-work(DAF)and a Intra-domain Adaptation(IDA)strategy.Firstly,DAF is proposed to adapt domains by capitalizing on the inherent correlations of semantic and depth information.Then a novel light-weight depth estimation network is designed provide additional depth information,and we fuse semantic and depth information by cross-task interaction,then align the distribution in depth-aware space between source and target domains.Finally,IDA strategy is proposed to bridge the distribution gap inside the target domain.To this end,a depth-aware ranking strategy is presented to separate target domain into sub-source domain and sub-target domain,and then we perform the alignment between sub-source domain and sub-target domain.Experiments on SYNTHIA-2-Cityscapes and SYNTHIA-2-Mapillary cross-domain tasks show that our method achieves significant improvement(46.7% m Io U and 73.3%m Io U,respectively)compared with the similar methods.In the aspect of real-time 3D reconstruction,The scene perception of intelligent devices includes 3D reconstruction and instance segmentation technology.At present,most of the work adopts the serial reconstruction first and then segmentation method.Although these methods can perform instance segmentation on the reconstructed 3D model,the defect of their slow speed always exists.In view of this,this thesis proposes a real-time 3D reconstruction and online instance segmentation method based on RGBD video stream,which can render the reconstruction and segmentation effect in real time.Firstly,the relative pose between adjacent image frames is optimized from the video stream collected by depth camera.In order to meet the synchronization and real-time requirements of reconstruction and segmentation,scene segmentation information is predicted in some frames,and the TSDF model containing segmentation label information is constructed by using the optimized relative pose.Secondly,the Ray Casting algorithm is improved to render the reconstruction results with label information in real time by using the pose information of the previous frame as the rendering perspective,and the Fast Location algorithm is proposed to obtain the 3D position and bounding frame of each instance under the rendering perspective.The final scene reconstruction result with segmentation information was obtained by using Marching Cubes algorithm.The experimental results show that the real-time scene reconstruction and online instance segmentation method proposed in this thesis not only ensures the accuracy of reconstruction and segmentation,but also improves the speed of scene perception and the precision of reconstruction.Among them,the instance segmentation accuracy is more than 95%,in terms of speed,compared with the first reconstruction and then segmentation method,the speed is improved by about 20% ～ 35%.Compared with the similar methods,the reconstruction accuracy is significantly improved. |
