Research On The Methods About Recognizing Vehicle Information On The Bridge And Warning For Avoiding Vessel Collision Based On Computer Vision

Bridge is an important part of the infrastructure,and its health condition is directly related to the social public security.Under the background of the current artificial intelligent age,how to use the advanced technologies to service for monitoring,operation and maintenance of the bridges has become a hot topic for the researchers.In this article,the existing problems in current video/image based methods about identification of vehicle information on the bridge and warning for avoiding vessel collision are researched based on computer vision technologies and deep learning algorithms.In the existing methods about identifying the vehicle information,there are problems that the identified vehicle information is simple and the robustness of the methods is weak.For solving the above problems,a method about identifying the comprehensive information of the vehicles is proposed based on image instance segmentation.And the proposed identification method is combined with the finite element computation of the structure for online monitoring the operation condition of the bridge.In the aspect of vessel collision warning,there are lack of effective methods about realtime vessel positioning and jointly warning vessel-bridge and vessel-vessel collisions.For these reasons,a homography based real-time vessel positioning method and a data-driven warning method about avoiding vessel-bridge and vessel-vessel collisions are proposed.The proposed warning method mainly includes the trajectory generative adversarial networks with multiple critics(TGANsMC)which is used for vessel trajectory data augmentation and a dual-task encoder-decoder network which is used for jointly warning vessel-bridge and vessel-vessel collisions.In this paper,the main research contents and innovations are as follows.(1)A method for identifying vehicle comprehensive information,including type,number of axles,3D bounding box,speed,length,current driving lane,and traffic volume,is proposed based on image instance segmentation which is realized by Mask R-CNN.In the identification of the vehicle information,a method for identifying the number of axles based on the Mask Io U between the vehicle and wheel is proposed.And the vehicle 3D bounding box is generated based on the high-quality segmented vehicle mask,which is more accurate than that generated by the traditional frame difference or background subtraction method.For calculating the vehicle speed and length,the control points on the road are determined by the lane lines and one scene vanishing point to calculate the homography.The length and speed of the vehicle can be calculated by combining the homography and the vehicle 3D bounding box.The 3D bounding box is also used for identifying the current driving lane.In order to obtain the reliable vehicle parameters,the tracking method SORT is used to associate the vehicle objects across frames,and then the vehicle parameters identified from multi-frames are post processed,so as to obtain more reliable vehicle parameters than the results identified from a single frame.Based on the proposed method,the system about identifying vehicle comprehensive information is developed and it is applied to Anqing Yangtze River highway bridge.(2)An online monitoring method about bridge operation condition is proposed,which integrates vehicle information identification,vehicle load distribution,finite element modeling and influence line calculation.Based on the proposed method about vehicle comprehensive information identification,the real-time spatial locations of the vehicles on the bridge are deduced firstly.Then,based on the identified number of axles of vehicles and the local vehicle load distribution,the load range of each vehicle on the bridge is determined.In order to simulate the response of the bridge,the finite element model of the bridge is established,and the response influence lines of the key positions of the bridge are calculated.After that,the combinations of the vehicle loads corresponding to the upper and lower bounds of the response of each key position of the bridge are determined according to the positive and negative regions of the influence line and the position and load range of the vehicles.Finally,each load combination is applied to the finite element model,and the response range of each key position of bridge can be calculated at arbitrary moment.This method can realize the online monitoring about operation condition of the bridge without installing other sensors on the bridge.(3)A homography based real-time vessel positioning method is proposed for solving the problem about positioning the vessels on the water in real time.In this method,the buoys on the water are used as the control points for solving the homography between the pixel coordinate system in the video images and the world coordinate system on the water.Combining the obtained homography with the recognition and tracking of the vessel,the vessel real-time positioning can be realized.The key in solving the homography is to obtain the world coordinates of the control points.In this paper,on the basis of geometrical similarity relation,the instantaneous world coordinates of all control points are obtained at one time with the aids of aerial photography by UAV and rectification of the oblique aerial image.Because the positions of buoys(i.e.,control points)are unstable on the water,the pixel coordinates of the buoys should be synchronized with the corresponding world coordinates in calculating the homography.In addition to the fixed camera,the proposed real-time vessel positioning method only needs an aerial image that contains the bridge and some buoys on the water,which is simple and easy to implement.(4)The trajectory generative adversarial networks with multiple critics(TGANs-MC)is proposed for augmenting the vessel trajectory data,especially the few abnormal trajectory data that have high risk about colliding with the bridge,so as to provide sufficient data for the data-driven warning of vessel-bridge collision.TGANs-MC adopts the Wasserstein distance based objective function with gradient penalty.The generator in TGANs-MC generates the fake trajectory samples based on gated recurrent unit(GRU).In the critic of TGANs-MC,1D convolution and 1D adaptive pooling are used to establish the variable-length sequence feature extraction module.For generating the multifarious trajectory samples,multiple critics with different combinations of the feature extraction modules are constituted to guide the updating of the generator.Because different critics have different levels of constraint on the generator,the trajectory samples with different patterns can be generated through the multiple critics based network.In training TGANs-MC,the strategy of curriculum learning is adopted to relieve the burden for both generator and critic in processing the whole trajectory sequence at the early stage of training.(5)A dual-task encoder-decoder network is proposed for warning vessel-bridge and vesselvessel collisions,whose innovation is to merge the trajectory anomaly detection which is for warning vessel-bridge collision and the trajectory prediction which is for warning vessel-vessel collision in one network.The encoder in the dual-task encodes the observed trajectory into a fixed-length trajectory information vector which is decoded for vessel trajectory anomaly detection and prediction in decoder.In the anomaly detection branch of the decoder,the fixed-length trajectory information vector is mapped to the risk degree of navigating,and the judgement about whether to issue the warning is made based on the risk degree.In the prediction branch of the decoder,the attention mechanism is adopted for relieving the burden of encoder to improve the prediction precision.If a distance between the different vessels' current positions or each predicted positions is less than the safety threshold,which indicates that the vessels have the tendency to collide,a warning will be issued.A strategy of three-stage training is adopted to optimize the parameters of the dual-task encoderdecoder network.The two branches of the network are seperatedly trained firstly,and then both branches are jointly trained.A real-time collision warning system is developed,which is applied on Guangdong Jiujiang bridge.And the vessels which have high risk about colliding with the bridge are identified in monitoring.