Research On Deep Network Camera Image Processor Based On Model Based Design

The work of this paper originated from the scientific research task of image processing of surveillance cameras in the lunar exploration project of our country.In a digital image processing system,the quality of the original input image of the camera has an important impact on the subsequent image quality improvement,intelligent analysis,and compressed storage of the system.The quality of the output image of the camera is largely determined by the performance of the Image Signal Processor(ISP).The traditional ISP system has a complex structure and a long processing pipeline.The order of the functional modules designed by different ISP chips on the processing pipeline is not the same.In most cases,it depends on the design experience of the engineer.So far,the ISP chip still has poor processing effects in some scenarios,and the processing Pipeline parameters cannot meet the pain points of all application scenarios.In response to the above problems,this paper considers the use of algorithms based on deep learning models to solve the problems of traditional ISP systems,and solves the problem of scenario adaptability through data-driven algorithm design.And deploy the algorithm to FPGA to realize the function and performance verification of the algorithm.Since the processing quality of ISP depends more on subjective quality,it is difficult to label data.At present,there are relatively few data sets for ISP.However,as the number of people involved in improvement continues to increase,the data set will gradually expand,and the deep network structure will also keep improving.In view of the fact that traditional FPGA design methods are not suitable for the continuous improvement of algorithm hardening tasks,this paper studies the design based on model definition(MBD,Model Based Design)design and verification method,apply this design method to ISP's deep network FPGA design and implementation,and complete the construction of the entire verification platform.The main innovations of the thesis include the following three aspects:(1)In-depth study of the traditional ISP framework and related processing algorithms,by comparing the processing modules and pipelines of different ISP processing pipelines,analyzed the shortcomings of traditional ISP design.On this basis,the ISP framework based on convolutional neural network is studied,combined with the advantages of traditional ISP,and the preprocessing of traditional ISP is added,and an improved design of ISP processor based on convolutional neural network is proposed.The performance was analyzed and compared.(2)In order to quickly realize the hardware verification of ISP under the condition that the data set is continuously expanded and the algorithm network structure is constantly updated,this paper proposes a design method for the deep network ISP design and verification platform based on MBD.The platform uses MATLAB's Simulink as the basic tool to realize complete test data generation and transmission,ISP neural network processing and processing result verification in MATLAB.All designs are completed in high-level language.(3)For hardware implementation,this article proposes a flexible and efficient convolutional neural network ISP system hardware acceleration architecture for the improved convolutional neural network-based ISP processor.Using the MBD design method,this paper has completed the model design of the proposed hardware acceleration architecture.In order to solve the problems of large data throughput,large number of multiplications and additions,poor versatility,and slow iteration speed in the convolutional neural network part of the ISP system for hardware implementation,this paper uses a model-based design method to design a convolutional neural network.The network hardware acceleration structure solves the difficulties of convolution operation,deconvolution operation,filling,pooling,and data storage in the hardware design process of convolutional neural network,improves the operation speed of convolutional neural network,and finally completes the overall ISP System hardware acceleration architecture design.This article uses the in-loop verification function in Simulink to complete the deployment of the hardware acceleration structure of the convolutional neural network ISP system proposed in this article on the ZCU106 board of the ZYNQ Ultra Scale+ series.Its operating frequency can reach 85 MHz,The processed image resolution can be up to 4K(3968×2944).(3968 × 2944).The computing power of the convolutional neural network module has reached 536.84 GOPS.For the key resources of convolutional neural network hardware acceleration,the architecture consumes 1548 DSP and 1244 BRAM,and the resource efficiency can reach 34.7% and 43.2%,respectively.This architecture can improve resource utilization while having high computing power.It has the characteristics of high efficiency,flexibility and high speed,and meets the needs of replacing traditional ISPs in the actual application process.