Image Super Resolution FPGA System Through Algorithm-hardware Co-design

Limited by the high price of UHD image capturing camera,UHD video content is severely scarce.Due to the limitation of the network bandwidth,UHD video is also difficult to transmit.The lack of UHD video content has become a key factor restricting the development of the UHD video industry.Image super-resolution(SR)technology can recover high-resolution(HR)images from low-resolution(LR)images,which is a method of generating UHD images.SR technology is used to solve the problem of matching the resolution between the image and the screen.SR often achieves state-of-the art restoration quality due to its end-to-end learning process.However,the SR neural network also has problems such as large model and complicated calculation,which make it difficult to deploy in the edge devices with limited resources.In view of the above background and problems,this paper uses FPGA as the SR neural network acceleration platform.The SR model is codesigned from both hardware and software aspects.In the condition of limited resources,high quality image SR can be achieved,which is of great significance to promote the application of SR technology.The work is summarized as follows:(1)We designed an image SR network suitable for deploying edge device.As the SR network intermediate feature maps do not decrease as they pass through the layers,the conflicts between the large number of parameters and the finite storage resources of the edge equipment,as well as the conflicts between the huge amount of computation and the shortage of computing resources are caused.To solve the above problems,we choose the model design method of hardware adaptability constraints to build a lightweight network basic structure.We use depth-wise separable convolution to replace the conventional convolution in the residual structure to reduce the number of parameters and the amount of calculation.And 1D row convolution is added into the network structure to further reduce the complexity.Moreover,in order to extract the high-frequency information in the pixel more effectively,we adopt the progressive feature distillation structure to carry out multi-level refining of the high-frequency features,which increases a little complexity but can greatly improve the network characterization ability.We use the sub-pixel convolution layer to realize the upsampling to form the ultra-lightweight image SR.Experiments show that our algorithm not only ensures image quality,but also has low complexity.The algorithm recreates FHD images in real time on a smartphone.(2)We designed a hardware accelerator for image SR based on FPGA.We optimize the algorithm and hardware for the high-efficiency calculation and real-time requirements of image SR.Firstly,fixed-point quantization of parameters is adopted to reduce the requirements of parameter memory and computing resources.And we use the layer fusion strategy to reduce the number of memory units.Secondly,we analyze the computing characteristics of each layer of the network.And the parallelism of the algorithm is fully exploited by the loop unrolling method,and each part of the network structure is mapped to a special accelerator module.Finally,according to the configuration of network structure,we customized the storage structure and data path,designed the whole accelerator module,and realized the mapping of SR algorithm to high-performance FPGA acceleration hardware.We developed and verified the accelerator quickly by using HLS technology.Experiments show that the accelerator achieves end-to-end algorithm acceleration and can recover high quality images with low resource consumption.(3)We have implemented an FPGA-based image super-resolution system.In order to further explore the application of image SR technology in industrial filed,we study the image super resolution FPGA System on Zynq.We divided the system,designed the platform mapping scheme and assigned the computing tasks.By combinings with VPSS IP for color space conversion and integrating other auxiliary modules,we implement image SR system on Xilinx ZCU104 FPGA evaluation board.The test results show that the images recover by the system have rich image details and visual quality.This work provides a research basis for the design of ASIC to improve the image quality.