Real-Time Image Processing Methods Based On Heterogeneous Computation For Super-Resolution Localization Microscopy

Super-resolution localization microscopy(SRLM)is a representative super-resolution optical imaging technique,which could achieve up to 20-30 nm spatial resolution and has gradually become an important research tool in life sciences for studying the inner structures of cells.In recent years,with the development and performance improvement of scientific Complementary Metal Oxide Semiconductor(s CMOS)cameras,it is now possible to enable larger field of view(FOV)and faster image acquisition speed in SRLM.Meanwhile,typical s CMOS cameras also bring huge computing pressure of realizing real-time multi-emitter localization or even high-density emitter localization at the maximum data throughput(2048× 2048 pixels @ 100 frames per second(fps)).Limited by the instruction decoder and the shared memory architecture of central processing unit(CPU)and graphics processing unit(GPU),current multi-emitter localization and high-density emitter localization methods suffer from several difficulties,including but not limited to repeated handling of redundant data,significant delay in the processing method and low computational parallelism,and haven’t realized real-time processing for images from s CMOS cameras working at maximum data throughput.Therefore,these reported localization methods are insufficient to satisfy researchers who want to have fast feedback on experimental results,real-time quality control and dynamic intervention on experimental process.Furthermore,these localization methods are unable to provide technical support for multiscale visualization of cellular functions.To meet the requirement of real-time image processing in SRLM,this thesis tried to develop technologies in three different aspects,including advanced heterogeneous computing platform,high-efficient image pre-processing method,and parallel scheduling method for big data flow.The main works were as follows.(1)Two different heterogeneous computing platforms were built for real-time localization of medium and low-density emitters or high-density emitters,and thus provided hardware support for new real-time localization methods.This thesis analyzed the real-time data processing requirements in SRLM and the architecture limitations in common computing platforms,and inserted field programming gate array(FPGA)hardware into the traditional CPU-GPU computing architecture.FPGA is able to read camera data directly,and is used to work with GPU computation that is superb in intensive computation.This thesis constructed an FPGA-GPU cooperative acceleration platform,which provides hardware support for real-time multi-emitter localization of images with medium and lowdensity molecule(< 0.6 μm-2),even when s CMOS cameras are working at maximum data throughput.Moreover,to solve the slow speed issue in processing raw images with highdensity emitters(up to 10 μm-2),this thesis considered new approaches for improving computational parallelism,and built a PCIe-based integrated heterogeneous computing platform.Results verified that these two computing platforms can provide good computing power for real-time image processing in SRLM.(2)A real-time multi-emitter localization method called HCP-STORM,which was implemented on FPGA-GPU cooperative computation platform,was developed for medium and low-density emitters.After investigating the distribution characteristics of fluorescence signals in images with medium and low-density emitters,the requirements in image preprocessing,and the advantages of heterogeneous platform,this thesis proposed a new image pre-processing strategy: utilize the sparse activation characteristics of fluorescent molecules,efficiently delete the useless information in raw fluorescence images via FPGA,output and classify only the effective region-of-interest(ROI).This strategy solved the long computation time problem that was originated from repeated handling of redundant data,and enabled 60 times reduction in data throughput,without losing effective information.A new localization method called HCP-STORM was developed by combining this image preprocessing with a series of maximum-likelihood-estimation(MLE)localization algorithms.HCP-STORM satisfied the requirements of the real-time multi-emitter localization for images with medium and low-density emitters from typical s CMOS cameras working at maximum data throughput.(3)A real-time localization method called AIO-STORM,which was implemented on PCIe-based integrated heterogeneous computation platform,was developed for highdensity emitters.To meet the high computation needs and solve the insufficient computation efficiency problem in high-density emitter localization,this thesis developed a real-time high-density localization method called AIO-STORM.AIO-STORM adopted the idea of task parallelism in computer science,was based on the HCP-STORM method,and was built on the PCIe-based heterogeneous platform.Through direct memory access(DMA)engine,this method realized direct data interaction between FPGA and GPU,so that time costs in CPU and Memory access could be minimized.In addition,the overall time costs were further reduced through maximizing the single load volume of images to reduce the number of data interactions,which took advantages of the fact that FPGA processing speed would not be reduced with the increased emitter density.Test results showed that,without sacrificing imaging quality,AIO-STORM was able to improve the data processing speed of QC-STORM(currently fastest multi-emitter localization method)by 4 times,and achieved a four orders of magnitude faster speed than the popular localization method called Thunder STORM.For raw images with emitter densities of up to 5.5 μm-2 and under a relative high data throughput(1024 × 1024 pixels @ 100 fps),AIO-STORM is still capable of real-time processing.In summary,this thesis developed two heterogeneous computing platforms for SRLM and the associated multi-emitter localization methods,improved significantly the image processing speed of SRLM with large FOV and fast imaging speed,thus provides technology supports for high resolution observing biological structures with large sample size.