Enabling large-scale storage and retrieval of whole slide images: A big data approach

Telepathology has the potential to transform the practice of pathology and be a game-changer for patients and pathologists. It can lead to wider, rapid access to expert pathologists across hospitals in the US, improve the daily workflow of pathologists, provide better diagnosis and treatment, reduce medical errors and enable hospitals to cope with constantly increasing caseload. There are certain technical challenges that must be overcome to enable telepathology on a large-scale in US hospitals. First, a glass slide can be scanned using advances in digital imaging to produce a whole slide image (WSI) of near-optical resolution. But WSIs are very large in size (about 6 GB per image). There is a need for cost-effective and scalable storage to host millions of WSIs and support thousands of requests from hundreds of pathologists per day. Next, the underlying networking infrastructure must be capable of transferring terabytes of image data per day.;As a pathologist may view a few hundred slides a day, it is necessary to provide access to WSIs in real-time, with minimal transmission delay and smooth viewing experience. In this work, we propose a software system for large-scale storage and retrieval of WSIs using Apache Spark and a cluster setup. Each WSI is partitioned using a space-filling curve and stored using Apache Spark's abstraction of a collection along with range partitioning. This enables us to place spatially closer partitions of a WSI together on a cluster node. During retrieval, partitions of a WSI are read and transmitted in parallel through the network. We conducted experiments on CloudLab using multi-gigabyte images and observed that our approach was 2 times faster than remote copy.