| Species diversity is an important measurement of ecological communities. Scientists believe that there is a strong relationship between species diversity and ecosystem processes. However efforts to investigate microbial diversity using whole genome shotgun reads data are still scarce. With novel applications of data structures and the development of novel algorithms, firstly we developed an efficient k-mer counting approach and approaches to enable scalable streaming analysis of large and error-prone short-read shotgun data sets. Then based on these efforts, we developed a statistical framework allowing for scalable diversity analysis of large, complex metagenomes without the need for assembly or reference sequences. This method is evaluated on multiple large metagenomes from different environments, such as seawater, human microbiome, soil. Given the velocity in growth of sequencing data, this method is promising for analyzing highly diverse samples with relatively low computational requirements. Further, as the method does not depend on reference genomes, it also provides opportunities to tackle the large amounts of unknowns we find in metagenomic datasets. |
