Current compute technologies and scientific methods for Big Data analysis today are demanding more compute cycles per processor than ever before, with extreme I/O performance also required. The capabilities of Intel Hyper-Threading that offers only 2 simultaneous threads per core limit the results of these recent advances in Big Data analysis techniques.
Despite multiple prior Hadoop Genome analysis attempts on an existing Intel based LSU HPC cluster, a large metagenome dataset could not be analyzed in a reasonable period of time on existing LSU resources. Knowing the extraordinary capabilities for big data analysis offered by IBM Power Systems, the LSU Center for Computational Technologies staff and researchers turned to IBM for help.
This presentation will talk about how IBM helped LSU render a 3.2TB Metagenome dataset in Hadoop in just 6.25 hours, using only 40 compute nodes , whereas the same analysis took 20+ hours on 120+ x86 nodes. This technology has direct parallel to Oil and Gas applications that can take advantage of simultaneous threading and a parallel file system.
