Authors: Harshvardhan, Brandon West, Adam Fidel, Nancy M. Amato, Lawrence Rauchwerger

Venue: In Proc. Int. Par. and Dist. Proc. Symp. (IPDPS)
DOI: 10.1109/IPDPS.2015.28
Link to Publication

Abstract:
With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores. 

@inproceedings{Harshvardhan-ahatpb-2015, 
 author = { {Harshvardhan} and B. {West} and A. {Fidel} and N. M. {Amato} and L. {Rauchwerger}}, 
 booktitle = {2015 IEEE International Parallel and Distributed Processing Symposium}, 
 number = {}, 
 pages = {799-808}, 
 title = {A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems}, 
 volume = {}, 
 year = {2015} 
}