Authors: Harshvardhan, Nancy M. Amato, Lawrence Rauchwerger

Venue: In Proc. IEEE Int.Conf. on Parallel Architectures and Compilation Techniques (PACT)
DOI: 10.1145/2628071.2671429
Link to Publication

Abstract:
With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques, which can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. Contribution. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into subgraphs that fit in RAM and uses a paging-like technique to load subgraphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores. 

@inproceedings{Harshvardhan-pbdgom-2014, 
 address = {New York, NY, USA}, 
 author = {Harshvardhan and Amato, Nancy M. and Rauchweger, Lawrence}, 
 booktitle = {Proceedings of the 23rd International Conference on Parallel Architectures and Compilation}, 
 doi = {10.1145/2628071.2671429}, 
 isbn = {9781450328098}, 
 keywords = {out-of-core graph algorithms, distributed computing, graph analytics, big data, parallel graph processing}, 
 location = {Edmonton, AB, Canada}, 
 numpages = {2}, 
 pages = {517–518}, 
 publisher = {Association for Computing Machinery}, 
 series = {PACT '14}, 
 title = {Processing Big Data Graphs on Memory-Restricted Systems}, 
 url = {https://doi.org/10.1145/2628071.2671429}, 
 year = {2014} 
}