Evaluating MapReduce for Multi-core and Multiprocessor Systems
Colby Ranger, Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, Christos Kozyrakis∗
Computer Systems Laboratory
Stanford University
Abstract
This paper evaluates the suitability of the MapReduce
model for multi-core and multi-processor systems. MapRe-
duce was created by Google for application development
on data-centers with thousands of servers. It allows pro-
grammers to write functional-style code that is automati-
cally parallelized and scheduled in a distributed system.
We describe Phoenix, an implementation of MapReduce
for shared-memory systems that includes a programming
API and an efficient runtime system. The Phoenix run-
time automatically manages thread creation, dynamic task
scheduling, data partitioning, and fault tolerance across
processor nodes. We study Phoenix with multi-core and
symmetric multiprocessor systems and evaluate its perfor-
mance potential and error recovery features. We also com-
pare MapReduce code to code written in lower-level APIs
such as P-threads. Overall, we establish that, given a care-
ful implementation, MapReduce is a promising model for
scalable performance on shared-memory systems with sim-
ple parallel code.
1 Introduction
As multi-core chips become ubiquitous, we need parallel
programs that can exploit more than one processor. Tradi-
tional parallel programming techniques, such as message-
passing and shared-memory threads, are too cumbersome
for most developers. They require that the programmer
manages concurrency explicitly by creating threads and
synchronizing them through messages or locks. They also
require manual management of data locality. Hence, it is
very difficult to write correct and scalable parallel code for
non-trivial algorithms. Moreover, the programmer must of-
ten re-tune the code when the application is ported to a dif-
ferent or larger-scale system.
To simplify parallel coding, we need to develop two com-
ponents: a practical programming model that allows users
to specify concurrency