Abstract
In this paper, we propose a novel task scheduling algorithm for Hadoop MapReduce called dynamic priority multiqueue scheduler (DPMQS). DPMQS i) increases the data locality of jobs, and, ii) dynamically increases the priority of jobs that are near to completing their Map phase, to bridge the time gap between the start of the reduce tasks and the execution of the reduce function for these jobs. We discuss the details of DPMQS and its practical implementation, then assess its performance in a small physical cluster and large-scale simulated clusters and compare it to the other schedulers available in Hadoop. Both real experiments and simulation results show that DPMQS decreases significantly the response time, and demonstrate that DPMQS is insensitive to changes in the cluster geometry.
| Original language | English |
|---|---|
| Title of host publication | 2016 IEEE International Conference on Communications, ICC 2016 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9781479966646 |
| DOIs | |
| Publication status | Published - 12 Jul 2016 |
| Event | 2016 IEEE International Conference on Communications, ICC 2016 - Kuala Lumpur, Malaysia Duration: 22 May 2016 → 27 May 2016 |
Publication series
| Name | 2016 IEEE International Conference on Communications, ICC 2016 |
|---|
Conference
| Conference | 2016 IEEE International Conference on Communications, ICC 2016 |
|---|---|
| Country/Territory | Malaysia |
| City | Kuala Lumpur |
| Period | 22/05/16 → 27/05/16 |
Bibliographical note
Publisher Copyright:© 2016 IEEE.