Keddah: Capturing Hadoop Network Behaviour

Jie Deng; Gareth Tyson; Felix Cuadrado; Steve Uhlig

doi:10.1109/ICDCS.2017.211

Keddah: Capturing Hadoop Network Behaviour

Jie Deng, Gareth Tyson, Felix Cuadrado, Steve Uhlig

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

2 Citations (Scopus)

Abstract

As a distributed system, Hadoop heavily relies on the network to complete data processing jobs. While Hadoop traffic is perceived to be critical for job execution performance, the actual behaviour of Hadoop network traffic is still poorly understood. This lack of understanding greatly complicates research relying on Hadoop workloads. In this paper, we explore Hadoop traffic through experimentation. We analyse the generated traffic of multiple types of MapReduce jobs, with varying input sizes, and cluster configuration parameters. As a result, we present Keddah, a toolchain for capturing, modelling and reproducing Hadoop traffic, for use with network simulators. Keddah can be used to create empirical Hadoop traffic models, enabling reproducible Hadoop research in more realistic scenarios.

Original language	English
Title of host publication	Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017
Editors	Kisung Lee, Ling Liu
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2143-2150
Number of pages	8
ISBN (Electronic)	9781538617915
DOIs	https://doi.org/10.1109/ICDCS.2017.211
Publication status	Published - 13 Jul 2017
Externally published	Yes
Event	37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017 - Atlanta, United States Duration: 5 Jun 2017 → 8 Jun 2017

Publication series

Name	Proceedings - International Conference on Distributed Computing Systems

Conference

Conference	37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017
Country/Territory	United States
City	Atlanta
Period	5/06/17 → 8/06/17

Bibliographical note

Publisher Copyright:
© 2017 IEEE.

Access to Document

10.1109/ICDCS.2017.211

Cite this

Deng, J., Tyson, G., Cuadrado, F., & Uhlig, S. (2017). Keddah: Capturing Hadoop Network Behaviour. In K. Lee, & L. Liu (Eds.), Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017 (pp. 2143-2150). Article 7980162 (Proceedings - International Conference on Distributed Computing Systems). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDCS.2017.211

Deng, Jie ; Tyson, Gareth ; Cuadrado, Felix et al. / Keddah : Capturing Hadoop Network Behaviour. Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017. editor / Kisung Lee ; Ling Liu. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 2143-2150 (Proceedings - International Conference on Distributed Computing Systems).

@inproceedings{f5b32b5c05e849cd892ba8cca3126ea6,

title = "Keddah: Capturing Hadoop Network Behaviour",

abstract = "As a distributed system, Hadoop heavily relies on the network to complete data processing jobs. While Hadoop traffic is perceived to be critical for job execution performance, the actual behaviour of Hadoop network traffic is still poorly understood. This lack of understanding greatly complicates research relying on Hadoop workloads. In this paper, we explore Hadoop traffic through experimentation. We analyse the generated traffic of multiple types of MapReduce jobs, with varying input sizes, and cluster configuration parameters. As a result, we present Keddah, a toolchain for capturing, modelling and reproducing Hadoop traffic, for use with network simulators. Keddah can be used to create empirical Hadoop traffic models, enabling reproducible Hadoop research in more realistic scenarios.",

author = "Jie Deng and Gareth Tyson and Felix Cuadrado and Steve Uhlig",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017 ; Conference date: 05-06-2017 Through 08-06-2017",

year = "2017",

month = jul,

day = "13",

doi = "10.1109/ICDCS.2017.211",

language = "English",

series = "Proceedings - International Conference on Distributed Computing Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2143--2150",

editor = "Kisung Lee and Ling Liu",

booktitle = "Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017",

}

Deng, J, Tyson, G, Cuadrado, F & Uhlig, S 2017, Keddah: Capturing Hadoop Network Behaviour. in K Lee & L Liu (eds), Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017., 7980162, Proceedings - International Conference on Distributed Computing Systems, Institute of Electrical and Electronics Engineers Inc., pp. 2143-2150, 37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017, Atlanta, United States, 5/06/17. https://doi.org/10.1109/ICDCS.2017.211

Keddah: Capturing Hadoop Network Behaviour. / Deng, Jie; Tyson, Gareth; Cuadrado, Felix et al.
Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017. ed. / Kisung Lee; Ling Liu. Institute of Electrical and Electronics Engineers Inc., 2017. p. 2143-2150 7980162 (Proceedings - International Conference on Distributed Computing Systems).

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

TY - GEN

T1 - Keddah

T2 - 37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017

AU - Deng, Jie

AU - Tyson, Gareth

AU - Cuadrado, Felix

AU - Uhlig, Steve

PY - 2017/7/13

Y1 - 2017/7/13

N2 - As a distributed system, Hadoop heavily relies on the network to complete data processing jobs. While Hadoop traffic is perceived to be critical for job execution performance, the actual behaviour of Hadoop network traffic is still poorly understood. This lack of understanding greatly complicates research relying on Hadoop workloads. In this paper, we explore Hadoop traffic through experimentation. We analyse the generated traffic of multiple types of MapReduce jobs, with varying input sizes, and cluster configuration parameters. As a result, we present Keddah, a toolchain for capturing, modelling and reproducing Hadoop traffic, for use with network simulators. Keddah can be used to create empirical Hadoop traffic models, enabling reproducible Hadoop research in more realistic scenarios.

AB - As a distributed system, Hadoop heavily relies on the network to complete data processing jobs. While Hadoop traffic is perceived to be critical for job execution performance, the actual behaviour of Hadoop network traffic is still poorly understood. This lack of understanding greatly complicates research relying on Hadoop workloads. In this paper, we explore Hadoop traffic through experimentation. We analyse the generated traffic of multiple types of MapReduce jobs, with varying input sizes, and cluster configuration parameters. As a result, we present Keddah, a toolchain for capturing, modelling and reproducing Hadoop traffic, for use with network simulators. Keddah can be used to create empirical Hadoop traffic models, enabling reproducible Hadoop research in more realistic scenarios.

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:000412759500213

UR - https://www.scopus.com/pages/publications/85027254859

U2 - 10.1109/ICDCS.2017.211

DO - 10.1109/ICDCS.2017.211

M3 - Conference Paper published in a book

T3 - Proceedings - International Conference on Distributed Computing Systems

SP - 2143

EP - 2150

BT - Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017

A2 - Lee, Kisung

A2 - Liu, Ling

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 5 June 2017 through 8 June 2017

ER -

Deng J, Tyson G, Cuadrado F, Uhlig S. Keddah: Capturing Hadoop Network Behaviour. In Lee K, Liu L, editors, Proceedings - IEEE 37th International Conference on Distributed Computing Systems, ICDCS 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 2143-2150. 7980162. (Proceedings - International Conference on Distributed Computing Systems). doi: 10.1109/ICDCS.2017.211

Keddah: Capturing Hadoop Network Behaviour

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this