Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints

Wei Wang; Baochun Li; Ben Liang; Jun Li

doi:10.1109/SC.2016.85

Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints

Wei Wang, Baochun Li, Ben Liang, Jun Li

Department of Computer Science and Engineering

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

43 Citations (Scopus)

Abstract

Providing quality-of-service guarantees by means of fair sharing has never been more challenging in datacenters. Due to the heterogeneity of machine configurations, datacenter jobs frequently specify placement constraints, restricting them to run on a particular class of machines meeting specific hardware/software requirements. In addition, jobs have diverse demands across multiple resource types, and may saturate any of the CPU, memory, or storage resources. Despite the rich body of recent work on datacenter scheduling, it remains unclear how multi-resource fair sharing is defined and achieved for jobs with placement constraints. In this paper, we propose a new sharing policy called Task Share Fairness (TSF). With TSF, jobs are better off sharing the datacenter, and are better off reporting demands and constraints truthfully. We have prototyped TSF on Apache Mesos and confirmed its service guarantees in a 50-node EC2 cluster. Trace-driven simulations have further revealed that TSF speeds up 60% of tasks over existing fair schedulers.

Original language	English
Title of host publication	Proceedings of SC 2016
Subtitle of host publication	The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	IEEE Computer Society
Pages	1003-1014
Number of pages	12
ISBN (Electronic)	9781467388153
DOIs	https://doi.org/10.1109/SC.2016.85
Publication status	Published - 2 Jul 2016
Event	2016 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016 - Salt Lake City, United States Duration: 13 Nov 2016 → 18 Nov 2016

Publication series

Name	International Conference for High Performance Computing, Networking, Storage and Analysis, SC
Volume	0
ISSN (Print)	2167-4329
ISSN (Electronic)	2167-4337

Conference

Conference	2016 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016
Country/Territory	United States
City	Salt Lake City
Period	13/11/16 → 18/11/16

Bibliographical note

Publisher Copyright:
© 2016 IEEE.

Keywords

Cluster schedulers
fairness
multi-resource allocation
placement constraints

Access to Document

10.1109/SC.2016.85

Cite this

Wang, W., Li, B., Liang, B., & Li, J. (2016). Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints. In Proceedings of SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1003-1014). Article 7877164 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 0). IEEE Computer Society. https://doi.org/10.1109/SC.2016.85

Wang, Wei ; Li, Baochun ; Liang, Ben et al. / Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints. Proceedings of SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2016. pp. 1003-1014 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

@inproceedings{cf7df04f08a24bcb96a62d880169426b,

title = "Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints",

abstract = "Providing quality-of-service guarantees by means of fair sharing has never been more challenging in datacenters. Due to the heterogeneity of machine configurations, datacenter jobs frequently specify placement constraints, restricting them to run on a particular class of machines meeting specific hardware/software requirements. In addition, jobs have diverse demands across multiple resource types, and may saturate any of the CPU, memory, or storage resources. Despite the rich body of recent work on datacenter scheduling, it remains unclear how multi-resource fair sharing is defined and achieved for jobs with placement constraints. In this paper, we propose a new sharing policy called Task Share Fairness (TSF). With TSF, jobs are better off sharing the datacenter, and are better off reporting demands and constraints truthfully. We have prototyped TSF on Apache Mesos and confirmed its service guarantees in a 50-node EC2 cluster. Trace-driven simulations have further revealed that TSF speeds up 60\% of tasks over existing fair schedulers.",

keywords = "Cluster schedulers, fairness, multi-resource allocation, placement constraints",

author = "Wei Wang and Baochun Li and Ben Liang and Jun Li",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016 ; Conference date: 13-11-2016 Through 18-11-2016",

year = "2016",

month = jul,

day = "2",

doi = "10.1109/SC.2016.85",

language = "English",

series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",

publisher = "IEEE Computer Society",

pages = "1003--1014",

booktitle = "Proceedings of SC 2016",

}

Wang, W, Li, B, Liang, B & Li, J 2016, Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints. in Proceedings of SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis., 7877164, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, vol. 0, IEEE Computer Society, pp. 1003-1014, 2016 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016, Salt Lake City, United States, 13/11/16. https://doi.org/10.1109/SC.2016.85

Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints. / Wang, Wei; Li, Baochun; Liang, Ben et al.
Proceedings of SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2016. p. 1003-1014 7877164 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 0).

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

TY - GEN

T1 - Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints

AU - Wang, Wei

AU - Li, Baochun

AU - Liang, Ben

AU - Li, Jun

PY - 2016/7/2

Y1 - 2016/7/2

N2 - Providing quality-of-service guarantees by means of fair sharing has never been more challenging in datacenters. Due to the heterogeneity of machine configurations, datacenter jobs frequently specify placement constraints, restricting them to run on a particular class of machines meeting specific hardware/software requirements. In addition, jobs have diverse demands across multiple resource types, and may saturate any of the CPU, memory, or storage resources. Despite the rich body of recent work on datacenter scheduling, it remains unclear how multi-resource fair sharing is defined and achieved for jobs with placement constraints. In this paper, we propose a new sharing policy called Task Share Fairness (TSF). With TSF, jobs are better off sharing the datacenter, and are better off reporting demands and constraints truthfully. We have prototyped TSF on Apache Mesos and confirmed its service guarantees in a 50-node EC2 cluster. Trace-driven simulations have further revealed that TSF speeds up 60% of tasks over existing fair schedulers.

AB - Providing quality-of-service guarantees by means of fair sharing has never been more challenging in datacenters. Due to the heterogeneity of machine configurations, datacenter jobs frequently specify placement constraints, restricting them to run on a particular class of machines meeting specific hardware/software requirements. In addition, jobs have diverse demands across multiple resource types, and may saturate any of the CPU, memory, or storage resources. Despite the rich body of recent work on datacenter scheduling, it remains unclear how multi-resource fair sharing is defined and achieved for jobs with placement constraints. In this paper, we propose a new sharing policy called Task Share Fairness (TSF). With TSF, jobs are better off sharing the datacenter, and are better off reporting demands and constraints truthfully. We have prototyped TSF on Apache Mesos and confirmed its service guarantees in a 50-node EC2 cluster. Trace-driven simulations have further revealed that TSF speeds up 60% of tasks over existing fair schedulers.

KW - Cluster schedulers

KW - fairness

KW - multi-resource allocation

KW - placement constraints

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:000405376800091

UR - https://openalex.org/W4245843336

UR - https://www.scopus.com/pages/publications/85017220109

U2 - 10.1109/SC.2016.85

DO - 10.1109/SC.2016.85

M3 - Conference Paper published in a book

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

SP - 1003

EP - 1014

BT - Proceedings of SC 2016

PB - IEEE Computer Society

T2 - 2016 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016

Y2 - 13 November 2016 through 18 November 2016

ER -

Wang W, Li B, Liang B, Li J. Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints. In Proceedings of SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2016. p. 1003-1014. 7877164. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). doi: 10.1109/SC.2016.85

Multi-resource Fair Sharing for Datacenter Jobs with Placement Constraints

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this