The statistical properties of host load

Article type: Research Article

Authors: Dinda, Peter A.

Affiliations: Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Tel.: +1 412 268 3077; Fax: +1 412 268 5576; E‐mail: pdinda@cs.cmu.edu

Abstract: Understanding how host load changes over time is instrumental in predicting the execution time of tasks or jobs, such as in dynamic load balancing and distributed soft real‐time systems. To improve this understanding, we collected week‐long, 1 Hz resolution traces of the Digital Unix 5 second exponential load average on over 35 different machines including production and research cluster machines, compute servers, and desktop workstations. Separate sets of traces were collected at two different times of the year. The traces capture all of the dynamic load information available to user‐level programs on these machines. We present a detailed statistical analysis of these traces here, including summary statistics, distributions, and time series analysis results. Two significant new results are that load is self‐similar and that it displays epochal behavior. All of the traces exhibit a high degree of self‐similarity with Hurst parameters ranging from 0.73 to 0.99, strongly biased toward the top of that range. The traces also display epochal behavior in that the local frequency content of the load signal remains quite stable for long periods of time (150–450 s mean) and changes abruptly at epoch boundaries. Despite these complex behaviors, we have found that relatively simple linear models are sufficient for short‐range host load prediction.

Keywords: host load properties, host load prediction, self‐similarity, long‐range dependence, epochal behavior

Journal: Scientific Programming, vol. 7, no. 3‐4, pp. 211-229, 1999