Affiliations: [a] University of Pennsylvania, Philadelphia, PA 19104, USA. E-mail: jsonch@cis.upenn.edu | [b] United States Naval Academy, Annapolis, MD 21402, USA. E-mail: aviv@usna.edu
Abstract: Many network security systems analyze large scale data collected from multiple collaborating domains or aggregated network vantage points. Scale is clearly beneficial for these systems, however it also makes them difficult to design and test. Large scale data sets can be difficult to acquire and may not contain important meta-information (e.g. ground truth). Further, their limited availability can make it extremely difficult to understand how well experimental results would reproduce in different conditions, or at different networks. In this article, we discuss using simulation to overcome these challenges. We present an augmented version of LESS, our recently proposed agent based simulator for evaluating large scale network security systems. LESS uses publicly available data sets and high level parameters to generate synthetic traffic that models large scale, multi-network scenarios. Essentially, LESS allows researchers to “scale up” the data and statistics about networks and attacks that they have access to, so that they can be used to test large scale network security systems. Researchers can also tune LESS’s high level parameters to better understand the sensitivities of their systems, and the reproducibility of their results. The version of LESS that we discuss in this article is extended to allow researchers to study an additional factor of system performance related to reproducibility: deployment location; by modeling the global Internet topology at the Autonomous System level. We demonstrate the applicability and benefits of LESS by tuning it with publicly available traces and then using generated records to reproduce and extend results from several recently proposed large scale security systems. In new experiments, we use LESS to study how deployment location affects large scale security systems. Our results demonstrate that LESS can evoke realistic performance from these systems with minimal tuning and provide insight into the network and topological factors that may affect the reproducibility of their evaluations.
Keywords: Data challenges, large scale security, simulation, agent based, stochastic
