Performance Testing for Cloud Computing




He, Sen

Journal Title

Journal ISSN

Volume Title



Cloud computing is increasingly popular due to its availability, elasticity, and cost-efficiency. Ideally, cloud users expect that the overall usage cost can be reduced by porting their applications to cloud platforms. However, the reduced amount heavily depends on the users' knowledge and familiarity of their application's performance on the cloud. To maximize the cost-benefit, users need to evaluate their applications' performance on the cloud before porting. Due to the multitenancy and black-box nature of cloud platforms, it's extremely difficult for users to accurately obtain the performance of their cloud applications. Usually, to ensure the accuracy of performance results, cloud users choose to test the applications' performance by extensively executing their applications on the cloud for a long time, however, it's time-consuming and cost-prohibitive. To help users to accurately obtain Infrastructure-as-a-Service (IaaS) cloud applications' performance without undue extra test runs, we present PT4Cloud and Metior which are based on non-parametric statistical approaches. PT4Cloud can provide reliable stop conditions to obtain highly accurate performance distributions, and Metior can help users to obtain single points of estimates at higher accuracy and lower testing cost. To demonstrate the effectiveness of PT4Cloud and Metior, six benchmarks from different application domains are evaluated on Amazon EC2 and Chameleon Cloud. In total, 33 benchmark configurations are executed over nine weeks to evaluate PT4Cloud and Metior. Both the two methodologies have been demonstrated to work properly on IaaS clouds. As non-parametric statistical tools, PT4Cloud and Metior are general enough to handle different types of cloud services. To demonstrate this, we also used PT4Cloud and Metior to explore the performance of serverless cloud platforms. Serverless cloud platforms, also known as Function-as-a-Service(FaaS), are growing in popularity due to rapid provisioning, easy scalability, and pay-as-you-go pricing model. However, performance testing is more difficult on FaaS clouds, as the resource abstract level of FaaS Clouds is higher than that of IaaS clouds, and users have less control over the execution environments. The performance fluctuations of FaaS clouds are from resource contention during the application execution stage and from random cold startup time during the environment initiation stage. Based on the observations, we developed PT4Cloud+ to measure the performance of FaaS cloud applications and the effectiveness of FaaS auto-scaling policies.


This item is available only to currently enrolled UTSA students, faculty or staff. To download, navigate to Log In in the top right-hand corner of this screen, then select Log in with my UTSA ID.


Cloud computing, Performance testing, Cloud platforms



Computer Science