If you ask two software testers, you will get three different definitions for each software testing term. This article discusses spike testing and how this type of testing relates to performance and load testing.
Author: Franco Martinig, Martinig & Associates
Some Software Testing Definitions
Spike, performance and load testing are three types of tests used to assess non-functional capabilities of the applications and websites under test, mainly on the server side. Performance testing is primarily concerned by the speed of availability of the application for the end-user. In the case of a website, software quality assurance teams will use tools like WebPageTest to detect issues related to the size of images or the time needed to reach for external resources like Google analytics scripts. Results might vary depending on the usage of browser caches or the distance between the user and the application resources.
Load testing is used to assess the behavior of the system when it is used simultaneously by many users. Software testers use online testing platforms that can reproduce the access of the applications by a certain number of virtual users with different mobile devices or browsers. Issues detected will range from the physical specifications of the servers like the disk or virtual memory size to the software concurrency problems that can occurs in writing log file or table access in a database. Software Testing Magazine has created a list of free online load testing tools available to perform this activity.
Spike testing is related to load testing, but it focuses on sudden and important increases in the application usage, a «spike». This situation can happen at an expected time like the start of ticket selling for a music festival, an online casino promotion such as the 888casino welcome offer or the Black Friday sale. A spike in traffic can also happen unexpectedly when some of your content reaches the front page of discussions websites like Reddit or hacker news.
A spike can also be created for bad reasons if somebody launches and denial of service attack on your website. This is a case where load testing meets server security. You should be able to make the differences between the normal user requests on your server and the traffic that is sent to break it. It is easier if the bad requests come only for a limited number of attacking IPs, but it is more difficult for a Distributed Denial of Service (DDOS) attack that try to imitate regular server requests.
A huge spike in traffic can have many bad consequences for your system, besides the loss of reputation and revenues caused by customers unable to buy your products. You can end up also with data losses or file corruption that can disrupt your operations on the long term.
Spike Testing Best Practices
What are the best practices to perform spike testing? The first one that is obviously that you should avoid performing spike testing on a production system as it might result in serious issues for your end-users or customers. You should have a dedicated environment for testing that reproduces the hardware and software characteristic of your production system. There are some examples of organizations testing in production, but you should avoid doing this if you don’t have the level of technological sophistication that allows you to get immediately back to normal when things go bad.
Defines realistic testing scenarios to achieve meaningful results. If you are selling tickets for a rock festival, your tests scripts should reproduce a full transaction starting from the home page to creating the ticket after a credit card payment for example. It is important that the pace of the actions are realistic and you are not just simply creating a huge number of simultaneous requests. A normal user will insert some «wait time» between the different steps of its transaction. If the user can follow different paths (for instance starting as an existing user that will just login or as new user that needs to register an account), this should be reflected in your scenarios. It is also important to cover the features that are most seldom used, like a user password recovery for instance. Knowing the past usage pattern of the users of your applications will help you create realistic scenarios.
Set up your monitoring. You will calibrate the external characteristics of your spike tests (number of users, requests scenarios, duration) in the tool used for automated testing, but you want to be able to follow closely how all the different parts of your system, both hardware and software, behave during the test. You have to gather data on CPU and memory usage or specific database status like file IO, buffer cache hit ratio or lock waits. It is also important to measure how long the system takes to recover after the spike.
Define some acceptable metrics threshold. If your application or website is not broken by your spike testing, you want to know how much your system is impacted with metrics like the average memory and CPU utilization or the time for web page loading. This will make you aware of your safety margin in the current context. If you see some evolution in your activity between the previous spike and the next one, you might want to act accordingly to preserve this margin. If your system was working at 80 % of capacity during the last Black Friday and you enjoy a 30 % growth of normal traffic during the following year, you might want to increase your treatment capacity so that you will not have a bad surprise in the upcoming November. Additionally, spikes also impact your physical organization, like the number of printers or people available to ship orders.
Conclusion
Performing spike testing might not be important if you just have a corporate website that inform about the activities of your organization, but stressing your systems with spike testing is a must if you work in the e-commerce domain or if your business has or create some promotional or unique traffic generating events. If you don’t do it, you might incur a painful loss of reputation, revenues and customers when your application will not be able to perform effectively under very high traffic, or even worse, simply crash.
Additional Spike Testing Resources
Performance Testing, Load Testing & Stress Testing Explained
The Importance of Stability and Reliability Testing in Software Development
Stress Testing vs. Soak Testing vs. Spike Testing: Best Practices
How to perform Spike Testing: Best practices, tools, and examples