Are we performance testing the right way?

High performing applications

With increasing competition and decreasing patience levels, performance has slowly and steadily become one of the most important aspects of any application. Performance in many cases can be a deal breaking concern.

Performance testing is primarily needed if we want to monitor the behaviour of an application under varying load and resource scenarios. There are various flavours of performance testing, however not all flavours are applicable at the same time. While most of the testing needs are met through the load or endurance testing. There might be few scenarios where advanced tests like a stress test or volume test may be needed.

What exactly do we need to test?

• Not every feature of an application experiences load. For instance for a use-case having frequency of less than once a minute, a routine manual or automation test should be able to identify performance bottle necks. Hence preparing a load test environment for the same is an overkill.
• Is low performance actually unacceptable? Not all features need to respond back in fraction of a second. Transaction including external system integration, like a central monitoring or validation system, or operations needing bulk conversion activities etc, require the extra time to ensure safety and security. Performance may not be a deal breaker here.
• Business vision: Some auxiliary features like banners or logos may be used in a higher frequency, however they may not be of importance from a business stand point. Performance testing of such features may not be the need of the hour.

How much is sufficient?

Testing with current load:

• Defining the unit of load is paramount priority when testing the performance. While the user base may go from hundreds to millions, its the most granular concurrency that should be considered.
• The hour, minute or second having the peak should be considered to identify the testable peak.
• If hundred user base show a ten transaction load at any given second, it should be tested with a higher load, as opposed to a thousand user base having not more than five transaction load at any given second.

Testing with extended load:

• Load extension should be inline with the vision of the enterprise. If the organisation has a vision for doubling its user-base on a visible time frame, a peak second load can be extended to 1.5 times the current load.
• A newly launched feature that is heavily marketed can see a spike in load with time. However the growth of the established featured would be gradual and phased. Hence an extended load test might be applicable in the first case and not in the second.

What test is applicable for you?

• Load test: Any application that can expect a load, which cannot be manually tested by a couple of testers is a candidate for load test. The primary objective of a load test is to identify the sanity of both the application and the environment with the anticipated load.
• Stress test: while a load test is sufficient to understand how the application will perform under the current load, a stress test will help identify how future ready the application is. It essentially identifies the breaking point of the application. Like we discussed earlier, a stress test should always be inline with the business vision of growth.
• Soak test: A soak test requires extended duration of test, for the expected load. Endurance test should be always inline with the business plan. If there are peak windows of only couple of hours during a day, a soak test of beyond that will not add any value. Ideally soak test should be performed with an average load and not a peak load, as the peak does not exist for the extended duration and hence can result in false failures.
• Spike test: A spike test is similar to a stress test, only difference being the load is suddenly introduced in the system and not added in a phased manner. A spike test can help in identifying the systems capability to balance load, in cases on an artificially introduced load, or a brut force attack. Both stress and spike test should emphasise on system stability rather than time taken.
• Volume testing: Unlike stress test, which focuses on an increasing user load of an application, volume test focuses on the stored data. Volume test is rather an indication of the efficiency of the underlying data store. In case of applications which do not own or persist data, a volume test might not be a correct use case.
• Scalability testing: This testing should primarily focus on horizontal scaling of application as per configuration, with planned increase in load. The primary intention is to validate the scaling theory proposed with increasing load. Scalability typically need not be an independent test of its own. It should ideally should be analysed as a part of the load testing cycle.

Attributes to test:

• Speed: This indicates the time taken for completing a well defined transaction. The identified transaction should not typically involve any human intervention. A typical candidate for speed test could be

Time taken for dash board to load, post submitting the password. Here the human factor to enter the username and password is eliminated

Time taken for congratulations page to load post submitting the transaction

• Scalability: This indicates the applications ability to add or release resources when the load on system fluctuates. Scalability can be measured for hardware, software or database, whatever is configured to be scaled.
• Stability: Stability is basically the sustenance of all monitored performance measures for an extended period of time. It basically ensures that the application performs satisfactorily for an entire business day, or season as applicable.
• Reliability: It is one of the key measures of any application. It ensures the correctness of the application throughout. It basically means it’s okay for an application to fail once in a while on other measures, however never on reliability.

Steps to follow:

For any performance testing activity to be successful or beneficial, a structured approach is essential:

Planning: A detailed plan need to be identified before are testing activity is resumed. At the end of this exercise the team should have:

• List of all identified scenarios
• Agreed load to be tested
• Agreed duration to be tested
• Acceptable measure ranges in terms of all attributes mentioned above
• Revalidation interval

Development: This step involves creating scripts and data relevant for testing.

• Identify and create all relevant data required for testing.
• Identify and create data flow within the application
• Map the testing scenario with its manual co-relation in the system
• Ensure appropriate data and application flows in terms of serial/parallel invocation. querying from database servers, cache’s or other complex operations as expected from the application

Execution: Ensure the test cases are executed with comparable environment where the application would otherwise run.

• Appropriate hardware and authentication mechanisms
• Appropriate load balancing configurations
• Adequate logging and monitoring support (also factor for overheads created due to additional monitoring needs)

Analysis: The benefit of the whole exercise lies in the effectiveness of the analysis framework.

• Identify performance breaches, along with their severity and frequency
• Mechanism to analyse the root cause of observed deviations.
• Tools to report and visualise captured logs.

Engineering mindset

Performance testing is a function of performance engineering. The output of the test should be able to feed into the performance tuning activity and overall improve the performance and experience of the application.

Performance engineering is everyone’s business, wether you are a developer, tester or product owner or any stakeholder aiming to make the product usable.

Happy performance tuning!!