Performance Testing for DevOps

In this blog post, I would like to discuss the need for performance testing within DevOps practices and how a practice such as continuous performance testing can add tremendous value to the software development and delivery process.

Note: If you are looking for technical details on how to build a continuous performance testing framework for your CI/CD pipeline. Please read my white paper building a continuous performance testing framework in there I have shared all technical details and advice to avoid pitfalls.

In general, the need for performance testing mainly arises for the following reasons.

1. Many enterprises have increased the frequency of application deployments through Agile and DevOps best practices. With the increased velocity comes the risk of quality and particularly performance related issues and bottlenecks creeping into your applications unnoticed.
2. Moreover, many enterprises are migrating their applications to public cloud infrastructures such as AWS, Azure or Google Cloud. As there are a lot of moving parts within these public cloud infrastructures, it’s important to realize that any changes these cloud providers make to their components such as load balancers, virtual machines or any others can impact the performance of your applications as well. Most cloud migration journeys had a lot of surprising issues related to the performance of applications. So, it’s very important to keep an eye on performance especially when your applications are hosted in the cloud environment.
3. Most of the customer-facing applications implement A/B testing or experiments to increase conversions. Particularly when your business teams are launching experiments without the need for the IT team, sometimes these experiments can impact the performance of the application.

Before we look at a solution to address these needs, let’s ensure the difference between load/stress testing vs continuous performance testing. Load/Stress testing is mostly performed on applications or platforms for the following main reasons.

• Performance fine-tuning of applications, platforms or databases and capacity planning to support upcoming major promotional events or sale days such as Black Friday or Cyber Monday.
• An application that is newly launched for the first time or major feature release or applications being migrated from internal data centers to public cloud service need load/stress testing to ensure the application can withstand the expected user traffic.

Mostly Load/Stress tests uncover performance issues that are experienced during heavy traffic by putting the system under stress.

However, continuous performance testing involves developing and deploying low volume load tests to capture performance metrics of critical application user flows, API end points and database queries of your systems without stressing them too much. These tests are deployed to execute on developer’s check-ins in a DevOps CI/CD pipelines and scheduled to execute on daily basis in your production environment or a similar test environment.

The key focus is to capture performance metrics such as latency, throughput, bandwidth, Time to First Byte (TTFB), application http response codes (404, 500), CPU, Memory, Disk I/O and others through your continuous performance tests. Doing this will help you to identify immediate performance issues on developer check-ins and over the time performance degradations, location-specific latency issues, and other errors.

Here is a diagram that show where your capture performance test can be deployed in your CI/CD pipeline.

In order to implement continuous performance testing effectively you need to make sure the following are already in place.

1. Have a list of your application’s critical pages, end to end user flow scenarios, API end points and database queries that you would like to test, measure and analyze on regular basis as automated performance test scripts.
2. You must establish baselines performance for the above list, by conducting dedicated load/Stress tests of your application, APIs and databases. Example: If your application is an e-commerce market place, having performance baselines for your critical pages such as home page, product details page, search page, and critical end to end user flows such checkouts, registration and critical actions such as add to cart, adding payment details are important.
3. It is important that you can execute your performance tests on daily basis in production environment along with real users’ traffic to capture performance metrics. Often performance metrics that are captured in environments other than productions differ because of various factors. But, ability to run your low volume performance tests on production will help you identify issues related to constant changes such A/B testing.

Once you have ensured the above prerequisites are taken care. You need a technical solution, a continuous performance testing framework that support development, maintenance and executions of your performance test scripts at any time and from anywhere. Your framework should focus on capturing the performance metrics of your critical pages, end to end user scenarios and critical user actions and send the data to central reporting infrastructure. Your framework should also analyze, report and provide dashboards on key performance metrics as shown below.

The benefits of doing continuous performance testing as part of your CI/CD process are,

1. Key performance metrics of your Web Applications, APIs and Database Queries are continuously captured and reported to uncover any performance issues or degradation trends.
2. When these small-scale load tests focused on capturing performance metrics are executed on developer check-Ins provides quick feedback to developers and help them to resolve any performance issues or bottlenecks in their code at a very early stage in development life cycle. We understand the cost of fixing bugs is cheaper in early stages than in production.
3. When these performance tests are deployed to capture and report metrics against baselines, it can help you identify latency issues, over the time performance degradations, location specific latency issues and other errors.
4. Continuous performance tests are very helpful for cloud hosted applications and services to measure long term stability and site performance on regular basis.

Kindly let us know if you have any further feedback and comments about performance testing as part of DevOps CI/CD process. Also, please read my white paper building a continuous performance testing framework if you are intrigued by this discussion.

Softcrylic help our clients in load/stress and continuous performance testing. We provide end to end consulting, design, development and implementation of performance testing as a part of CI/CD process. Please reach out to us if you need more information or help to build a continuous performance testing framework.

Read more from our Test Engineering & Operations practice here.