Safe Software Testing in Production

Performing software testing in production is not a taboo anymore in our era of cloud computing and DevOps. This article discusses different types and approaches to do software testing in production and associated contexts, during which one or another kind of functional testing is useful. Also, we will describe in detail about the effective organization of the safe testing of various services in production.

Author: Nataliia Syvynska, TestMatick, https://testmatick.com/

It should be mentioned from the very beginning that content of this article concerns only those systems and services which delivery is managed by software developers. Also, it should be highlighted that use of any kind of testing available here – is a very difficult task, that constantly needs a lot of changes in methods for testing, software creation, and designing.

Let’s start.

Why do we need testing in production if it can be easily transferred to staging?

Different QA engineers understand staging cluster in different ways. For some web studios and companies, software testing on staging is a compulsory stage that officially verifies the final release of the product.

Many famous organizations and companies consider staging as a small copy of the workspace. In this case, developers need to provide and synchronize the processes. Moreover, they have to provide a function of the distinct increments of stateful system (for example, a database). Only confidential information can be an exception here. This information can help to get to know the personality of the real user (it is necessary for compliance with the requirements of  HIPAA and  PCI regulations).

The main problem of this approach is the need to use a different database which includes some real information about production space. Traditionally, the difference extends to such moments:

• The volume of the staging cluster;
• The fact that on the staging, a cluster of small size is used, so, parameters of configuration during testing will be different from what will be used for production;
• The number of operational connections which are performed by the stateless command, or method of reuse of TCP-connection by the special proxy server;
• Deficiency of monitoring process on staging.

All differences that were listed above cannot be considered as very serious arguments against application staging, as opposed to the anti-pattern system, which should not be faced with. But, sincere intention to do everything quickly and qualitatively requires plenty of engineers’ work. Production environment constantly changes and is influenced by many factors, so that attempting to reach a universal decision for quality assurance on the production leads nowhere.

Philosophy of testing in production

Currently, the term “testing in production” is inextricably related to some stereotypes and negative connotations (“harmful programming”, unit tests deficiency, tester’s negligence concerning the perception of product functionality, etc.).  Testing the web product functionality in production has a similar reputation, if it is done carelessly and not accurately. It cannot replace traditional testing before the release in production and of course, it cannot be considered as a simple task. Moreover, it is possible to assert that positive and qualitative testing at the stage of production requires some factors such as a special level of the knowledge in the automation field, a good understanding of long-present practices, as well as an opportunity of systems designing with sample orientation to the given type of testing.

In order to organize a safe and productive process of testing web product in production, it is important not to consider this as a generic term that means a simple set of techniques and tools. So, the concept “testing in production” includes a lot of techniques and tools which often are applied in three main stages. Which ones? We will discuss this later.

Three stages of production

Traditionally, discussions concerning production exist exclusively in the context of the development of ready functionality in production, the process of monitoring or different urgent situations when something went wrong.

Stage 1. Development

When the  process of testing is the way to check the current achievements by comparing the best ratings, the accuracy of tests is guaranteed only if that method of checks is maximally identical with the way of real web product application in the release.

In other words, testing should be executed in the environment which simulates a working context as much as possible.

And the best imitation of a production workspace is an actual workspace. For all tests in a workspace, it is necessary that positive result of each of them does not touch the final user. But it is possible only if the product will be developed in a workspace, and the user cannot interact with parameters of the tested product.

Of course, potential opportunity to monitor software in a real environment without access of final user has plenty of advantages, when it comes to testing. First, this decreases the necessity to support separate environments for tests, development, and staging, which should be constantly synchronized with production.

Stage 2. Release

Release is a process that directs a production environment to entirely new software modification. Paying attention to all this, potential risks that are connected with the shipping of new threads also relate to the new software release.

At the stage of software development or pre-production, we can easily perform software builds just every hour or to send automatically potential launches of previous tests passing. For big programs and services which are oriented to customers, it is possible to start the release only from one cluster, and then to increase gradually its scales, until all clusters are updated.

It is important! – For the most important objects of web infrastructure, the term of their implementation can be easily increased to a few days, depending, for example, on the geography of potential users.

In such context, the words “release” and “launch” have the same meaning, and they can be easily “connected” into a simple concept – development (for example, canary deployment or blue-green deployment).

Moreover, unsuccessful release can become the catalyst of small or big break-downs in the work of already set service. At this stage, it is important to do hotfix or backhitch in time, if it is clear that the new version of the product doesn’t function correctly.

The period of launch works best when it is maximally automated and is being done incrementally. Similarly, as failure or service hotfix are really useful when the periodicity of the errors and the amount of demands corresponds automatically with database ratings.

Stage 3. After release completion

If the release period was smooth and new product version processes data inside the production environment without failures, we can consider it successful and positive. After a successful release, the new stage comes which can be characterized as “after release”.

Any, even the most difficult system, gradually loses its productivity. It will be logical to assume: it doesn’t mean that it can be done without no failure or hotfix. Instead, it is necessary to find such deterioration and to do debugging manually if this is possible. Because of this, testing of release seems to be a not classical procedure, but banal debugging or process of collecting the analytical data.
It is a common opinion that any component of the system should be developed recognizing the fact that no big and modern system functions perfectly at 100%. But, it is important to remember that any failure should be taken into account at any stage of assembling, testing or monitoring the software functioning.

Integration testing as the kind of the most effective software check in production

Traditionally, integration testing is done by the continuous integration server in a specially allocated environment for each development branch. Copy of all service topology is also displayed for test tasks of all services which potentially can work jointly. And that is not very effective. First, there is no way to extend the test environment to be similar to the current production environment, even if tests start in the same Docker container which is applied in production. Basically, this is true, that test suites can be started in the test environment.

It doesn’t matter how the test is done, the most likely, it executes only one connection to overhead service (Docker) if it is in the production environment, where it can process few parallel connections at the same time (as a rule, TCP-connections).

Mostly, such problems arise because, after launch, more than half of tests develop new database or space of cache-keys on that unit, where this test is done. With the help of this kind of software testing, it can be shown well that system functions reasonably for the specific inquiry.

An integrated approach is not really effective at imitation of big, well-known types of failures, to say nothing of varieties of partial system failures. There is even the kind of exhaustive testing, which helps to anticipate beforehand unpredictability of performance of some command since it is impossible to find this out during the complex check of the whole system.

But all this doesn’t mean that integration testing is not effective. It is possible only to add that application of integration testing in the artificial environment doesn’t make sense. It should be examined, in order to know accurately that a new series of launched product:
* doesn’t interfere in any way the interaction with overhead services;
* doesn’t influence negatively on purpose and the task which are necessary to overhead services.

The first necessity is reached with the aid of contract testing. It can help to ensure the correct functioning for interfaces at the stage of pre-production, where the development of all service topology is not required.

Conclusion

This article can be summarized in this way: software testing in production can seem to be a very difficult and time-consuming task which goes outside the most QA specialists’ area of technical knowledge. Even so, such testing cannot be considered as an enjoyable activity, because it is connected with plenty of risks. However, if you do it following all prescriptions and rules, you can achieve 100% of reliability in the functioning of distributed systems which today can be met practically everywhere!