Performance tests (part 2): What can you expect from a performance test?

Performance tests sometimes have a myth around them, depending on the personal experience people have made with them. For some it’s just a waste of time and effort, because the last test they conducted has not delivered any value. For others it’s the best since sliced bread.

With performance tests it’s the same as with every test: You need to ask a question and model an appropriate test case, then you get an answer for it. A performance test cannot tell you things about your system, which you have not asked. Especially it cannot give you any guarantee, that you system is rock-solid and fast (even if you ask this question :-))
It is not a silver bullet for any problem. But it is an important step to a point where you can start to trust your system to function properly.

So, what’s this “asking”? In Performance testing you ask questions by designing and executing a test case. When your test case is like this “Execute 100 requests to this specific URL with 10 concurrent threads, the response should be received within 500 milliseconds, no errors are allowed”, the question could be like this “Can my system copy with 10 concurrent users and respond to them within 0.5 seconds with no  errors”. The execution of the performance test can then answer the question with “yes” or “no”, depending on the outcome.

If the outcome is “no”, we obviously have to change this and work on our systems and our code to get the answer we are aiming for: “ok”. But our success is limited then. Even in the “ok” case we cannot infer, that with 15 concurrent users the system is still working ok, but slower than with 10 concurrent requests. This is an assumption we often do because we have only a limited set of validated results, and technically we should validate them all. So every test case answers exactly one question, gives us only one piece of information about the performance behavior of the system. For all other questions we have to create different performance test cases. (There will be a follow-up article on this topic…)

So, as a conclusion, the quality of the results of your performance result is directly depending on the questions you ask. For this reason you should invest a good amount for the preparation of your performance tests to define the right questions.