The overall performance test is usually grouped into smaller test chunks (I call them scenarios), which replicate certain aspects of the daily operation like:
- normal operation
- many people login into the system during the start of the business hours
- your marketing campaign got viral and now huge traffic spikes come over from Twitter/Facebook/G+/…
- how can your system handles traffic when some are down for maintenance?
- you are preparing a huge amount of new content on authoring
- (and many more …)
Collect the scenarios you know you will have, and detail them out. If you have historical data on them, use them. If you don’t have such data, start with rough estimations. You will adjust them afterwards anyway by any feedback of your production system.
A performance test scenario consists at least of these items:
- Objectives what should be tested (the “questions” of part 2); this objectives should detailled and clear, usually they are oriented on the aspect of the daily operations you just identified.
- A number of test cases, which are executed during the performance test; in most real-life situations multiple different activities are carried out in parallel, and your performance test scenarios should reflect that.
- Systems and environments, where the tests are executed and can deliver meaningful results.
- Facilities to monitor the systems, which are involved in the test execution; it should be described, which information is considered valuable for this test scenario and which should preserved for analysis.
- Instructions how to interpret the monitoring data gathered throughout the test execution.
- Instructions how the test should be executed.
- In the end there should be a clear answer if the objectives have been met.
All of the aspects should be covered. And now it is clear, why good performance test is nothing which can done without preparation and within 1 day. Such adhoc tests usually never deliver proper results, because they heavily focus on the test execution and less on proper preparation of the test cases and the interpretation of the results.
Even if you plan much more scenarios than you will ever execute: Just thinking of them, planning and defining them often gives you valuable input for your application design. As in many cases functional test planning makes you think about system, about expected behaviour and error handling. It’s the same here.
(Applying the TDD mechanisms to performance tests would be an interesting exercise …)
Things on a content management system 