Activities Involved in Workload Modeling

Performance testing is a complex activity as it consists of various phases and each phase has several activities in it. Workload modeling is one of the most important parts of the performance testing activity and it’s not simple by any means. Some of the activities necessary for identifying the performance test workload model are listed below:

1. Test Objectives Identification

Not just the performance testing but in any activity you put efforts which are aligned to your objectives. Identifying your test objectives means to examine what actions you need to take in order to successfully achieve those test objectives. So before we formally start working on any application’s performance testing, first step is to identify its test objectives in detail. For an E-commerce web application, following are some of the examples of performance test objectives:

• Response Time: Product search should not take more than 3 seconds.
• Throughput: Application server should have the capacity of entertaining 500 transactions per second.
• Resource Utilization: All the resources like processor and memory utilization, network Input output and disk input output etc should be at less than 70% of their maximum capacity.
• Maximum User Load: System should be able to entertain 1000 concurrent users by fulfilling all of the above defined objectives.

2. Application Understanding

Complete understanding of the AUT with all its features is the basic step in any testing activity. You can’t thoroughly test an application unless you have its complete understanding. Same is the case with the performance testing. Performance testing starts with planning and planning starts with application understanding. You explore the application from performance perspectives and try to get the answers of the following questions:

• How many types of users are using this application?
• What are the business scenarios of every user?
• What is the AUT current and predicted peak user load for all its users’ actions over time?
• How the user load is expected to grow with time?
• In how much time a specific user action will achieve its peak load?
• For how long the peak load will continue?

Performance testing teams can also collaborate with network team to check the server logs to get the answers of some of the above mentioned questions. They can also interview the marketing team and application stakeholders for some of the answers.

3. Key Scenarios Identification

It’s neither practiced nor required to simulate all user actions in performance tests due to the budget and time constraints. Performance testing teams always select limited number of user actions which have greater performance impact on the application. In one of our previous Defaultpapers (Load Testing Scenarios Identification) we have discussed this topic in detail. Following are the examples of few scenarios which should be selected while conducting the performance tests:

• Measureable Scenarios: A basic criterion for selecting any user scenario for performance testing is it should be fully measureable.
• Most Frequently Accessed Scenarios: Application scenarios which are mostly accessed by the users when they browse through the application.
• Business Critical Scenarios: Application core scenarios which contain its business transactions.
• Resource Intensive Scenarios: User scenarios which consume more resources as compared to typical scenarios.
• Time Dependent Frequently Accessed Scenarios: Such application scenarios which are accessed on specific occasions only but are accessed very frequently.
• Stakeholders Concerning Scenarios: Application features about which the stakeholders are more concerned such as AUT newly integrated modules.

Some of the most desired performance testing scenarios of an E-commerce application could be,

• Browsing product catalog
• Creating a user account
• Searching for a product
• Login to application
• Order Placement

4. Determining Navigation Paths of Key Scenarios

Once you have identified all the AUT scenarios which should be included in a performance test, next step is to figure out each scenario’s all possible paths which a user can opt to successfully complete it. Any application users most probably have different level of domain and technical expertise and it’s quite obvious that they will follow different steps to complete a specific scenario(s). Performance testing teams identify all possible paths which users could follow to successfully complete the identified scenario and also figure out the frequency of each path to decide whether it should be included in performance test or not? Application response for the same scenario can greatly vary depending upon user navigation path and it’s strongly advised to test the selected scenario’s all major paths under load. Following are the few guidelines which could be followed to identify any scenario’s navigation paths:

• Figure out AUT paths which can be used to successfully complete more than one identified scenarios and have major performance impact.
• Read manuals (design and user) to find out identified scenario’s all possible paths.
• In case of production application, check log files to find out the users’ navigation patterns to complete an identified scenario.
• Explore the application and try to find out all possible paths of a scenario by yourself.
• Another approach could be to provide application access to new and experienced users and ask them to complete certain scenarios and observe their actions.

5. Identify Unique Test Data

Unfortunately, identifying selected scenario’s all possible navigation paths doesn’t provide all the information required to successfully simulate them in a performance test. Several bits of information is still required to accurately simulate the workload model and preparing the required test data is one of them. You can never achieve accurate test results with improper or inefficient test data. One can develop an initial idea about the required test data by getting answers of the following queries:

• While navigating through a specific path, how much time a user spend on a page?
• Which conditions force the user to alter the navigation path?
• What input data should be required for a specific path?

You will also require a greater amount of test data and you need to keep an eye on the data base state if you wish to effectively run the test. Following are the few considerations which should be followed while executing a performance test where multiple navigation paths are tested for identified scenarios:

• Make sure that you have all the required test data as you will need more test data when you are testing scenarios with all navigation paths.
• Avoid using same test data for multiple users as it will produce invalid results.
• Periodically test the database status during the test execution and make sure it’s not overloaded.
• Include some invalid test data as well to simulate the real users’ behavior because users sometimes provide invalid values as well.

6. Relative Load Distribution Across Identified Scenarios

Now that you have understood the application, identified its key test scenarios, their navigation paths and test data required for each identified scenario, next step is to figure out the distribution of all the identified scenarios in your workload model. It’s quite obvious that in any application few scenarios are executed more frequently as compared to others. Moreover, for some applications’ scenarios the execution also depends on relative time. For example, for an online billing application where bill payments are made only during first week of the month, the administrator will be using the application mainly for accounts updating and importing billing information etc. in last 3 weeks of the month but as the first week starts, the major chunk of site users will be using it for bill payments. So you need to figure out which AUT scenarios will be executed at what time and what will be their execution percentage in the overall workload. Following techniques could be helpful in identifying relative distribution of your identified scenarios:

• Check out the server log files to identify users’ trends on application if it is in the production environment.
• Consult with the sales and marketing teams to figure out which features they believe will be mostly used.
• You can also interview the existing and potential clients to find out in which features they are most interested.
• Another approach could be to share a beta release among a smaller segment of users and check out their trends and behaviors on application from server log files.
• In case none of the above mentioned approaches work then use your experience and intuitions to get the answers of your questions.

7. Identify Target Load Levels

One last step involved in workload model completion after performing all the above mentioned activities is to identify the normal and peak load levels of every selected scenario to test the application for expected and peak load conditions. Mainly depending upon the application you need to identify the monthly, weekly, daily and hourly average load targets for every selected scenario. You need to know the following information in order to identify the target load levels for an application:

• What are the current and expected normal and peak user requests level?
• What are the application key test scenarios?
• What is the distribution of user requests on the identified scenarios?
• What are the navigation paths for all the scenarios along with relative utilization of every scenario?

8. Workload Design Miscellaneous Options

Once you have identified the key test scenarios, their relative distribution and the target load levels the last thing is to figure out different options for your workload like browser mix, network mix, think time and pausing between the iterations to simulate the workload as per the real environment. All performance testing tools are equipped with following options:

• Browser Mix: List down all the browsers which you want to include in your test and specify the load distribution across all browsers to verify what will be the system response in all these browsers.
• Network Mix: There are various internet connections available these days and people use almost all of them based on their availability and constraints. So it’s better to include major list of network connections in your test and appropriately distribute the load on them.
• Think Time: As real users always take some time while taking different actions, it’s very important to include the think time in your test based on the application users’ comfort level in using the application identified scenarios. Ignoring the think time can invalidate the test results.
• Pause Time: There is always a certain pause before the user receives a response from server and send a new request which can be cater to with pause time between the iterations.

Challenges Involved in Workload Modeling

We have discussed all the activities involved in workload modeling and its quite obvious that the workload modeling for performance test is not a piece of cake. Performance testing teams face various challenges while performing all the above mentioned activities. Following is a list of some of these challenges:

• Complete application access in planning phase before executing the performance test
• Getting help from the marketing and network teams for their input on application performance critical scenarios and their workload distribution
• Accessibility to relevant server logs
• Applying the data mining techniques on server logs to extract the relevant information
• Presenting the workload models to the stakeholders in an effective manner to get their approval on it.