Case Studies – Identifying Performance-testing Objectives

Case Study 1

Scenario

A 40-year-old financial services company with 3,000 employees is implementing its annual Enterprise Resource Planning (ERP) software upgrade, including new production hardware. The last upgrade resulted in disappointing performance and many months of tuning during production.

Performance Objectives

The performance-testing effort was based on the following overall performance objectives:

• Ensure that the new production hardware is no slower than the previous release.
• Determine configuration settings for the new production hardware.
• Tune customizations. 

Performance Budget/Constraints

The following budget limitations constrained the performance-testing effort:

• No server should have sustained processor utilization above 80 percent under any anticipated load. (Threshold)
• No single requested report is permitted to lock more than 20 MB of RAM and 15-percent processor utilization on the Data Cube Server.
• No combination of requested reports is permitted to lock more than 100 MB of RAM and 50-percent processor utilization on the Data Cube Server at one time.

Performance-Testing Objectives

The following priority objectives focused the performance testing:

• Verify that there is no performance degradation over the previous release.
• Verify the ideal configuration for the application in terms of response time, throughput, and resource utilization.
• Resolve existing performance inadequacy with the Data Cube Server.

Questions

The following questions helped to determine relevant testing objectives:

• What is the reason for deciding to test performance? 
• In terms of performance, what issues concern you most in relation to the upgrade?
• Why are you concerned about the Data Cube Server?

Case Study 2

Scenario

A financial institution with 4,000 users distributed among the central headquarters and several branch offices is experiencing performance problems with business applications that deal with loan processing.
Six major business operations have been affected by problems related to slowness as well as high resource consumption and error rates identified by the company’s IT group. The consumption issue is due to high processor usage in the database, while the errors are related to database queries with exceptions.

Performance Objectives

The performance-testing effort was based on the following overall performance objectives:

• The system must support all users in the central headquarters and branch offices who use the system during peak business hours.
• The system must meet backup duration requirements for the minimal possible timeframe.
• Database queries should be optimal, resulting in processor utilization no higher than 50-75 percent during normal and peak business activities.

Performance Budget/Constraints

The following budget limitations constrained the performance-testing effort:

• No server should have sustained processor utilization above 75 percent under any anticipated load (normal and peak) when users in headquarters and branch offices are using the system. (Threshold)
• When system backups are being performed, the response times of business operations should not exceed 8 percent, or the response times experienced when no backup is being done.
• Response times for all business operations during normal and peak load should not exceed 6 seconds.
• No error rates are allowable during transaction activity in the database that may result in the loss of user-submitted loan applications.

Performance-Testing Objectives

The following priority objectives focused the performance testing:

• Help to optimize the loan-processing applications to ensure that the system meets stated business requirements.
• Test for 100-percent coverage of the entire six business processes affected by the loan-manufacturing applications.
• Target database queries that were confirmed to be extremely sub-optimal, with improper hints and nested sub-query hashing.
• Help to remove superfluous database queries in order to minimize transactional cost.
• Tests should monitor for relevant component metrics: end-user response time, error rate, database transactions per second, and overall processor, memory, network, and disk status for the database server.

Questions

The following questions helped to determine relevant testing objectives:

• What is the reason for deciding to test performance? 
• In terms of performance, what issues concern you most in relation to the queries that may be causing processor bottlenecks and transactional errors?
• What business cases related to the queries might be causing processor and transactional errors?
• What database backup operations might affect performance during business operations?
• What are the timeframes for back-up procedures that might affect business operations, and what are the most critical scenarios involved in the time frame?
• How many users are there and where are they located (headquarters, branch offices) during times of critical business operations?

These questions helped performance testers identify the most important concerns in order to help prioritize testing efforts. The questions also helped determine what information to include in conversations and reports.

Case Study 3

Scenario

A Web site is responsible for conducting online surveys with 2 million users in a one-hour timeframe. The site infrastructure was built with wide area network (WAN) links all over the world. The site administrators want to test the site’s performance to ensure that it can sustain 2 million user visits in one hour.

Performance Objectives

The performance-testing effort was based on the following overall performance objectives:

• The Web site is able to support a peak load of 2million user visits in a one-hour timeframe.
• Survey submissions should not be compromised due to application errors.

Performance Budget/Constraints

The following budget limitations constrained the performance-testing effort:

• No server can have sustained processor utilization above 75 percent under any anticipated load (normal and peak) during submission of surveys (2 million at peak load).
• Response times for all survey submissions must not exceed 8 seconds during normal and peak loads.
• No survey submissions can be lost due to application errors.

Performance-Testing Objectives

The following priority objectives focused the performance testing:

• Simulate one user transaction scripted with 2 million total virtual users in one hour distributed among two datacenters, with 1 million active users at each data center.
• Simulate the peak load of 2 million user visits in a one-hour period.
• Test for 100-percent coverage of all survey types.
• Monitor for relevant component metrics: end-user response time, error rate, database transactions per second, and overall processor, memory, network and disk status for the database server.
• Test the error rate to determine the reliability metrics of the survey system.
• Test by using firewall and load-balancing configurations.

Questions

The following questions helped to determine relevant testing objectives:

• What is the reason for deciding to test performance?
• In terms of performance, what issues concern you most in relation to survey submissions that might cause data loss or user abandonment due to slow response time?
• What types of submissions need to be simulated for surveys related to business requirements?
• Where are the users located geographically when submitting the surveys?

How to calculate sessions per hour in performance testing? - Little's Law

How to calculate sessions per hour in performance testing? - Little's Law

Calculating Sessions per hour: For this discussion, we will be focusing on a session as the total time for the user group to finish one complete set of transactions. We may wish to know the number sessions that will be completed for any given number of virtual users.

• Example 1:- If a baseline test shows that a User Type takes a total of 120 seconds for a session, then in an hour long steady state test this User Type should be able to complete 3600 / 120 = 30 sessions per hour. Twenty of these users will complete 20 x 30 = 600 of these session in an hour. In other cases, we have a set number of sessions we want to complete during the test and want to determine the number of virtual users to start.
• Example 2:Using the same conditions in our first example, if our target session rate for sessions per hour is 500, then 500 / 30 = 16.7 or 17 virtual users. A formula called Little's Law states this calculation of VirtualUsers in slightly different terms.
• Using Little's Law with Example 2:
  wV.U. = R x D where R = Transaction Rate and D = Duration of the Session
  If our target rate is 500 sessions per hour (.139 sessions/sec) and our duration is 120 seconds, then Virtual Users = .139 x 120 = 16.7 or 17 virtual users.

Performance case studies

Case study 1:
We need to test the application which has 50,000 user traffic in 1 hour and each user transaction compromises of 7 page assuming each transaction takes 30 min.

Solutions First of all each user will not perform activity in 30 minutes. He will be in the state of think time (most of the time). Let’s take each page time taken to complete is 10 sec.

Also 50,000 user will perform 50,000 transactions/iterations.
1 user -> 7 Pages-> 1 page (10sec)-> 7pages=>7*10 =>70 seconds
1 user in 1 hour will have go 3600*70=>51 iterations
So for 50000 transactions/iterations => 50000/51 => 981 users approx.

So if your tool capability is not upto 50,000 user and you need to test application for 50,000 users , Go by explained strategy or if you have your tool capable with 50,000 user load, then go by original no. of users.

Other way solution of this problem can be that ask your web server administrator that how much no. of http hits application is going to get in the peak time of the activity and then go by goal –oriented scenario in Loadrunner.