|
Lakmal Nanayakkara Posts: 10
5/13/2011
|
Hi,
I want to test load test using StressStimulus. One of client ask me to do a load test and idenfy how the application behave under increasing load. I've tested with 500 users but I didn't see any issue in web application.
I wonder about the results. When the testing in progress I manually login to it and see any bottlneck I can see inthwe web app but I didn't see.
So what could be reasons for this? Is the application give error message when web application is crashing?
What do you think would be good configuration for my requirements.
Here are the test summary I got.
Note that testing web application is hosted in remote server and stressstimulas plugin install in my local machine. Can we run the with that way (web allocation in remote host and stresstimulus plugin in another machine).
Test Summary:
Test Case Summary
Base Pages: 1
URLs : 5
Test Parameters
Think time between base pages: 1s
Think time between iterations: 0s
Load pattern: Step load
Start with: 5 user(s)
Increase by: 100 user(s)
Every: 5 second(s)
Maximum: 500 user(s)
Browser Type:
Network Type: Cable/DSL 384k
Complete test after: 500 seconds
Warm-up time: 0 seconds
Test Run Information
Start Time: 13:07:33
End Time: 13:52:26
Warm-up (s): 0
Aggregate Session time (s): 411549
Test run duration (s): 2692.45
Overall Result
Status: Aborted
Max User Load: 500
Total bytes sent: 4,844,020
Total bytes received: 111,778,778
Bytes sent/sec: 1,799.112
Bytes received/sec: 41,515.622
Test Iterations
Total test iterations: 1003
Total iterations completed: 614
HTTP Transactions (sessions)
Total requests: 3,523
Requests Failed: 4
Requests/sec: 1.3
Avg. response time (s): 123.8
Thanks
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
5/13/2011
|
Hi Lakmal,
It looks like your test stopped. It was set to run for 500 sec, but you aborted it in > 2692 seconds. The issue with stopping the progress bar was fixed in the hot fix that we released yesterday (see this post). It may be the same one that you are having, so please try the fix until the next (official) beta release is out. The hot fix can be downloaded here.
Cheers,
-Vadim
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
5/13/2011
|
Lakmal,
My previous comment was about the test which you’ve pasted. I missed the attachment.
Here is my response about the test in the attachment (below I pasted your Key Indicators graph):
The test was set to run 250 seconds and after that wait for pending responses. It ramped up to 500 users in the first ~10 seconds (1). Then at ~ 250 s it stopped sending requests (2) and was waiting for another ~82 s to receive all pending responses (3). Test completed successfully in 332 seconds. All 2304 requests were returned by the server without errors. However, the average response time was 156.1 s (4), which is unacceptably high.
Note: The current version of StresStimulus does not track long response time as errors. Base on your input, in the next version we will add page timeout tracking.
Below are the answers to your questions:
Q:“So what could be reasons for this? Is the application give error message when web application is crashing?”.
A: What makes you believe that there are errors or crashing? I do not see any errors in the attachment. Your server/application is slow. The average response time (blue graph) grew during the first 10 s due to users ramping. It continued to grow throughout the test (with constant 500 users) because of growing server request queue.
Q:”What do you think would be good configuration for my requirements.”
A: To troubleshoot the performance you should use more granular user ramping. I suggest starting with 10 users and incrementing by 10 users every 10 seconds. Watch at what load level the response time reaches unacceptable level.
Q:” Can we run the with that way (web allocation in remote host and stresstimulus plugin in another machine).
A: Not sure I fully understand the question. StresStimulus can be installed on more than one machine.
Q: I've tested with 500 users but I didn't see any issue in web application. I wonder about the results. When the testing in progress I manually login to it and see any bottlneck I can see int hwe web app but I didn't see.
A: Not sure I fully understand the question, but there could be many reasons to bad performance, such as not optimized page code or database design or insufficient hardware resources on the server etc. In this test you should watch the number of queued requests as well as server handling time.
Cheers,
-Vadim
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
5/19/2011
|
Hi Vadim,
Thanks for the quick reply.
I installed your tool with hot fix and ran the test for 3 times. Hewewith I've attched the document with screenshots. According to results, I've seen different results than ealier.
Now i'm bit confused with these results.
Do you have any idea about this? Seems now our application working fine now without any modifications.What could be the reasons for this strange behavior?
Thanks,
Lakmal
Vadim Kleyzit said:
Lakmal,
My previous comment was about the test which you’ve pasted. I missed the attachment.
Here is my response about the test in the attachment (below I pasted your Key Indicators graph):
The test was set to run 250 seconds and after that wait for pending responses. It ramped up to 500 users in the first ~10 seconds (1). Then at ~ 250 s it stopped sending requests (2) and was waiting for another ~82 s to receive all pending responses (3). Test completed successfully in 332 seconds. All 2304 requests were returned by the server without errors. However, the average response time was 156.1 s (4), which is unacceptably high.
Note: The current version of StresStimulus does not track long response time as errors. Base on your input, in the next version we will add page timeout tracking.
Below are the answers to your questions:
Q:“So what could be reasons for this? Is the application give error message when web application is crashing?”.
A: What makes you believe that there are errors or crashing? I do not see any errors in the attachment. Your server/application is slow. The average response time (blue graph) grew during the first 10 s due to users ramping. It continued to grow throughout the test (with constant 500 users) because of growing server request queue.
Q:”What do you think would be good configuration for my requirements.”
A: To troubleshoot the performance you should use more granular user ramping. I suggest starting with 10 users and incrementing by 10 users every 10 seconds. Watch at what load level the response time reaches unacceptable level.
Q:” Can we run the with that way (web allocation in remote host and stresstimulus plugin in another machine).
A: Not sure I fully understand the question. StresStimulus can be installed on more than one machine.
Q: I've tested with 500 users but I didn't see any issue in web application. I wonder about the results. When the testing in progress I manually login to it and see any bottlneck I can see int hwe web app but I didn't see.
A: Not sure I fully understand the question, but there could be many reasons to bad performance, such as not optimized page code or database design or insufficient hardware resources on the server etc. In this test you should watch the number of queued requests as well as server handling time.
Cheers,
-Vadim
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
5/19/2011
|
Hi Lakmal,
I agree that last three tests look pretty consistent. When compare with the older test, the results are different, but the loads are different too. I compared the results in the attached Excel. Not sure which numbers specifically confused you.
The older test ran for 250 seconds with ramping 50 + 50 /every 1 s ->500. The later test ran for 500 seconds with ramping 10 + 10 /every 10 s ->500. While the final number of VUs in both tests are the same, the older test is twice as short and it was ramping users 50 times faster. As a result the average number of VU in the older test was close to 500, while in the new test is probably in the area of 250. The average response in older test is ~5x slower (156.2s vs. 29.4s) probably as a consequence of much higher average load. This does not seem to be inconsistent.
Am I missing anything?
Cheers,
-Vadim
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
5/20/2011
|
Thanks Vadim.
Do you think these results are 100% accurate?
yesterday I forgot mentioned I've experienced progress bar stop issue again with your hot fix.
Can you tell me when we can get a stable build? I mean the full build ?
Thanks for your support.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
5/20/2011
|
You’re welcome, Lakmal.
StresStimulus is very accurate. It sits on the top of Fiddler, which accuracy was verified by millions of users. Additionally, the accuracy of StresStimulus is very easy to check. If you display StresStimulus’s traffic in Fiddler, the performance indicators in Statistics tab will match StresStimulus’s numbers.
Install the latest version that we released yesterday and let me know if you still see the progress bar stop issue. We are very excited with the beta progress. StresStimulus is used by thousands of users, and the feedback is very positive, so I we are getting close to releasing the retail version.
-Cheers,
Vadim
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
5/23/2011
|
Thanks.
How can I validate the results with Fiddler?
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
5/23/2011
|
Also I want to test how my web application behave when users are using different search criteria
User 1
Search for all
User 2
Search by age
User 3
Search by dept.
etc..........
How can I do test this?
2. Also I noticed below error sevral times.What could be the reasons and solutions for it?
15:53:25:9271 Fiddler.Network.ProtocolViolation - [#9777] Content-Length mismatch: Response Header indicated 45,271 bytes, but server sent 2,705 bytes.
15:55:02:9777 Fiddler.Network.ProtocolViolation - [#11421] Content-Length mismatch: Response Header indicated 45,271 bytes, but server sent 31,525 bytes.
3. Today I updated my installation to latest one (It was popped-up for new update). But still I can see progress bar issue.What will be the reasons? Even after I clicked 'Abort' it's not aborting correctly.Only option I've is kill the process of Fiddler.
Thanks,
Lakmal
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
5/23/2011
|
Hi Lakmal,
- On your first question: you have couple of options:
Option 1: if all three types of searchers are performed from the same page that submits different request parameters, then you can use parameterization. Create a CSV file with 3 records, one for each type of searches. When define data-binding, use VU-bound type.
Option 2: if these 3 types of searchers use different pages, you can use separate machines for each test case. FYI: Support for multiple test cases running from a single StresStimulus instance is already on our feature-request list.
These warnings suggest that your server created Response Header “Content-Length” that does not match the actual length of the two responses as measured by fiddler.Look into these 2 responses in fiddler to confirm that Content-Length header is incorrect and if this is the case then this is the server issue.
FYI: StresStimulus does not use content lengths header for its load test statistics. So this server discrepancy should not impact accuracy of the load test.
But still I can see progress bar issue.What will be the reasons? Even after I clicked 'Abort' it's not aborting correctly.Only option I've is kill the process of Fiddler.
Can you describe the issue and provide more information, such as progress bar screenshots, which counters on the progress bar are advancing and which stopped, does this happen all the time or sometimes, provide as much details of the test case as possible. I would be happy to debug if you can email me the test case files.
Best,
-Vadim
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
5/23/2011
|
Lakmal, I will provide the respond in a separate thread and will let you know.
-Vadim
Lakmal Nanayakkara said:
Thanks.
How can I validate the results with Fiddler?
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
6/1/2011
|
Hi Vadim,
Did you get a chance for look into above? could you pls send me thread related to above.
thanks for your help given,
Lakmal
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/2/2011
|
Hi Lakmal,
Sorry for not getting back to you sooner. I am thinking of a comprehensive way to describe a methodology of StresStimulus result validation with Fiddler. My challenge is to describe all of it in one post, because there are too many metrics elements in StresStimulus’ summary report, request / page details grids and multiple graphs. So I would like to come up with a series of blogs that will describe how virtually every element reported by StresStimulus can be independently verified by Fiddler metrics, perfmon data and other sources. This however will take some time.
So I wanted to ask you if there any specific number in your load test result that you need to verify. If you can indicate such parameter in your test result and send it along with saz file containing all test sessions, then I will show how these two are connected. In the absence of the saz file I will also try to explain how the result was received.
As I said, the blogs on load test validation are coming.
Cheers,
-Vadim
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
6/24/2011
|
Hi Vadim,
Today I ran load test for another web application and got the below results.
Again I did see abnormal behavior. According to results of our web application, it should be slow with 500 Vuser load.
While running the load test in one machine, I connected to same web application using other machine. At that time response time was around 500 seconds. That means it takes around 8 minutes for a request. At that peak time I logged from another machine to a same web application (while running the load test) and I expected to see extremely slow response because load test is already running.
But I saw application is running properly without any delay. So I’m confused with accuracy of the tool.
I think when another user login from another machine he should experienced the slow reponse.But it is not happening.
Please explain this. see attached document.
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
6/24/2011
|
Lakmal Nanayakkara said:
Hi Vadim,
Today I ran load test for another web application and got the below results.
Again I did see abnormal behavior. According to results of our web application, it should be slow with 500 Vuser load.
While running the load test in one machine, I connected to same web application using other machine. At that time response time was around 500 seconds. That means it takes around 8 minutes for a request. At that peak time I logged from another machine to a same web application (while running the load test) and I expected to see extremely slow response because load test is already running.
But I saw application is running properly without any delay. So I’m confused with accuracy of the tool.
I think when another user login from another machine he should experienced the slow reponse.But it is not happening.
Please explain this. see attached document.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/24/2011
|
Hi Lakmal.
The problem with this test is that your client got overloaded, with CPU Utilization reaching 100% after about 350s after test started. Load test only OK if the client CPU utilization stays below 90%. This is a general rule of thumb, not specific to StresStimulus. This post explains how running load test on overloaded client gives wrong result: The "request response time" measurement.
When you reached 500 users, between 50 and 350 sec, but CPU utilization stayed low, the response time seems to be good. It increases after reaching 100% CPU utilization. I assume that the red line on the Peformance Counters graph is CPU utilization (default). 
Couple if suggestions to address this issue:
- Restart Fiddler to make sure that there is no issue with its process.
- Make sure that in the StresStimulus main menu -> Options -> Purge response bodies set to All or Non-Errors.
- See if you can close any programs that uses resources on this machine.
- If none of these steps will help, you may need to reduce the number of virtual users or split them between several machines.
-Cheers,
Vadim
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
6/27/2011
|
Thanks Vadim.
Today i ran the test again and saw some licening issue when trying use the StressStimulas tool. Then I downgrade to lower version and ran the test again.
But I saw two isseus
1. Now I can't test urge response bodies set to All or Non-Errors.Its automatically set to default one when change it
2.Now I can't see performance counters grap or values. It displayed empty. This is strange.
Please help me on regard these issue.
Thanks,
Lakmal
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/27/2011
|
Hi Lakmal,
We resolved the issue with Pro beta activation (see this thread) . To fix, re-install the latest version of StresStimulus from http://www.stresstimulus.com/. Then, make sure that you use Pro edition - the label on the bottom of StresStimulus should show this:
To switch to Pro, click the green item "Click to Use Pro Edition". Otherwise you will experience the limitations of the free edition that you mentioned.
-Cheers,
-Vadim
|
|
|
0
link
|
|
Lakmal Nanayakkara Posts: 10
6/28/2011
|
Hi Vadim,
Yes. After re-installed my problems was disappered.
Do you have any proper tutorials (setp by step) to do a load test? If yes please share the link with me.
Thanks,
Lakmal
|
|
|
0
link
|