|
Huong Nguyen Posts: 69
7/20/2011
|
Hi Vadim,
The CPU graph keeps on increasing. SS's option is to purge response body for non-errors. In my test for 256 SS, there were 2 errors, 582 total timeout. Initially, I ran SS with wireshark in the background. Then, halfway through the test, I closed wireshark. I expected the CPU usage to drop, but it keeps on increasing. http://screencast.com/t/gyNm1JAadYr.
Previously, with 128SS, the graph looked like this: http://screencast.com/t/BH6mOC4iVLi. The 2 graphs doesn't look similar so I can't compare. For the 256, the ramp up is 1 user every 2 seconds, until 256 users. The 128 users test, the ramp up is 1 user every 4 seconds, until 128 users.
Can you tell me the reasons for the increasing CPU utilization? How do I know if it's a code-based change that caused it or it was just due to the increase in virtual users? What I should look into? Etc.
Thank you in advance.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
7/20/2011
|
Hi Huong,
I assume that the CPU utilization increase was not cause by another process unrelated to StresStimulus (while I recommend double-checking it, just to make sure). It seems that that are two factors that are causing a CPU utilization increase in the current test as compared to the previous one.
First is the doubling of the number of VUs. That increases the CPU utilization requirement for the load engine.
Second (assuming that both graphs are displayed with the same scale factor), in the current test, your server responds faster. The response time (the blue line) in the current test seems to be under 5 seconds, while in the previous test it was more likely between 15 and 20 seconds. A faster server causes the client to work faster to keep up with responses arriving at a faster pace.
It should be easier to comparatively analyze two tests when only one factor was changed. In your case, you changed the code base and also the number of VUs. I recommend running the second test with the same 128 VUs to make the code change performance impact more apparent.
This question reminds me of your other recent question about maintaining the same workload to compare two tests. We will develop an option to maintain a constant iteration duration regardless of the server response time, in v1.0 (which is less than 2 weeks away). With this option, your second test may run with 250 VUs without overloading the CPU. This should be a great feature! Thanks again for suggesting it!
Cheers,
-Vadim
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
7/21/2011
|
Hi Vadim,
Yes, it seems like doubling the vu count is the cause. I didn't really expect the graph to look like that. How do I debug what is the reason why doubling vu count/request count/etc. causing the CPU usage? I would expect the CPU to be high when I run close to 1500 vu. Right now, it's like that only for 256 vus. This means that I will need 4+ machines to simulate 1500, instead of 1 machine. This is not ideal for me, since it makes configuration, etc. much harder. Please advise.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
7/21/2011
|
Huong,
By looking on your graph again I noticed that 55 out of 60 minutes of your test (or 11 out of 12 measurements) the CPU utilization stayed below 80%. The total processor time counter reads an instant (not averaged) value, and the one high reading could be caused by a sporadic spike unrelated to StresStimulus. To better understand was is going on, I suggest doing 2 things: decrease the sample rate from 5 min to let say 30 sec, and add the Fiddler process %CPU counter. Here is the example of Fiddler vs. the entire computer CPU utilization and (unrelated to Fiddler) spikes. This is an indication that there are programs that can be stopped to give StresStimulus more resources.
We will actually add the Fiddler %CPU counter as a default in v1.0
Going back to your points, doubling the number of VUs is a factor, but I would not disregard a reduction of the server response time as another factor.
Predicting a number of VUs that a single machine can handle for a particular test is difficult because it depends on many conditions. To name a few, hardware capacity of the client machine, size of your test case, speed of your web server, and think times. In attempt to increase the number of VUs per machine, the users can consider such measures as using a realistic think times instead of zeros, breaking down test cases (if possible), stopping unnecessary programs, and using the faster machines. However, none of these steps can guarantee that a machine can handle the desired number of the VUs for a particular test. Therefore, the right course of action is to determine through testing how many VUs a machine can handle with CPU utilization not exceeding 85-90%, and based on that and the required number of VUs, calculate how many machines is necessary. 1500 VUs in StresStimulus Pro RC is the capped number of VUs allowed. This number can be reached in some but not all tests. Adding more machines to a load test is common.
If you can send me the test report of those two tests and a new graph with Fiddler CPU use, I can check if I can add anything else to this.
Cheers,
-Vadim
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
7/24/2011
|
Hi Vadim,
I've been monitoring the client and CPU utilization is for Fiddler only, there's no other running programs.
We are using recorded think time.
I can't break down the test cases anymore.
The specs for client VM is XP x64 SP 2, @2.13 GHz, 6 GB RAM. 2-core.
I'll try to add more core to the VM. I'll also post the logs requested after I finish the runs I need to do (because I don't want to change settings while comparing test results).
Looks like if indeed I need to add computers, that will be 4 computers per test. I'm already using 6 computers, and it will make me go to a total of at least 14 computers. This number is overwhelming. I would really like ability to configure max user count, iteration steps, etc. from command line (text file). Also, the ability to mix test cases can bring this count down to around 12. This also concerns me regarding licensing.
I'll update you when I start getting the logs for you. Thank you.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
7/26/2011
|
Hi Huong,
The 6 GB RAM is not a bottleneck, since StresStimulus uses less than 2 GB in your tests. Your 2-core CPU @2.13 GHz per VM seems to be powerful enough to handle more than 250 users (while it also depends on the size of your test case). We have many examples of running a greater number of VUs with less CPU utilization. For example in this post the user provided a report of three tests with a 500 VU step load where the average CPU utilization was 20-40% (with some unrelated spikes - http://community.stresstimulus.com/forum/attachment/download). I wonder whether you use custom scripting. If yes, can you disable it just to check what its impact on the performance is? Scripts in general degrade SS performance. Adding command line functionality is on our list.
I am looking forward to the logs that you will send.
If you send me (in the private message) the required number of VUs for each of your concurrent test scenarios and the number og VM you think you would need, I can help you to optimize the licensing option.
Best,
Vadim
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
8/18/2011
|
Hi Vadim,
It’s due to my script. Without script, SS runs 128 vu, for 1 hour using only 0-5% CPU.
Will follow up with you after I finish testing my converted script into Fiddler addin.
Thank you.
|
|
|
0
link
|