|
Tom Mancine Posts: 6
3/2/2012
|
***Resolved on 3/20/12***
I am using StressStimulus to perform load testing. We are trying different values of think time to see how the system responds to requests arriving at different rates. However, I have noticed some unpleasant behavior. When I run with n users and a think time of 1s, three pages will have average times of say, 4.5s, 7s, and 2.5s. If I run these tests with the same n users, but a think time of 5s instead of 1s, I would expect those times to go down. However, they not only go up, but they all go up by about 4s each.
My assumption was that the think time should not be counted by the system when computing how long a page took to render. Was this assumption incorrect?
I have recently upgraded to StressStimulus Pro Version 1.5.4442 and am still seeing this behavior, which I first observed in v1.2.x.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/3/2012
|
Hi Tom,
We are looking if your result can be replicated. I will let you know what we find.
Thanks,
-Vadim
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/3/2012
|
Tom,
We were able to replicate your result and fixed the issue causing it.
Your assumption that the think time is not included into the page response time is correct. However, I'm confirming that in some situations, which apparently took place in your case, the load engine was injecting an additional think time in the middle of the page, which understandably was counted toward the page response time. As a result, increasing think time caused increasing page response time that, as you correctly noted, should not be the case.
The revision with the fix is released. Please download it from http://www.stresstimulus.com/default.aspx#dwnld (as auto-update will not detect it for now) and check if you're satisfied with the result.
Thank you for noticing and for posting this issue on the forum.
-Vadim
|
|
|
0
link
|
|
Tom Mancine Posts: 6
3/5/2012
|
Thank you for the very quick turnaround time! In the new version, the think time no longer appears to affect the individual requests, which is great news. However, it is also computing that one of the last pages in the test takes 80 seconds on average, when inspecting the session timelines shows that none of them took more than about 10 seconds. It seems to me like maybe some of the extra time is ending up there, although I don't know enough of the internals of the tool to speculate on how that could be happening. Would this be a possible consequence of the change, or should I create a separate forum topic to cover it?
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/5/2012
|
Tom,
We were trying to check your hypothesis that there was a consequence of the change, but were unable to replicate this behavior. Everything seems to work fine on our end.
I would be able to answer your question, if after running the test you can submit it to our supports repository using the "upload support case" tool located in the StresStimulus menu -> Support Options. We do not need to run the test against your siteto check the performance metrics.
Thanks,
Vadim
|
|
|
0
link
|
|
Tom Mancine Posts: 6
3/12/2012
|
I have just finished uploading a sample test case; I included the URL of this discussion in my comments. Thanks in advance for your assistance!
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/13/2012
|
Hi Tom,
We found the issue with the long page response times in your test. It was caused by the same bug that was injecting the think time in the middle of the page. Apparently it was not fixed for the pages using multiple hosts in this specific case. It is fixed now. Please install the latest version and let me know if it is OK now.
I appreciate your help with tracking down this issue.
-Vadim
|
|
|
0
link
|
|
Tom Mancine Posts: 6
3/13/2012
|
Vadim,
You guys are really great! I'll download the new version and get back to you as soon as I can. Thanks!
-Tom
|
|
|
0
link
|
|
Tom Mancine Posts: 6
3/15/2012
|
Although we still do see a variance of a few tenths of a second between 1s think time and 5s think time, we feel comfortable with the numbers we are getting now. Thank you very much for all of your help!
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/15/2012
|
Hi Tom,
Thanks again for your help with tracking down this issue! I am glad that the fix worked for you.
If you compare the 2 tests with 1s think time and 5s think time respectively, then a few tenths of a second discrepancy in response time should be statistically fine, as it is caused by random factors that are always causing slight deviation.
Cheers,
-Vadim
|
|
|
0
link
|
|
Tom Mancine Posts: 6
3/18/2012
|
I agree that the few tenths of a second are completely negligible. 
I am now having some other strange problems, however, which I hope you can help with, also related to think time. Specifically, I have a test case that reports a certain page taking significantly less time (10s vs. 16s) with 100 users than with 40 users when I set the think time to 5sec. We would definitely rather run our tests with 5s think time than 1s, because we feel it represents a more typical usage pattern.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/19/2012
|
Hi Tom,
I received your support cases. We will analyze and I will let you know what we found.
Thank you,
-Vadim
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
3/20/2012
|
Hi Tom,
We resolved the issued that you described. You are correct. The response time with 100 VUs should be higher than the same with 40 VUs. This was the case when you compare the median of the response times for the dashboard.aspx page (2.5 s with 40 VUs vs. 7.0 s with 100 VUs). The issue with average response time, was that StresStimulus was holding a few requests for a very long time before sending. As a result while the median time was correct, the average time was impacted.
We've fixed the issue, and now, when you use 5s think time, the average response time should be much faster for 40 VUs than for 100 VUs.
Please install the new version and let me know if you confirm the fix.
Thank you!
-Vadim
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
4/22/2012
|
Since after the fix we can no longer replicate this issue and there is no evidence to the contrary, I will marked the issue as resolved. if you have any update, please feel free to post here.
|
|
|
0
link
|