|
Huong Nguyen Posts: 69
6/15/2011
|
Hi Vadim,
I ran the test for the Vendor scenario (from the email that I sent you).
When I ran it with 1 iteration, it finished in about 12.5 minutes.
I'm seeing that when I tell it to run for 1 hour, with 288 users, 0/288 iterations finished. I expected 1152 iterations to be started by this time. What are the reasons for the iterations not being able to finish? I'm also seeing a lot of more errors than I saw previously, I'm attaching the error logs and test summary.
NOTE: This doesn't happen to all of my tests, some of the ones with less user count does iterate correctly.
Look forward to your help. Thank you in advance.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/16/2011
|
Huong,
It looks like everyone of 288 users was able to start only one iteration, but none of them completed.
The reason for an iteration to not complete is insufficient test time to send and receive all of its requests. For example, User 01 had a full hour for its 773 iteration to complete, but apparently this period was insufficient. Errors in responses do not affect incomplete iterations, but slow responses do.
Note: the total number of issued requests was 111,894, or less then 400 per user (111,894/288 = 389). User 01 probably had sent more than 400 requests, but apparently less than 773.
To troubleshoot, display all the first user's requests in Fiddler after the test completion. The Fiddler Timeline tab should show which transactions were returned slowly and at what point. One explanation is that, as the number of VUs ramped-up, it took forever for the server to respond. A large number of timeouts is an indirect evidence to it.
Also, make sure that the client CPU utilization was below 85-90%, otherwise it is possible that the client was slow to send requests quickly enough. To reduce the client load, in the StresStimulus main menu -> Options -> Purge response bodies set to All.
To display the first user's sessions, before staring the test in Other Test Options select First User (or Ask) and select All sessions.
Please let me know whether it make sense and what did you find out.
-Vadim
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
6/16/2011
|
Hi Vadim,
Thank you for the explanation, it confirmed my suspicions.
1. I displayed the 1st user's request: 421 requests.
2. Client CPU utilization is very good. http://screencast.com/t/nIPg9w6Iw3
3. It's confirmed that requests are getting slower and slower. http://screencast.com/t/ecYDlZlylaHw
4. First 200 response in red (which I assume to look like an error due to timeout) is at 7:26:05 PM, test started at 6:36:46 PM - around 3000 seconds. Seems like the Key Indicator graph also support this, because around 3000 seconds, the avg response is low, I'm guessing due to the requests taking longer to return...
Do you disagree with any of these analysis?
From this, I have some questions/requests:
1. What role does the request timeout play in this? I thought that the purpose is to abort the request if it was taking too long, but it seems like the value is only there to help SS determine if a request reached the timeout threshold or not?
2. What should I ask the developer to look into to improve the response time, so that certain requests doesn't take as long to return, with load?
3. I'm not sure how to analyze page and request details node, can you explain?
4. Can we have an export for the whole test results node? It seems like page/request details are not exportable right now, and the graphs, test summary have to be saved separately.
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
6/16/2011
|
One more question:
5. If a request is slow, it should only affect the current virtual user's subsequent requests, correct? For example, if another virtual users can get requests quickly, virtual user 2 (which starts after 1), might finish the iteration before user 1 does?
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/16/2011
|
Huong, see my answers below:
1. I displayed the 1st user's request: 421 requests.
Consistent result. FYI: In the final v0.9 beta we will have one more grid in the Test Results section, called Iteration Details that will show the number of iterations and request per VU, so you would be able to get this number without going to Fiddler. We decided to add it after I figured that you would want to look at this information, and it is not easily available in the alpha.
2. Client CPU utilization is very good. http://screencast.com/t/nIPg9w6Iw3
Selfishly, I am glad to see that client utilization stays below 8.5% at all times, while memory utilization for the process is below 20%. StresStimulus load engine is designed to be very lightweight to handle a large number of VUs in heavy test cases.
3. It's confirmed that requests are getting slower and slower. http://screencast.com/t/ecYDlZlylaHw
Consistent result.
4. First 200 response in red (which I assume to look like an error due to timeout)...
StresStimulus shows in red the requests that are timed out.
Do you disagree with any of these analysis?
No. All of the points are correct.
1. What role does the request timeout play in this? I thought that the purpose is to abort the request if it was taking too long, but it seems like the value is only there to help SS determine if a request reached the timeout threshold or not?
There is no timeout in HTTP or Fiddler. There is no standard for timeout in web server or browser, and it is up to the respective vendors. To avoid blocking VUs for a very long time, StresStimulus allows users to setup a timeout that is handled as follows: It does not abort timed out requests to allow receiving the response at a later time that may be helpful for troubleshooting and debugging. However StresStimulus removes the block from TCP connection and allows VU to send next depended requests without waiting for the timed out response. So in that respect from the load testing standpoint StresStimulus does abort the waiting for a response.
2. What should I ask the developer to look into to improve the response time, so that certain requests doesn't take as long to return, with load?
You should run several more load tests to get more information for the developers. Based on this load test determine what is your server operational ceiling, meaning with how many users the response time becomes unacceptable. Let say, it is 140 users. Do more testing below operational ceiling to find bottlenecks. For example, run 3 tests (using warm-up time) with 100, 120 and 140 users, Sort detail grid to determine the slowest pages and requests. The developers might want to know that. They also may want to see more performance counters such as server memory, CPU, database and ASP.NET (e.g. ASP.NET\Requests Queued or NET CLR Memory). ASP.NET performance optimization is my favorite topic, but there are typically no short answers.
3. I'm not sure how to analyze page and request details node, can you explain?
The meaning of each column is described in tool tips that popup when mouse over the grid headings. Request details are result of direct measurement on HTTP level. Page details are result of aggregation of requests. The details could be helpful when certain requests or pages are slower than others to determine the bottlenecks.
4. Can we have an export for the whole test results node? I... and the graphs, test summary have to be saved separately.
Question. In What format would you like to see this report? Text (are images important)? PDF(hard to import in MS Office)? Excel (proprietary)? HTML (images will be in separate files – inconvenient to email)? XML (are images important)? Do you have any preferences?
It seems like page/request details are not exportable right now,
Select the entire grid, copy and pasted in Excel in tabular form.
5. If a request is slow, it should only affect the current virtual user's subsequent requests, correct?
Totally correct for primary requests, because until the response is received all other requests for that user are blocked. Partially correct for dependent requests. Based on selected browser type, each VU has a certain maximum number of available concurrent connections per host (IE7 has 2, IE8 has 6). One slow response will keep one connection, leaving remaining connections for other requests. But if all available connections are waiting for slow responses, then all requests will be blocked before some responses will start coming back. Again, this behavior describes a single user. If you will try IE8 instead of IE7, the client will use 3X more TCP ports and potentially can be faster. But if the server is the only bottleneck then this will not make a big difference.
For example, if another virtual users can get requests quickly, virtual user 2 (which starts after 1), might finish the iteration before user 1 does?
Correct.
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
6/16/2011
|
Hi Vadim,
Thank you for the response, I will look into your suggestions and will ask you for more help when I'm stuck.
"Question. In What format would you like to see this report? Text (are images important)? PDF(hard to import in MS Office)? Excel (proprietary)? HTML (images will be in separate files – inconvenient to email)? XML (are images important)? Do you have any preferences?"
I would like the export to be HTML, everything zipped up. I've used TestComplete before, and they have everything exported to mht format, which was pretty convenient, but having HTML pages, in a zip folder works for me too.
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/17/2011
|
Thanks for your thoughts on the report..
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/27/2011
|
Update: the Consolidated Test report is in development. Will be released in early July
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
6/29/2011
|
The Consolidated Test report is completed in v0.9 Update.
|
|
|
0
link
|
|
Huong Nguyen Posts: 69
7/7/2011
|
Thank you! I used it today and it's great!
|
|
|
0
link
|
|
Vadim @StresStimulus Administrator Posts: 583
7/7/2011
|
Thanks for the update, Huong
-Vadim
|
|
|
0
link
|