Timeouts For Smoother Performance Testing 20. January 2016 StresStimulus Performance Testing (0) In a nutshell, every browser type has a certain limit of opened HTTP connections (e.g. 2 for IE6; 6 for IE8). If the number of pending requests (the ones that are issued, but yet to be received) exceeds this limit, the browser blocks and queues subsequent requests, until at least one of the pending requests is received. After that, the browser will issue one request from the queue for every received response, so the number of the opened connection stays within the limit. Request blocking only slightly slows down the browser because a typical website returns responses quickly. In instances where the browser will come across a very slow request, for example a large image, the user may abandon this request by continuing navigation without waiting for the response to complete. Why timeouts are necessary for smoother load testing: The detrimental impact of slow responses on load tests can be much more severe than on regular browsing. The most important part of a performance testing tool is the load engine, whose main job is to accurately emulate the behavior of hundreds or thousands of virtual users (VUs) with browsers. Under load, all responses slow down, especially those that are not optimized. Due to a large number of iterations and VUs, even a relatively small number of un-optimized responses can create a sufficient number of pending requests to completely block all requests. Unlike real users, virtual users cannot (and should not) make a decision as to which slow requests can be abandoned. As a result, the entire test can be halted. This is similar to how a few cars broken in different lanes can create a gridlock for the entire highway. Just like the timely removal of those broken cars can prevent gridlocks, the timely removal of the request blocking caused by slow requests can prevent halting the test. So, what can fix the problem? HTTP does not impose timeouts on long responses, and while web/application servers typically have a default timeout (for example, ASP.NET’s timeout is 110 seconds), there are no standards requiring web servers to timeout ever. This means that the load test can stop at any moment, and user will simply think that it hanged. That is where StresStimulus timeouts come in handy. How do they work: StresStimulus timeouts remove the obstacles created by slow requests to allow the other requests to continue the job. A timeout event is registered when the response was not received after reaching the timeout threshold. StresStimulus does not abort the timed out requests. Instead, StresStimulus marks such pending requests as “timed out”, removes the block from the TCP connection as if the request were received, and allows the blocked VU user to proceed with its request sequence. The timed out response will be dealt with separately when it arrives, which is helpful for troubleshooting and debugging. How to configure them: You can disable timeouts under Build Test Case settings. Timeouts work on the request level, not on the page level. The timeout threshold can be changed for every page (see below), and this value will be imposed on every request of the page. In v4.5 the default timeout threshold is 60 seconds. How they are tracked and reported: Every timeout is tracked in real time and is reported on the graph and the progress bar. At the end of the test, the number of the timeouts is reported for every request and every page. The page is considered timed out when at least one of its requests is timed out. StresStimulus treats timeouts somewhat similar to the errors. Timed out sessions are displayed in the grid in red, even though the response status code can be 200. Timed out response bodies by default are not purged to simplify troubleshooting. Unlike regular requests that have request and response events, the timeout requests additionally have a timeout event between those two. The responses of timed out requests completed during the test are counted toward the test metrics (average response time and size). Changing the timeout threshold may impact the number of timeouts in the test report. If this seems to be confusing, remember that the timeout mechanism is only necessary to allow the test to continue or complete sooner. You should not translate the number of timeouts in the load test to a real world user experience, since there are no timeouts in the real world. If the number of timeouts is relatively high, it can impact the accuracy of your test results. In this case I recommend a few options: Disable timeouts.Determine which requests are causing timeouts and optimize their performance.Reduce the amount of load.