|
Brianna Blanchard Posts: 76
6/17/2013
|
I need to look at the PerfMon data for a ramped load test, but only after the load has reached full capacity. Is there a way to do this automatically? I know that performance metrics are not taken during warm-up time, but the ramping up of users is very rapid during the warmup time (up to the full 100 users in less than 2 seconds), and I don't see a way to change it.
If not, would manually processing the data in the PerfMonCurves table in SQL give a reasonable approximation? I would be graphing the data taken after the full load was reached and finding the average and max. The exactness of the figures is not a priority: within a few percentage points is fine.
|
|
|
0
link
|
|
Max @StresStimulus Administrator Posts: 101
6/17/2013
|
I am not sure I fully understand your question. You can setup all perfmon counters in the beginning and ignore the values collected before the load test reached full capacity.
If ramping up users during warmup time is too rapid then increase the warmup time and the Step Duration in the Load Pattern.
|
|
|
0
link
|
|
Brianna Blanchard Posts: 76
6/17/2013
|
In a previous post, I was informed that the Key Indicator graph was intended as a rough insight as a test was running, and that manually processing the data would result in incorrect metrics. I was concerned that manually processing the PerfMon data would have a similar problem.
However, it would be more convenient if StresStimulus automatically waited until the full load was reached before taking data. It appeared to me that that might be the intent of the warmup time, but it seems you cannot set a ramped load for the warm up and then a steady load for the actual test. My initial question was basically if either a) there was another setting to delay the start of data collection, or b) some additional parameters for the warmup time to make it function as I needed. Since it seems you cannot control warmup seperately from the load during the test, at this point I need an explanation of warmup behavior in relation to the step load settings.
It appears that the fuctionality of the warmup feature is as follows (I am using v 2.5): when "step load" is used, the warmup ramps up to the number of uses specified in "Start VU". The rate at which is does this is not the same as the rate specifed for the step load, but rather seems to depend on the length of the warmup and the number of start VUs. Using the same Step settings and varying only the start VU and the warmup time, I've seen rates of anywhere from adding 50 users in less than 2 seconds to taking all allotted 15 seconds to add the specified 20 users, to jumping up to 20 before 30 more users over the next 30 seconds. Additionally, when the actual test begins when using a ramped load, the test does not actually begin with the number of users specified by "Start VU", but instead just starts at 1 and begins ramping.
When "steady load" is used, the warmup starts at some number below the total VU, but very quickly (over less than 2 seconds) ramps up to the full number of VU.
|
|
|
0
link
|
|
Max @StresStimulus Administrator Posts: 101
6/18/2013
|
In a previous post, I was informed that the Key Indicator graph was intended as a rough insight as a test was running, and that manually processing the data would result in incorrect metrics. I was concerned that manually processing the PerfMon data would have a similar problem.
Unlike Key Indicators, which have two datasets (data points and moments of sending / receiving requests), Perfmon data points only has one dataset - data points. Average values, such as average CPU utilization, are calculated at the end of the test are based on this dataset. These numbers have accuracy of Windows instrumentations
However, it would be more convenient if StresStimulus automatically waited until the full load was reached before taking data.
We might develop this feature if enough people ask for it...
It appeared to me that that might be the intent of the warmup time, but it seems you cannot set a ramped load for the warm up and then a steady load for the actual test.
The idea of warmup time is to warmup the server (i.e. database or alike) and not StresStimulus. You are correct that you can't set ramped load for warm up and steady load for the test.
My initial question was basically if either a) there was another setting to delay the start of data collection, or b) some additional parameters for the warmup time to make it function as I needed.
Besides warmup, there is no setting to delay data collection. The only parameter of warmup time is it's duration.
Since it seems you cannot control warmup seperately from the load during the test, at this point I need an explanation of warmup behavior in relation to the step load settings.
It appears that the fuctionality of the warmup feature is as follows (I am using v 2.5): when "step load" is used, the warmup ramps up to the number of uses specified in "Start VU". The rate at which is does this is not the same as the rate specifed for the step load, but rather seems to depend on the length of the warmup and the number of start VUs. Using the same Step settings and varying only the start VU and the warmup time, I've seen rates of anywhere from adding 50 users in less than 2 seconds to taking all allotted 15 seconds to add the specified 20 users, to jumping up to 20 before 30 more users over the next 30 seconds. Additionally, when the actual test begins when using a ramped load, the test does not actually begin with the number of users specified by "Start VU", but instead just starts at 1 and begins ramping.
You are right. The warmup rampup starts with 1 user and ramps up to the Start VU value adding equal users every second. After the warmup is over a new set of users is created for the actual test. We are looking into enhancing this approach in v 3.0
When "steady load" is used, the warmup starts at some number below the total VU, but very quickly (over less than 2 seconds) ramps up to the full number of VU.
This is by design. StresStimulus lauches users (even in Steady Load) with a small delay to not overload the system.
|
|
|
0
link
|
|
Brianna Blanchard Posts: 76
6/18/2013
|
You are right. The warmup rampup starts with 1 user and ramps up to the Start VU value adding equal users every second. After the warmup is over a new set of users is created for the actual test. We are looking into enhancing this approach in v 3.0
I am not seeing the behavior you described: the users are sometimes evenly added, and other times abruptly at the start. I've even seen two different ramping patterns within a single run (as described earlier, where 20 users were add immediately, then 30 added slowly). I have captured a video (attached as well as linked) showing one of the odd behaviors I'm seeing. Perhaps this is a bug?
I'm also still confused by the fact that the ramped load test does not actually start at the number of users specified in "Start VU". Is this number only used when determining where to stop ramping up?
The idea of warmup time is to warmup the server (i.e. database or alike) and not StresStimulus.
As a note, in case this comes up when discussing adding new features to warmup time, I am trying to warm up the server and not StresStimulus. I want to capture data (perfmon and "key indicator" type data) only when the server is at peak load, but I want it to get to peak load in a somewhat realistic manner (e.g. not adding 100 users in under 2 seconds). At this point, I will be getting around the limitation by extending the time at peak load and decreasing the ramp time to get a higher percentage of data points at the load I want to analyze.
Also, thank you for the clarification on the accuracy of PerfMon data.  I'm glad to know I can just use the data that is saved.
|
|
|
0
link
|
|
Max @StresStimulus Administrator Posts: 101
6/18/2013
|
I am not seeing the behavior you described: the users are sometimes evenly added, and other times abruptly at the start. I've even seen two different ramping patterns within a single run (as described earlier, where 20 users were add immediately, then 30 added slowly). I have captured a video (attached as well as linked) showing one of the odd behaviors I'm seeing. Perhaps this is a bug?
You are right. There seems to be a bug in the ramp-up process of warm-up. This will be fixed in the next release. The accuracy of ramping-up users during the warm-up has no affect on any test metrics no metrics are collected during the warm up time.
I'm also still confused by the fact that the ramped load test does not actually start at the number of users specified in "Start VU". Is this number only used when determining where to stop ramping up?
Like I stated before, users are never started concurrently to avoid overloading the system. Users are launched about 1ms apart. However, if in step load you set the Over property, users will launch over a period of time.
|
|
|
0
link
|
|
Brianna Blanchard Posts: 76
6/18/2013
|
Like I stated before, users are never started concurrently to avoid overloading the system. Users are launched about 1ms apart. However, if in step load you set the Over property, users will launch over a period of time.
I understand that they won't all start at the exact same time; that's not what I was asking about. The description of "Start VU" is "The intial number of VUs in the beginning of the test". Based on that description, if I set "Start VU" to be 50, then have a ramp rate of 2 users/sec up to 100, I expected 50 users to be added basically instantaneously (about 1 ms apart is fine), then after than point added according to the ramp rate. From your statement regarding the Over property, I take it this is not how Start VU was intended to work.
Upon some further investigation, it looks to me the Start VU is used as follows:
- Starting from 1, Start VU number of users are added at a rate of Step VU Increase/Over;
- When Start VU is reached, no more users are added until the end of a step (as determined by step duration; It will be the first step if Start VU < Step VU , or a later step if Start VU > Step VU)
- After the end of this step, the ramping continues at the same rate, but now adding Step VU Increase during each step.
Is my characterization correct? If so, it seems to me that "the initial number of users in the beginning of a test" is not an accurate description of how Start VU is used. I would make a feature request for the functionality I was expecting, but if the warmup functionality is expanded, than it seems likely that it would be a redundant feature.
|
|
|
0
link
|
|
Max @StresStimulus Administrator Posts: 101
6/19/2013
|
You are correct. This behavior is caused by the same bug in handling user rampup, which I mentioned earlier. It affects the rampup during the warmup and in the initial step of the test. I reported this bug in Bugs & Issues category here and will report back when it is fixed.
Thanks a lot for pointing it out!
As I mentioned before, in v 3.0 managing users will be more consistent. After the warmup is over the same set of users will be used for the actual test. It sounds like this approach would be more useful to your situation, since you will be able to set a warmup time until you reach peak load, after which metrics will start to be calculated.
|
|
|
0
link
|
|
Brianna Blanchard Posts: 76
6/19/2013
|
The 3.0 functionality sounds like what I'm looking for Glad to know that's in the pipeline.
Thanks for reporting the bug on the the other forum.
|
|
|
0
link
|
|
Max @StresStimulus Administrator Posts: 101
6/20/2013
|
OK. Since this feature will be helpful to you I posted it on the Suggestions & Feature Requests forum here, so you can track when it is released in 3.0 (hopefully in a few weeks)
Also, I renamed this post to better reflect its content.
|
|
|
0
link
|