This post, written by the StresStimulus team, is a reprint of the article originally published in the Spring 2016 issue of Methods & Tools magazine.
Beware of the nasty problem
You just installed a new load testing tool and are very excited to
start testing the performance of your web application. You selected this
tool because its training video showed how easy it is to use. Just
navigate through a test scenario while the tool records it. Then the
tool will emulate real world impact on the server infrastructure by
replaying this scenario with as many virtual users (VUs) as you need and
monitor how well your application performs under load. While this may
seem pretty simple, testing may not go as you planned. The website under
test can refuse to cooperate with the VUs. Errors may be thrown or
functionality may seem to be broken. This means you are likely dealing
with a correlation issue.
What this article is about: this two-part article presents a
deep dive into dealing with correlation errors often found when load
testing Web and mobile applications. Part 1 reviews performance testing
errors and explains how to recognize correlation issues. Published in
the next issue, part 2 will introduce a comprehensive method to fix
these problems.
Who should read this: Developers, software quality assurance
specialists and performance engineers who want to master performance
testing and avoid typical mistakes in handling testing issues.
What is correlation? It is a common mechanism in web applications consisting of two aspects as shown in figure 1.
- While communicating with multiple clients, the server generates
dynamic values such as cookies, tokens or IDs that are included in
responses. Every user session receives a unique set of values.
- The server then expects to receive these values back in subsequent requests.
The name "correlation" points out that the dynamic values in the
requests are related to the corresponding values in the previous
responses. A web browser naturally correlates the dynamic values. In
load testing tools however, the clients are virtual users and not
browsers, so correlation must be handled using other means.
Why is correlation necessary? Here are several examples:
-
Session ID. The server communicates with multiple clients at
any given moment. Because the HTTP protocol is stateless, the server
cannot determine which client sent a particular request by just using
protocol metadata. Therefore, the server generates a unique ID when it
establishes communication with this client, to match requests to the
client who sent them. The client should return this ID on all subsequent
requests.
Figure1 - How Correlation Works
-
Security tokens. The authentication process issues a security
token that is sent back and forth between the server and client.
Examples are Single Sign-on Security Token and AntiForgeryToken.
-
Application context or state. In client/server systems, the
server needs to have the application context of every user to process
its requests correctly. The application context may include various data
about past user activity. For scalability purposes, application context
might not be stored on the server, but rather be included in all HTTP
roundtrips. One example of a dynamic parameter that carries application
state is the "__ViewState" parameter in ASP.NET applications. On every
postback to the server, the web browser must return the same _ViewState
value that it received in the previous response.
In all of these examples, if the clients fail to return the expected
dynamic values generated on the server, the application will break.
How web browser implements correlation. The physical browser
is designed to carry-over values from responses to the subsequent
request thanks to several embedded mechanisms implementing correlation
logic. Here are a few examples:
-
Hyperlinks. Often time the server renders a web page with a
hyperlink containing a query string with a dynamic parameter. When a
user clicks on the link, the browser sends a request with the query
string containing this parameter.
-
Cookie correlation. The server sends cookie information to the client using the Set-Cookie header. The browser will include the cookie in every subsequent request to the server.
-
Web form hidden fields. By default, the browser will submit hidden fields unchanged to the server when a user posts back the form.
-
Programmatic correlation. The client’s code can include the
custom logic of storing dynamic server values and re-injecting them to
requests when required. Such logic can be implemented in the webpage’s
JavaScript, or client-side code executed in Silverlight, Adobe Flex or
another environment.
What is the problem with the virtual users? The essential
component of load testing is realistically emulating user-base impact on
a servers’ infrastructure by instantiating hundreds or thousands of
virtual users. A VU is a lightweight software object that generates HTTP
traffic. Its main job is to be indistinguishable from a real user from
the server’s perspective. Unlike a real client, they are not designed to
execute JavaScript or fully emulate the browser’s behavior. Therefore,
VUs do not implement correlation mechanisms. Instead, they just replay
the traffic as it was recorded while using a small footprint, so
hundreds or even thousands of users can be emulated from a single
machine. Therefore, correlation must be created manually.
Who correlates your test? Some load testing tools can
automatically create the required rules in the test configuration or
script. This method is called autocorrelation. Not all errors, however,
can be fixed this way, and the remaining issues must be addressed by
applying manual correlation techniques. This article describes a
comprehensive manual correlation method that simplifies it and makes it
more efficient.
Ten reasons why correlation errors are tough
Among load testing errors, correlation errors are the toughest. Here is why:
1. Not obvious. If you are a novice in performance testing, it
may not be obvious why correlation is necessary. It appears that the
end-user knowledge of the application under test should be sufficient to
record a scenario and run the test. Deep understanding of application
internals seems to be unnecessary for successful load testing. After
all, you are a tester, not a developer. The problem is that dynamic
variables that are at the center of correlation are hidden from the
end-user and live under the hood.
Side note: Does it mean that before testing
application, you have to learn how it is designed? Fortunately, no. The
comprehensive method of manual correlation described in Part 2 of this
article will allow you to correlate applications of any complexity like a
pro without spending time on learning application logic.
2. Correlate or not to correlate. Your test runs with errors.
Does it mean that you need to correlate it? It depends on the type of
errors you’re dealing with. Broken correlation causes some but not all
errors in the test. Non-correlation related errors cannot be fixed by
correlation and vice versa. It puts correlation errors in a class by
itself as you can see in load test error classification below.
Unfortunately distinguishing them from other errors can be challenging
as they look similarly.
3. Invisible errors. If you don't receive errors in your test,
it can be bad news. While HTTP errors are easy to find because their
response code is in the 400 or 500 range, some application errors can
escape detection. If the VU incorrectly replayed business transaction
messages, error pages return by the server can have a valid response
code 200. Such error can be easily missed unless you monitor responses
for specific known content indicating a failure. If you don’t know that
the problem exists, you don’t know that it should be fixed.
4. Unclear goal. Since correlation errors are difficult to
detect, it is also difficult to determine if the correlation rules that
you created fixed the error.
5. Deal breaker. Correlation is not necessary to make your
test better. It is necessary to make your test work. Running a
performance test before all correlation issues are resolved is largely a
waste of time. Because broken correlation causes errors in some
transactions, they will fail and complete sooner. If such errors are not
fixed, the average response times in the test report may be largely
inaccurate. Relying on such test result can cause troubles for your team
or clients.
6. Autocorrelation - false expectations? Some tools like
Jmeter and most other open source tools do not offer autocorrelation. It
means that even the most obvious dynamic variables like ViewState in
ASP.NET websites must be correlated manually. High-quality commercial
tools provide an autocorrelation feature. Does this mean that the
correlation problem is solved? Not really. Some top-notch tools can
reliably emulate standard behavior embedded in the web browser such as
web forms or cookie correlation. But the biggest challenge is presented
by applications where the dynamic variables are handled programmatically
by a client-side JavaScript or other client runtimes such as
Silverlight or Adobe Flex. Load testing tools would need to reverse
engineer such client logic to crack down hidden correlation rules, and
this is not easy. According to [1], the automatic correlation in load
runner works in only 5% of cases and manual correlation is more
advisable. If you expect that autocorrelation will resolve all errors,
you may be disappointed as some of the errors have to be rectified
manually.
7. Inconsistent result. To make things worse, you can never be
sure that a script that successfully handles one application will also
work with your next application or even the next version of the same
application. Sometimes simple upgrading underlying framework to the next
version will break your test script. Sometimes a script that worked
last week stops working on Monday even though no changes were made on
the server, except that over the weekend, the server was rebooted and
some values that seemed to be static last week are actually dynamic keys
that are re-generated on reboots. This unpredictability makes manual
correlation even more inevitable.
8. Platform specific vs. systematic approach. There are some
publications dedicated to correlation load tests of particular web
frameworks or popular Enterprise applications, such as PeopleSoft
, Oracle
BI, Cognos reports, SharePoint and Microsoft CRM Dynamics. They list
dynamic parameters belonging to these platforms and provide other
helpful platform-specific hints. This may create an impression that the
key to successful correlation is learning specific platforms. However,
it diverts attention from focusing on a systematic and comprehensive
approach that works in all environments.
9. Manual correlation is poorly defined. While there are some
useful publications dedicated to correlation, they largely lack a
systematic description of the manual correlation as a process. For
example, a vague explanation like "It may take some searching to find,
but careful study of a detailed log will eventually yield the session
id" [2] provides some hints to the solution but not the solution itself.
10. Correlation-is-getting-harder trend. As web applications’
versatility evolves over time, so does their complexity. Developers
"hide" more often dynamic variables in nontraditional parts of the HTTP
message like custom headers. An increasing variety of application
protocols and encoding methods create even more challenges for
correlation. Having a comprehensive approach to correlation becomes,
even more, essential.
Simplifying the problem
Fortunately, you do not have to know how dynamic values are created
or how they’re used in the web application to understand and solve the
correlation challenge. Instead, we can use a "black box approach" to
simplify the problem by focusing exclusively on significant aspect and
ignoring complicated and unessential details.
Three properties of the correlation:
- The server creates a dynamic value and includes it in response.
- The client sends back the same value in one or several subsequent requests.
- The value must change either across the VUs or across the iterations of replaying the test scenario.
Correlating the load test means to emulate such client behavior by creating correlation rules.
What is a correlation rule? It is an algorithm for extracting
dynamic values from a server response and using it to substitute some
recorded values in the subsequent requests. To create such rules you
need to ascertain:
- which responses contain a dynamic value and how to find it;
- which requests should use this value and what search-and-replace pattern should apply.
Correlation rules should be added to the test configuration or test script so the VUs can correctly handle dynamic variables.
To start fixing correlation errors, you need to know how to distinguish them from other errors.
Load test errors
There are three types of errors that you may encounter in connection with load testing (Figure 2):
1. System Errors are caused by the application, framework, or
network. They are not related to the test configuration. In an ideal
world, these errors must be exposed during functional testing and
addressed before the performance testing cycle. In reality, you may run
into them while recording a test scenario or verifying the script. It is
important to recognize them quickly and report to the system
stakeholders. If you falsely mistake system errors for test errors and
try to address them by tweaking the test configuration, you will be
wasting time.
2. Performance Errors, like the previous group, are thrown by
the system under test. However, they appear only on elevated load
levels. Tracking down such errors is one of the goals of load testing.
For example, a concurrency error happens when an application cannot
handle more than N concurrent users. Once the load emulation exceeds N
users, some transaction will fail. Performance errors should be
addressed last. A detailed discussion about fixing them is outside of
the scope of this article.
3. Load Test Errors are entailed by incorrect test
configuration and should be fixed by correcting test script,
configuration or execution. This group can be broken down into four
categories:
a. Recording Errors are caused by using an incorrect or old recording. Here are three examples:
- If you record traffic from the web browser that has primed cache,
some requests may be missing in the recording. To avoid such issues,
before recording, always clear the browser cache or launch a browser in
Private mode [3].
- If you started recording when the web browser displays one of the
pages, your script would miss some initial actions, such as session
initiation handshaking. When recording, always start a fresh browser
instance.
- If after script recording, the application was modified, and this
caused script verification errors, you need to either fix your script
manually or re-record. Fixing it may include editing, repositioning,
deleting existing or adding new requests.
Figure 2 - Load Test Error Classification
b. Test Environment Errors. The incorrect or incomplete configuration of the test environment can induce some errors. Here are two examples:
- After recording the scenario over HTTPS, replaying it with a single
user, triggers multiple problems due to an issue with certificate. To
fix it, install a certificate on the browser or mobile device.
- After recording the scenario, the application under test changed
host, port or schema (HTTP to HTTPS and vice versa). It should be
addressed by appropriate tool settings.
c. Correlation Errors are caused by missing correlation parameters.
d. Non-Correlation Errors are all remaining test errors.
The last two categories of errors are the most challenging. Let’s review non-correlation errors first.
Non-Correlation Errors
These errors have to be resolve first to leave the toughest
correlation errors for the end. Here are a few examples of
non-correlation errors:
-
Unique constraint. Your test scenario involves user entry of
unique data. When replaying the script with a single user, the system
will throw an error because of a unique constraint violation. These
types of errors are easy to replicate by repeating the same user entry
in the browser. To fix this issue, parameterize the recorded entry
values to substitute them with unique values on replay. See the
differences between parameterization and correlation below.
-
User session limit. Some applications allow one session per
authenticated user. If a real user tries to log in from a second
browser, the system will either reject authentication or log him off in
the first browser. When you replay a test scenario recorded on such
application, it will work with one VU. However, it will break with two
or more VUs. To resolve this issue, parameterize the recorded user
credentials with an array of credentials, so every VU is treated as a
unique user. (This is one of few non-correlation issues that does not
prevent successful correlation and, therefore, can be addressed after
correlation is complete.)
-
Client dynamic parameters. In some applications, the client
can also originate some dynamic values, such as a random GUID or a
timestamp. Requests with such values must be parameterized to avoid
replay errors.
Correlation vs. Parameterization. Parameterization helps to
resolve some of the discussed non-correlation errors. It is also used
for reasons not related to fixing errors. For example, you recorded a
test scenario of a person entering her name and contact information to
create a customer profile. To emulate such scenario more realistically,
dynamic data that varies on different test iterations should replace the
recorded values entered in the web form fields.
Similarly to correlation, parameterization replaces the recorded
value with dynamic data. However unlike the former, the latter does not
use values generated on the server [4]. Instead, it uses predefined data
sets, random data, or other information originated on the client such
as a timestamp.
To start fixing correlation errors, you need to identify them first. The next chapter describes how to do that.
Correlation Errors
If a stale recorded value is included in the request in place of a
dynamic parameter during the test scenario playback, the server will
encounter an application failure and will send back an appropriate
response. It's hard to predict how a particular application will react
to such correlation error. For example, it can return an abnormal HTTP
response code in the 400 or 500 range that is detectable by the tool,
but it may also try to inform a real user about the abnormality and
return a valid HTTP response with a warning message, an alert image or
text reporting an unhandled exception. It would be desirable to see the
content of the error message in the web browser. Unfortunately, this is
not feasible, as web browsers do not make correlation errors. So you
don’t know what content to expect in the responses, and therefore, it is
virtually impossible to track down all such errors automatically by
monitoring their content in the tool.
To overcome this challenge, we recommend using more broadly defined
criteria to identify the responses potentially triggered by correlation
errors.
Signs of potential correlation errors:
- Recorded response code was 200, but the replayed response code is in the 400 or 500 range.
- Recorded response code was 200, but the replayed response code is 302, or vice versa (Fig. 3)
- Recorded and replayed responses were redirected to different URLs.
- The size of the recorded and replayed response are substantially different (more that 50-100%).
- The expected transaction did not complete. For example, the file was
not created, the email was not sent, or a database record was not
created.
Figure 3 - Compare Recorded and Replayed Response
As much as responses to correlation errors vary from application to
application, the resolution of correlation errors is not
application-specific. This means that you do not need to know the
application’s design to efficiently resolve correlation errors and that a
platform-independent correlation method can be developed.
Identifying potential correlation errors is the first step in fixing
them. Published in the next issue of Methods & Tools, the part 2 of
this article will describe step-by-step tool-independent correlation
method that works on all platforms.
References
[1] Correlation in LoadRunner: Ultimate Guide. http://www.guru99.com/correlation-in-loadrunner-ultimate-guide.html#automatic
[2] Automated Test Script Correlation. http://www.testingperformance.org/performance-articles/correlation
[3] How do I set my browser to Incognito or Private mode? http://www.computerhope.com/issues/ch001378.htm
[4] What is the difference between parameterization and correlation in HP LoadRunner.