A Good Plan Gone Sour

Your organization is getting ready to launch a new Dynamics CRM deployment. Prior to that its performance has to be tested to make sure that the level of responsiveness and scalability expected by stakeholders will be delivered. Sounds like a good plan.

The first step is to make sure that your testing solution works. Using a load testing tool at hand, you record a simple scenario where a user logs in and then creates a record. Then the tool replays captured traffic and checks for errors. Now get ready for the bad news: unexpectedly, the test fails and no record is created by the test.

This situation will most likely occur whether you test online or on-premise deployment. It will happen whether your application is Dynamics CRM 2011, Dynamics 365 or any product version released in between them. It will occur with virtually any test scenario, authentication method and regardless of what load testing tool you use.

The core problem causing test failure is broken correlation. What is correlation, you may ask? This post will answer this question and help you learn how to overcome Dynamics CRM load testing hurdles. This is the first in a series of posts sharing tips on load testing Dynamics CRM using Visual Studio, JMeter, and LoadRunner.

Why Dynamics CRM is Resisting Scenario Testing

Load testing a web application starts from recording user actions with a tool that captures Web traffic and creates script.  By load testing here we mean a method of recording and playing back test scenarios on a protocol level that is used in top performance testing tools like Visual Studio, LoadRunner or JMeter. 

The tool then emulates how multiple physical users are accessing the application by instantiating virtual users replaying the script. At the same time, the tool monitors application performance to determine how user load impacts application responsiveness. 

Notably, along with application data, the recorded traffic includes dynamic tokens and IDs created by the server to maintain user session security and application integrity. These dynamic parameters are the root cause of the problem.

Correlation Challenge

Correlation in performance testing is a mechanism of extracting dynamic values from server responses and using them to substitute recorded values in the subsequent request during test execution. To learn more about correlation, check this article.

Many top load testing tools automatically correlate recorded scripts and replay them without errors. This, however, is not going to happen with Dynamics 365 scenarios. What makes this Microsoft ERP platform such a tough cookie when comes to load-testing? This question goes to the heart of its architecture.

• First, it is designed to allow broad customization capabilities provided in the product itself and by third-party plugins, workflows, and other enhancements. Every instance of Dynamics CRM is different. To maintain application stability, its architecture utilizes built-in entity IDs to uphold system integrity in a broad variety of configuration options and components used.
• Second, Dynamics CRM is built for utmost security. It supports a variety of authentication methods and technologies to operate securely in the online environment as in the on-premise environment. Here is an example:

WRPC tokens

As an example of secure architecture, let's review a mechanism of a Web remote procedure call (WRPC tokens). These tokens are used to validate the identity of user sessions, connections, organizations, and to guarantee content integrity. To assure successful correlation strategy, load testing tools or test script developers need to know the rules of sending and receiving such tokens. Unfortunately, these rules are not systematically documented. Instead, they are experimentally discovered by some investigative performance engineers and developers, and scattered in several blogs and forum posts. Here's what we learned after analyzing dozens of Dynamic CRM test scenarios. 

At different points in a user session lifecycle, the server generates WRPC tokens associated with different Dynamics CRM services. The tokens then get re-generated from time to time. To make things worse, each token is timestamped. Its validity is determined on the Server by matching up a token value, with its timestamp and the service it is associated with. If that’s not obscure enough, then note that tokens are sent between the browser and the server in a dissimilar matter. One token can be included in a JSON or XML response body, another will be included in JavaScript and the third is injected into a custom header. The tokens are then returned back to the server in an equally not-systematic manner. They can be included in the request body, header or even URL. In some cases, token values undergo some encoding on the client, which makes it even more difficult to discover the rules of handling WRPC token correlation.

During a load test, every time you see an INVALID_WRPC_TOKEN error, be aware that your load-testing engine simply failed to extract a token value from a certain part of a response, provide appropriate encoding if necessary, and then include it in certain parts of subsequent requests.

There are a number of blogs that discuss the challenge with the INVALID_WRPC_TOKEN error during performance testing. One workaround was to disable WRPC tokens. That was possible in older versions of Dynamics CRM. However, according to some reports confirmed by our testing, in the latest Dynamics CRM versions, a registry hack to disable WRPC tokens does not always work.

Light at the end of the tunnel

Is there a way to overcome the Dynamics CRM correlation challenge to construct reliable test scripts that can be used for error-free performance testing?

Fortunately, yes. The rest of this blog is dedicated to sharing some rules that can help you to correlate your test script regardless of what load testing tool you use.

Untangling the Dynamics CRM Correlation Spaghetti

Keep your tool

The bad news is that correlation rules for Dynamics CRM are not publicized. They are also complex. And there are many of them. The good news is that they can be revealed, documented and learned. Also, they are not tool-specific. This means that if you know the necessary rules for your scenario (learn some of them in this blog) you will likely be able to script them in the load-testing tool you currently have.

To prove this assumption, we conducted the following study:

First, we documented correlation rules for a simple test scenario as explained below. Then we asked experts in Visual Studio, JMeter, and LoadRunner, to record and configure the same scenario. The script at first did not work in any of these tools. But after the documented rules were applied, all of them were able to successfully complete the test.

Side note: In a series of subsequent blogs, we will provide a detailed account of the steps necessary to complete test script configuration in each of these tools. Also, stay tuned for the final blog where we will compare the usability and effectiveness of reviewed tools for performance testing Dynamics CRM.

With this said, not all load tools are created equal. For example, a tool that can fully auto-correlate your test script will save you a lot of time. But even when the tool available to you has insufficient auto-correlation and switching to a more appropriate tool is not possible, you can still make it work with some manual effort.

Methodology

In our journey to discover the Dynamics CRM correlation rules, we used StresStimulus, our load testing tool designed to automatically correlate Dynamics CRM products. A benefit of StresStimulus is that it clearly exposes all correlation rules it finds and helps you to visualize them and implement in other tools.

For the purpose of this paper, we used an on-premise instance of Dynamics CRM to create such schema. If you use an online instance, then your correlation schema can be slightly different.

Tested Scenario

We used a simple test scenario of creating a lead consisting of the following steps (Fig. 1)

1. Login to open Dashboard
2. Navigate to Sales –> Leads
3. Click New
4. Complete the lead form and Save

After recording the scenario, the test case script is displayed on a test case object tree (Fig 2) that includes transactions, requests, as well as extractors and parameters created by auto-correlation.

When replaying the script, an identical second lead is created. This suggests that auto-correlation is correct and complete. Recording a lead scenario in Dynamics CRM online is shown in this video.

Extractor - Parameter Tree

For the purpose of studying auto-correlation, one of the most helpful views is the Extractor-Parameter object tree shown below. It helps to visualize relationships between server responses with extractors and client requests with parameters.

Objects are presented on the tree within 3 hierarchical levels (Fig. 3):

• L1: The top level of the tree shows three responses (# 13, 231 and 276) used to create extractors.
• L2: The 2nd level shows extractor information including names and extraction rules. 
• L3: The 3rd level of the tree shows parameters created from each extractor and the request numbers where the parameter is created.

Correlation topology schema

Let's take a closer look at the portion of the extractor-parameter tree in the orange rectangle. It is presented in the correlation topology schema (Fig. 4).

Here is how to read this graph:

1. Three extractors are created from response 231:  
   • WRPC token for INLINEEDITWEBSERVICE, 
   • WRPC token for APPGRIDWEBSERVICE, 
   • WRPC token timestamp.
2. WRPC token extractor for INLINEEDITWEBSERVICE and its time stamp are used to create parameters in request 276 to InlineEditWebService.asmx.
3. WRPC token extractor for APPGRIDWEBSERVICE and its time stamp are used to create parameters in requests 278 and 280 to AppGridWebService.ashx.
4. An extractor of Entity ID for the lead is created from response 276.
5. This extractor is used to create parameters in requests 278 and 280.

Exposing Auto-correlation Rules

WRPC Tokens Extractors

Most often, WRPC tokens are included in the server response body as a part of JSON or JavaScript. In response 231 from Data.aspx they are included in the JSON body as follows (extracted values are shown in red)

"tokenData": [{
"Timestamp": "636325309991805260",
        "Token": "JR6oM0xZEeeAzAAVXQKjAPt6BMXpNlh19ftE4MHWc7eKce3SHduLtU7nmQOkfsYn",
        "Url": "\/CRMDB\/APPWEBSERVICES\/INLINEEDITWEBSERVICE.ASMX"
    },

Extractor editor for WRPC token timestamp is shown on Fig. 5. It uses regular expression:

"?Timestamp"?:\s?"([0-9]{12,})"

The extracted value is an 18-digit timestamp.

To extract a value of the WRPC token, the following regular expression is used:

"?Token"?:\s?['"]([a-zA-Z0-9\\/+]{24,})['"]

WRPC tokens parameters

Both, WRPC token value and timestamp, are sent in the request custom headers CRMWRPCToken and CRMWRPCTokenTimeStamp respectively.

Parameter editor for request 276 is shown on the Fig. 6.

Entity ID Extractors

Entity ID generated on the server is included in response body (#276) of InlineEditWebService.asmx request as a part of JSON or JavaScript (extracted value of Entity ID is shown in red).

<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><ExecuteResponse xmlns="http://schemas.microsoft.com/crm/2009/WebServices">
<ExecuteResult>
{
    "_entity": {
        "Id": "{411FA5C4-5B4C-E711-80CC-00155D02A300}",
        "TypeCode": "4",
        "TypeName": "lead",
        "TypeDisplayName": "Lead",
        "isRecordHierarchyEnabled": "False",
        "RowVersion": "552135",
        "Name": "Penny Shep"
    },
.........
}
</ExecuteResult></ExecuteResponse></soap:Body></soap:Envelope>

Extractor editor for Entity ID is shown on Fig. 7. It uses a regular expression:

"([0-9A-F]{8}\-{0,1}[0-9A-F]{4}\-{0,1}[0-9A-F]{4}\-{0,1}[0-9A-F]{4}\-{0,1}[0-9A-F]{12})"

Entity ID Parameters

Entity ID is sent as a part of an XML message in the request body in the <oId> attribute.

Parameter editor for request 278 is shown on the Fig. 8. The recorded value is shown crossed out. It is replaced by preceding highlighted extractor.

Conclusion

While performance testing of Dynamic CRM/365 is not trivial, it is achievable with the performance tool you already have, even if it failed in the past. To complete your test script, use the correlation rules described in this article. If your scenarios require additional rules, expose them by recording it in StresStimulus, downloadable here. Then use these rules as a blueprint to manually create extractors and parameters in your tool and take advantage of performance testing that finally works.

Visual Studio, JMeter, and LoadRunner users: stay tuned for coming posts with step-by-step script configuration guides.