Avoid Disjointed Experiences While Experimenting

AB TastyBlogExperimentationAvoid Disjointed Experiences While Experimenting

As your marketing, product, and engineering teams run experiments it is critical to think through the entire customer journey, avoid disjointed user experiences, and consolidate the data needed to make decisions. 

Have you considered how to keep consistency between anonymous users and logged users? And what about keeping cross-device consistency? Furthermore how will you reconcile data between anonymous users and logged users?

In fact this is quite an important topic as it concerns the integrity of your brand and product experience. Your users might feel a bit lost if the experience you provide them is not the same before and after logging in. Same goes for the ones that will start their journey on a desktop and then later decide to continue that journey on a smartphone.  And we are not the only one saying this. According to the fourth edition of the state of the connected customer study from Salesforce: In 2020 74% of the customers have used multiple channels to start and complete a single transaction, 66% of them have used multiple devices to start and complete a single transaction. And finally 76% of customers prefer different channels depending on context. So keeping consistency between these channels is fundamental.

Let’s take an example of what could happen if you don’t consider the consistency of the user experience: Let say that you have decided to run experimentation on one CTA that appears on both your website and in your mobile app. You want to A/B test the color of this button and would like to try 2 different colors, Green and Blue. It could have been something else, like a new payment or delivery option, but we’ll stick to the color of a button to keep it simple. You decided to allocate 33% of your users on the original variation (red), 33% on Variation 1 (green) and 33% on variation 2 (blue).

In the best case, a user experiencing this on a desktop while being anonymous could see a color for the button e.g. green and the same after logging in. He would have a 1/3 chance to see the same variation twice. 

But what would happen if he decided to continue the journey on another device? Like a smartphone. If you are lucky, he can see the same color as on his desktop, which is another 1/3 chance. And finally, he would see the same color as his desktop after logging in, because at least you could reconcile the user’s experience when he is no longer anonymous.

Here is a visual representation of the best situation where an anonymous user would have a 33% chance of seeing the same variation once logging in and an 11% chance of seeing the same variation if transitioning from one device to another.

Now in the worst situation.

A user could see the red variation while being anonymous, then the green variation when logged in. He could then see a green, or blue variation when transitioning to the smartphone. Without reconciling the experience. This is more likely to happen than in the best case scenario.

Well, at AB Tasty, we’ve thought about it too and in Flagship, we launched the Experience Continuity feature. This native feature requires 2 steps only to make you feel relieved about maintaining consistency in your users’ experience and having consolidated data : 

  • 1.toggling on the feature in your settings

  • Integrate the visitor_id reconciliation into your code following our new SDK method or Decision API workflow.

And that’s it –  you’re ready to go ! 

For the curious and tech savvy that wonder how the magic works, here is a short explanation:

A visitor landing on your website/app will belong to some variations. Those variations are (with and without the Experience Continuity enabled) stored on our side. When the visit signs into his account, we will ensure the same variation is experienced even if the visitorID changes. Now, what will happen if your visitor is using another device? As soon as he will sign in, we will check if he has already seen a variation, if yes, we will assign him the same one. If not, the experience will be kept from his anonymous to known session. It’s true that they may have a different variation before they connect to another device, but at least you make sure that they will have the same experience once they connect, regardless of their device.

Now with Experience Continuity, an anonymous user has a 100% chance of seeing the same variation once logging in. A user who transitions to another device and becomes anonymous again would now have a 33% chance of seeing the same variation but once logging in would go back to having a 100% chance.

Check out this visual representation of the different scenarios that can happen : 

Case 1:

Case 2:

In term of data visualization here is what the SDK will send to the API if Experience Continuity is activated: 


Anonymous session

anonymousId ⇒  null

visitorId ⇒  anonymous 1

VariationID ⇒ Green

Logged session

anonymousId ⇒  anonymous 1

visitorId ⇒  Jon Snow 

VariationID ⇒ Green

And then in your report instead of having two different visitors you will see only one the anonymous1 that have been transferred to Jon snow.

Now if he consult your product with another device here is what might happen: 

Anonymous session on another device

anonymousId ⇒  null

visitorId ⇒  anonymous 2

VariationID ⇒ Blue (might be green)

Logged session

anonymousId ⇒  anonymous 2

visitorId ⇒  Jon Snow 

VariationID ⇒ Green because we recognized the visitorId.

How are we able to reconcile this data ? The Decision API stores in a distributed key value database the variation seen by the anonymous visitor and links it to the logged user upon login. When the visitor logs in on another device, the Decision API retrieves the variation value from the previously assigned variation.

Learn more about Flagship and don’t sacrifice experience for experimentation!


Share on Linkedin
Share on Facebook
Share on Twitter

You might also like...

Enable your product team to put users in the center of product strategy, while enabling engineering to deploy code faster.
Trusted by
Request a demo
Don't throw away hard-earned traffic.
Make your website work overtime - so you don't have to.
click-through rate
conversion rate
average order value
Trusted by
Request a demo