Split Tests
You can use split tests to check the performance impact of new performance rules, security rules, or third parties.
Split tests are different from the A/B tests that Yottaa's sales engineers perform when you first install your Yottaa product. With split tests, you can apply any performance or security rule to a percent of traffic in order to test its impact. You can also use split tests to compare the impact of two different third parties if you are unsure which one to install on your site.
You create split tests, and define the percentage of traffic to apply each test variation to, on the Context Intelligence Settings page. To access this page, click Settings > Context Intelligence.
You can apply a split test to a new or existing rule using the Create Performance Rule and Create Security Rule sliders.
On the Performance Rules page, you can click Test Rules to open the Test Rules slider. The Test Rules slider allows you to test each variation of the split test using WebPage Test or another third-party tool.
Example 1: Testing a Performance or Security Rule
The site administrator for the for the eCommerce company Mugs R' Us wants to make a performance rule to allow Facebook to load before onload. Currently, their site has a rule to load all third parties after the page has loaded , except for a select few that are essential to the initial loading of the site.
The site administrator creates a split test where 40 percent of traffic is called Load Facebook Normally and 60 percent is called Optimize Facebook. They then create a sequencing rule to load Facebook normally, and apply that rule to the Load Facebook Normally variation.
The site administrator monitors page delay violations to see how often Facebook causes page delay violations. Third parties can only cause page delay violations when they load before page load, so those violations are only caused by the Load Facebook Normally variation.
The site administrator also uses the Test Rules slider and Web Page Test to compare page load time for the two variations of the split test. They find that the site loads faster when Facebook is loaded after page load, so they delete the rule that sets Facebook to load normally.
Example 2: Testing a Third Party's Impact
The eCommerce company Rugs for All wants to test the impact of installing a Customer Reviews app on their site. The site admin starts by creating a split test calling 25 percent of the traffic Customer Reviews and 75 percent No Customer Reviews. They install the third party through a JavaScript injection rule, and apply the rule to the 25 percent of traffic . that they've labeled Customer Reviews.
The system administrator then monitors violations and anomalies to see how often the third party is causing issues. When they determine that the third party does not have a negative performance impact, they apply the JavaScript injection rule to all traffic.
Example 3: Comparing Third Parties
The eCommerce Company Best Maple Syrup wants to install a new chat widget on their site. They are trying to decide between ChatApp A and ChatApp B. The site administrator creates a split test where 50 percent of the traffic is called ChatApp A and 50 percent is called ChatApp B. They install each third party using a JavaScript injection rule. They apply each JavaScript Injection Rule only to its associated variation from the split test.
The system administrator then monitors violations and anomalies and sees that ChatApp B causes more performance risk violations than ChatApp A. The system administrator also tests each variation using the Test Rule slider and Web Page Test and sees that the variation with ChatAppB tends to load more slowly. They delete the ChatApp B JavaScript injection rule and apply the ChatApp A JavaScript injection rule to all traffic.
.