Evaluating Split Tests

When you set up split tests to evaluate rules or third parties, you can use Yottaa dashboards to evaluate whether the third parties are slowing down your site. You can also use the Test Rules Slider and an external tool to check which variation of your split test is better for your page load time. Finally, you can set a custom dimension in your own analytics tool to the split test variation using the Rapid+JS API. Then you can use your tool to determine which variation is preferable. For help with this process, contact Yottaa Customer Success.

Yottaa recommends waiting until about 24-48 hours after you publish a split test rule to evaluate it, depending on how much traffic you assign the rule to.

To check a third party's impact on your website using Yottaa dashboards: 

  1. Open the Service Flow dashboard by clicking Inventory > Service Flow.

  2. To find the third party in the list, enter the name of the third party in the Custom Search box.

    The blue bar shows the average time it takes the third party to load, and whether it loads before events like onload and largest contentful paint. The Impact column shows the impact the third party has on page load time. If the rule you are testing controls whether the third party loads before onload, you can use this number to decide whether to keep the rule as-is.

  3. Click the arrow next to the third party's name to see the details view.

  4. Click Third-party Details to open the Third-party Details page.

    • For third parties that load before onload, the Page Delay Violations box shows whether the third party takes longer than 750 milliseconds (or your custom threshold) to load.

    • For third parties that load after onload, the Performance Risk Violations box shows whether the third party takes longer than 1500 milliseconds (or your custom threshold) to load.

    • The Size Violations and Load Failure Violations boxes show whether the third party is loading large files or images, and how often it fails to load altogether.

    • The Service Flow by Page Category graph shows the third party's loading behavior on different page categories.

      The violation rate data boxes show the third party's violation rate site-wide, so if the third party only loads on a small percent of pages, take this into account when evaluating performance.

      For example, if your split test only allows the third party to load on 25 percent of pages, and the violation rate is 1 every 4 pages, you can estimate that the third party would have a violation rate of about 1 per page if applied site-wide.

You can use the Test Rules slider and an external tool to compare metrics like onload time between your split test variations.

To evaluate a split test using the Test Rules slider: 

  1. Do one of the following: 

    • If you are evaluating a performance rule, click Configuration > Performance Rules to open the Performance Rules page.

    • If you are evaluating a security rule, click Configuration > Security Rules to open the Security Rules page.

  2. Click Test Rule.

    The Test Rule slider opens.

  3. Click Published Actions.

    A set of options appears.

  4. Do one of the following: 

    • If the rule is applied to a specific page on your site, under Test URL, enter the page's URL.

    • If the rule applies site-wide, accept the defaults.

  5. Under How to Preview, right-click the link and click Copy Link.

  6. Open Web Page Test, or another external tool for site evaluation, and enter the link. You can compare the results with your overall site performance, or, if you made another rule for the other variation of your split test, you can repeat this procedure and compare the results.