SEO AB Testing, John Lawson

SEO AB Testing

Find out whether an on-page change won you organic traffic. Change half the pages in a section, leave the other half alone, and this tool compares the two. It learns how your variant pages tracked your control pages before the change, predicts the traffic the variant pages would have had without it, and tells you if the difference is real.

Your test

Loaded: 400 product pages split 50/50. Title tags were rewritten on the variant half on 9 September and organic clicks measured for 30 days. A third series, extra_control, stands in for any outside measure the change can't affect, such as an untouched section of the site or a Google Trends index. A core update hit the whole site in August. EXAMPLE

Variant pages The bucket of pages you changed.

Control pages

Pages you left untouched. Add another unchanged section of the site here too if you have one.

Change went live

Measure until

Confidence

Model draws

Checking analysis server…

Running an SEO split test

Pick a section of similar pages built from one template: product pages, category pages, location pages. A few hundred pages or more gives a steadier signal.
Split the pages, not the traffic. Randomly assign half to variant and half to control. Sort by traffic and alternate down the list so both halves get a similar mix of big and small pages.
Check the buckets before you launch. Load your history with the planned split and set the change date to 30 days ago, as if a test had run. You should see no effect. If you do, reshuffle.
Change only the variant pages. Leave control pages exactly as they were.
Export daily clicks per page from Search Console covering 100 days before the change and everything since. Upload it with a column saying which bucket each page is in.
Allow for recrawling. Google needs days or weeks to see the change, so effects ramp up. Run for at least 3 to 4 weeks.

Getting reliable results

Control pages should move with variant pages before the change. A correlation above 0.7 in the controls table is a good sign.
Keep the change off control pages. Sitewide template edits, shared components and new internal links from variant pages to control pages can all leak the change and hide the effect.
Use enough history. Aim for at least three times as many days before the change as after it, such as 100 days before and 30 after.
Avoid other changes during the test. A migration, redesign or big content push affecting only one bucket will distort the result. Sitewide events like core updates are fine, because both buckets feel them.

Ready

Run the analysis to see whether your change moved organic traffic.

Data checks

Variant page traffic vs. expected

Variant pages Expected without the change 95% range

Did the change work?

Cumulative extra clicks

Range of outcomes

The change is significant once the whole shaded area lifts clear of the zero line, above it for a win or below it for a loss. The shaded area widens over time because the forecast gets less certain the further it runs from the data it learned from.

Daily difference

Effect summary

Control pages

Weight is how much the model relied on each control to predict the variant pages. A control with low correlation and low weight is not helping.

Test report

CausalImpact's own report