Note: New feature as of 0.10.22.4

Overview

This feature allows you to run experiments where segments of your traffic get the page rewritten with different settings in order to figure out which filters work best for your site. It reports to your Google Analytics account, storing data in a custom variable.

Configuration

To run an experiment you must set several options in the pagespeed.conf file. First, you turn on this feature and tell it your Web Property ID:

    ModPagespeedRunExperiment on
    ModPagespeedAnalyticsID UA-XXXXXXX-Y

This will enable several filters needed for experiments including the Insert Google Analytics filter to insert a tracking snippet into each page. If you already have a tracking snippet on your pages, you should remove it so that the one mod_pagespeed inserts is the only one.

To disable experiments and experiment tracking you can set RunExperiment to 'off':

    ModPagespeedRunExperiment off

    # If you still want mod_pagespeed to automatically insert the Google
    # Analytics tracking snippet, you need to enable the insert_ga
    # filter.
    ModPagespeedEnableFilters insert_ga

Once you have turned on RunExperiment and set your Google Analytics id you can set up an experiment. For example, to test how much mod_pagespeed is speeding up your site you can apply optimizations for only half your visitors:

    ModPagespeedExperimentSpec id=1;percent=50;default
    ModPagespeedExperimentSpec id=2;percent=50

Each ModPagespeedExperimentSpec defines an experimental treatment or specification with a semicolon-separated list of settings. Some settings are required:

id

A positive integer, unique across all experiments. You can't reuse ids from one experiment to the next; every id must be unique. If you stop running an experiment that had three experiment specs 1, 2, and 3, then your next experiment should start with id 4 or higher.

percent

An integer between 0 and 100 indicating what fraction of users should receive this treatment. If the sum of percent over all treatments is less than 100 then the remaining users will be assigned to 'no experiment' and continue seeing what they usually would. If the sum exceeds 100 it will report an error when parsing the config file.

ExperimentSpec 1 above adds the optional setting default, described below, which tells mod_pagespeed to apply just the filters and settings it would normally apply. ExperimentSpec 2 specifies nothing and so turns off all filters.

To report on these settings, mod_pagespeed will inject JavaScript into the page to send data to your Analytics account. It will set one of the following for each visitor:

_gaq.push(['_setCustomVar', 1, 'FuriousState', 'Experiment: 1']);
_gaq.push(['_setCustomVar', 1, 'FuriousState', 'Experiment: 2']);

In addition to id and percent described above, there are additional optional settings intended to give you a way to test almost any configuration you could set in the pagespeed.conf:

default

Apply only the filters and settings that would normally apply. Warning: this setting cannot be used in combination with any others: it overrides any other configuration specified in the experiment spec and the other settings are silently ignored. This means that in default,enabled=remove_comments the enabled=remove_comments would have no effect. Note that while all the other settings are in the form key=value, as in percent=42, this setting has no 'value' and is just default.

level

Set the ModPagespeedRewriteLevel.

enabled

A comma-separated list of filters explicitly enabled. For example, enabled=rewrite_images,inline_js. Equivalent to ModPagespeedEnableFilters.

disabled

A comma-separated list of filters explicitly disabled. Equivalent to ModPagespeedDisableFilters.

css

The maximum size CSS file to inline, in bytes. Equivalent to ModPagespeedCssInlineMaxBytes.

js

The maximum size JavaScript file to inline, in bytes. Equivalent to ModPagespeedJsInlineMaxBytes.

image

The maximum size image file to inline, in bytes. Equivalent to ModPagespeedImageInlineMaxBytes.

ga_id

The Google Analytics ID you would like the data of this experiment reported to. If not specified, this experiment will use the Analytics ID specified with ModPagespeedAnalyticsID.

slot

Google Analytics provides five slots for custom variables. Here you specify which custom variable slot you would like to use. The default is slot 1, but you can change this setting globally with:

ModPagespeedExperimentVariable 2

Visitors are assigned to ExperimentSpecs on a week-to-week basis. This means that when you finish one experiment and start another, it will take up to a week for repeat visitors to be assigned to the new experiment.

Examples

Run an experiment on 30% of visitors where half get the default configuration and half get no filters. Use Google Analytics ID UA-XXXXXXX-Y to run the experiment, and default to logging the experiment information into custom variable slot 1:

    ModPagespeedRunExperiment on
    ModPagespeedAnalyticsID UA-XXXXXXX-Y
    ModPagespeedExperimentSpec id=1;percent=15;default
    ModPagespeedExperimentSpec id=2;percent=15

Run an experiment on 30% of visitors where one third get the default configuration, one third get a new configuration which has the core filters plus inline_preview_images and remove_comments without rewrite_javascript, and twice the normal CSS inlining limit, and one third get no filters. Use Google Analytics ID UA-XXXXXXX-Y and custom variable slot 3:

    ModPagespeedRunExperiment on
    ModPagespeedAnalyticsID UA-XXXXXXX-Y
    ModPagespeedExperimentVariable 3
    ModPagespeedExperimentSpec id=3;percent=10;default
    ModPagespeedExperimentSpec id=4;percent=10;level=CoreFilters;\
enabled=inline_preview_images,remove_comments;\
disabled=rewrite_javascript;css=4096
    ModPagespeedExperimentSpec id=5;percent=10

While the RunExperiment feature supports testing many changes, as shown above, the discipline of changing only one variable at a time helps you see the effect and know where it's coming from. The only drawback is that sometimes two filters are more useful in combination, such as minification and inlining, where minifying brings resources down to the inlining threshold.

There is also a tradeoff with the experiment percentages: a larger percentage means you will collect sufficient data for meaningful results more quickly but it also affects more of your users. If you're testing manually with query parameters to be sure your settings aren't making things worse for you, the downside of running with large experiment percentages is low. While you may not have determined the ideal settings right away, you're unlikely to have made things much worse.

Reporting

After you have been running an experiment for 24 hours Analytics will have results for you. The screenshots below walk through the process of creating a custom report to view them.

1. Set up advanced segments

   1. Advanced segments let you label fractions of your traffic. To view speed results broken down by experiments you need to add segments for each experimental group. Click here to import a segment for a sample experimental group.
   2. If prompted, log into Google Analytics:
      
   3. Select the profile that your experiment data was collected under, enter a name for the segment, and click Create Advanced Segment:
      
   4. On the next screen you can adjust this custom segment to fit your situation. If you used a custom variable slot other than the default, which is 1, change Custom Variable (Key 1) and Custom Variable (Value 01) to the appropriate values. Where it says Experiment: 1 replace 1 with the experiment id you used:
      
   5. Click Save Segment.
   6. Repeat these steps for the rest of your experiment ids.

2. Add advanced segments to the speed report

   1. Click All Accounts.
   2. On the dropdown menu click the name for your profile. In the screenshot below this is labeled your profile name [DEFAULT].
      
   3. Click Standard Reporting:
      
   4. Click Content
      
   5. Under Content click Site Speed and then Page Timings to bring up the Site Speed Page Timings page:
      
   6. Click Advanced Segments.
   7. Under Custom Segments you will see the segments you added above. Check the boxes next to them and click Apply:
      
   8. This takes you to back to the Site Speed Page Timings page.

3. Examine experimental results

   1. The Explorer tab shows average timing information both overall and, below, per-url:
      
   2. Averages can be misleading, however, because one user who suffers abnormally high latency before the onload event will have an outsize effect on the average. Unless your sample is very large, such users can make an experiment falsely appear to be a success or failure. To overcome this problem a histogram is helpful. Click the Performance tab to see page load times grouped into intervals.
      
   3. To see more fine grained timing detail, click the + icon next to any histogram interval to expand it.
   4. In this experiment mod_pagespeed appears to have moved about 8% of visits from the 1-3 second category to the 0-1 second one. For a simple mostly-text site running only the core set of filters, this is a reasonable result. As a next step the webmaster of this site might examine the filter list to determine if there are filters that are not in the core set which might be good targets for future experimentation.

Technical Implementation

When a visitor first arrives on a site with this feature enabled mod_pagespeed chooses an experiment from among the ExperimentSpecs. If you are running your experiments on less than 100% of your visitors, some will be assigned to "no experiment", which mod_pagespeed represents as id=0. It generates the page in accordance with the ExperimentSpec and then sets a _GFURIOUS cookie valid for one week, so that on future page loads it knows which ExperimentSpec to apply to this visitor:

   Date: Thu, 10 May 2012 14:19:43 GMT
   Server: Apache/2.2.14 (Ubuntu)
   Accept-Ranges: bytes
   Set-Cookie: _GFURIOUS=3; Expires=Thu, 17 May 2012 14:19:43 GMT; Domain=.www.example.com; Path=/

You can test this on your site with the curl program:

   $ curl -D- -o /dev/null http://yoursite
   ...
   Set-Cookie: _GFURIOUS=1; ...
   ...
   X-Mod-Pagespeed: ...
   ...

If you don't see a Set-Cookie header in the output, this feature isn't set up properly. If you don't see an X-Mod-Pagespeed header, mod_pagespeed isn't running at all. On each page mod_pagespeed adds some JavaScript to set a Google Analytics custom variable:

   _gaq.push(['_setCustomVar', 1, 'FuriousState', 'Experiment: 4']);

While it does try to detect when a page already is using Google Analytics and add to it only what it needs, this is imprecise and can get it wrong. We recommend that you let mod_pagespeed insert the only snippet. If you need to call other tracking methods, for example to set the sample rate for your site, you can do this by adding to the _gaq array within <head>. The code to do this for setting the site sampleRate to 80% would be:

   _gaq = _gaq || [];
   _gaq.push(['_setSampleRate', '80']);

Last updated November 12, 2012.