This article was originally published on SitePoint.

Speed index is a relatively new method of measuring web page performance that was originally developed for webpagetest.org. It works by measuring how much of the above-the-fold content is visually complete over time until it is 100% complete (the part that is visible to the user when loaded initially).

The reason speed index is such a good metric is because it does a far better job at measuring how the user perceives the performance of a page. Take my blog for example, I’ve spent a long time optimizing it to get the speed index as low as possible, yet most pages actually take 2-3 seconds to load fully due to MathJAX’s numerous requests and work it does. Unless there is actual mathematical markup above-the-fold, this additional work will be completely ignored by speed index. Ignoring this makes sense from a user perception perspective because by the time the user has scrolled down to the part of the page where the math is used, it has likely loaded anyway. Optimizations for speed index also typically improve the mobile experience because they’re focused on getting something up and usable as soon as possible.

Calculating Speed Index

To measure the speed index of a page a video capture needs to be taken of the page load. Each frame is compared against the final loaded frame to determine their percentage completeness.

Plotting the visual progress of a page over time gives a graph something like this,

The area under the curve represents the visual complete progress, but since it is unbounded it doesn’t make a good score to measure as it will keep increasing. Instead, the speed index is the area above the curve which will approach 0 as the page gets faster.

Measuring Visual Completeness

The simplistic approach to measure visual progress is to compare the pixel at each position with the corresponding pixel in the completed frame. This turns out to have some issues though, particularly around new elements triggering a layout that shifts other elements around. Using this method, if a photo was positioned even a single pixel off its final position it would mark the entire sector as incomplete.

webpagetest.org measures visual progress by taking a histogram of colors and calculating the difference with the final frame. While this does have issues of its own, it fixes the problem outlined above.

Frames Per Second

webpagetest.org captures in 10 frames per second currently but that could change in the future, basically the more frames that are measured the more accurate the result will be.

Paint Events

There is another more recent method of calculating visual progress that uses browser paint events emitted via the remote debugging protocol. This removes the need to create a video recording of the page load which could potentially negatively impact the page load itself. This method is for Chromium powered browsers only however so it cannot be applied to all browsers.

Telemetry

Chromium have developed a testing framework written in Python called Telemetry that is responsible for generating all the results on the Chrome Performance Dashboard. It can also be used to run page load tests on URLs of your choosing to generate speed index and other valuable metrics like navigation timings.

Telemetry works with Chromium-powered browsers and is slowly increasing its support to browsers that implement the WebDriver API (see CR implementing IE support).

Setting Up Chromium

The instructions to get the Chromium source are available on the Chromium website, you don’t need to run gclient sync to synchronize the submodules or build the project since we’re only interested in Telemetry which is all contained within the main repository.

Creating a Telemetry Page Set

Telemetry works by running measurements on page sets. Measurements determine how a page is run and what metrics to gather, page sets are a collection of pages to run and the configuration under which to run them.

You will need to create a page set at ./tools/perf/page_sets/ with the URLs you are interested in measuring. Here is an example that measures two URLs on my blog:

from telemetry.page import page as page_module
from telemetry.page import page_set as page_set_module

class GrowingWithTheWebPageSet(page_set_module.PageSet):
  def __init__(self):
    super(GrowingWithTheWebPageSet, self).__init__(
        # Forces Math.random to use the same seed to improve result consistency
        make_javascript_deterministic=True,
        # Sets the browser to use the generic desktop user agent
        user_agent_type='desktop')

    urls = [
        'http://www.growingwiththeweb.com',
        'http://www.growingwiththeweb.com/data-structures/fibonacci-heap/overview/']

    # Add the pages to the page set
    for url in urls:
      self.AddPage(page_module.Page(page_set=self, url=url))

Running Telemetry

Once the repository has been downloaded the following command will run the page set.

# arguments
# use the page_cycler measurement
# use our custom page set
# use the system browser
# repeat the pageset 5 times
# allow testing of live sites (not the default)
# tell page_cycler to record and report speed index

./tools/perf/run_measurement \
  page_cycler \
  ./tools/perf/page_sets/growing_with_the_web.py \
  --browser=system \
  --pageset-repeat=5 \
  --use-live-sites \
  --report-speed-index

Results

This will run the measurement on your page set and report back various metrics in addition to a HTML file containing the results of this run and previous runs. Here is a snippet from the results of a run (warm means the site resources are cached):

RESULT cold_speed_index: http___www.growingwiththeweb.com= [1062,799,666] ms
Avg cold_speed_index: 842.333333ms
Sd  cold_speed_index: 201.525019ms
RESULT cold_speed_index: http___www.growingwiththeweb.com_2014_06_fibonacci-heap.html= [626,794,656] ms
Avg cold_speed_index: 692.000000ms
Sd  cold_speed_index: 89.599107ms
...
RESULT warm_speed_index: http___www.growingwiththeweb.com= [422,456] ms
Avg warm_speed_index: 439.000000ms
Sd  warm_speed_index: 24.041631ms
RESULT warm_speed_index: http___www.growingwiththeweb.com_2014_06_fibonacci-heap.html= [462,500] ms
Avg warm_speed_index: 481.000000ms
Sd  warm_speed_index: 26.870058ms

The framework is incredibly powerful, providing things like network speed emulation and recording of page sets for replay later to completely remove network noise. Explore more capabilities on the official site or by running ./tools/perf/run_measurement help list.

How Fast is Fast Enough?

Paul Irish, a Google Chrome Developer Advocate, recommends a speed index of under 1000 and to deliver the above-the-fold content in the first 14kb of the page.

Wrap Up

I’ve introduced the speed index metric and tools that can be used to measure them. Start tracking the speed index of your web site and other important metrics today with web page test or Telemetry.

More posts tagged Optimisation