New Test Methodology
The scripts were basically identical, except they fed their data into 2 different IndexTools accounts. This allowed us to more accurately measure the impact of placement on counting of visitors to the site. While we implemented the two scripts sitewide, we focused our analysis on the Page Views to the home page.
The reason for this is that one key element of the study was to measure the delay between the execution of the two scripts. This would be hard to do in a rigorous manner across all the pages of the two sites, each of which is quite large and has highly varied types of pages. However, a specific high traffic page is easier to measure with some precision.
Our results in this test confirmed what we saw in the initial Shootout report. Here is the raw data for City Town Info:
Here are the raw results for Tool Parts Direct:
Here are the definitions for my cryptic labels in the results:
- Delta: Top Page Views – Bottom Page Views
As you can see, there is once again some loss of data in the results. So we have confirmed our theory that we formulated in the original Shootout report.
In addition, with this test, you can see how the longer delay that you see on Tool Parts Direct resulted in an increase in lost page views from 2.42% to 4.53%. Therefore it is reasonable to expect that a slower loading page would see a larger impact.
As any practitioner of analytics will tell you, the Data Quality Sucks, Let’s Just Get Over it (well maybe they won’t all use Avinash’s words, but they will tell you essentially the same thing). The bottom line in analytics remains that it is its relative measurement capabilities that matter the most. In other words, you need to focus on trends in the data.
For example, if you make a set of changes to your site you want to measure the scope of the impact those changes had. This is simply done by looking at your data before and after the changes were made. If the metrics you chose to look at improved, then it’s a sure bet that the changes you made will help your business. This is what is so wonderful about web analytics – you get much richer data than is available to measure the results of traditional marketing campaigns.
When someone comes to your site, and won’t wait 3 or 4 seconds, this is most likely due to a complete lack of interest that the person has in what they saw there. Of course, even pages that normally load quickly can have moments when they won’t load. Your web server could be down, for example. The nature of the lost data in this scenario is largely related to situations you can’t influence.
But, as the total time of delay scales up, you start getting situations where someone sees the link they want on your page while it is still partially loaded. You might not want to lose the ability to track those people. For example, they may be coming to your site from a PPC campaign.