clipped from: norvig.com   

Alexa Toolbar and the Problem of Experiment Design


Despite these caveats, the difference in the stats is quite profound. For example, I get about twice the pageviews of mattcutts.com, but his Alexa pageview ranking is about 25 times more than mine (I got this by looking at the 1 year, most highly smoothed graph on Alexa, and then squinting to guess at the mean, so that's yet another source of error in my study.) What that means is that people with the Alexa toolbar installed are 25 times more likely to view a page on Matt's site versus mine, but overall, all users view twice as many pages on my site. That's a 50 to 1 difference introduced by the selection bias of Alexa. Presumably this is because Matt's site is really appealing to a core group of SEO enthusiasts, many of whom also like the Alexa toolbar.