Too Many Cooks

Stats and damn lies

So, you want to know who is looking at your blog? This is a very brief guide to explaining how the logfiles work and what they mean for your blog. But first, a quick caveat - I find logfiles really interesting right up to the point that I start having to do it for a job. I've spent a great deal of time, and a considerable amount of fustration over my many years trying to devine some secrets of human behaviours out of these damn things. With that out of the way, here goes nothing:

Every request made to stonesoup.co.nz is dutifully logged. This describes what was requested (eg which page or image), who requested it (ie IP address), when, and some other stats. A raw unprocessed logfile entry looks like this:

203.97.2.242 - - [23/Apr/2003:00:10:45 -0500] "GET /chinashop/ HTTP/1.1" 200 20774 "http://www.stonesoup.co.nz" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"

Roughly speaking, the first string of numbers is IP address that made the request, then you can see the date, then there is what was asked for (chinashop index page) followed by some other details and then, you'll note, a URL. This URL is the referrer. It is from this page that this person clicked from to go to iona's blog. The remaining stuff describes the browser and the operating system.

A logfile is made up of thousands of such lines. The webfarm hosting service keeps the log file going for a week at time. They delete the old entries and start from scratch every tuesday morning. If I want to do any processing myself (which, in my nerdy way, I do) I need to grab these files every Monday.

Fortunately all is not lost. You can see some rudamentry analysis of the entire stonesoup site at www.stonesoup.co.nz/wusage - there is a weekly history that goes back to the very first day of stonesoup. Unfortunately, this quite high level and not entirely applicable to just your blog. You can type in the following:
http://www.stonesoup.co.nz/wusage/weekly/2003/04/20/directories/directories.html

(Remember to change the date every week in the URL). From there you can pick out your blogs directory and get an idea of what your most popular page is.

Again, this doesn't really tell the whole story. This is why I do some logfile processing myself. Just so I can find out fun filled stats like this one:
view referral graph

The correct, and proper thing to do, would be for me to find a way of automatically downloading the monday logfile, split it into the blog directories and processing each one individually and then uploading the results for you to view. I might look into once the search engines kick in...

Oh yeah. Search engines. Right now, because of all the pings we are sending out to places like weblogs.com and blo.gs we are getting loads of spider traffic. This results in links to our blogs, the more links we get the more likely we will be picked up by google. When that happens, all hell will break loose. Expect to see some strange and wonderful search queries. Just ask Iona about all that Frodo and Buffy fanfic she's been publishing.

Still loads more work to do though. And as always with these stats, they create more questions than they do answers.

Next lesson: robots.txt and why you should give a rat's ass.

Posted by kim at April 24, 2003 10:52 AM