My first blog post defined Quality as Value (which originated from Jerry Weinberg but I heard it from James Bach). I still really like that definition but have been thinking about the expanded definition that I’ve been seeing more and more; Quality is Value to some Person who Matters. (I’m attributing the expanded definition also to James though I don’t have a direct reference to it. Instead I have a definition of a bug from a deck of Cem Kaner’s which if you work backwards give you the definition of a Quality. Actually, he uses stakeholder instead of person, but I like person better).

So why am I thinking about this now? I’m trying to concoct the testing strategies I’m going to try and one of the things I need to figure out is the ratio to test with each web browser. My fallback position on this is usually to look at the browser market share and use that as a pretty good slice.

But.

I’ve figured out that if you are going to use such a heuristic though that you really should know the situations where it might fail. For instance, if you are testing a site that is for mac power users you likely care more about Safari and Firefox than you do Internet Explorer. But even that is rather gut-feely which may or may not be acceptable [to you / management / the customer]. What you (hopefully) have though is the server’s access logs which is as close to am empirical source as you are likely going to get. Access logs are usually pretty safe as far as privacy leakage so you likely can get them if you asked.

Here is the browser percent breakdown from 10h 20m of access log from one of our properties:

	Internet Explorer 6: 31
	Internet Explorer 7: 14
	Internet Explorer 8: 0
	Firefox 2.x:         8
	Firefox 3.x:         1
	Safari 2.x:          1
	Safari 3.x:          6
	Opera:               0

Interesting.

What this shows me is that even though IE7 has the global lead in market share, our customer base is (currently) overwhelmingly using IE6 so any testing should be skewed towards that browser. Yes, in an office full of Macs with near 100% Firefox saturation making it work on non-IE browsers might give us a warm-and-fuzzy feeling in that technologically cool and standards compliant manner but it is (nearly) completely irrelevant; we don’t matter. In our environment, at this snapshot in time, the people who matter are IE users, 2:1 in favor of version 6.

(The script I used to parse the log is below the cut)

#!/usr/bin/ruby

# buckets for totals
browsers = {"ie6" => 0, "ie7" => 0, "ie8" => 0, "ff2" => 0, "ff3" => 0, "s3" => 0, "s2" => 0, "opera" => 0}
bots = {"msn" => 0, "slurp" => 0, "googlebot" => 0, "become" => 0}

# put the gateway ip's for your office here
office = ["x.y.z.a"]

total = 0
File.open("/Users/adam/work/parser/adam.log", "r").each do |line|
    # one crazy regex -- token[9] is the agent string
    token = /^(.*?)\s+(.*?)\s+(.*?)\s+(\[.*?\])\s+(\".*?\")\s+(\d+)\s+([\d-]+)\s+(\".*?\")\s+(\".*?\")$/.match(line)

    # remove the office gateway
    if office.index(token[1])
        next
    end

    # counter
    total += 1

    # grab the user agents
    # ie6
    if token[9].index("MSIE 6")
        browsers["ie6"] += 1
    # ie7
    elsif token[9].index("MSIE 7")
        browsers["ie7"] += 1
    # ie8
    elsif token[9].index("MSIE 8")
        browsers["ie8"] += 1
    # ff2
    elsif token[9].index('Firefox/2')
        browsers["ff2"] += 1
    # ff3
    elsif token[9].index('Firefox/3')
        browsers["ff3"] += 1
    # safari
    elsif token[9].index('Safari')
        # 3.x
        if token[9].index('Version')
            browsers["s3"] += 1
        # 2.x
        else
            browsers["s2"] += 1
        end
    # opera
    elsif token[9].index('Opera')
        browsers["opera"] += 1
    # msn bot
    elsif token[9].index('msnbot')
        bots["msn"] += 1
    # yahoo slurp
    elsif token[9].index('Slurp')
        bots["slurp"] += 1
    # googlebot
    elsif token[9].index('Googlebot')
        bots["googlebot"] += 1
    # becomebot
    elsif token[9].index('BecomeBot')
        bots["become"] += 1
    else
        #puts line
    end
end

puts "Total hits: " + String(total)
puts "\tInternet Explorer 6: #{((Float(browsers["ie6"]) / total) * 100).round}"
puts "\tInternet Explorer 7: #{((Float(browsers["ie7"]) / total) * 100).round}"
begin
    puts "\tInternet Explorer 8: #{((Float(browsers["ie8"]) / total) * 100).round}"
rescue ZeroDivisionError
    puts "\tInternet Explorer 8: 0"
end 
puts "\tFirefox 2.x:         #{((Float(browsers["ff2"]) / total) * 100).round}"
puts "\tFirefox 3.x:         #{((Float(browsers["ff3"]) / total) * 100).round}"
puts "\tSafari 2.x:          #{((Float(browsers["s2"]) / total) * 100).round}"
puts "\tSafari 3.x:          #{((Float(browsers["s3"]) / total) * 100).round}"
puts "\tOpera:               #{((Float(browsers["opera"]) / total) * 100).round}"