Posted on June 25, 2008 in Quality by adamNo Comments »

Guy Kawasaki has posted his newest interview in his On Innovation column at Sun. This time he interviews Ariane de Bonvoisin of The First Thirty Days which deals with change. Question 3 is Why are some people and businesses better at change than others? and what struck me is that you could replace change with testing in most of her answers. Or at the very least the traits describe I would consider helpful in testers.

  1. They have a positive belief about change and are generally optimistic. I call these people “change optimists.”
  2. They believe in the change guarantee: that something good always comes from change.
  3. They know that they possess a “change muscle”–that they are strong, capable, powerful, and intuitive enough to handle any change that comes into their lives or that they want to initiate.
  4. They refuse to become paralyzed by “change demons”–negative emotions that arise during change.
  5. They don’t resist change–choosing instead to accept the reality of their situation.
  6. They understand that their thoughts, the words they say and the feelings they allow themselves to experience during change have a direct affect on how easily they move through the transition.
  7. They believe that life has a deeper meaning than what can easily be seen or felt, that something greater is at play, and that no change is arbitrary.
  8. They surround themselves with a support team to help them move through change.
  9. They refuse to get stuck during change. They keep moving and take care of themselves mentally, physically, and emotionally.
Posted on June 25, 2008 in Process, Quality by adam1 Comment »

One of the key things I promote these days is the constant, continual peer evaluation. This helps solve a large part of the problem with automated unit tests which is that they cannot understand context. The most efficient way I have experienced to do this is the Buddy System. This is a big part of the Jonah Methodology even if they do not promote it on their site (though I think they should).

Remember back when you were a kid on a class trip and you had a buddy you had to stick with for the day (in theory to keep each other out of trouble)? The similar idea applies when developing software. The XP community has been using the Buddy System for years in the guise of Pair Programming. I’m pretty sure that the XPers have produced studies to show productivity games of having pairs of coders sharing a single keyboard but I don’t buy it. What the Buddy System does is take the benefits of having two sets of eyes on a change but without the full time consumption of pair programming.

Here is how the process works:

  • Developer A produces some code
  • Developer A asks Developer B to Buddy them
  • Developer B then sits with A for a session (more on this in a bit)
  • Developer B provides feedback on the change and when it is acceptable,
  • Developer A then commits their change recording in the the commit notes who the buddy was (you can enforce this using a pre-commit hook if you are using subversion

So what sort of conversations occur during a session?

  • Show it working
  • Which exceptions are you anticipating and how are they handled?
  • Do the existing unit tests all continue to pass?
  • Which new tests have you added?
  • Any Security issues?
  • Can the page / process handle non-English text data?
  • Are Corporate/Industry/Company standards met?
  • Does it build clean?
  • How have you improved the existing code?

Obviously the nature of the change will dictate which questions are asked but those are the big ones in most cases.

The concern people typically have when hearing about this the first time seems to be that it will make their development cycles longer. To borrow from the TDD crowd, it should actually reduce the time (or at the very least maintain it) since there will (in theory) be less rework / debugging associated to changes which have been buddied.

A smart build system coupled with the Buddy System are pretty strong foundations to build a quality system around.

Posted on June 19, 2008 in BITW, Quality by adamNo Comments »

I’m pretty sure that cruisecontrol.rb does not hand svn:externals properly so I’ve have to explore regexes in Ruby. In doing this I found the super handy Rubular online regex testing site. It is certainly easier to debug these things in realtime in a browser than kicking off a full cc build.

There is however a tiny glitch in it.

In HTML, multiple whitespaces in text get merged into a single one. If you want/need multiple ones then you need to insert the ‘not breaking space’ HTML entity code; & . The results of Rubular appear to be neglecting this.

Using the regex

(.*-r[ ]{2,}\d+\s+|\s?)(.*)

against the string

geokit -r       45 svn://rubyforge.org/var/svn/geokit/trunk

results in the following matches

  • geokit -r 45
  • svn://rubyforge.org/var/svn/geokit/trunk

Remember the rule about whitespace?

The first match should have been: geokit -r        45 

The devil, as is often the case, is in the details.

Posted on June 19, 2008 in Quality, Video by adamNo Comments »

I found a nice little video via a thread on /. called How To Teach a Healthy Dose of Skepticism. The video itself is called Here Be Dragons – An Introduction to Critical Thinking by Brian Dunning who runs the Skeptoid podcast. The title is somewhat misleading as it is directed to critical thinking in marketing (more or less) rather than in general, but it is still worth the 30 minutes or so to watch. Oh, and it is free. Of course, you’ll want to make your own notes, but here are mine.

  • With a made up concept and a few words the unknown becomes simple and satisfying
  • Pseudoscience – an idea that claims to be real but is not backed by any real science / evidence
  • If we don’t test / experiment we don’t learn / progress
  • Common warning signs to identify pseudoscience
    • Appeal to authority – white lab coat, celebrity endorsement, mentions certification
    • Ancient wisdom – We shouldn’t care that the ancient Chinese thought something works, where is the proof that it actually works
    • Confirmation bias – When we remember ideas that match our beliefs and forget those that don’t
    • Confuse correlation with causation – Just because two events look the related they might not be
    • Red herring – Irrelevant distractions that no way address the item under discussion
    • proof by verbosity – it’s not the quantity of information, it is the quality of the information
    • Mystical energy – energy is defined as measurable work capability; anytime you hear energy being used, substitute the real definition and see if it makes any sense?
    • Suppression by authority – there is no business reason to suppress science or invention; new things make money
    • All natural – lots of natural things are harmful too and the non-harmful ones are often synthetically created to be more effective/easy to produce
    • Ideological support – good science is done in the lab, not the courts/media/blogs
  • Apparently Brian sleeps with a clip-on mic?
  • The Law of Large Numbers disproves ‘psychic connection’ / ‘precognition’
  • There is a nice explanation of the triple-blind FDA trial
  • Why do smart people think crazy things

    Of course, you have to think critically about everything presented in the video.

Posted on June 17, 2008 in BITW by adam1 Comment »

In what could be the best bug-in-the-wild ever, it appears that a recent patch to Age of Conan had the unintended consequence of changing how endowed the female avatars are. But players, mostly the teenage male demographic naturally, have been told by the publisher not worry as they “are working on a fix for this and your breasts should be back to normal soon.”

Some lessons:

  • Always keep an eye out for accidental breakage; even in seemingly unrelated (body) parts
  • Know your audience and what they want. That knowledge can help you tailor your release checklists

I’m not sure what is most entertaining about this bitw

  • The outcry
  • That it got missed
  • That “check breast size” is now on a checklist somewhere
  • A PR person came into work and had to spend the day wordsmithing responses to complaints about digital bust sizes

I can’t claim to have discovered this; the original article that passed by me is at Wired.

Posted on June 16, 2008 in Quality by adamNo Comments »

I took my son to see Speed Racer on opening weekend and since then have had the new version of the soundtrack rattling around my head (which isn’t necessarily a bad thing). One line that keeps coming to the fore is a clip where a bad guy says “You have beaten us Speed Racer, but I solemnly swear that you will pay for our defeat. We will race you again. Soon. And we will beat you!”.

This is analogous to a tester’s battle with bugs in their product. We can spend hours/days/weeks/months/years searching for bugs in the code we are responsible for, but when the next release rolls around there will be a fresh batch of bugs that try to trip you up. They just do not give up.

There is also a line from Speed in the song where he laments “Why do you try to win this race fair and square?” which is appropriate. They don’t because they that is not what they do. Same with bugs. They are there because they always have, and always will be. And the sooner you accept this, you won’t beat yourself up (too badly) when something slips your net and makes it into the wild.

Posted on June 11, 2008 in Quality by adamNo Comments »

I’ve started to move my svn hooks from theory to reality. I’ll fix the earlier post with any corrections I may discover, but here is the main pre-commit script

#!/bin/sh
REPOS="$1"
TXN="$2"
SVNLOOK=/usr/bin/svnlook

# All checks passed, so allow the commit.
for f in `find $REPOS/hooks/scripts -name "pre-*"`
do
    if [ -x "$f" ]
    then
        $f $REPOS $TXN $SVNLOOK
        if [ $? -eq 0 ]
        then
           continue
        else
           exit 1
        fi
    fi
done

exit 0

And here is pre-non-empty.sh which makes sure that the commit message is not blank.

#!/bin/sh
#Make sure that the log message contains some text.

REPOS=$1
TXN=$2
SVNLOOK=$3

$SVNLOOK log -t "$TXN" "$REPOS" | grep "[a-zA-Z0-9]" > /dev/null 
if [ $? -ne 0 ]
then
   echo "Commit messages cannot be blank" >&2
   exit 1
else
   exit 0
fi

Of course, it doesn’t stop people from keyboard bashing to get around the check, but that will somewhat be handled by other pre-commit scripts and casual monitoring of the log.

Posted on June 9, 2008 in Quality by adamNo Comments »

Part of my current role is to establish the policies and procedures around testing and quality in general. One of things I like to do is tie version control commits with items in the bug system (I track features in there as well).

We’re using Subversion as the versioning system and FogBugz as the bug system. Now, the way to do the coupling is to download a script from your instance and save it to hooks/post-commit (unix). This is pretty easy, but in our case limits all the rules in post-commit to be perl. This might not be a problem except that we are a ruby shop, the person currently admin-ing the svn instance does things in shell and my brain’s first language is python. It also means that all the rules are in one file. I don’t really like either of those limitations.

A disclaimer before going any further. This is still theoretical as it hasn’t been deployed yet but it should work. Should being the magic word.

To work around these limitations, here is the post-commit script I’m going to use

#!/bin/sh

REPOSITORY="$1"
REVISION="$2"
SVNLOOK=/usr/bin/svnlook

# All checks passed, so allow the commit.
for f in `find $REPOSITORY/hooks/scripts -name "post-*"`
do
    if [ -x "$f" ]
    then
        # scripts need to notify people who care themselves
        $f $REPOSITORY $REVISION $SVNLOOK
    fi
done

What is does is loop through all the files in a scripts sub-directory and execute any of them that start with ‘post-‘. By adopting this simple naming convention anyone can add their own post-commit scripts in whatever language they are most comfortable with. The only requirement is that they exit with either a 0 or 1 which is how svn decides whether something is okay or not.

What the FogBugz post-commit script does is modify the bug record saying ‘change x’ so you can see what code changed as a result of this bug.

post-commits are only half the story though. pre-commit checks are just as important and are implemented in pretty much the same way only using ‘pre-‘ instead of ‘post-‘. I currently have 2 pre-commit scripts.

  1. #!/bin/sh
    
    REPOS=$1
    TXN=$2
    SVNLOOK=/absolute/path/to/svnlook
    
    if $SVNLOOK log -t "$TXN" "$REPOS" | egrep -i "^BugzID: [0-9]*$" > /dev/null
    then
        exit 0
    else
        echo "You must associate a check-in with a FogBugzID."
        exit 1
    fi

     

  2. #!/bin/sh
    
    REPOS=$1
    TXN=$2
    SVNLOOK=/absolute/path/to/svnlook
    
    if $SVNLOOK log -t "$TXN" "$REPOS" | egrep -i "^buddy: [\w]*$" > /dev/null
    then
        exit 0
    else
        echo "You must have a buddy check your commit."
        exit 1
    fi

What these two scripts do is check the commit message for a line that says ‘bugzid: id’ (which is necessary for the FogBugz integration) and one that says ‘buddy: name’ (for the buddy system I am trying to implement — more on that in a later post).

One other thing about this system is that if you want to turn off a rule for whatever reason, you just remove the execute bit from it and it is not going to fire rather than having to move or rename the file.

Update: Modified the post-commit script to not care about the return code of the scripts it calls since svn itself ignores (the commit has already happened). If the author of the script that is failing cares to be informed they will have to put the proper notification stuff in themselves.

Posted on June 3, 2008 in Quality, Ruby by adam3 Comments »

My first blog post defined Quality as Value (which originated from Jerry Weinberg but I heard it from James Bach). I still really like that definition but have been thinking about the expanded definition that I’ve been seeing more and more; Quality is Value to some Person who Matters. (I’m attributing the expanded definition also to James though I don’t have a direct reference to it. Instead I have a definition of a bug from a deck of Cem Kaner’s which if you work backwards give you the definition of a Quality. Actually, he uses stakeholder instead of person, but I like person better).

So why am I thinking about this now? I’m trying to concoct the testing strategies I’m going to try and one of the things I need to figure out is the ratio to test with each web browser. My fallback position on this is usually to look at the browser market share and use that as a pretty good slice.

But.

I’ve figured out that if you are going to use such a heuristic though that you really should know the situations where it might fail. For instance, if you are testing a site that is for mac power users you likely care more about Safari and Firefox than you do Internet Explorer. But even that is rather gut-feely which may or may not be acceptable [to you / management / the customer]. What you (hopefully) have though is the server’s access logs which is as close to am empirical source as you are likely going to get. Access logs are usually pretty safe as far as privacy leakage so you likely can get them if you asked.

Here is the browser percent breakdown from 10h 20m of access log from one of our properties:

	Internet Explorer 6: 31
	Internet Explorer 7: 14
	Internet Explorer 8: 0
	Firefox 2.x:         8
	Firefox 3.x:         1
	Safari 2.x:          1
	Safari 3.x:          6
	Opera:               0

Interesting.

What this shows me is that even though IE7 has the global lead in market share, our customer base is (currently) overwhelmingly using IE6 so any testing should be skewed towards that browser. Yes, in an office full of Macs with near 100% Firefox saturation making it work on non-IE browsers might give us a warm-and-fuzzy feeling in that technologically cool and standards compliant manner but it is (nearly) completely irrelevant; we don’t matter. In our environment, at this snapshot in time, the people who matter are IE users, 2:1 in favor of version 6.

(The script I used to parse the log is below the cut)
(more…)

Posted on June 1, 2008 in Quality by adam1 Comment »

To paraphrase Hugh MacLeod, Testing is never finished; it is ended.