Posted on April 30, 2008 in Python, Quality by adamNo Comments »

One of the first things I learned in testing is to use ‘real’ test data. First, a story.

My first job as a tester is was verifying financial software; typically for reporting capital positions to central banks. So there I was, on-site testing one for a Caribbean Trust company (think Tax Haven for wealthy Canadians) and using numbers I could nicely (easily) calculate in my head or scrap piece of paper as input. Things like $10, $5, $7.50, etc. Of course, at some point the person in charge of the company saw me do this and freaked. And as it turned out, rightly so. To open an account there takes a large monetary commitment, so their capital positions were many, many more digits than I was using. He was appeased, though still skeptical, with numbers like $100000000.00, $50000000.00 and $75000000.50.

This is ‘real’ test data because it adheres to its internal rules. All data has rules. Even ‘free form text’ has rules. The trick is of course figuring out what they are.

The good thing about rules, is that once you know them, you can exploit them.

I’ve been around a new testers (or people who have been temporarily conscripted to be testers) enough to have notice that there are patterns when creating test data. Take a ‘name’ field for instance. A new tester will often use their name first, their spouse’s name next, then their kids, the rest of the family, characters from TV shows and ending up with movie characters then they get stuck. The trap they have stumbled into is that while they did create data that met the rules of the field but their thoughts were influenced by the field label (‘name’).

Let’s pretend these are the rules around the ‘name’ field:

  • minimum 2 characters
  • maximum 60 characters (that is the column size where it will e stuffed into in the database)
  • spaces and hyphens are acceptable
  • case is preserved in the data base

But doesn’t ‘dofdsiiIOIDFk’, ‘dsklfjewojf-k’ and ‘dsfjslkfjl sjflksjdfiew’ also satisfy those rules? Of course they do, they are just hard to pronounce. Guess what? The system doesn’t care. All it cares about is that it is getting something that passed a set of rules that describe it. Once you have this epiphany you can start to interesting test data generation.

I’m not sure whether this is the fuzzy line between model-based automation or dynamic data driven testing but the theory is that you have the script do all the thinking for you. Here is a python script which will create unique test data forever (or at least long enough that it might as well be forever).

import random, string

name = []

valid = string.letters + " -"
min = 2
max = 60

how_many = random.randint(2, 60)
for x in xrange(0, how_many):
print "".join(name)

Here is a sample of what it generated

  • sWZidlWaWQ
  • EIMZpdFvYzhZINKQoByWWVxbqGXhhIU gp-FZR neMIgZfIaOsn
  • cdVaKADxlDJxABlMCF GmpqmyvQThDCUnLjfWp

Great. So what?

Well, now you can use your brain for thinking up interesting scenarios to test, not test data. The data is often just a means to an end in most cases.

You could also make this a function in a module some place and have your automation rig call it for test data instead of some hard coded value? Now you’re really getting somewhere.

As a summary…

  • All test data has rules
  • Anything that meets those rules is ‘real’
  • Rule identification can be hard
  • Once you know them, you can exploit them
Posted on April 28, 2008 in Quality, Selenium by adam2 Comments »

It seems that the posts that get the most attention these days are those that deal with Selenium. I’m not sure how I feel about that, but here is another one. Here is an ant file which will check a bit of environment stuff, launch the selenium server, run my tests (a custom metaframework), and then stop the server. Previously I was just running this whenever someone asked me to, but it was decided that it should be run as part of the nightly build which is controlled through ant.

<project name="Run Test" default="run_test" basedir=".">

    <!-- sensible defaults; override as necessary -->
    <property name="config_file" value="c:\path\to\framework\config.xml" />
    <available file="${config_file}" property="config.ok" />

    <!-- see for an explanation -->
    <property name="test_filter" value="ro_cas" />

    <target name="run_test" depends="check_jvm, check_config" description="Start Proxy ; Run Tests ; stop Proxy">
        <antcall target="start-server"></antcall>

        <exec executable="jams.bat" >
            <arg value="-c ${config_file}"/>
            <arg value="${test_filter}" />

        <antcall target="stop-server"></antcall>

    <target name="check_jvm" depends="get-jvm" unless="jvm.ok">
        <fail message="You need to use at least Java 1.5 for the selenium tests" />

    <target name="get-jvm">
        <condition property="jvm.ok">
                     <equals arg1="${}" arg2="1.4"/>

    <target name="check_config" unless="config.ok">
        <fail message="The config file you specified does not exist" />

    <target name="start-server">
        <!-- this is a 'known location' for this -->
        <java jar="../server/selenium-server-1.0-SNAPSHOT-standalone.jar"

            <arg line="-proxyInjectionMode"/>

        <waitfor maxwait="30" maxwaitunit="second">
                 <socket server="localhost" port="4444"/>
                 <!-- this url will 403, so we say that it should start counting errors at 404 to skip -->
                 <http url="http://localhost:4444/selenium-server/core/index.html" errorsBeginAt="404"/>

    <target name="stop-server">
        <get taskname="selenium-shutdown" 
            dest="result.txt" ignoreerrors="true" />
        <echo taskname="selenium-shutdown" message="DGF Errors during shutdown are expected" />

Like most things these days, I can’t claim the original work for everything involved; I just the following together:

Posted on April 26, 2008 in Quality by adamNo Comments »

It seems like the notion of ‘schools’ of thought is about to infect the development community with Martin Fowler’s recent post on the subject. The test community has been grappling with it in various forms for a number of years largely because of Bret Petticord’s presentation on the subject.

I used to be quite enamored with the notion of schools but I’m now starting to think they are almost more dangerous than useful:

  • Schools are often exclusionary
  • School membership is exclusive; much as you cannot be both a Christian and a Muslim you cannot be both a Context and Factory member (in testing parlance)
  • Schools characterize a certain set of beliefs at a specific point in time, by labeling them a school they become a fixed reference point and can cause splintering and sub-schooling.
  • Schools can be used in a derogatory manner

In a recent post on on software-testing (okay, from February) Chris McMahon mentioned he was trying to start a new school (which to my knowledge would be the 6th officially titled school of testing). I think in a way, everyone should be creating their own school having exactly one member; themselves.

Heck, I created the Adam School of Testing two years ago which has evolved significantly from what is listed there in the time since. This evolution is both natural and healthy but also absolutely necessary for the growth of a school — or in my model personal growth of the tester.

This post has been bouncing around my head for awhile a couple things conspired to make it fall into place.

The first was a post by David Byrne (ex-lead singer of Talking Heads wherein he mentioned of Paul Simon that He had made something that didn’t sound like any of his sources or inspirations, yet couldn’t have been made without them. To some degree this is missing from a lot of testing discussions where the Cult of Personality of some can overpower. I believe that those in the upper ranks of the testing elite (whatever that means) all have this mashup of ideas and beliefs at their core.

The next item also ties into that quote. Last weekend at the AST board meeting James was talking on a break about the important of community and colleagues to bounce ideas off of for mutual learning (or something to that effect) but was having trouble articulating in a concise fashion when I showed him the quote which worked quite well. During the course of the conversation he said that I was a good example of this as I’m not going to blindly accept something because person X said it was so. Instead I’ll keep what I agree with, reject what I don’t and challenge people when I’m undecided. I also reserve the right to change my mind.

At the same meeting, it appeared there are differences in opinion about what AST is and where it should go. (Thankfully the similarities are greater.) This surprised me since the founders and leaders of AST are closely associated with the Context-Driven School. If they can’t get all their ducks in a row then how can anyone else in a large (or even greater-than-small) collective of people?

All this circles back nicely to the notion of individual schools. Only you can have lived your life and have experienced your experiences and been influenced by your influencers. Being true to your own beliefs and ethics is going to cause you greater success in your career than proclaiming yourself a member of the Agile school or Context one (for example) just because someone you respect is also a member.

(And yes, I am aware that tailoring a schooled to yourself is the most extreme form of contextual adaptation out there thus in lines with the Context-driven approach, but that seems just a bit too weaselly to me)

Posted on April 24, 2008 in Quality, Video by adamNo Comments »

I tend to think that the security of a system should be tested before any other area (usually) so it’s not surprising that Neil Daswani‘s talk What Every Engineer Needs to Know About Security and Where to Learn It caught my eye. Unfortunately, if this is truely what every engineer needs to know about security then it is no surprise why so much insecure coade is floating around. Seems what every engineer needs to know is how to buy his book and/or visit his learn security website.

I did take some notes through.

  • I did not know, or did not know I knew, that XSS, SQL Injection and classical buffer overflows are in the same bucket of attack class; command injection. For some reason I thought they were independent but this does make a certain amount of sense.
  • According to the Security Focus Vulnerability database the following types of problems are increasing
    • Design Errors
    • Boundary Conditions
    • Exception Handling
  • Using the same data set, input and access validation problem rates are holding steady
  • Like most things, there is more than one such data set and they all measure things differently, however the top 4 problems seem to be
    • XSS
    • Various Injection holes
    • Memory corruption
    • DoD
  • Regardless of where the data is coming from there is an increase in the number of detected vulnerabilities. Of course, does that mean that we’re just better at detecting these or is the number being written increasing? (Likely both)
  • Neil thinks every engineer should be knowledgable in the following areas
    1. Secure Design – least priviledge, fail-safe stance, weakest link, etc.
    2. Technical Flaws (that can result in attacks) – cause, effect
  • Universities don’t teach security. A sweeping statement, but generally true
  • Security is a process, not a productBruce Schneier
  • Recommended Courses
  • Recommended Books
  • Recommended sites

Direct link here

Posted on April 24, 2008 in Quality by adamNo Comments »

I’m a big fan of YSlow for catching the low hanging performance fruit. The problem I encounter though is communication of my findings. Especially when trying to record trends over time. To solve that problem, and to act as a checklist for Yahoo’s other best practices for improving web performance I’ve created a worksheet which I’m making available to anyone who feels it would be helpful to them.

Posted on April 17, 2008 in Quality by adamNo Comments »

One of the traditional methods of generating the volumes of data to check for overflows is the Shoe Test (and its variation, the Book Test) wherein you put your shoe on the keyboard and let it press a key for awhile.

Here is a picture (technically it is a LOLCat) which illustrates this technique.

Posted on April 15, 2008 in Quality by adam2 Comments »

I clearly remember Captain Oddenes saying that going north at this time of the year is quite contrary to what his sailor experience would have recommended. But since the routing agency had all sorts of satellites and computers to help them to predict the weather, – Captain Oddenes decided to take their advice and go north, instead of following his own instincts and years of experience! – from The Storm in context of going straight through a hurricane. Complete with 22+ meter waves. There are a bunch of other great pictures there too.

The moral of the story? Have faith in your instincts; especially when you have trained them appropriately. The main difference between a new tester and one that has been doing it for 10 years is that they have had time to train their intuition. To think like a tester.

Posted on April 10, 2008 in Quality, Selenium by adamNo Comments »

File this one under ‘I wish someone had written this post already.’

Today I started exploring why my Selenium tests were not working in IE. Until now I have been able to get away with using Firefox and still be able to execute scripts even though we technically only support IE but the only feature that is a good candidate for automation in this last release doesn’t work in FF due to heavy javascript. Switching the browser from *chrome to *iehta should just work but nothing is ever that simple.

Right away I encountered an issue where selenium claims that it could not find my user-extensions.js even though the path I handed it was correct. A bit of searching showed that this has been a problem for at least 11 months

Anyhow, 3 hours later I have produced a patch which solves the problem. It is in the but report and below the cut.

Some lessons learned:

  • Open-source codebases are an absolute mismash of styles – The chrome and iehta launchers are significantly different in style both at a code level and philosophical level. In production software I would log this as a bug and hopefully would be picked up by static analysis tools and code reviews
  • Knowing how to code is increasingly important for testers – Okay, I knew this one, but it reinforced it. I could have waited forever for someone to fix this, but I did it myself. Just imagine how cool the tools would be that we could be using if everyone who used them could fix problems and add features.
  • Being able to Build your own lightsaber is critical
  • Firefox, being the darling of geeks, gets all the lovin’ when it comes to feature completeness – I’m amazed that passing in a custom user-extensions file to iehta wasn’t a huge priority for people. Again, in production software this would have been run up the priority list quite quickly.