Posted on November 6, 2008 in GLSEC2008 by adam2 Comments »

This was my half-day tutorial. I think it went okay and have yet to hear anything to say that it sucked so that’s a positive. We’ll see what the evaluation forms say. I’ve posted the slides I used on slideshare for anyone who wants them. I’ve also embedded them below along with links to the posts which have the actual scripts. When looking at the slides, when you get to a slide that says ‘How to Use’ you should go to the relevant script to have things make some sense.

Here are the individual posts that I wrote while building this presentation (you can get all the scripts from this presentation from the individual posts):

Posted on November 6, 2008 in GLSEC2008 by adam1 Comment »

I’ve known of Jason Huggins for a couple years and we had exchanged emails a couple times about Selenium over that period. This was the first time we had met in person though. Another person off the list.

Anyhow, he was at GLSEC to do the closing keynote in which he compared the evolution of movies (silent to talkies) to how people learn how to use an application (rtfm to rtfv) with the advent of screencasts. Even if you take away the points for using a Three Amigos clip, there were a couple mind blowing ideas presented; especially if you are a small Agile shop trying to show progress to client’s on a iteration-by-interation basis. (Steve, this is directed at you.)

What if you engineered your automated tests to build a screencast that the client could then run to show completion and functionality? We have the technology (here too) to do most of it. All that is left is the glue to put it all together.

These could also be given to the end user as a video of how to use the product at which point the tests really are the documentation.

If I didn’t have so much already on my plate I would totally go off hacking right now…

Posted on November 6, 2008 in GLSEC2008 by adamNo Comments »

From a most-useful-in-my-job perspective, Jake Scruggs’ talk on metrics was it. He is the author of metric_fu which is a gem I keep meaning to incorporate into our code somehow. Here are the notes:

  • One of the first slides had a fortune cookie fortune: a can of worms won’t open itself
  • Hiring a bunch of smart people does not guarantee beautiful code
  • Review old code to see if that hot piece of code holds up 6 months later
  • Carlin’s law: everyone slower than me is stupid, and everyone faster is crazy
  • Code coverage
    • Coverage numbers are relative to the language and/or project for completeness
    • A good heuristic for number of tests: # tests = # paths
  • complexity measurement
    • Another cool tool (with a great url / logo) is Flog
    • See Jake’s post on the score values to help you interpret what they tell you. What, you mean 394.7 isn’t good?!?!
    • Complexity isn’t (always) bad, you just need to know where the complex parts are
  • A ruby cyclomatic complexity tool is Saikuro which I haven’t played with yet. But will.
  • Metrics are a guideline, they do not tell you what to do
  • Things that score better metrics wise might be worse for your code
  • Explain your cool (complex) code
  • inject can be evil from a complexity perspective, though I didn’t write down why. This post explains what inject does and it just seems like thinly veiled ruby black magic from here. Of course it could really be quite obvious and I’m just tired and/or suspicious of all things for black magic these days.
Posted on November 6, 2008 in GLSEC2008 by adamNo Comments »

I almost titled this write-up ‘Write your tests for a 3rd grader’ but the established pattern is to use the actual talk name, so didn’t. This talk was the only one I had a really strong negative reaction to.

This was essentially an experience report by Holly Barsamian of Steelcase on outsourcing their testing; and by testing she means QTP script creation and execution. This in itself wouldn’t have been a bad thing as there were some useful lessons learned such as

  • It takes about six months into this sort of relationship to really become efficient
  • Interview the staff the agency assigns to your project and don’t rely on your account manager to get clueful people

What really got me worked up was when we started talking about the test cases they employed. If you think of classical, dumbed down, pesticide paradox prone scripts then you have the idea. To be doubly sure I heard correct I asked whether she meant “In the firstname field put Adam and the lastname field put Goucher. Then press ok” to which point she said yes. Okay, so she does very restrictive and not very powerful test cases. So do lots of people. But what really amazed me was when she followed up with “As a rule of thumb they write them for a 3rd grader”. <BOGGLE> I know people treat outsourced resources as virtual slaves, but seriously. At least pretend to respect the people ‘testing’ your application. I have no doubt that there are dumb people in India just as there are in Canada or Spain or any other place, but if you are interviewing the people working on the application then you shouldn’t be the equivalent of a 3rd grader. Doesn’t seem like they were efficiently using their outsource resources at all for testing. (Maybe for creation of QTP scripts, but even then I would want my outsource resources creating data driven or model driven tests to truly get what I would call efficient, well written scripts).

Oh to have had LAWST facilitation…

Posted on November 6, 2008 in GLSEC2008 by adamNo Comments »

Michael Cloran is the CEO of Interactions and has a wicked cool application. It is hard to explain, but essentially they are a ‘smart’ IVR system where ‘smart’ means they know when people are better at doing something than a computer; so they dump it to a human to make the decision. That so doesn’t really do it justice, but it is pretty jaw dropping in it coolness. I can’t decide if it would be really, really hard to test that or relatively simple (with a whole bunch of devil-in-the-details stuff). Anyhow, as always, here are the things I found interesting.

  • Identity theft is a serious problem in call centers. In some locations, loan sharks are known to circle call centers and then when their ‘clients’ can’t make their payments they can pay in a broken leg or in stolen identities. To prevent this, the call centers are totally paperless so you can’t write things down, but smuggled camera phones thwart this policy. Reminds me a lot of what the diamond mines used to be like in South Africa (I’m a deBeers history buff)
  • All business logic decisions handled by computer, not people. People make mistakes and are not consistent. Computers (barring the bug thing) do not.
  • The only role humans have is to determine the intent of bits of a call which the system cannot automatically determine.
  • Time delays count a lot (audio is sped up when played to the humans so the system has caught up to the speaker for real-time communication)
  • To succeed, tell a compelling story to/and solve an annoying problem
  • Game theory plays a large part of how they get the humans to be productive and invested in the system
    • The interface is like a video game where the intent analysts get points for correct interpretation
    • ‘Correct’ is determined by submitting it to more than one analyst
    • But of course some people started gaming the system by intentionally getting incorrect answers to some things when new people came on board as there was common error patterns. (slick)
    • Thus proving it’s very hard to incentivize the correct behavior
  • They didn’t know what they didn’t know which meant they made what they thought were rational decisions at the time but turned out to be not so good a couple years later. (The whole known knowns, known unknowns and unknown unknowns thing)
  • There is never enough time to fix things later
  • Some mistakes they made
    • Not taking short pain of complete refactor
    • Lack of integrated testing carried forward
  • They use VersionOne to track stories, etc. (I had never heard of it)
Posted on November 2, 2008 in GLSEC2008 by adamNo Comments »

One of a tester’s key tools is the ‘tail’ command, especially when coupled with a -f which means that it will continually display the appended contents of a file providing a realtime view of a log for instance. Unfortunately this functionality does not exist for the windows logging system, Event Viewer. The consistency oracle says that are application should log to this system so we are out of luck when it comes to realtime views of logs without constantly refreshing the pane (which is pain).

This script illustrates how to tail the event log by using Python’s ability to tap into COM. A very handy thing indeed.

import wmi, sys, threading, pythoncom

class event_watcher(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)

  def run(self):
    pythoncom.CoInitialize()
    c = wmi.WMI(privileges=["Security"])
    # If you want to limit the types of entries, add the 'Type=' argument
    # 'Type' can be Error, Warning, Information
    my_watcher = c.watch_for(notification_type="Creation", wmi_class="Win32_NTLogEvent")

    while True:
      event = my_watcher()

      # to see the full event log entry, uncomment below
      #print event

      # some handy information which could then be redirected to a
      # file and be easily parsed.
      print "%s in %s log: %s" %  (event.Type, event.Logfile, event.Message)

watcher = event_watcher()
watcher.start()

COM is massively powerful on windows-based systems, but it does add a platform dependency to your code. To be extra safe, I should have checked that it was on windows and failed gracefully, or just fall back to that platform’s tail but I trusted myself to not run it on a non-windows machine. Your audience will of course differ.

My original write-up of this recipe is here.

Posted on November 2, 2008 in GLSEC2008 by adamNo Comments »

An oracle for testing purposes can be a number of things: a person, a specification (like an RFC), a comparable product depending on what you are trying to do. If the oracle happens to be an algorithm then I tend to script it up in order to

  • Have a separate implementation to use as a guide
  • Be able to determine if the application is giving me the correct thing
  • Create test data to use that meets the algorithm

The first 2 are scripting the algorithm up as an oracle instance and the third lets me not have to think about the validity of my test data letting me concentrate on more important parts of the testing effort.

Here is an example that I created to verify or generate an account number. In this scenario the usernames are not sequential, but instead are 9 digits long, cannot start with a 0, can only have numbers and the last digit is a mod-10 check digit. If run with an argument then it will verify the validity of the account number, otherwise it will create a valid one for you.

import random, sys, string

def verify_number(to_verify):
    # length check
    if len(to_verify) != 9:
        # raise an exception with some context
        msg = "Invalid account length"
        raise SystemError, msg # yes, I know, SystemError isn't really the right error

    # only numbers
    for char in to_verify:
        if char not in string.digits:
            print "Accounts can only have numbers"
            sys.exit(1)

    # cant start with zero
    if int(to_verify[0]) == 0:
        print "Accounts cannot start with 0"
        sys.exit(1)

    # checkdigit
    run_total = 0
    for run in to_verify[:-1]:
        run_total += int(run)
    mod = run_total % 10
    if int(to_verify[-1]) != mod:
        print "Check-digit failed"
        sys.exit(1)

    print "Account is valid"    

def generate_number():
    account_num = []
    account_num.append(str(random.randint(1,9)))
    for ix in range(0,7):
        account_num.append(str(random.randint(0,9)))
    run_total = 0
    for run in account_num:
        run_total += int(run)
    mod = run_total % 10
    account_num.append(str(mod))
    print "Your new account number is %s " % "".join(account_num)

if __name__ == "__main__":
    if len(sys.argv) == 2:
        # show a try/except block in a real example
        try:
            # verify provided number
            verify_number(sys.argv[1])
        except SystemError:
            print caught
            sys.exit(1)
    elif len(sys.argv) == 1:
        # generate number
        generate_number()
    else:
        print "usage: my_script.py [account number]"

This happens to be in Python, but I once wrote a number of generators like this in Perl and had them all in a nicely dynamic CGI page which would give you account numbers for any available algorithm. It was pretty slick. I should try to find that code…

Posted on November 2, 2008 in GLSEC2008 by adamNo Comments »

The most common question I get when talking to people who are new to scripting is Which language should I use?. The answer is of course is ‘it depends.’ One thing you want to avoid is taking someone’s answer as gospel. Everyone’s situation is different and they have their own biases. Here are some things off the top of my head you might want to consider.

  • Do you know the language?
  • Could you learn it?
  • Is it supported on all your (current) platforms?
  • Are those platforms all considered first-class citizens?
  • Can it interact with your database?
  • Does it control your other tools? (Selenium, Watir, etc.)
  • Can you recycle code (Java jars or .NET assemblies) from elsewhere in your application?
  • Does it support Unicode as a core data type?
  • Is there seamless interaction with the operating system?
  • Is there an active member community (to whom you can turn to for help)

Inevitably the answers to the above will lead you to Perl, Python and Ruby. Here is my take on each.

  • Perl – Has been around forever (I’ve been using perl for more than half my life) and has been ported to every possible platform. There are tonnes of people to help you script and there is a wealth of libraries to do almost anything you can conceive. Perl code is also however known for being completely unmaintainable (if you are not really, really, careful).
  • Ruby – Even though it may not truly deserve the label, it is the New Hotness (to quote Agent J from Men in Black). Most of the interest in Ruby seems to be a direct result of the popularity of Rails. If you are a Rails shop, then choosing this is a safe bet. I can only think of a handful of people I know who use it as a general scripting language though and the version 1.8 is the most prevalent in the real world (for Rails support) and it doesn’t handle Unicode/UTF-8 properly. 1.9 apparently does though.
  • Python – Is known for producing maintainable code, has been ported all over the place and has lots of support for interacting with other systems (not as much as Perl, but more than Ruby). It is also easy to learn (which is a bit bonus). It is going through a bit of versioning pain right now as version 3.0 is not backward compatible with the 2.x versions so things are going to be a bit sketchy for a while with 3rd party modules.

Each has its own strengths, and weaknesses. I tend to recommend people learn Python really well and become familiar with both Perl and Ruby from a scripting context. And don’t be afraid to change languages if it is better suited for a particular task. Stubbornly holding onto a language when it is not the right one for the task is a trap people fall into too often.

Posted on November 2, 2008 in GLSEC2008 by adamNo Comments »

One thing I have been interested in for a couple years is the notion of metaframeworks which are at their core wrappers which combine different other frameworks for a common reporting and execution. It’s on the upper size limit of what can be considered a recipe, but I’ve written them at 3 different companies now, so I’m including it.

The key thing about a metaframework is its extensibility and ability to run different types of tests. My choice of language for this recipe is Jython. Here are some recipes I have utilized when building these.

Posted on October 30, 2008 in GLSEC2008 by adam1 Comment »

Another script I have previously posted was about parsing one of our Apache’s log files to determine the actual browser share on our site. Sure, you can find the aggregate numbers online from a pan-Internet perspective, but when choosing test data for our site I really only care about the numbers as they relate to our site. This is another important piece of the scripting puzzle: make it work for your own needs first. Only then should you make it generic. And only if you have time.

I’m sure there are more scalable and/or efficient ways of gathering and displaying the data (dynamically adding keys to the hash for example) but the script is both unrefactored and one of the first ones I wrote in Ruby so there is a lot of newbie-isms in there as well.

The script itself is in Ruby (as mentioned above) and is shown in the midst of a larger discussion on the definition of Quality.

Next Page »