Posted on June 30, 2010 in Uncategorized by adam1 Comment »

Cem Kaner was at TASSQ this evening to talk about applying test techniques to the evaluation of the models we build into software, to help us decide whether we are building the right thing, to address validation and accreditation of software instead of burying our heads in verification. And he did, though in going into the meeting I had focused on the sentences before and after that one which mentioned ‘automated exploratory testing’ which, as an automation consulting tester I was keenly interested in hearing Cem’s thoughts on the topic. So with that in mind I thought this was the poorest (personally) of the talks I have seen him deliver. Until that is, Paul Carvalho phrased it as a case study, as an experience report. With that frame of reference, it was an excellent talk which showcased in depth, methodical test probing of various financial models in a way only someone with his training and background could provide. Here are my notes.

  • There are green bananas and ripe bananas and rotten bananas and big banana and little bananas. But by and large, a banana is a banana — on commodity level software testing. The lesson is don’t be a banana — though a preferred alternative fruit was not provided
  • There are a number of levels of testing; five that fit on the slide are
    1. checking
    2. basic exploration
    3. systemic variation
    4. business value
    5. expert investigation

    I first interpreted this as a linear progression, but as he described them it appears as it was more 1, 2, then people tend to specialize in either 3, 4 or 5.

  • The last three in the list allows a tester to ‘earn their keep’
  • The decision to automate a regression test is a matter of economics, not principle
  • Automation is a starting point to say it is ready to test, not that it is ready to ship
  • The most important slide from the deck I think was on the seven risks to any model.
    1. model – the model is theoretically incorrect
    2. characterization – the model is correct, the spec is wrong
    3. comprehension – we misunderstood the model. our code accurately implements the wrong model
    4. implementation – our code inaccurately reflects our intent
    5. execution / environmental – platform too slow, volume issues, etc
    6. tool – the test tool misleads
    7. scope – the model is properly developed but it is not appropriate to today’s circumstances
  • Implementation is the easiest to find and the least business value
  • Computer programs are solutions to people’s problems
  • If you understand the subject matter of the business and use that expertise in test design then you have true business value
  • There is skill in empirical research that we have as testers that can be applied to any area that uses software. and that software is just a representation of a human model
  • Focus your work on answering the questions you are trying to get answered rather than supporting your automation

Even with the correct context likely in place for the presentation, I have two complaints; both minor in the grand scheme of things.

  • In his ‘opening rant against regression testing’ he said as agile development has gradually failed. Maybe it is because part of what I do is agile coaching-ish, but I felt it came off wrong and detracted from his point.
  • The slides are terrible. I mean, they are loaded with information, but they are massively overloaded. The deck is not the presentation, the presenter (Cem in this case) is the presentation. Had it been anyone less trained and practiced in public speaking they could have just read from the slides and been coherent. Of course, I get the opposite complaint about by decks so lets call it a wash.

The title page of the deck mentioned that he will be doing a presentation as a keynote at this year’s CAST which seems like a good enough excuse to plug it. I won’t be there as my wife is at a conference in Alabama at the same time and then it is a Agile the following week, but it should be good. (As always.)

Posted on June 26, 2010 in Uncategorized by adam2 Comments »

Nolan Ryan pitched in a different era, where pitchers were not on pitch counts and were not ‘coddled’. For the non-baseball readers, a pitch count is just that — the number of pitches thrown, and when a magic number (usually around 100) is reached the pitcher is pulled regardless of whether they still have command and strength to remain. During his career as a pitcher he threw 222 complete games and worked hard to stay in form. Well, he is now the president of the Texas Rangers and is re-instilling that sense of work ethic to the team. This change is documented in the May 24, 2010 Sports Illustrated in Nolan Ryan’s Crusade — the contents of which I can’t find on the SI website.

But here are the interesting parts that apply to testing and culture (painstakingly typed out even).

  • What’s wrong with pitchers today? For one thing, they don’t learn to think for themselves anymore. Coaches started calling all the pitches in high schools and colleges. How do they know, sitting on the bench, what the guy on the mound has confidence in?
  • Pitchers have been pampered. I’d go to spring training, and al they”d do was throw on the side. Now how in the world do you learn how a hitter’s going to react to your pitches without a hitter in there?
  • Today a quality start is six innings. What’s quality about that?
  • Their ceiling has been lowered. It’s up to us to jack it back up
  • Texas hurlers have embraced Ryan’s challenge to raise their expectations and take ownership over their starts.
  • Backing away from a pitcher’s limits too far doesn’t make a pitcher less vulnerable; it makes him more vulnerable. And pushing the envelope, while it may lead to a catastrophic event, is more likely to enhance the pitcher’s durability than to destroy it.
  • …on the first day of spring training last year Maddux stood in front of his pitchers and said, “Pith counts are limits. You have no limits.”
  • The Rangers instead believe that not all 100-pitch games are created equal. Some are more stressful on the arm than others, and if the pitcher is cruising late in the game, there’s no reason to hive him the hook.
  • Ryan ordered that pitchers throw live batting practice to hitters from Day One of spring training — a routine almost unheard of in Big League camps…
  • They were the first team willing to think outside the box and now they’ve started a chain reaction
  • It is important to have people who are willing to make a commitment
Posted on June 13, 2010 in Uncategorized by adamNo Comments »

This blog has been running for about 4.5 years (and over 3 different locations) and until now it made sense to have all topics under the umbrella of ‘The Brand of Adam’ (feel free to read that with lots of echo). But now that I’m part of the core Selenium team as well as a Selenium consultant its time for a bit narrower focus here, to hit a much larger audience overall. So here is the new plan.

  • adam.goucher.ca – This will be ‘testing’ focused with the usual book reviews, podcast summaries and other things I think people interested in testing mind find useful.
  • element34.ca – This is my ‘work’ site and will be catching anything dealing with Selenium. Which kinda makes sense since I now make my living off of Selenium and want potential clients to go to that site.
  • seleniumhq.wordpress.com – This is the ‘official’ blog of the Selenium project and I’m going to start posting the Smattering of Selenium content there so it can reach more than just those who have stumbled upon my blog.

I think this split better serves the [core] audiences of the content better, and in the long run might actually expose it to more people.

Posted on June 9, 2010 in Uncategorized by adam1 Comment »

First, why? Just because your application is written in Java does not mean you have to write your Selenium framework in it. JRuby or Jython are far better options for you. But let’s say you are determined to do it, what are some things you need to keep in mind? Here are two that I have encountered this week.

Properties Files

Java’s native method for configuration information is the java.util.properties class. So if you are going to pass information into your framework, and let’s face it, its a framework so you will, use properties files. Not Excel, csv, yaml or some crazy propriety format. Use properties. Not only is it built into the language but it is a text format so can be checked into version control (and diff’ed, etc.).

getResourceAsStream

This builds off of the first point. Do not put paths into your code! Java has this thing called the classpath. You know, that thing that tells the JVM where to look for things — like properties files! Conveniently enough, you can parse a properties file as a stream from a file that was located in the classpath.

package ca.element34.flyingmonkey.environment;
 
import java.io.InputStream;
import java.util.Properties;
 
import org.apache.log4j.Logger;
 
public class ReadEnvironment {
  static Logger log4j = Logger.getLogger("ca.element34.flyingmonkey.environment.ReadEnvironment");
 
  public Properties ReadEnvironment() throws Exception {
    Properties environmentProps = new Properties();
    InputStream is = this.getClass().getResourceAsStream("/environment.properties");
    environmentProps.load(is);
    log4j.info(environmentProps);
    return environmentProps;
  }
}

Some things to keep in mind with this:

  • The directory that contains the environment.properties file needs to be in the classpath. Don’t put the file itself in the classpath. You have no idea how many times I made that mistake when I was at HP…
  • You need to have the leading / in the filename. Without it, the package hierarchy is searched and not the classpath
  • You will likely have a number of environments that your framework needs to interact with, so have an environment.properties.development, an environment.properties.staging, an evironment.properties.production, etc. But don’t have checked in an actual environment.properties file. Instead, create a symlink to the file on the local machine.

Part of my role as a Selenium Consultant is to dive into a team’s code and spot things that will likely hurt them in the long run. My typical technique is to run the framework on a platform they are not using — it should work. And when it doesn’t, these two tricks will go a long way in moving it a lot closer to working.

And this isn’t to pick on just Java. Python and Ruby have similar native tricks for accomplishing this as well.