Posted on January 27, 2010 in Uncategorized by adam5 Comments »

So I went to the monthly gathering of TASSQ this evening. I figured that I should probably go to a meeting or two before I’m slated to talk at the March one and now that I’m independent I should make myself known in various circles. What I discovered appeared to the a bastion of ‘old school’ qa (not testing).

Now, I know this post is going to sound like a bash, and it’s not really meant to be one. It is more rooted in incredulousness. I know it is hard to get up in front of people and expose your ideas and ways of doing things to others. But you have to remember, that when you do, you open yourself to this sort of thing.

Oh yes, and it is also important to realize (as most who read this blog likely do already) that I’m philosophically aligned with the Contextual, Agile and Craftsman schools of thought.

What was the name of the talk?

Factory School Testing Revisited / Redacted / Re-energized. aka Kickin’ it ‘Old School’

Gah! Right there I knew I was in trouble.

Anyhow, here are the notes / impressions / ideas I had during it.

  • One of the stated goals was to have a topic that resonated with folks in the room. I’ve only been to a couple TASSQ meetings before, but if this is the topic of interest then it kinda says that the group is not concerned with new ideas.
  • I haven’t worked in a regulated environment in a long time so I could be wrong, but don’t the regulatory bodies care about types of testing you are going to do before hand and the results after? I seemed to get the impression that one of the large arguments for creating a large body of test cases ahead of time was to show the ‘depth’ (or something) of your ‘to be done’ activities. Well, that doesn’t prove anything. What matters is what has been done. I could say that as part of the testing I’m going to build a rocket and go to the moon! Results are what matter.
  • Also provided on the same slide was the notion of resource variability. ‘We have high turnover of QA people so we need stacks of test cases to bring the replacements up to speed’ is my paraphrase. Well, if you have high turnover, then I would say that is a smell that you are doing something wrong. And what about turnover in test management? I’m guessing it has the exact opposite pattern — the entrenched stay entrenched and those that challenge or deviate from plan are removed.
  • He had a nice metaphor about Christmas and testing. Testing is like Christmas morning; when you wake up there are gifts lying under the tree. But they are wrapped and you can’t see what is under the paper. And sometimes you like what you get, but sometimes it doesn’t fit and you have to return it. This can sometimes hurt the feeling of the giver, but often they understand. And there is no guarantee the replacement you get is going not have a page missing or some other thing wrong with it that you don’t discover until the next day. So the cycle repeats itself.
  • I’m not the best speaker, but there is likely a business opportunity waiting to be exploited in teaching testers how to present to small-ish audiences of strangers. How to speak clearly, to the audience, without fidgeting and with a nice slide deck. I wonder how my Lessig-esque deck is going to go over in March. Care to bet the comments that come back are that I need more bullets?
  • Not once were the people involved referred to as ‘testers’. It was always ‘QA’. What you name something is critically important.
  • What I think was most scary was that I can remember a time when everything that was said would have rang true to me. I’m not saying that Rapid Software Testing threw me from my horse as I was likely travelling down that road anyways, but apparently the big Canadian banks need some heavy introductions to Context techniques and ideas.
  • I sat next to Fiona Charles who offered up to a discussion of metric a rule-of-thumb I hadn’t heard before. 3:3:1 – For every three test cases you write, you will find one bug. For every three bugs you find, one will be severe enough to warrant fixing. If you application is producing more bugs that the model, then you need to look at the quality of the application. If however you are finding less bugs than that, you need to look at the quality of the test cases.
  • If metrics and counting test cases executed is so important, isn’t it rather ironic to encourage testers to go ‘off-script’? You number are wrong(er!) then by your own admission.
  • I did like that it was stressed during the metric part that it is not the number exactly that matters on the graph. It is the pattern that it depicts that counts. Oh, and that it might take 20 minutes to generate the numbers daily; of which 15 minutes of that is writing the summary and story (context) around them.
  • My big issue with the Factory is around its reliance on test cases. So I asked a couple questions at the end about it (in as non-obnoxiously as I could). And through that I had to explain the Mine Field problem. Sigh. I know I associate with people who value constant learning as a core attribute, but c’mon! Learn the core theories of your craft! Someone in the audience added that in real-time/safety-critical systems they cover the system 100% thus removing all the mines. Except, that is impossible. You’ve covered every scenario and path you could conceive of at the time. But unless you test every scenario, every interaction, by every person, ever to use your product, ever, then you don’t have 100% coverage. Even if you have 100% code coverage, which I admit is likely what he meant, you can’t account for oracles like emotions at the time of use.
Posted on January 27, 2010 in Uncategorized by adamNo Comments »

The Stack Overflow podcast #71 has finally bubbled to the top of the queue. It is kinda two different podcasts smooshed into one. The first half is clips from the speakers from the SF DevDay event, and the second half is the usual gang of suspects goofing off around a microphone. The redeeming part of this particular episode was Jeff Atwood’s on-stage portion. Here are the bits you should care about. (Or at least I did…)

  • An interesting interview idea – Prepare a list of the smartest people in your field and present that list to the candidate. How many people can they identify? Do they argue for people you missed or that shouldn’t be present on the list? Seems like an interesting way to weed out the people who do the job just for the mortgage
  • The Stack Overflow Team Motto: Feel free to fail early, often and frequently as possible in really painful ways
  • Don’t phone in
  • Be passionate for the ‘craft’. But not just the craft of constructing the software, but for delivery of the solution
  • Would you sign your name on the product? The Amiga 1000 hardware developers did.
  • Jeff recommends Coders at Work and provided some of the rationale behind it through quotes:
  • Before you ask for help on something talk through the problem to a stuffed animal first
  • Can you deploy something new everyday? Do you make it even a tiny bit better every day?
  • The Train of Shame – the error logs, in real-time(ish) to show you what is busted. So important that you figure out and implement this first before you do any production code. Start and end you day by watching this.
  • Take bits from different stacks as necessary. Don’t just stick with only one solution family
  • The big engineering problem is going from 1 to 2 servers. After that, it is ‘easy’ to add more.
  • With technology, use the lightest thing you can get away with
Posted on January 25, 2010 in Uncategorized by adam1 Comment »

Part of the reason for the 1.0.4 release of Se-IDE last week was support for the Firefox 3.6 series of browsers. Now, Mozilla has a fairly aggressive upgrade process, but you can’t always trust that people have the same / current version of the browser. That means you need to test against multiple versions of Firefox.

This post explains my trick of having multiple Firefox versions on a single Mac OS machine.

  1. Download as many releases as you want / care about
  2. Rather than install it into Applications like you normally would, install it into a folder in Applications that relates to the version.
  3. Start each one and uncheck the Preference that would have the browser check for updates

Now you have many different versions of Firefox installed. There are likely to be some headaches around plugins, etc. but those could likely be mitigated by having a different profile for each one.

What about with Selenium? To do that we need one extra step

  1. Mac applications are directories called Something.app (so Firefox.app). So what you want to do is create a symlink from /Applications/your-version-of-firefox-dir/Firefox.app to /Applications/Firefox.app

Now you can run your Se-RC tests on different versions of Firefox by changing where the symlink points to. This is because how Se-RC finds Firefox on the system. Here is the Firefox 2 locator. Three is very similar.

private static final String[] USUAL_UNIX_LAUNCHER_LOCATIONS = {
        "/Applications/Firefox-2.app/Contents/MacOS",
        "/Applications/Firefox.app/Contents/MacOS",
        "/usr/lib/firefox", /* Ubuntu 7.x default location */
};

In this trick we’re making use of the middle location.

Posted on January 25, 2010 in Uncategorized by adam4 Comments »

Making the rounds through twitter right now is Sikuli. I became aware of it through someone claiming it a Selenium killer. Well, not quite.

The ‘magic’ of Sikuli is that it used the actual content on the screen to find elements rather than a native API or XPath by taking screen captures and looking for image areas to interact with.

Yes, that’s correct, it used the way an object is rendered on the screen to determine whether it is interacted with or not. Didn’t industry decide this was a bad idea a number of years ago? Here is why this project fails the sniff test for me.

  • What about when I create a script on one resolution and play back on another?
  • What about different record and playback OSes?
  • Accessibility enhancements on/off?
  • Image reuse? Without this you have to change every location when something changes. And things always change.
  • How do you data-drive something that requires you to have image captures of the run? Data driving is rather important when it comes to minimizing code duplication yet increasing test data.
  • And since you have to have visited each screen in a certain manner whilst writing the script, you start to run afoul of both the Minefield Problem and Pesticide Paradox
  • And and if you can’t data-drive a script, is there any chance that scripts would be written in a model-based manner? Highly unlikely.
  • Can you record actions? I would wager that Selenium’s success can be largely attributed to the Record functionality of Se-IDE. A quick spin through their site implies that manual script creation is the only option. People who have done the market research have told me that most QTP users stay in the keyword, non-script, part of the environment.

Of course, the goal of Sikuli isn’t to solve these problems. It is a research project between two groups at MIT. This is said with some derision in my voice. Yes, some things are well suited to make the transition from Academia to Commercialization. Google came out of Stanford for instance, but it is/was at its core, a new algorithm to (re) attempt to solve an understood problem. The issue with this is that the divide between Academia and the Real World in Testing is mammoth. I’m not sure why this is exactly, but I often just shake my head when reading papers on testing by people ensconced in a University somewhere. Yes, there are some exceptions to this, but they are exceptions rather than the rule.

Now, contrary to how it may seem, I’m not completely bashing Sikuli, more just pouring some cold, hard reality against it. I understand its appeal; since it uses screen captures is can interact with anything that displays on the screen. That is close to a Holy Grail if ever there was one, but they way in which they accomplish it is terminally flawed I fear.

Posted on January 23, 2010 in Uncategorized by adam10 Comments »

One place I see Selenium users stubbing their toes on time and again is around automating HTTPS connections. I’m about to tell you that you should save yourself a lot of pain and just not do it.

Having a secure connection in production is an extremely good idea. Especially at places like checkout and authorization. But in testing? Not so much.

There is exactly one situation where you need to be running HTTPS in testing, and that is if you are using certificates as an authentication means. For every other test scenario, in test, you can run things just as HTTP. Not only can you watch the network traffic in the clear over HTTP but you don’t have to worry about the trust issues certificates introduce.

And there is almost always trust issues.

Why?

Because ‘real’ certificates cost money. The whole notion of security is based around trust, and the way to establish that trust is through money and verification. If you look in your browser, it will list somewhere all the CA (Certificate Authority) root certificates that it trusts by default. Those vendors pay non-trivial amounts of money to get in there. And then they charge non-trivial amounts of money end consumers to sign a certificate with their root certificate. Production sites will (almost) always go that route since your customers are likely to be a little concerned about all the browser warnings about things being wrong with the certificate.

In test though, machines are often rebuild and used for multiple purposes so companies are loath to spend the money on real certificates. When manual testing, this isn’t anything more than a nuisance dialog to dismiss, but when automating, it is much more annoying.

The problem is that the error box is not part of the application, but is part of the browser. In Selenium 1.x there are some workarounds, and I believe Selenium 2.x and WATiR solve this by low-level control of the browser.

The easier solution though, is to just not use HTTPS in the testing environment. Sure, your testing machines won’t be as ‘production-like’ as they might otherwise be, but you have to ask yourself, does that materially affect the quality of your testing?. In most situations, it won’t. In fact, I would argue that it increases the breadth of testing available to you; both manually and in an automated fashion.

edit: To be clear, the workaround mentioned are just some of the possible solutions in Selenium 1.x for dealing with HTTPS pages. There are a lot of situations where it will Just Work. Especially in RC where the server is a proxy and certificates get injected on the fly. I would still say in those cases to turn off HTTPS though; it is really hard to figure out what is going on at a network level when it is encrypted.

Posted on January 21, 2010 in Uncategorized by adam1 Comment »

Been a busy couple days in Selenium-land. On Tuesday I pushed a new version of Selenium-IDE (see the official announcement) and yesterday Sauce Labs pushed Sauce IDE. I’ve not been coy about my relationship with Sauce Labs and don’t want this to be a generic fluff post, but I think Sauce IDE will eventually be the model for building a commercial product around Selenium-IDE. And the reason for this is/will be packaging.

Firefox extensions are, at their core, just a zip file with an extension Firefox knows what to do with and have a minimum common set of configuration files (install.rdf, chrome.manifest and such). One of those configurations in the Se-IDE and Se-IDE plugins is to set the em:type to 2 which tells Firefox that it is an extension (rather than a theme or whatnot).

The ‘classic’ way for a businesses to commercialize an open source project is to essentially fork it and start along their own, potentially divergent path. One of the goals of starting to develop an API for Se-IDE was to explicitly avoid all this splintering.

The Firefox system for extension is extremely helpful in this regard. There are very few things a commercial entity cannot do to Se-IDE through overlays. This means they don’t need to fork. I fully anticipate that situation where extra helper methods, or even some refactoring will be necessary to succeed long term, but in general, companies don’t need to change the code Se-IDE code. They just write an extension.

There is still a bit of a delivery problem though. My model for the plugin idea for Se-IDE has been the relationship between Firebug and YSlow. When you go to the YSlow page it says to install Firebug, then install YSlow. This is fine for the alpha geek demographic, but is less than appealing from a corporate perspective. What you want is a single installer.

The folks at Firefox thought of that too! Enter the multi-item installer.

What this lets you do is package multiple .xpi files in a single .xpi file. Which means, in a commercialized Se-IDE perspective, you can ship the version of Se-IDE unmodified as well as your modifications as an extension in a single file. Here is the install.rdf of a multi-item xpi banged out to package Firebug with a couple plugins as a make-sure-it-works experiment.

<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
     xmlns:em="http://www.mozilla.org/2004/em-rdf#">
 
  <Description about="urn:mozilla:install-manifest">
    <!-- nsIUpdateItem type for a Multiple Item Package -->
    <em:type>32</em:type>
 
    <em:id>firebug@goucher.ca</em:id>
    <em:name>Firebug Multi-installer</em:name>
    <em:version>0.0.1</em:version>
    <em:creator>Adam Goucher</em:creator>
    <em:description>Proof is in the pudding</em:description>
 
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id> <!-- firefox -->
        <em:minVersion>2.0</em:minVersion>
        <em:maxVersion>3.6.*</em:maxVersion>
      </Description>
    </em:targetApplication>
  </Description>
</RDF>

Now, Sauce IDE isn’t quite there yet, but that is the endgame. Right now, it is about 60% there; Se-IDE was forked, but the modifications were made in a plugin-esque manner so they could be separated once it was all working. It’s working now, so the splitting will commence shortly meaning the fork will be retired and an official Se-IDE xpi will be bundled as part of the multi xpi.

Plugins are where a lot of future development of Se-IDE will be focused. Heck, some existing functionality might be extracted into an extension to clean up the code / core. Plugins are only half the story though when it comes to commercial entities adopting Se-IDE as the automation platform they build upon. They need simple distributions to appeal to the mass market. Thankfully, there are ways to solve that problem.

So let’s start getting those commercial distributions of Se-IDE going!

Posted on January 13, 2010 in Uncategorized by adam1 Comment »

As part of its 100 Best Global Brands series, Business Week had an article on The Great Trust Offensive. Trust is an idea we need to be very aware of, not just from an outward corporate perspective, but from internal corporate and personal career ones as well.

Here are three key quotes:

  • In the world of branding, trust is the most perishable of assets
  • If you aren’t open with me, then I won’t trust you
  • Stay relevant to the current mood or risk been seen as spin merchants

Part of why I think agile-ish things work is that by breaking down the silos and building an environment where communication can happen, then trust has a chance of being achieved.

The article ends with a warning though which resonates with my theory that agile will fail to reap maximum benefits if any part of the organization is actively resisting (intentionally or otherwise).

Trust-related marketing only works if there is a message that people want to believe in. You cannot spin an audience that doesn’t want to be spun.

Posted on January 12, 2010 in Uncategorized by adamNo Comments »

There was an article in the paper about picking and nurturing the next generation of leadership. In the banking context, the type of program being discussed is commonly known as The Fast Track since it is just that up the corporate ladder. Most testing and/or agile teams don’t have a fast track, but the last three paragraphs are pretty good (emphasis is added by me).

It has to be a two-way street, says Suzanne Gagnon, a professor of organizational behaviour at Desautels Faculty of Management, McGill University in Montreal. “The executives have to listen to the employees being developed and also give them that autonomy to do that work on their own, to make mistakes and if they fail, to learn from the failure,” she says.

Prof. Gagnon, whose research includes the impact of internal politics within organizations, also believes that in order for such a program to truly succeed, it needs to be implemented with great care. For example, the selection of high-potential individuals has to be transparent and as objective as possible.

We have developed a leader profile over the years that details exactly what competencies are critical to us and what experiences are important,” Mr. Toda says. That profile is used both to identify and develop training for potential leaders. “We need to have people who are effective leaders and that means effective collaborators and team players, so bringing people together from a variety of backgrounds and cultures and walks of life onto a team like this gives them some sense of the nature of the bank that they might not otherwise have had.”


The last paragraph deserves some further commentary. While a profile that you look for is a handy, and likely necessary thing when dealing with a corporation the size that they are, this has groupthink warning flags all over it. Which is fine in some situations, but not where change is necessary. The ability to change, and the acceptance of change are two very important parts of how companies succeed.

And maybe those attributes are part of the profile, but if my experiences with large companies, including Canadian banks is accurate, then it is more about deference to seniority, political connections and race/gender profiling.

Posted on January 11, 2010 in Uncategorized by adamNo Comments »

Canadian Business Magazine ran an article that I found while cleaning my laptop bag called Presenting to the board: how to handle the heat. I couldn’t quickly find it on their site to link to, so here are the tips from Carol Stephenson.

  • Understand the board – Focus on three things: the general context of the organization right now, the primary focus of the meeting, backgrounds and preferences of the members of the board.
  • Bring the right people – You don’t need to bring the whole team to back up your position; if you are the one doing the presentation, you are the one accountable for the issue/topic.
  • Keep the numbers simple – Don’t try to impress with fancy quantitative analysis. That is not what the board wants. What they want is your analysis of the numbers and your recommendation.
  • Articulate key assumptions – Walk through any key assumptions, then document the rest in an appendix or somewhere. They are relevant to the whole, but not the presentation.
  • Present your answer first – Tell the answer, then back it up.
  • Read your audience constantly – Don’t just barge through your presentation
  • Be transparent about risks and challenges – Don’t sugarcoat.
  • Address questions immediately – When a director asks, answer promptly.
  • Frame the outcome – End by asking for something. The board level issues are closed by: approving, denying or asking for further study. Frame for this sort of response.
  • Role play your presentation – Not just the classic ‘practice, practice, practice’, but field tough questions as well. If it works for trial lawyers, why not you?
  • Admittedly, a lot of us don’t have the need or opportunity to present to a board, or if we do it is not a ‘traditional’ board; more of a ‘founders and financiers’ informal one, but these are important things to keep in mind anyways regardless of the presentation.

Posted on January 10, 2010 in Uncategorized by adamNo Comments »

Formatters in the Selenium-IDE context are what lets you record or export in a particular language. The default is Selenese (HTML), but Se-IDE also ships pretty decent variety out of the box as well. They are very important to the success of rolling out Selenium in an organization by letting you record in Se-IDE and then modify the script in some sort of Se-RC client.

But what if you are a business, or even another open source project that leverages Selenium with some sort of unique bit of vendor/product-script? You could create a formatter, post it on your website and have people load it into their browsers, sure. Or, you could distribute it as a plugin which lets you update it when bugs are fixed in it and track its download and usage.

Selenium-IDE (as of about half an hour ago) has support to let you do the latter. And here is how you add in your formatter. How you create a formatter, well, that is something for another time.

Most of this should be old hat by now.

Add to the chrome.manifest

# custom format(s)
overlay chrome://selenium-ide/content/selenium-ide-overlay.xul chrome://preflight/content/formats/format-loader.xul

And create the overlay

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
 
<overlay id="preflight_format_loader_overlay"
         xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
         xmlns:html="http://www.w3.org/1999/xhtml">
    <script type="application/x-javascript" src="chrome://selenium-ide/content/api.js"/>
    <script type="application/x-javascript" src="chrome://preflight/content/preflight.js"/>
    <html:script type="application/javascript">
        var ide_api = new API();
        ide_api.addPluginProvidedFormatter("preflight", "Preflight Formatter", "chrome://preflight/content/formats/preflight-formatter.js");
    </html:script>
</overlay>

The arguments for addPluginProvidedFormatter() are as follows:

  • The id of the formatter. This is used internally.
  • The name of the formatter. This is what the user sees.
  • The chrome url of the formatter.

And of course we want to cleanup after ourselves. Add this to the quit-application section of your observer (that was started in part 4).

var current_ppf = branch.getCharPref("pluginProvidedFormatters");
if (typeof current_ppf != "undefined") {
    var split_ppf = current_ppf.split(",");
    for (var ppf = 0; ppf < split_ppf.length; ppf++) {
        if (split_ppf[ppf].search("chrome://preflight/content/formats/preflight-formatter.js") != -1) {
            branch.setCharPref("pluginProvidedFormatters", current_ppf.replace(split_ppf[ppf], ""));
        }
    }
}

Now, let’s start getting those formats in there. I’m looking at you Selenesse.

Update 1/12/2010 – changed the token you split on from a ; to a , in the observer

Next Page »