For the last couple years there has been a growing collection of people rallying around the software craftmanship. From an outside perspective, it appears that they are dipping into the classic apprentice – journeyman – master model for growth (without the inclusion of guilds which is often the point of fear with this model). From its title, Apprenticeship Patterns – Guidance for the Aspiring Software Craftsman would be aimed only a the beginning of the road and only for developers. Both would be wrong. This book is of value to anyone in a knowledge role and at any point in their development.

Dave Hoover and Adewall (Ade) Oshineye have collected together a series of of almost 40 ‘patterns’ that you can use, and likely already are, to assist your growth. Each pattern follows the same structure by defining the context, giving a sample scenario where the pattern might solve a particular type of problem, a solution and action that the reader can take immediately to start implementing the pattern. Because of this structure it is easy to jump at random throughout the book and [start to] digest a pattern in a single (brief) sitting. The real power of the patterns for people starting out is in the Action section I think, though I admit that I have not followed through on most. These are the ‘get off you butt and do something about your growth’ motivators.

Having embarked on my path already I found that I was already implementing a number of these patterns which I think completely validates their inclusion in the book. But it also meant that the Solution section was what got my attention the most with lots of ideas how to refine my implementation of the patterns. Flipping through my copy of Apprenticeship Patterns I can find only 2 or 3 patterns that I did not underline or star or otherwise mark up.

My favourite patterns (both from the book and the real world)?

• Breakable Toys – Experience is built upon failure as much as (if not more than) success
• Find Mentors and Kindred Spirits and Rubbing Elbows – Seek out those who have gone ahead of you and strive to learn from them
• Record What You Learn and Share What You Learn – Start writing. Start writing now!

People who know me likely won’t be that surprised with that list. Your list, almost by definition will be different as you have walked a different (though perhaps similar) road in your journey.

My only knock against the book is likely its price. It is 130 pages and $37.99 CDN. That is steep, though I suppose in-line with current pricing structures since Beautiful Testing is 320 pages for $62.99. To their credit, the patterns could at one point be all found online (minus some polish that comes from the editing process) but I couldn’t find them at time of writing. But a book like this really should be consumed in dead-tree format where you can make notes in the margins. (If someone finds the wiki link again, please post it.) And at the risk of sounding like I’m pimping my book, Amazon has been long running a deal where you can save if you buy Apprenticeship Patterns and Beautiful Testing together. While likely related to release dates and ish similar breadth of content, I’m ridiculously happy to see it.

The Agile community is in the midst of an obsession of Shu Ha Ri, and much to my annoyance I admit that there are different ways that people learn at different stages of their learning. I don’t know if there is going to be ‘Journeyman Patterns’ follow-up, but I hope there is. But even if there is, Apprenticeship Patterns is staying on the bookshelf where it is nice and handy.

I expect that the majority of plugins that people create are going to be around adding custom functionality or formats, but sometimes you will want to tie Se-IDE to you service. (Much like Sauce Labs does with Sauce IDE.) Typically that tie-in needs some sort of credentials.

There are two different ways to do this. First, you could just store things as a preference and be done with it, but that has a bit of a problem around it. Namely, it gets stored in the clear in your profile’s prefs.js file and (depending on your permissions regime on disk), anyone could read it and get your credentials. This is not quite ideal.

The better way to store your credentials is through Firefox’s Login Manager.

Again, Preflight doesn’t have need for a password right now, but if it did, here is how you would protect it.

Add a new overlay to selenium-common.xul in the chrome.manifest

overlay chrome://selenium-ide/content/selenium-ide-common.xul chrome://preflight/content/overlays/preflightCommonOverlay.xul

And in the overlay, add a js file that is going to have out secure password manipulation code.

<overlay id="common_overlay" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <script type="application/x-javascript" src="chrome://preflight/content/js/secure.js"/>
</overlay>

Here is the contents of secure.js. I’ll explain them afterwards.

var passwordManager = Components.classes["@mozilla.org/login-manager;1"].
                       getService(Components.interfaces.nsILoginManager);
var nsLoginInfo = new Components.Constructor("@mozilla.org/login-manager/loginInfo;1",
                      Components.interfaces.nsILoginInfo, "init");
 
function storeUsernameAndKey(username, key) {
    var logins = passwordManager.findLogins({}, "chrome://preflight", null, 'Username and API Key');
 
    if (logins.length == 0) {
        var extLoginInfo = new nsLoginInfo('chrome://preflight',
                               null, 'Username and Password',
                               username, key, "", "");
        passwordManager.addLogin(extLoginInfo);
    } else {
        var extLoginInfo = new nsLoginInfo('chrome://preflight',
                               null, 'Username and Password',
                               username, key, "", "");
        passwordManager.modifyLogin(logins[0], extLoginInfo);
    }
}
 
function retrievePreflightUsername() {
    var logins = passwordManager.findLogins({}, "chrome://preflight", null, 'Username and Password');
    if (logins.length == 1) {
        return logins[0].username;
    } else {
        return "";
    }
}
 
function retrievePreflightPassword() {
    var logins = passwordManager.findLogins({}, "chrome://preflight", null, 'Username and Password');
    if (logins.length == 1) {
        return logins[0].password;
    } else {
        return "";
    }
}

This is almost straight copy-and-paste from the Mozilla docs, but with an additional layer of wrapping added. The part that you will want to change is the chrome url. It should be the url for your plugin.

The reason why I wrapped things like this is that it allows us to do things like

var u = retrievePreflightUsername()

from anywhere in your addon’s js. (This works because we included secure.js at a very high level in the app.)

Some things to be aware of when using this

• This does not protect your credentials from other plugins. They can just as easily make the same calls as you did and get the information. All it does it protect it from prying eyes on disk.
• This method of dealing with password was introduced in Firefox 3.0. As a result, if you are using this you need to set the minVersion in install.rdf appropriately.

The world is full of people who don’t read English, so it makes sense that your application should cater to more of the world than just them. Firefox provides a means to display these other languages through its use of locales.

There are a couple of ways to play with locales, but the easiest way is the Quick Locale Switcher plugin.

Se-IDE currently ships with support for four locales

• en-US – US English
• fr-FR – France french (as compared to Canadian french)
• ja-JP – Japanese
• pt-BR – Brazilian portuguese

The reason I list these here is that, like most things, you need to make a chrome.manifest change for each locale you support. And since the plugin your custom plugin supports those four, yours should too.

locale preflight en-US chrome/locale/en-US/
locale preflight fr-FR chrome/locale/fr-FR/
locale preflight ja-JP chrome/locale/ja-JP/
locale preflight pt-BR chrome/locale/pt-BR/

Now, Firefox knows where to look for its language strings.

Believe it or not, if you have been following this series you are already used to making use of the localization framework. Recall that we have been putting the following line in our .xul files?

<!DOCTYPE prefwindow SYSTEM "chrome://preflight/locale/options.dtd">

Through a bit of internal magic, the ‘locale’ part of that gets switched into whichever locale you have selected so in my ‘normal’ case the url is really chrome://preflight/chrome/locale/en-US/options.dtd. With that url loading in the overlay we can now pull locale specific strings from it.

Right now, it only has one string in it.

<!--en-US-->
<!ENTITY performpreflight "Perform preflight checks">

To reference that string you use the entity name preceded by a & as shown in this snippet of overlay.

<hbox align="center">
  <label control="name" value="&performpreflight;"/>
  <checkbox preference="pref_perform" id="perform" />
</hbox>

Ideally, every string that is presented to the user in your overlays would be handled like this. A good way to test this is through a technique I call LOUD which I will let you research on your own a bit.

But referencing DTDs in xul is only half of the L10N story as far as our plugins go. Plugins are a combination of XUL and JS.

The way you localize the JS is very similar to the XUL only instead of using .dtd files you use a .properties file. Now my Preflight Checks plugin is still too simple to have JS that displays something to the user, so I can’t really make use of this. But essentially, this is what you need to know…

• Property files have the syntax of key=value
• In your overlay, use the stringbundle element to inject it into the DOM
  

  <stringbundle id="options.strings" src="chrome://preflight/locale/options.properties" />

• Get a reference to the bundle from the DOM and then extract the string
  

  var strbundle = document.getElementById("options.strings");
  var nofilesfound=strbundle.getString("notFoundAlert");

That’s it. It can be a bit of a pain when creating screens to now have to put text into multiple files when working, but it is an absolute pain to have to find all your strings after the fact, so just get in the habit of writing L10N clean code from the start.

Oh, and here are two links that you will want to have handy when doing this step are

• XUL Localization
• Property Files

So I went to the monthly gathering of TASSQ this evening. I figured that I should probably go to a meeting or two before I’m slated to talk at the March one and now that I’m independent I should make myself known in various circles. What I discovered appeared to the a bastion of ‘old school’ qa (not testing).

Now, I know this post is going to sound like a bash, and it’s not really meant to be one. It is more rooted in incredulousness. I know it is hard to get up in front of people and expose your ideas and ways of doing things to others. But you have to remember, that when you do, you open yourself to this sort of thing.

Oh yes, and it is also important to realize (as most who read this blog likely do already) that I’m philosophically aligned with the Contextual, Agile and Craftsman schools of thought.

What was the name of the talk?

Factory School Testing Revisited / Redacted / Re-energized. aka Kickin’ it ‘Old School’

Gah! Right there I knew I was in trouble.

Anyhow, here are the notes / impressions / ideas I had during it.

• One of the stated goals was to have a topic that resonated with folks in the room. I’ve only been to a couple TASSQ meetings before, but if this is the topic of interest then it kinda says that the group is not concerned with new ideas.
• I haven’t worked in a regulated environment in a long time so I could be wrong, but don’t the regulatory bodies care about types of testing you are going to do before hand and the results after? I seemed to get the impression that one of the large arguments for creating a large body of test cases ahead of time was to show the ‘depth’ (or something) of your ‘to be done’ activities. Well, that doesn’t prove anything. What matters is what has been done. I could say that as part of the testing I’m going to build a rocket and go to the moon! Results are what matter.
• Also provided on the same slide was the notion of resource variability. ‘We have high turnover of QA people so we need stacks of test cases to bring the replacements up to speed’ is my paraphrase. Well, if you have high turnover, then I would say that is a smell that you are doing something wrong. And what about turnover in test management? I’m guessing it has the exact opposite pattern — the entrenched stay entrenched and those that challenge or deviate from plan are removed.
• He had a nice metaphor about Christmas and testing. Testing is like Christmas morning; when you wake up there are gifts lying under the tree. But they are wrapped and you can’t see what is under the paper. And sometimes you like what you get, but sometimes it doesn’t fit and you have to return it. This can sometimes hurt the feeling of the giver, but often they understand. And there is no guarantee the replacement you get is going not have a page missing or some other thing wrong with it that you don’t discover until the next day. So the cycle repeats itself.
• I’m not the best speaker, but there is likely a business opportunity waiting to be exploited in teaching testers how to present to small-ish audiences of strangers. How to speak clearly, to the audience, without fidgeting and with a nice slide deck. I wonder how my Lessig-esque deck is going to go over in March. Care to bet the comments that come back are that I need more bullets?
• Not once were the people involved referred to as ‘testers’. It was always ‘QA’. What you name something is critically important.
• What I think was most scary was that I can remember a time when everything that was said would have rang true to me. I’m not saying that Rapid Software Testing threw me from my horse as I was likely travelling down that road anyways, but apparently the big Canadian banks need some heavy introductions to Context techniques and ideas.
• I sat next to Fiona Charles who offered up to a discussion of metric a rule-of-thumb I hadn’t heard before. 3:3:1 – For every three test cases you write, you will find one bug. For every three bugs you find, one will be severe enough to warrant fixing. If you application is producing more bugs that the model, then you need to look at the quality of the application. If however you are finding less bugs than that, you need to look at the quality of the test cases.
• If metrics and counting test cases executed is so important, isn’t it rather ironic to encourage testers to go ‘off-script’? You number are wrong(er!) then by your own admission.
• I did like that it was stressed during the metric part that it is not the number exactly that matters on the graph. It is the pattern that it depicts that counts. Oh, and that it might take 20 minutes to generate the numbers daily; of which 15 minutes of that is writing the summary and story (context) around them.
• My big issue with the Factory is around its reliance on test cases. So I asked a couple questions at the end about it (in as non-obnoxiously as I could). And through that I had to explain the Mine Field problem. Sigh. I know I associate with people who value constant learning as a core attribute, but c’mon! Learn the core theories of your craft! Someone in the audience added that in real-time/safety-critical systems they cover the system 100% thus removing all the mines. Except, that is impossible. You’ve covered every scenario and path you could conceive of at the time. But unless you test every scenario, every interaction, by every person, ever to use your product, ever, then you don’t have 100% coverage. Even if you have 100% code coverage, which I admit is likely what he meant, you can’t account for oracles like emotions at the time of use.

The Stack Overflow podcast #71 has finally bubbled to the top of the queue. It is kinda two different podcasts smooshed into one. The first half is clips from the speakers from the SF DevDay event, and the second half is the usual gang of suspects goofing off around a microphone. The redeeming part of this particular episode was Jeff Atwood’s on-stage portion. Here are the bits you should care about. (Or at least I did…)

• An interesting interview idea – Prepare a list of the smartest people in your field and present that list to the candidate. How many people can they identify? Do they argue for people you missed or that shouldn’t be present on the list? Seems like an interesting way to weed out the people who do the job just for the mortgage
• The Stack Overflow Team Motto: Feel free to fail early, often and frequently as possible in really painful ways
• Don’t phone in
• Be passionate for the ‘craft’. But not just the craft of constructing the software, but for delivery of the solution
• Would you sign your name on the product? The Amiga 1000 hardware developers did.
• Jeff recommends Coders at Work and provided some of the rationale behind it through quotes:
  • Douglas Crockford – Don’t major in CS; major in English
  • Joshua Bloch – Recommends Elements of Style. If you can be a really good communicator, thats the kind of program you will write.
  • Joe Armstrong – Advice to universities: produce graduates that can write and argue coherently
• Before you ask for help on something talk through the problem to a stuffed animal first
• Can you deploy something new everyday? Do you make it even a tiny bit better every day?
• The Train of Shame – the error logs, in real-time(ish) to show you what is busted. So important that you figure out and implement this first before you do any production code. Start and end you day by watching this.
• Take bits from different stacks as necessary. Don’t just stick with only one solution family
• The big engineering problem is going from 1 to 2 servers. After that, it is ‘easy’ to add more.
• With technology, use the lightest thing you can get away with

Part of the reason for the 1.0.4 release of Se-IDE last week was support for the Firefox 3.6 series of browsers. Now, Mozilla has a fairly aggressive upgrade process, but you can’t always trust that people have the same / current version of the browser. That means you need to test against multiple versions of Firefox.

This post explains my trick of having multiple Firefox versions on a single Mac OS machine.

1. Download as many releases as you want / care about
2. Rather than install it into Applications like you normally would, install it into a folder in Applications that relates to the version.
3. Start each one and uncheck the Preference that would have the browser check for updates

Now you have many different versions of Firefox installed. There are likely to be some headaches around plugins, etc. but those could likely be mitigated by having a different profile for each one.

What about with Selenium? To do that we need one extra step

4. Mac applications are directories called Something.app (so Firefox.app). So what you want to do is create a symlink from /Applications/your-version-of-firefox-dir/Firefox.app to /Applications/Firefox.app

Now you can run your Se-RC tests on different versions of Firefox by changing where the symlink points to. This is because how Se-RC finds Firefox on the system. Here is the Firefox 2 locator. Three is very similar.

private static final String[] USUAL_UNIX_LAUNCHER_LOCATIONS = {
        "/Applications/Firefox-2.app/Contents/MacOS",
        "/Applications/Firefox.app/Contents/MacOS",
        "/usr/lib/firefox", /* Ubuntu 7.x default location */
};

In this trick we’re making use of the middle location.

Making the rounds through twitter right now is Sikuli. I became aware of it through someone claiming it a Selenium killer. Well, not quite.

The ‘magic’ of Sikuli is that it used the actual content on the screen to find elements rather than a native API or XPath by taking screen captures and looking for image areas to interact with.

Yes, that’s correct, it used the way an object is rendered on the screen to determine whether it is interacted with or not. Didn’t industry decide this was a bad idea a number of years ago? Here is why this project fails the sniff test for me.

• What about when I create a script on one resolution and play back on another?
• What about different record and playback OSes?
• Accessibility enhancements on/off?
• Image reuse? Without this you have to change every location when something changes. And things always change.
• How do you data-drive something that requires you to have image captures of the run? Data driving is rather important when it comes to minimizing code duplication yet increasing test data.
• And since you have to have visited each screen in a certain manner whilst writing the script, you start to run afoul of both the Minefield Problem and Pesticide Paradox
• And and if you can’t data-drive a script, is there any chance that scripts would be written in a model-based manner? Highly unlikely.
• Can you record actions? I would wager that Selenium’s success can be largely attributed to the Record functionality of Se-IDE. A quick spin through their site implies that manual script creation is the only option. People who have done the market research have told me that most QTP users stay in the keyword, non-script, part of the environment.

Of course, the goal of Sikuli isn’t to solve these problems. It is a research project between two groups at MIT. This is said with some derision in my voice. Yes, some things are well suited to make the transition from Academia to Commercialization. Google came out of Stanford for instance, but it is/was at its core, a new algorithm to (re) attempt to solve an understood problem. The issue with this is that the divide between Academia and the Real World in Testing is mammoth. I’m not sure why this is exactly, but I often just shake my head when reading papers on testing by people ensconced in a University somewhere. Yes, there are some exceptions to this, but they are exceptions rather than the rule.

Now, contrary to how it may seem, I’m not completely bashing Sikuli, more just pouring some cold, hard reality against it. I understand its appeal; since it uses screen captures is can interact with anything that displays on the screen. That is close to a Holy Grail if ever there was one, but they way in which they accomplish it is terminally flawed I fear.

One place I see Selenium users stubbing their toes on time and again is around automating HTTPS connections. I’m about to tell you that you should save yourself a lot of pain and just not do it.

Having a secure connection in production is an extremely good idea. Especially at places like checkout and authorization. But in testing? Not so much.

There is exactly one situation where you need to be running HTTPS in testing, and that is if you are using certificates as an authentication means. For every other test scenario, in test, you can run things just as HTTP. Not only can you watch the network traffic in the clear over HTTP but you don’t have to worry about the trust issues certificates introduce.

And there is almost always trust issues.

Why?

Because ‘real’ certificates cost money. The whole notion of security is based around trust, and the way to establish that trust is through money and verification. If you look in your browser, it will list somewhere all the CA (Certificate Authority) root certificates that it trusts by default. Those vendors pay non-trivial amounts of money to get in there. And then they charge non-trivial amounts of money end consumers to sign a certificate with their root certificate. Production sites will (almost) always go that route since your customers are likely to be a little concerned about all the browser warnings about things being wrong with the certificate.

In test though, machines are often rebuild and used for multiple purposes so companies are loath to spend the money on real certificates. When manual testing, this isn’t anything more than a nuisance dialog to dismiss, but when automating, it is much more annoying.

The problem is that the error box is not part of the application, but is part of the browser. In Selenium 1.x there are some workarounds, and I believe Selenium 2.x and WATiR solve this by low-level control of the browser.

The easier solution though, is to just not use HTTPS in the testing environment. Sure, your testing machines won’t be as ‘production-like’ as they might otherwise be, but you have to ask yourself, does that materially affect the quality of your testing?. In most situations, it won’t. In fact, I would argue that it increases the breadth of testing available to you; both manually and in an automated fashion.

edit: To be clear, the workaround mentioned are just some of the possible solutions in Selenium 1.x for dealing with HTTPS pages. There are a lot of situations where it will Just Work. Especially in RC where the server is a proxy and certificates get injected on the fly. I would still say in those cases to turn off HTTPS though; it is really hard to figure out what is going on at a network level when it is encrypted.

Been a busy couple days in Selenium-land. On Tuesday I pushed a new version of Selenium-IDE (see the official announcement) and yesterday Sauce Labs pushed Sauce IDE. I’ve not been coy about my relationship with Sauce Labs and don’t want this to be a generic fluff post, but I think Sauce IDE will eventually be the model for building a commercial product around Selenium-IDE. And the reason for this is/will be packaging.

Firefox extensions are, at their core, just a zip file with an extension Firefox knows what to do with and have a minimum common set of configuration files (install.rdf, chrome.manifest and such). One of those configurations in the Se-IDE and Se-IDE plugins is to set the em:type to 2 which tells Firefox that it is an extension (rather than a theme or whatnot).

The ‘classic’ way for a businesses to commercialize an open source project is to essentially fork it and start along their own, potentially divergent path. One of the goals of starting to develop an API for Se-IDE was to explicitly avoid all this splintering.

The Firefox system for extension is extremely helpful in this regard. There are very few things a commercial entity cannot do to Se-IDE through overlays. This means they don’t need to fork. I fully anticipate that situation where extra helper methods, or even some refactoring will be necessary to succeed long term, but in general, companies don’t need to change the code Se-IDE code. They just write an extension.

There is still a bit of a delivery problem though. My model for the plugin idea for Se-IDE has been the relationship between Firebug and YSlow. When you go to the YSlow page it says to install Firebug, then install YSlow. This is fine for the alpha geek demographic, but is less than appealing from a corporate perspective. What you want is a single installer.

The folks at Firefox thought of that too! Enter the multi-item installer.

What this lets you do is package multiple .xpi files in a single .xpi file. Which means, in a commercialized Se-IDE perspective, you can ship the version of Se-IDE unmodified as well as your modifications as an extension in a single file. Here is the install.rdf of a multi-item xpi banged out to package Firebug with a couple plugins as a make-sure-it-works experiment.

<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
     xmlns:em="http://www.mozilla.org/2004/em-rdf#">
 
  <Description about="urn:mozilla:install-manifest">
    <!-- nsIUpdateItem type for a Multiple Item Package -->
    <em:type>32</em:type>
 
    <em:id>firebug@goucher.ca</em:id>
    <em:name>Firebug Multi-installer</em:name>
    <em:version>0.0.1</em:version>
    <em:creator>Adam Goucher</em:creator>
    <em:description>Proof is in the pudding</em:description>
 
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id> <!-- firefox -->
        <em:minVersion>2.0</em:minVersion>
        <em:maxVersion>3.6.*</em:maxVersion>
      </Description>
    </em:targetApplication>
  </Description>
</RDF>

Now, Sauce IDE isn’t quite there yet, but that is the endgame. Right now, it is about 60% there; Se-IDE was forked, but the modifications were made in a plugin-esque manner so they could be separated once it was all working. It’s working now, so the splitting will commence shortly meaning the fork will be retired and an official Se-IDE xpi will be bundled as part of the multi xpi.

Plugins are where a lot of future development of Se-IDE will be focused. Heck, some existing functionality might be extracted into an extension to clean up the code / core. Plugins are only half the story though when it comes to commercial entities adopting Se-IDE as the automation platform they build upon. They need simple distributions to appeal to the mass market. Thankfully, there are ways to solve that problem.

So let’s start getting those commercial distributions of Se-IDE going!

As part of its 100 Best Global Brands series, Business Week had an article on The Great Trust Offensive. Trust is an idea we need to be very aware of, not just from an outward corporate perspective, but from internal corporate and personal career ones as well.

Here are three key quotes:

• In the world of branding, trust is the most perishable of assets
• If you aren’t open with me, then I won’t trust you
• Stay relevant to the current mood or risk been seen as spin merchants

Part of why I think agile-ish things work is that by breaking down the silos and building an environment where communication can happen, then trust has a chance of being achieved.

The article ends with a warning though which resonates with my theory that agile will fail to reap maximum benefits if any part of the organization is actively resisting (intentionally or otherwise).

Trust-related marketing only works if there is a message that people want to believe in. You cannot spin an audience that doesn’t want to be spun.