Posted on November 25, 2008 in Selenium by adam2 Comments »

So I admit to not being too active on the forums so I may be blowing smoke here, but Selenium seems to be losing traction in the market right now. I don’t think we’re at the turn-off-the-lights-when-you-leave phase, nor do I think it will ever really come to that given the number of existing users, but from a going forward perspective I think there might be some pain.

I use Selenium as part of larger tools that I build. I might buy a tool if it did everything I wanted, but like to build my lightsabers. Let’s follow the the Star Wars analogy for a bit.

  • Leadership – At the top of the Jedi heap is the Jedi Council who lead and guide the rest of the herd and in that group there is a single, identifiable, recognized leader. Watir has Bret Pettichord, but Selenium has …? We need someone to come out and take ownership of Selenium, coordinate the parts and releases and plan out the roadmap.
  • In-person gathering – The Jedi Council would often get together to work on problems and assign tasks either in person or using remote presence technology. Skype, IRC, forums, etc. are all great, but nothing beats in-person communication. I seem to think that the Selenium developers got together at Google once, but that sort of even needs to happen more often; twice a year maybe? The Watir community is having an event January which is not necessarily just the developers, but I would bet that the core group will be there.
  • Corporate Support – All member planets / systems in the (Old) Republic contributed to the expenses of the Jedi. Watir now has the backing of WatirCraft. Selenium has…? Well, ThoughtWorks is where it came out of, but are they actively promoting it or employing core members of the team? Google poached Jason Huggins from ThoughtWorks and seemed like it could be the corporate sponsor, but he left. So who is there?
  • Outsourcing – When things got tough and the Republic needed a lot of (relatively) cheap bodies to do the grunt work, they called in the clones. (Well, sorta. But it could be argued that the outsourcing much like the clone army often backfire). But what offshore outsourcing firms have Selenium expertise? Most I have seen / talked with are HP/Mercury shops that if pressed will tell you they have a couple people on staff with some clue. But clue to what level? I think this is a potentially untapped market just ripe for exploitation
  • Consulting / Training – Aside from ThoughtWorks, I couldn’t come up anyone who is billing themselves as a consultant or offering training on selenium. (PushToTest does to some degree, but that is in the context of their TestMaker product). Look at the industry around Mercury toolset. Training and value-add is often cited as the path to success with open source, so where is it with Selenium? (Sorry, no Star Wars metaphor on this)
  • Books – There is a Short Cut on using Selenium, but that seems to be about the extent of commercial documentation. A consultant who has written the book on Selenium could make a decent living I think.
  • So how do we address these? Or do we want to? (And now I have to go re-subscribe to all the Selenium mailing lists since I seem to be jumping back in.)

Posted on November 23, 2008 in Uncategorized by adamNo Comments »

I’ve had twitter for awhile now, but have only used the RSS aspect of it which can make for disjointed reading. But while I am still more in the ‘Don’t quite get it’ camp, I’ve started to actually publish my own tweets. You can join the madness here.

Posted on November 23, 2008 in Ruby, Selenium by adam1 Comment »

For awhile now I have been advocating that building an automated test is a four phase process. The phases are:

  1. Record
  2. Add checks
  3. Data drive
  4. Make it smart

In this post I’ll illustrate these steps in automating our One Minute Calculators using Selenium. Because this is tutorial in nature, not all aspects are automated; just enough to illustrate. It is also a huge post, so it is behind the break rather than make the main page huge.

(more…)

Posted on November 21, 2008 in Quality by adam2 Comments »

I’ve come to realize that I have a tonne of content that is in powerpoint and not in the blog as entries which can be a pain if I can’t get to the slides (that are hosted on sourceforge), so I’m going to start recapping some of it. Here is the first one…


First, let me start by clarifying why rules in in quotes. These rules are very much heuristic in nature and should not be taken as gospel; it is quite easy to think of scenarios when they would be inappropriate to follow. They are merely guidelines that have I have successfully applied when thinking up test cases ahead of time. They also help when thinking about whether something will be automated or not.

  1. One, and only One thing – This rule is states that a test case should only be measuring one, and only one thing. Imagine you have a test case that does two things: A and B. If A passes, but B fails, does the test case pass? Or does it fail? It is more like ‘sorta’ which is not really a state you want to be in. Similarly, if A fails, but B passes, does the ends justify the means and it is a pass? Likely not. Likely, sorta, maybe, kinda are not things I like having to report.
  2. Self-sufficient – Test cases should not be dependent on other test cases. Much like the previous rule, if you need 5 tests to pass before yours can even run, you are setting yourself up for a tangled web of test case dependencies. This doesn’t mean that your test case can’t have a number of preconditions as part of the test setup, but those are included in the test case. And these setup steps do not have to be overly prescriptive. ‘Create a valid user’ is enough detail for a lot of tests. An extra benefit of this is that tests can be run in a random order which is a good check for whether they can be run in parallel if/when they are automated.
  3. Black or White – Black and white is a nice place to be. The success (or fail) criteria of a test should be very clear. Again, fuzzy results are not fun. (Of course, the definition of what is a pass or a fail is contextual and should be specified somewhere.)
  4. Valid test data – All data has rules around it, though they might not be obvious at first. Explicit rules are things like the column definition in the table and things like password strength rules. This only half the problem, and the less important half at that. They other half deals with the implicit rules that you might not be able to get without questioning the actual users. I learned this one first hand while in the Bahamas on my first business trip for my first job. I was testing central bank reporting software for the offshore Trust subsidiary of a major Canadian bank and using numbers like 5, 10, 25, 100 as inputs. Technically they met the explicit rules and were really easy to do math on without a calculator but when the head of IT for the company saw what I was doing he hit the roof. To open an account there, clients were plunking in most cases over $1 000 000. My $5 test was not exercising the application in any sort of valid way. I added a bunch of zeros to my input numbers and redid all the tests. Lesson learned.
  5. Repeatable – Tests should be constructed in such a way as to be repeatable, both immediately and at some point in the future. This often means that you reset the environment to its prior state whether through removing entries in the database or simply clearing your cookies.
  6. Write for the the correct audience – Do not write your tests for 3rd graders unless the people executing your tests are 3rd graders. Odds are testers doing the tests know the system, so you don’t have to explain how to use the system. ‘Setup a password server’, ‘Create a new question’ or ‘Login as a user with administrative rights’ might not mean anything to you right now, but you are not in the situation for it to. If you were, then it would.
  7. Brain on? – The best testing is done with the tester’s brain firmly locked in the ‘on’ position. This means not boxing them in with things like ‘Start the application. In the firstname field input ‘Adam’ and in the lastname field input ‘Goucher’…’. This was the old school way of designing test cases and too often employed by teams who thing poorly of their testers. The only excuse for tests of the brain ‘off’ sort is when there was a specific triggering condition, then you should have the test case specify that specific condition.
Posted on November 19, 2008 in Uncategorized by adamNo Comments »

In the process of writing up the Deep Bruise Heuristic I realized there was another lesson in there as well.

Other testers (or players) are not you.

Just because you would have done something some way does not mean that they someone else would do so as well. I would have cleared the crowd to find a shot or pass had I been the one fighting for the ball, but that is because I cannot shoot backward or underhand. He could which, unfortunately for me.

When I talk about tricks and hard-won realizations to new testers, one of the things I stress is that you are not like the customers / end-users of the system so any ‘pretend to be the customer’ stuff is not much use. As testers, we know the existing system (usually). We know the goal of the product. We know what it is supposed to do. We know all the hacks and shortcuts we needed to use in order to work around problems or incomplete features. We know X. We know Y. We know Z. The customer does not. (Or usually does not to avoid and absolute statement)

When I talk about structuring a test team, a large part of what I advocate is for a diverse team to compensate for everyone’s biases and blind spots which is essentially this idea. Not diversity for diversity sake, but when it makes sense. This makes sure that testers are not all like you.

Both these things have a root in uniqueness and the danger of equating yourself as the same as someone else. Take a step back from a problem and double-check your surroundings if you find yourself thinking someone would have tested something because it is something you would have tested. For instance, I have thought a lot about how to test i18n in a web application and have presented at a conference before on that thinking so I like to think I know what I am doing. If someone says they have tested i18n to me I need to make sure I don’t automatically assign a level of testing that I would have done. I need to ask clarifying questions to come to the proper understanding of the situation.

Posted on November 19, 2008 in Uncategorized by adamNo Comments »

Named heuristics are fun. Here is one that has been make itself apparent to me over the last week which I have started to think of as the ‘Deep Bruise’ heuristic:

Sometimes a really deep bruise is necessary to actually learn something.

I play lacrosse, which even though it is a non-contact league (I’m not crazy) there are still plenty of opportunities to inflict pain on yourself. Take last week for instance.

One of the things you are not supposed to do as a player when in the offensive end is get in the lane of a shot from someone else on your team. When on defense, sure, but not offense. Guess what I forgot. I was to the left of the goalie and the ball was in front of him. There was someone defending me where I was so I decided I should be on the other side to both escape them and possibly wrangle a zone coverage penalty (they had already been warned). So I start running in front of the crease to get to the other side — and cut through the lane. Now, the player on my team trying to get the ball had his back to the goalie and I figured when he got the ball he would take steps forward to clear the crowd then turn around and either shoot or pass. Except he got the ball and fired it behind him (see where this is going?) and straight into my leg. I’ve been hit in worse places with greater velocity, but it did sting a bit and it was completely my fault.

Five days after the fact, the bruise is really only now starting to appear which meant it was pretty deep (and means most of my thigh is a weird yellow/brown/purple which is oh so pleasant) and it somewhat sore.

I like to think that the next time I decide to be on the other side of the net for a play that I will run behind the net rather than cut through the shooting lane. I don’t need another bruise which could be a lot worse next time (there was someone on my team who plays professionally and can shoot FAST).

Now for the ‘Deep Bruise’ heuristic:

You will miss bugs when testing. And when you do both your product and your reputation take a bruising. The trick is of course to turn those bruises into a learning experience and not to get bruised in the same way again.

Posted on November 18, 2008 in Uncategorized by adam2 Comments »

When I talk to testers about the tools I find useful for testing I always end up talking about Firefox. If you are testing a web-based application, you owe it to yourself to get up to speed on Firefox-the-tool as compared to Firefox-the-browser.

The power of Firefox is in its extension ecosystem. Here are the ones I have installed, though I am sure there are other interesting ones that I haven’t discovered.

  • Web Developer – Clearing your cookies from the toolbar is worth this extension alone but you can also validate your html/css and highlight certain elements on the page.
  • Firebug – Firebug is likely the single most useful plugin for Firefox. I use it to troubleshoot html generation and css rendering issues as well as checking for proper form validation by removing client-side validation from the rendered page. It will also show you the XPath location for items on the screen if you are using that in your automation setup.
  • YSlow – YSlow is a plugin for Firebug which ranks your page based on a set of heuristics that were learned by people at Yahoo and later in a (couple of) books. I’m less hot on this than I was a year ago though as a lot of sites don’t need all the things it suggests; like a CDN. It is still a useful tool for a suggestion perspective though.
  • Live HTTP headers – Right after Firebug, Live HTTP headers is plugin I install. It lets you see the non-rendered communication between the browser and the server; cookies, expiration headers, etc. There are a tonne of testing possibilities once you can see this exchange.
Posted on November 6, 2008 in Uncategorized by adam2 Comments »

This is going to be a rather ironic post since I don’t consider myself (yet) all that great a presenter and it is not meant to be (overly) critical of people, but, I’ve been thinking about this stuff for a bit and today solidified some of it. What follows are things that detract from presentation (at conferences or elsewhere) and/or things I have learned when doing my speaking.

  • The Lessig style of preparing slide decks is nice
  • The quality of your template affects the outcome of your slides dramatically
  • Do not overrun the boundaries of your template in distracting ways
  • Too much content on a slide is too much content on a slide
  • Trying to code live rarely works and results in silly type-o’s more often than not
  • If you are running applications or scripts during your presentation make sure they have been tested on the machine being used during the presentation. All of them. Not just all but a couple.
  • Have resource external to your deck pre-loaded
  • If you are presenting on a topic, you should absolutely own your content. You might not have thought of it originally and you might be rehashing it to a new audience, but have internalized it enough that you can convince people of your ‘expertise’. You don’t have to be the ultimate expert, just make the audience feel you are more expert than them
  • Check that you have your various personal security-blankets with you before leaving the house for the conference / event
  • Do not print out your slides and hold them in front of your face and read things. If you need documentation to help you then just have a couple pages with cues. Or if something is a long quote then have just that quote. Reading from your slides without elaboration is not instilling the belief of expertise.
  • If showing books either in their dead-tree format or in your deck, don’t accompany them with ‘I used these to do my research for this presentation.’ Cite them as ‘for additional information see these…’ The former conveys that you don’t know enough to really be presenting. That is heuristic of course, but again, if you are at the front of the room you are the expert.
  • Printing out large decks with a nice binding makes things start out on a professional note. (I wish I had done that along with code listings for my tutorial)
  • Whenever possible, learn about the demographic of the audience and tailor appropriately.

And yes, I know I am guilty of more than a couple of these. (I did say there was some irony about me writing this.) But I am less guilty of them now than I was a year ago; especially with my QA101 course which I can do without the crutch of a deck and be just as effective.

Posted on November 6, 2008 in GLSEC2008 by adam2 Comments »

This was my half-day tutorial. I think it went okay and have yet to hear anything to say that it sucked so that’s a positive. We’ll see what the evaluation forms say. I’ve posted the slides I used on slideshare for anyone who wants them. I’ve also embedded them below along with links to the posts which have the actual scripts. When looking at the slides, when you get to a slide that says ‘How to Use’ you should go to the relevant script to have things make some sense.

Here are the individual posts that I wrote while building this presentation (you can get all the scripts from this presentation from the individual posts):

Posted on November 6, 2008 in GLSEC2008 by adam1 Comment »

I’ve known of Jason Huggins for a couple years and we had exchanged emails a couple times about Selenium over that period. This was the first time we had met in person though. Another person off the list.

Anyhow, he was at GLSEC to do the closing keynote in which he compared the evolution of movies (silent to talkies) to how people learn how to use an application (rtfm to rtfv) with the advent of screencasts. Even if you take away the points for using a Three Amigos clip, there were a couple mind blowing ideas presented; especially if you are a small Agile shop trying to show progress to client’s on a iteration-by-interation basis. (Steve, this is directed at you.)

What if you engineered your automated tests to build a screencast that the client could then run to show completion and functionality? We have the technology (here too) to do most of it. All that is left is the glue to put it all together.

These could also be given to the end user as a video of how to use the product at which point the tests really are the documentation.

If I didn’t have so much already on my plate I would totally go off hacking right now…

Next Page »