Posted on September 30, 2009 in Uncategorized by adam1 Comment »

For a couple unrelated reasons, I’ve been thinking of which role in an organization has the greatest impact on Quality.

The first obvious answer is that testers do, since they find and advocate for bugs while acting as an agent for the user. Except, that implies gatekeepr-ism, which is not desirable (and been explored a couple times iirc.)

The next obvious answer is that the developers do. After all they are the ones putting in the bugs in the first place. All a tester does is discover them. But even the developer can’t solely be held responsible for the code when what they are developing is a giant ball of knotted yarn.

So what is the most important role for Quality?

I’m starting to think it is the Product Owner. Think about it for a second. The Product Owner:

  • Talks to the customer
  • To find out what they need
  • Interacts with both development and test
  • By prioritizing work
  • And accepts or rejects it before it gets to the end user

All of those thing have a HUGE impact on what gets released. If the product owner is all over the place, then the product will be too. If the product owner only pays lip service to quality, it will be evident when it gets released. If the product owner doesn’t have the pulse of the consumer / market then they will be leading the project down the wrong road.

There is a whole industry around training testers to think better and to write better automation to make them more efficient. There is an even bigger industry around training developers to write (more | any) tests, loosely-coupled objects, etc.. What I don’t have any insight into is what we are doing to make our Product Owners better in terms of Quality.

I suspect this might be a potential role for an evolved tester. Usually the roadmap is junior, intermediate, senior, lead, manager. Perhaps there is a fork in the road somewhere that takes you up into the business side of things where you can drive a Quality Agenda from.

Food for thought.

Posted on September 30, 2009 in Uncategorized by adamNo Comments »

The kids at Sauce Labs have put up a video from the last SF Selenium meet-up where the speaker was none other but Mr. Continuous Deployment himself, Timothy Fitz. You might remember the kerfuffle he raised in the testing world earlier in the year. If not, then you can see Michael Bolton’s thread on it here, here, here, here, here and here. I had hoped that Continuous Deployment (CD) would not get widespread adoption, but he is still flogging it so I decided to momentarily stop avoiding it and watch the video of the presentation.

Before I list my thoughts and notes I took while watching the video I think I’ll outline my mindset to frame them. CD to production is reckless and borderline irresponsible. ‘Automated tests do not a test strategy make’ to paraphrase Yoda. That said, there are some cool things he is doing that I can see be quite worthwhile exploring; especially if you are rolling out to dozens, or hundreds of machines. I suspect that some of them are almost required at that sort of scale. CD to a test environment for humans to test is an absolutely fantastic idea though. We should do more of that.

With that out of the way, the notes:

  • Don’t fix bugs on features no one uses. I would go further and question why something is in the product at all that is not used.
  • At it’s core, CD means ship more frequently
  • Regarding automated testing: now we’re playing for keeps
  • That is a huge problem for me though. That implies that we know what the automated tests should look like to replicate human behavior the first time. We, as humans, are very bad at looking into crystal balls regarding the future.
  • CD implies at a very deep level that tests will be executed in parallel. That can only be a good thing in my books.
  • As is the idea to run tests in random order even when there is no commit-triggered build in order to find race conditions.
  • Debug all failures. No excuses. Your automated tests are the only thing between your IDE and the customer.
  • The most powerful idea in Timothy’s talk was about creating an Immune System for your application. I completely want to steal this idea. Essentially you instrument the application out the wazoo and it will automatically back out changes that cause a rush of white blood cells. Things like ‘number of registrations per minute’ or ‘conversion rate in 30 minutes’ are monitored; real business drivers not low-level server stuff. Well, likely in addition to, not as complete replacement.
  • The crux of CD is that you are putting code in front of real users as soon as humanly possible so they can derive the most value from it. This is standard Agile theory, pumped up on steroids to clown-like proportions as you could be delivering code 30+ times a day easily.
  • But customers are also the primary human means of problem discovery as well. Which is the principle problem with CD from a ‘tester’ perspective. I don’t want my customers to experience annoyance, frustration or confusion from my application and take their money to a competitor and tell their friends why they couldn’t do X as advertised on the system. CD seems to have accepted that ‘customers will find bugs’ and are ok with it. I accept that they will find bugs (perfect software is a myth, etc.), but I’m not ok with letting it happen with just a shrug.
  • Lock the repo when something fails since you are now in an unstable state.
  • Integrate the operations team into the development team. Again, nothing new here. We want to develop a culture of ‘the product team’ not silos.
  • We don’t have QA people writing tests, we have engineers writing tests. Ummm, ya. I’m having deja-vu here. This sounds a lot like early Agile/XP where testers were not needed because ‘the bar is green dammit!’. What is state-of-the-art now? Oh ya, having ‘agile testers’. Guess that didn’t work so well, did it.
  • Wow. What a shock. CD is also related to Lean thinking. That metaphor has taken a beating recently in the testing world.
  • You have to make sure your application/platform can handle every edge case. Which of course means you can predict them all.
  • You can’t just start doing CD; your application needs to be built around it.
  • At 58:30 he says that another option is to deploy not to production, but to a production-like environment. That’s the winning idea
  • Have ‘dials’ for all features. The volume of the dial determines how much of the cluster gets the new feature.
  • Gradual deployment is also a component of CD. That is the great backstop that lets your pull things out of production if they don’t work.
  • In response to a question about whether to use it in a ‘mission critical’ (which I’m going to take as ‘someone could die’) he admits that CD is likely not right in that situation. This reinforces the notion I’ve had from seeing other stuff on CD that it is fine for ‘toy’ applications (where things like even large social media things like Twitter and Facebook could possibly be lumped as ‘toy’), but if you are in the ‘enterprise’ or higher then you need to really, really think twice about doing ‘pure’ CD
  • Once a failure is rolled back from production you need to write unit and acceptance tests for it so you don’t find that problem in that way again. Fine. But what about all the other ways?
  • Engineering problems quickly become people problems. Ain’t that the truth.
Posted on September 29, 2009 in Uncategorized by adam3 Comments »

We (testers) know, acknowledge and understand that our role is not to be the gatekeeper of the product. We provide information to the [business] stakeholders who can then make a decision based on their (usually) larger world view.

But what if management hasn’t latched onto this idea yet? Or you yourself are just starting to see the light?

Well, that is a bit trickier.

First, I would say that the person you need to talk to is your boss. Not the team, nor your counterpart in development, but the person who holds your job in their hands. Explain your idea of your what your role should be and explore the differences between what they think it should be. Ultimately, their opinion trumps yours. And if you think otherwise, then be ready for an unpleasant review next time around.

Depending on how good you are at manipulating existing structures to your will, you could slowly change their opinion to be the same as yours.

But chances are good that they will say ‘Sorry, you are the QA/Test person and its your job to be in charge of quality’. And short of finding a new job (more on that in a second), well, you are the gatekeeper whether your like it or not. On the upside though, this has been confirmed by management. Never underestimate the power of such a confirmation. You have been given the soapbox to stand on and a stick to wield.

But it only works if you wield them. Remember, you were given approval, instruction actually, to do so.

This I find is not always comfortable for people. Unfortunately, the ability to do so becomes more important the more senior you get. A key characteristic of a test ‘lead’ is the ability to go nose-to-nose with their equal on the dev team and not go hide in the corner with their tail between their legs. But such is the daily agenda of a gatekeeper.

One way I’ve found to head this off is point out what you perceive the role to entail right in the interview process. That way everyone understands what the role is going to be like and both can say ‘no thanks’ if it does not meet the desires of either party.

Being a gatekeeper is one of those things that seems like a fun job requirement as you absolutely have impact on the quality of the product. It is also very stressful since we as testers rarely have all the information about the larger world our software operates in to make a fully informed decision and blame for production failure has an easy target to aim at.

Posted on September 28, 2009 in Uncategorized by adam6 Comments »

Pending approval per the IP Agreement we all bound ourselves to, my detailed notes will be showing up later in the week, but I’m giving myself permission to publish my own stuff.

My contribution to TWST this year was to throw my archetypes idea against the wall in a room full of testers I respect and see what sticks. From some of the conversations afterwards, it seems like a fair bit did. Or at least enough to warrant further investigation and though. There was some furrowed brows in the audience too though so I’ll have to ferret out that confusion.

One thing I didn’t mention during my talk, or my slides but came up afterwards is that that a particular item can have multiple archetypes at play. For example, a Window is likely an archetype. But the content in that Window might be a Form or Report.

Speaking of my slides, here they are.

Posted on September 27, 2009 in Uncategorized by adamNo Comments »

As someone who blogs (a lot), is starting to appear with greater frequency in conference programs and who has a book coming out, I tend to get a fair amount of unsolicited job inquiries appearing in my mailbox. Some I dismiss completely, but others I will at least open up the company web site. What I am looking for in small, startup-ish companies is not the technology, or the industry, or anything along those lines.

What I am looking for is whether I can determine what they do without having to use the word ‘and’.

In my experience, quality requires disciple in all aspects of an organization. Including its mission and reason.

We help individuals, [and] non-profit organizations, [and] cities, and enterprises do (this), [and] (this), and (this) is a pretty strong indicator that quality is being sacrificed in favor market reach.

And really, that is a perfectly valid strategy from a business perspective. But from a quality one it is atrocious. Those three markets have very different drivers and feature requirements. By targeting all three what you are essentially doing is splitting your attention three different ways resulting in three 1/3 completed products. Instead of that strategy, why not pick one and absolutely dominate it in terms of features and quality. Then take on the next.

Similarly, the three actions mentioned above are also a concern, but not necessarily as bad a one. It all depends on where in the growth curve the company is. For a brand new company it is a huge problem. You should absolutely kick ass with one action before expanding into another.

But! By constraining focus to a laser point are you not turning your back a large part of the market?

Yes. If quality is your goal, you will be rewarded.

Imagine that the market for your multi-limbed product is 100M per year and split into four, clear, sub-markets of 25M each. You could try to fight in all of them and win some battles but generally tread water as you have to spread your efforts across four distinct audiences and this results in say 20M and a reputation in the market as meeting requirements, but meh customer experience or joy. Or, you could focus in on a single segment, kicking ass along the way, still make 20M per year as the dominant force and have an awesome reputation in the market.

As someone who is deeply rooted in the idea of delivery a quality product, the latter option is much more compelling a story.

To a business owner though I completely understand why it would not be. Narrow focus companies tend to be smaller in terms of potential revenue and have a slower growth curve; two items that often run counter to why the company was started. Narrow companies are (likely) not going to change or save the world either. They can however change or save a specific segment of it which hopefully is enough.

So how about your company? How many ‘Ands’ do you need to describe it? And what can you do to decrease the ands and increase the quality?

Posted on September 22, 2009 in Uncategorized by adamNo Comments »

I’m a believer in actively managing your career. After all, no one else is qualified to do it but you. So two articles in Business Week’s September 14th issue caught reminded me of something.

Sometimes your hire-ability depends not on who you are, but your current and past employers.

Article one is The Best Places to Launch Your Career

  • … she entered a three-month management training program …
  • While traditional perks such as pensions and health insurance still have their place, more companies are finding inventive ways to attract, retain, and motivate their youngest employees—using everything from work-from-home programs to faster promotions to financial benefits that kick in a few years down the road. Says Adam Kling, a workplace consultant with RHR International: “They’re using those and other perks to help offset what you’re seeing in your monthly paycheck.” Or more to the point, what you’re not seeing in your monthly paycheck.
  • The Big Four firm still attracts more than 3,000 highly sought-after accounting students each year with extensive training and mentoring programs, performance bonuses, and the promise of face time with top executives—including an annual trip to Walt Disney World (DIS) for all U.S.-based interns, where they get to mingle with the powers that be.
  • To make the most of their recruiting efforts, many companies are scaling back their scope. In recent years, Philip Morris USA dropped nearly 50 campuses from its program, leaving it with just 34. Philip Morris President Craig A. Johnson says the quality of new hires is up as a result, and that more interns are being converted into full-time hires: 47% in 2007, up from 32% in 2005.
  • For many companies, though, the bigger problem—far bigger than recruiting—is retention.
  • Indeed, one reason high pay alone is no longer enough to guarantee loyalty is that many members of Gen Y, who have been entering the workforce since 2004, have other priorities. For them, issues such as community service and serving the greater good are among the most important, according to the 2008 Universum USA survey of U.S. undergraduates.
  • By making a big impression when it matters most—in the first year of employment, when a lot of entry-level hires jump ship, or at the three-year mark, when boredom and frustration often set in—employers can get their young charges over the hump and, with luck, motivated to stay on for many more years.
  • While many companies award promotions only when a vacancy exists, several, including Philip Morris USA, have shifted to a system that considers employees for new positions whenever they’re ready—vacancy or not—thereby removing one of the biggest obstacles to promotion.
  • And Whirlpool is now giving employees a chance to fast-track their careers by offering them opportunities to work on special projects that will enhance their skills, thus making them eligible for promotions earlier. The projects can be anything from a 60-day stint with HR to six months spent designing a new appliance. The pilot program was started in July, but the company expects it to be popular with young employees and to improve retention, which is already the best among consumer products companies in our ranking. “The more opportunities we can provide our employees to engage in meaningful work and challenging assignments, the more quickly we can help them achieve career goals,” says Jeff Beavers, Whirlpool’s director of global university relations.

Reading between the lines then, training and non-monetary carrots combine for low(er) turnover and thus better reputation the company has in the market. Better company reputation, the better you look for having worked there. If your employer has high turnover, no training plans or other schemes to better you, it is a warning sign.

Article two is The Companies Headhunters Avoid and reinforces the deductions from the first. In fact they are back-to-back in the dead-tree version.

  • The question: Which companies do they largely avoid recruiting from? Some of the most cited are known to be in turmoil …
  • Recruiters also singled out companies that are widely viewed as successful.
  • The conclusion among headhunters is that the very attributes that make Coke a great company—an iconic brand and an unmatched global distribution system—also make it too easy for young managers to rise without having to develop the entrepreneurial skills necessary to compete in other arenas.
  • Whether it’s their quirkiness, poor leadership development, or political culture, these players have become the corporate equivalents of the Hotel California: You can check in and enjoy your stay, but the risk is that you can’t leave.
  • Three of the companies named as problematic by recruiters made this year’s ranking of best places to start a career.
  • One trait that puts a company on the blacklist is excessive bureaucracy.
  • … recruiters take issue with what one describes as a “patrician culture of conflict avoidance,” which tends to make some alumni ill-equipped to handle crisis
  • One peril is a tendency at some companies to relegate managers to narrow duties, thereby fostering limited skill sets.
  • Several Silicon Valley headhunters say they’re now hesitant to recruit from two of the tech sector’s most successful companies because of their testosterone-driven, take-no-prisoners cultures. Managers who thrive there, they say, are often bad fits anywhere else.
  • … a culture that seemed to reward executives more for political skills than results.

It is accepted that most upper-level positions in business/management do not get advertised on Craigslist but are instead handled by an agency of some sort. As testing evolves, this will become more normal in this field as well. But even before that, hiring managers will see your resume and see your employer. Is that going to tilt the scales in your favor or against? It might be worth seeing what the market has to say about your employer and then seeing how you can offset or escape it if necessary.

Posted on September 21, 2009 in Uncategorized by adam1 Comment »

When talking about ‘ideal’ [testing] team structures, I tend to stress the need for a diverse set of backgrounds. Not to the exclusion of specific needs of course, but in general, a more diverse team is going to find a greater variety of bugs due entirely of how different view the world.

If a baker walks into a kitchen, the first thing they are going to notice is if the counters are clean. If an artist walks into the same kitchen they are going to notice what colour the walls are and if the paintings are crooked.

That is from this week’s Writing Excuses (by Mary Robinette Kowal specifically).

Bringing it back to testing.

If a scripter tests a product, the first thing they are going to notice are the hooks that make automated checks easier and more effective. If a victim of identity theft tests the same product they are going to notice the security and privacy features.

Posted on September 21, 2009 in Uncategorized by adam7 Comments »

Much like in software, when you release a book into the hands of the masses it is largely out of your control. Part of the lack-of-control with a book is the review process where you send out free copies to people who have a platform in hopes they give it a nice review (thus garnering more interest and sales). I’ve been doing this the last week for Beautiful Testing and I get books sent to me periodically. James Whittaker’s Exploratory Software Testing is one such book that landed in my mailbox, but the review is not going to be positive.

Lack of control sucks.

I really wanted to like this book. Exploratory testing is an extremely powerful technique which is largely passed from person-to-person rather than through books. This lack of textbook presence keeps it out of University curriculums. But when reading it, the only word I kept thinking of was cheat.

At its core, Exploratory Software Testing is about the useful metaphor of a tester as a tourist and the software their destination. An application, like a city, can be broken down into a number of districts (business, historical, tourist, entertainment, hotel and seedy) and for each district a number of ‘tours’ can be had (guidebook, money, landmark, intellectual, fedex, after-hours and garbage collectors for the business district for example). This metaphor is fantastically developed and is quite useful in developing a testing strategy.

But is that metaphor enough to justify a book? No. Not really. I wouldn’t dream of pitching SLIME as an entire book though I think it is just as useful to keep in mind as being a tourist. More on that in a second.

But first…

I took James Bach’s Rapid Software Testing August of 2005 (during hurricane Katrina) and during it there was a discussion of ‘tours’ you can take through an application. Specifically Mike Kelly’s FCC CUTS VIDS was mentioned. Mr. Whittaker joined Microsoft in 2006 and started to develop the idea of tours of there (on the soccer pitch according to Alan Page who did the Foreword) which ultimately culminated in this book. Now, there is nothing wrong with taking an idea and evolving it. We’ve all done it. But when you do that, you are obligated to say that is what you did and to cite the sources. The lack of background material in Exploratory Testing is a concern. Not only is there no mention of people who did work on the Testing-as-Touring idea, but there is no mention of people who are known thought leaders in exploratory testing. You mean to say you can write a book on exploratory testing without mentioning James or Jon Bach, Cem Kaner or Michael Bolton? Really? What does that do to your credibility on the subject?

The lack of citation is actually not my biggest criticism of Exploratory Testing. Like I said, I think the metaphor is wonderful and a useful addition of testing lore. My biggest complaint is that the book even got published.

Exploratory Software Testing is 224 pages in length and if we operate on the premise that the tour metaphor is the main focus of the book then it should be the majority of the content. Right? Well, James’ work on it lasts for 39 pages (chapters 4 and 5). Total. In fact, the original work ends on page 143 at which point he reprints a bunch of StickyMinds articles and his MSDN blog for the remainder. Even that content before the reprints isn’t all his. The chapter after the tour metaphor is finished is mini-essays (34 pages) by people inside Microsoft who have used tours successfully on their project.

In short, there isn’t enough content to warrant publishing a book that retails for $39.99 USD ($47.99 CDN). Take the tour metaphor one more step and bind it as a pocket tourist guide that sells for $15 and you have a winner. But for this price and packaging, unless you have an unlimited book budget, I can’t recommend that you pick it up. Eventually someone will post a description of each district and its tour and a few minutes of google-fu will give you the same value and still have enough money in your pocket for a week’s work the lunches.

Having been through the book process once this feels a lot like the publisher saw that they could sell a bunch of books based on the author having had successful books in the past and a title that is somewhat misleading but popular in their target audience’s lexicon rather than producing something of lasting value.

Here is another thing from the publishing world; they really don’t like publishing a book with the same(-ish) title as an existing one. Especially in the same subject matter space. It is a shame that this book is what will forever carry the mantle for exploratory testing.

Posted on September 18, 2009 in Uncategorized by adamNo Comments »

For a (long) while the phrase Page Object has been floating around the periphery of knowledge, but it wasn’t until today that I actually looked it up and actually figured out what it is. And now that I have that knowledge, it seems to me that there are (at least) 3 different patterns for creating a (Selenium) script.

  1. Feature Flow – These are the classic type of scripts where you automate a flow through an application in a long series of steps.

    selenium.type(“Search”, “monkey”)
    selenium.click(“sButton”)

    For additional maintainability and reuse, you will evolve things to include helper methods/fixtures. I would guess that the vast majority of scripts are of this type.
  2. Page Object – If Feature Flow is ‘functional’ programming, then Page Objects are OO. The webdriver project (which is what is going to power Selenium coming version 2.0) has what appears to be the definitive reference on Page Objects. Essentially, each Page is an Object.

    myPage = searchPage()
    myPage.search(‘monkey’)

    with all the implementation details are tucked into the search method code.
  3. DSL – This method is an abstraction layer on top of the implementation details which typically makes the script more consumable by non-technical persons, but also to hide lots of details from technical persons that they don’t want to worry about either.

    go_to search_page
    search_for monkey

What I realized is that they are not really exclusive. You can use bot Feature Flows and Page Objects as the underlying logic for a DSL. And in fact, the implementation details that are encapsulated in the Page Object methods will look like Feature Flow code.

There has got to be other patterns also out there that I haven’t heard of. But what are they?

Posted on September 14, 2009 in Uncategorized by adam1 Comment »

Selenium currently has the lion-share of the open-source browser driving frameworks/systems. But if it is improve or even maintain that lead I think one thing that needs to happen is the formalization of both the project and the corporate entities that benefit from it.



I see a path to potential success looking a bit like the above venn diagram.

On the left is company A who uses Selenium as a core technology for their products and/or services. But they have also developed extension to the core product. Those extensions are their value-add; it is what separates that company from others and belongs to them. That I think should not be contributed back to the main project. Now, if they fix bugs in the server or refactor or otherwise improve the core then that should be contributed.

Same thing for the company B on the right who just happens to have two novel improvements that they keep as part of their corporate IP.

Area Se (C — you have no idea how funny I think that is) is the missing links right now. Companies A and B (and likely D – J) already exist. Area Se is something akin to The Selenium Foundation which would be a not-for-profit which holds the copyrights and IP for the main project. Their role would be to act as the central marketing position for member companies, manage the mailing lists, web site, etc. They would also, ultimately, be responsible for the project roadmap and managing releases. I think this organization already exists, but I’m not sure it really has taken the bit and started running.

This wouldn’t have to be a big organization; likely just a single person (Managing Director) initially, voted into a 2-year term by board members (initially I see such a board being made of representatives from Thoughtworks, Google, Sauce Labs, Browsermob and the community at large) and a big enough budget salary-wise to let someone dedicate 100% of their time on the task.

I had thought Watir had pulled off this bit of re-organization with WatirCraft, but with its shuttering recently it looks like that momentum was lost.

Neither of A or B are precluded from offering their smaller bubbles back in into the the core projects (IDE, RC, Grid) at some point in the future if they desired. But at the same time, the Foundation wouldn’t be obligated to accept them if they were too specific to a certain problem.

Of course, the venn diagram starts to break down pretty fast once you get more than two participants, but I think the model in-general resembles something close enough to the Debian / Ubuntu one that it still will scale out. The devil is of course in the details and the political will must exist among the existing vendors in order to make it happen.

Next Page »