Posted on August 30, 2007 in Quality by adamNo Comments »

From an article on Caterpillar in the August 20, 2007 Fortune magazine: …automation alone is not a substitute for disciplined processes and workflows.

Seems to be pretty applicable to other domains than just manufacturing, including testing.

Witht that I’ll go back to writing selenium based functional tests against a product which does not have any unit tests that run outside of the IDE, has no notion of seed data, a monster-in-the-closet of a build system, routinely has database changes without notification… Not that I don’t think this is completely back-asswards, but one of the chief reasons for automation I list in my intro to qa course is because the boss says to.

Posted on August 28, 2007 in Quality by adamNo Comments »

I’ll be your mirror, reflect what you are – The Velvet Underground

Anu Arora is Michael Hunter’s Five Questions With… interview this week. These are really great as it gives different testers’ perspectives to the same questions (even if it is weighted more to MS employees, but that can likely be attributed to his employment there). One thing in particular jumped out at me in Anu’s response. Right at the end she says

I started looking at my job as more then – I am a mere tester, my job is to just show the mirror. I became a quality advocate.

This is the first time I have seen a tester refer to their job as being a mirror, but it is quite appropriate. I believe, contrary to some, that Quality is primarily the responsibility of the developer. If I find a bug, then there was a breakdown in the process somewhere upstream. Would bigger / stronger / faster unit tests have found this? Would having clearer requirements have prevented this? Would static analysis or source code inspection have discovered it?

I suspect this is the part where we start splitting hairs over the definition of someone who does Testing and who does QA. But ignoring that, when you act as a mirror you say things like “I found a couple of these types of bugs this release, if we do X, I think we might be able to catch those before we go through the hassle of an install, etc” or “I notice that your path coverage has decreased, how can I help to get it back up?”. Of course, if you look in the mirror and only see ugliness, you are going to stop looking in the mirror. This is what happens when testers are constantly running with their heads cut off and being abusive to their development team.

That is why it is also important to reflect back the good things your development team does. “Wow, it was hard to find new bugs in that last build” or “Your coverage jumped 15% and your cyclomatic complexity dropped 5. Way to go!”.

Posted on August 28, 2007 in Quality by adamNo Comments »

Through the joys of RSS I keep track of the QA/Test jobs in my area (the theory is that companies I deem as ‘cooler’ will be posting there vs. other places — or at least in addition to). Adding onto Guy Kawasaki’s How to get a job on Craigslist and How to not hire someone via Craigslist I’ll add a couple tips of my own.

Title your job appropriately – There is a post currently advertising for ‘Rational Testers’. Now, I’m going to go out on a limb and guess that this means that you are skilled in the IBM Rational toolset, but that is not explicitly called out so it might mean that they want someone that is agreeable to reason. If it is the former, then I guess I can hence forth label myself as an irrational tester since I have no experience there.

Spelling / Grammar counts – There is also an ad in the services section promoting someone who will do testing for companies. Their ad is riddled with spelling and grammatical mistakes. I’m certainly not perfect in either of those regards, but you can be darn sure that if I was posting an ad about how well I’m going to find issues in your product that I’m going to make sure it is grammatically correct. Yes, I know, not everyone has English as their first (or second, or third) language, but Toronto is a predominantly English market so that is the context you are stuck with.

Don’t raise flags – The first thing I do when I am evaluating a person or organization is spend 5 minutes digging everything I can up on them in Google. The person offering the testing services lists himself on other testing boards as being in India on various testing boards and in fact is currently looking for an apartment in Bangalore. If I am looking for services on Craigslist, odds are I expect them to be local; especially when they are flagged as being ‘Toronto’. The Testing market is pretty tight right now (at least here; finding available people with a clue is hard and I have a couple companies sniffing around me now) so if your ad or my searching raises even a yellow flag I’m going to pass you over and move onto the next candidate / company.

Posted on August 28, 2007 in Quality, Video by adamNo Comments »

Here are links to the lightning talks from GTAC2007 (or at least the ones that landed in my RSS box). It looks like if users went over their time limit they got pelted with rubber balls…

Posted on August 28, 2007 in Housekeeping by adamNo Comments »

I’m not sure if this is a good idea or not, but what the heck.

If you are on MSN and feel like adding me to the list of people you can contact / be contacted, my MSN address is Just put in the message that you saw this post so I know you are not trying to add this imposter as they do on Facebook.

Posted on August 27, 2007 in Quality by adam2 Comments »

Sometimes, it is nice to have a full script to check for something across your codebase. For instance, when you are on windows, or you are in a toolsmith role and have a team of non technical people to support. Other times though, you can just revel in the power of unix.

adam@feisty$ find . -name "*.pm" -exec grep -i -f dev_patterns.txt -H -n {} \\; > dev_notes.txt

does more or less the same thing as my developer notes script.

As a bit of explanation, what this does is…

  • Search (find)
  • Starting in the current directory (.)
  • every perl module (*.pm)
  • Once found, do a case insensitive search (-i)
  • For each of the patterns in dev_patterns.txt (-f)
  • And display the file name (-H)
  • And line number (-n)
  • And pump the output to a file (>dev_notes.txt)

Depending on your variant of unix, your flags might vary.

Posted on August 27, 2007 in Interviews by adam1 Comment »

I have been thinking again about the process of interviewing from both a the interviewer and interviewee.

I was talking to someone the other day about a pending interview they were going to have and I mentioned that they should treat it as a chance to decide whether they wanted to work for the company and less about whether the company would hire them. The theory being that if you got through the resume filters they are pretty sure they would tolerate having you work for them and are just doing a check to see the organizational fit and whether you are trying to snow job them on your resume. I don’t remember the questions I suggested, but here is a slightly more thought out list.

  • Pay frequency
  • When if first pay cheque
  • Is there a career ladder
  • What is the employee turnover
  • What is the burn rate
  • Education / training budget
  • What are crunches like
  • Average amount of overtime
  • Ability to work from home
  • Is home internet connection subsidized
  • Company leash (cell, pager)
  • Measurable objectives
  • Bonus calculation
  • Eligible for current year’s bonus
  • Vacation carryover rules
  • Benefits availability
  • Benefit eligibility period
  • Travel requirements
  • Expense policy
  • Corporate CC
  • How projects are tracked
  • What bug system
  • How is time tracked
  • General time policies
  • What a typical release looks like
  • Interesting projects in the 6 – 12 month timeframe
  • Composition of group you are interviewing for
  • Office hours
  • Dress code
  • Guiding philosophies of organization
  • Resistance of company to change
  • Role of self in company now, 1 month, 6 month, 12 month

Most, if not all of the above are all about me style questions. Just because you get an offer does not mean you have to take it if the answers to the above do not meet your expectations. If your list is not well developed you might find yourself in a place that wants you, but you don’t want to be.

What’s on your list?

Posted on August 25, 2007 in Quality, Video by adamNo Comments »

When I was at HP, I’d sometimes be able to core dump our application. Of course, when you are logging the corresponding bug the developers really like when you put in the stack trace to help them isolate the problem and the way we would do that is through gdb. This gave me a peek at what the system was doing. Giving me peeks is dangerous. I eventually figured out how to attach to the running process and see what the system is doing.

Now, depending on where you draw the boundary of responsibility of the tester, knowing how to use the debugger might be part of their job description or might just be a fun, geeky thing to do.

Bryan Cantrell works for Sun and appears to the grand high wizard of DTrace. (He is also a Beautiful Code contributor.) DTrace appears to be a debugger for running processes, but on steroids. I was hoping that this would be a DTrace overview / howto, but instead it appears to Bryan showing off his favorite toy. At first I wasn’t impressed, but it ends up being really entertaining with little bits of geek history and insight into the guts of solaris which makes it worth watching even if you are not going to learn DTrace from it. (Oh ya, and he types really well; yes, I have typing envy) Here are the non-DTrace notes I took

  • Use prstat which is better than top on solaris (and might explain why top is not installed on solaris by default)
  • We’ve gotten sloppy in our coding practices because system resources are so cheap and plentiful. Bryan uses the phrase embarrassment of riches
  • In the server room, you do work when there is work to do. The end. Desktop apps tend to break this (see previous point)
  • A debugging technique: when a tool (mpstat, top, prstat) produces a columnar output, work on getting the the columns all nicely lined up. Once you get the system in that state, work on the original problem.
  • Anonymous memory is ‘cheap’, but performance is not a good reason to use it. malloc is more effective in this context
  • There is even instrumentation for python

Direct link here.

Posted on August 25, 2007 in Quality, Video by adamNo Comments »

Patrick Copeland is the worldwide director of test engineering at Google and was the opening keynote at this year. In it he discusses the culture inside of Google in respect to testing, which given that he is the one responsible for it is eminently qualified to discuss.

  • The testing culture in based on 3 components
    1. People – Give them tools, not answers
    2. Science – Use technology to solve problems
    3. Improve – Measure everything
  • They spend a lot of effort evangelize to themselves about testing
  • The bulk of Google’s testing is developer testing
  • Even though they organizationally hate things like CMMi, RUP, Agile from a constraining perspective (choosing instead to borrow bits and pieces from each as appropriate), they have developed their own such process; the Test Certified Ladder
    • level 1: setup tools
    • level 2: learn techniques and technologies that enable team to write tests for all new code
    • level 3: improve suite and get all code tested
  • Test Mercenaries (personally I think this is the coolest group inside Google, but my coding is definitely not up to the challenge of this group after this slide and from when I was talking to their recruiter
    • Introduce technologies
    • Refactor code
    • Train individuals on testing and techniques (This I could do)
    • Test mercenary involvement is time limited engagement (1 or 2 quarters)
  • They have a shared testing vocabulary throughout company
  • The only exhaustive testing there is, is so much testing that the tester is exhausted! – Bill Hetzel
  • The role of test engineering
    • Firmly part of engineering
    • Involved and complementary
    • Improving products and systems
    • Uses science to solve problems
    • Rewards big risks
  • Constantly as yourself ‘Where can we help?’
  • Make the default assumed state be one of failure, not one where the system is working nicely
  • Use descriptive errors
  • Inject problems (have a switch which makes the app think the network or disk has flown south) and monitor resolutions
  • The whole team thinks that quality is important; quality is owned by the product team, not just by testing
  • Don’t think only about bits and pieces of the application, think about it as a system
  • Identify and fix the root causes (root cause analysis)
  • swiss cheese model
  • Try to run tests in parallel
  • Security testing happens way up front in the design phase (which I also recommend)

Direct link is here.

Posted on August 24, 2007 in Quality, Video by adamNo Comments »

The videos from GTAC (Google Test Automation Conference) are starting to appear. Follow the link to find them.

Or if you want to track them via RSS, put this into your feed reader. Unfortunately they are all YouTube videos which means that they don’t come down the pipe as enclosures, but it’s still nice to have things being discovered for you.

Seems there is now a playlist too.

Next Page »