(I’m blatantly repackaging this from Joey)

Even if you’re not a jazz musician or even a jazz fan, you might still get some mileage from the advice that legendary jazz pianist Thelonious Monk gave to jazz saxophonist Steve Lacy. Lacy took down some notes…. Joey has pictures of the notes, but he typed them out. I know what my click-through rates are, so I’m reproducing his reproduction here.

• Just because you’re not a drummer, doesn’t mean you don’t have to keep time.
• Pat your foot and sing the melody in your head, when you play.
• Stop playing all those weird notes (that bullshit), play the melody!
• Make the drummer sound good.
• Discrimination is important.
• You’ve got to dig it to dig it, you dig?
• ALL REET!
• Always know….(MONK)
• It must be always night, otherwise they wouldn’t need the lights.
• Let’s lift the band stand!!
• I want to avoid the hecklers.
• Don’t play the piano part, I’m playing that. Don’t listen to me. I’m supposed to be accompanying you!
• The inside of the tune (the bridge) is the part that makes the outside sound good.
• Don’t play everything (or every time); let some things go by. Some music just imagined. What you don’t play can be more important that what you do.
• A note can be small as a pin or as big as the world, it depends on your imagination.
• Stay in shape! Sometimes a musician waits for a gig, and when it comes, he’s out of shape and can’t make it.
• When you’re swinging, swing some more.
• (What should we wear tonight? Sharp as possible!)
• Always leave them wanting more.
• Don’t sound anybody for a gig, just be on the scene. These pieces were written so as to have something to play and get cats interested enough to come to rehearsal.
• You’ve got it! If you don’t want to play, tell a joke or dance, but in any case, you got it! (To a drummer who didn’t want to solo)
• Whatever you think can’t be done, somebody will come along and do it. A genius is the one most like himself.
• They tried to get me to hate white people, but someone would always come along and spoil it.

Sure, some are kinda a stretch to apply to testing, but the spirit of most of them do pretty easily.

Contracts are a powerful tool in the battle of ‘WTF are my requirements’ as they are by definition a series of commitments between the two parties. Depending on the size of the organization and your role in it you might be able to see the contract, at which point you can test them. If the keeper of the contracts doesn’t want you to know the financial details, ask them to give you a redacted version. The money is not what we are interested in really, it is the clauses. Here is how I test a contract (off the top of my head while on the train):

Additions welcome.

I’ve come to realize that I have a tonne of content that is in powerpoint and not in the blog as entries which can be a pain if I can’t get to the slides (that are hosted on sourceforge), so I’m going to start recapping some of it. Here is the first one…

First, let me start by clarifying why rules in in quotes. These rules are very much heuristic in nature and should not be taken as gospel; it is quite easy to think of scenarios when they would be inappropriate to follow. They are merely guidelines that have I have successfully applied when thinking up test cases ahead of time. They also help when thinking about whether something will be automated or not.

1. One, and only One thing – This rule is states that a test case should only be measuring one, and only one thing. Imagine you have a test case that does two things: A and B. If A passes, but B fails, does the test case pass? Or does it fail? It is more like ‘sorta’ which is not really a state you want to be in. Similarly, if A fails, but B passes, does the ends justify the means and it is a pass? Likely not. Likely, sorta, maybe, kinda are not things I like having to report.
2. Self-sufficient – Test cases should not be dependent on other test cases. Much like the previous rule, if you need 5 tests to pass before yours can even run, you are setting yourself up for a tangled web of test case dependencies. This doesn’t mean that your test case can’t have a number of preconditions as part of the test setup, but those are included in the test case. And these setup steps do not have to be overly prescriptive. ‘Create a valid user’ is enough detail for a lot of tests. An extra benefit of this is that tests can be run in a random order which is a good check for whether they can be run in parallel if/when they are automated.
3. Black or White – Black and white is a nice place to be. The success (or fail) criteria of a test should be very clear. Again, fuzzy results are not fun. (Of course, the definition of what is a pass or a fail is contextual and should be specified somewhere.)
4. Valid test data – All data has rules around it, though they might not be obvious at first. Explicit rules are things like the column definition in the table and things like password strength rules. This only half the problem, and the less important half at that. They other half deals with the implicit rules that you might not be able to get without questioning the actual users. I learned this one first hand while in the Bahamas on my first business trip for my first job. I was testing central bank reporting software for the offshore Trust subsidiary of a major Canadian bank and using numbers like 5, 10, 25, 100 as inputs. Technically they met the explicit rules and were really easy to do math on without a calculator but when the head of IT for the company saw what I was doing he hit the roof. To open an account there, clients were plunking in most cases over $1 000 000. My $5 test was not exercising the application in any sort of valid way. I added a bunch of zeros to my input numbers and redid all the tests. Lesson learned.
5. Repeatable – Tests should be constructed in such a way as to be repeatable, both immediately and at some point in the future. This often means that you reset the environment to its prior state whether through removing entries in the database or simply clearing your cookies.
6. Write for the the correct audience – Do not write your tests for 3rd graders unless the people executing your tests are 3rd graders. Odds are testers doing the tests know the system, so you don’t have to explain how to use the system. ‘Setup a password server’, ‘Create a new question’ or ‘Login as a user with administrative rights’ might not mean anything to you right now, but you are not in the situation for it to. If you were, then it would.
7. Brain on? – The best testing is done with the tester’s brain firmly locked in the ‘on’ position. This means not boxing them in with things like ‘Start the application. In the firstname field input ‘Adam’ and in the lastname field input ‘Goucher’…’. This was the old school way of designing test cases and too often employed by teams who thing poorly of their testers. The only excuse for tests of the brain ‘off’ sort is when there was a specific triggering condition, then you should have the test case specify that specific condition.

I spoke at the University of Toronto this week for an hour to about 40 of Greg Wilson’s third year students. It could have used more (any) rehearsal and as usual I had a major case of the nerves but overall I think it was a success and I got to the points I wanted. This is the mini-essay version of that talk (primarily so it can be sent to the students)

Being Picky about Terminology

Even though the introduction used the term QA a half dozen times, what I do and what I talked about is actually testing. Why? Well, let’s look at the term QA a bit.

• Quality – The definition I currently use is ‘Value to some person that matters’. For more build-up on this concept see here and especially here.
• Assurance – Most people who do QA cannot actually achieve or influence this part of the term. Can you actually assure the quality of the product? Do you make staffing decisions? How about schedule ones? Do you have final say on the release metrics and which bugs get fixed? Odds are you don’t and won’t. So you aren’t really assuring much.

Okay then. What is testing then? Well, the AST has defined it as

• An empirical technical investigation conducted to provide stakeholders with quality related information
• A cognitively complex activity that requires critical thinking, effective communication and rapid self-directed learning

(If memory servers, Cem Kaner heavily influenced that definition)

The key words in the first bullet are provide quality related information. That is primary motivator for all testing activities. Notice how neither point says anything remotely like ‘improve the quality’. The Quality is actually your (the programmer) responsibility. As Michael Bolton has been saying recently, testers are there to defend the Quality of the code.

Oracles

Whether you are a tester or a programmer, you need to know what your oracle is when you are testing. This is really an unfortunate term given the size of Larry Ellison’s company, but this is not Oracle as in the database, but Oracle as in the Oracle of Delphi (who could talk to the Gods but was really stoned on sulphur fumes). An oracle in testing terms is the principle or mechanism by which you recognize a problem (BBST). This could be a person, another application, your professor, the TA, or even internal.

Ask Questions

A large part of testing can be summarized as ‘asking questions of the application’. Ask a lot of different questions. If you ask the same questions all the time, you won’t learn anything new.

But don’t limit yourself to asking questions only of the product itself. Ask about the things circling the product. Such as the requirements. The requirements for the assignments you have received certainly have enough information to do something with (both create and test). But really, the requirements kinda suck. But that is okay. Requirements in the real world do generally suck. And they likely always will be. The world is a dynamic place which is undergoing constant change. As such, the requirements will change constantly too.

Your job though is to question every line of them though. Ask the obvious question, you might not get the obvious answer. Ask the crazy questions, that is often enlightening. Ask why we are bothering with this feature, that will help clarify your mission.

Mission

Knowing your mission is important to testing. What information do the people you are testing for asking you to find? Doing security testing might be fun, but if your employer is wanting performance information then you are off mission. It doesn’t mean you can’t do security checks, but make sure you have given them the performance data they were asking for first.

Complete Coverage

Complete coverage is technically possible if you are doing TDD in its purest form. However, that number is often abused. Yes, you have touched 100% of the lines of code, but you have not touched 100% of the code with all possible values and in all possible contexts. This is part of the ‘Impossibility of Complete Testing’ which is also covered in the BBST course linked to above. Anyone who tells you otherwise is lying or just confused.

This doesn’t mean there is not value in coverage tools, or even aiming for 100% coverage. They are a useful class of tools to tell you where you have under tested and also lets you determine when your coverage has change (for the better or worser) when then lets you question whether that can be lived with or not. (There is that questioning concept again…)

Code Review

Ask anyone who has written a book about the value of an editor. Just as writers fill in the blanks between what they meant to write and what they actually did write, programmers will do this with their code. While having every line of your code reviewed by a peer is a bit overkill, it is certainly valuable for anything of any sort of complexity. Learn how to review code.

Random Bits

• Don’t fall into the trap of thinking ‘No one will ever do that…’. No one will ever do that who is acting in a sane manner who does not think they are clever or are trying to find holes and/or problems with the system and are are only ever using it in the manner it was envisioned. Well, the real world is messy. Someone is almost guaranteed to do that at some point. Hopefully you won’t look too dumb when they do.
• You are nothing like the the person who will be using the code you produce. Well, most of the majority of the time at any rate. Think of the person who calls you to ‘fix their computer’. If they could use you application, then you are doing pretty good.
• Bugs tend to cluster. When you find one in a piece of code, spend some time to poke around and see what else is lurking.

That was more-or-less the hour except some discussion are possible tests for a term assignment. I likely should have written this before the class shouldn’t I. 🙂

I’ve been neglecting my 5 questions with… folder recently. Here is the catch-up; with commentary. Of course.

Amol Kher

• Models are what developers will hate to love or love to hate – Or both?

Ethan John

• I had no idea that [testing] was something I could do for a living. – This is such an important problem for the testing community. One which I have no idea how to fix.

Michael Corning

• The first thing a tester needs to know about testing is that you cannot test what you do not understand – Oh I so don’t agree. Maybe if you tacked on a ‘or does not want to understand’ I could go along with the statement. As it is now it is equivalent to saying that all programmers need to know Assembly. Heck, a lot of programmers nowadays don’t even know C and the ‘joy’ that is pointers and manual memory management.
• In my mind, the tester is an auditor. The function of the auditor is to attest. In financial auditing, the auditor attests with his or her signature that the financial results fairly represent the financial results of business operations. From this attestation, other decision makers make their decisions. Both the auditor and the tester provide information to the eco-system. – Attestation implies a certain level of guarantee. Again, to cite the diagram in the BBST Foundations material, there are so many factors that attesting to the quality is a trap — at best. An opinion, sure. But attestation? Nope.
• I do my best to use the machine to generate test cases for me. – All hail the machine. This to me is the point you want to evolve your automation towards

Keith Braithwaite

• Write tests to capture requirements. Count tests written to measure scope. Count tests passing to measure delivery. Use tests to drive design. Use tests to communicate between teams, with customers. Use tests to meet regulatory loads and show compliance without mountains of paper. Oh, and to find defects too. But that last one is just the cherry on the cake. – Count tests to get a useless number; I can write a million tests that provide useless information but still shows 7 figures in the count. Use tests to confuse the customer; maybe he deals with different people in different organizations, but the people on conference calls I have been on would go glassy-eyed in under a minute if presented with tests. Which isn’t bad as they don’t need to know how to read the tests. This applies too to requirements. I’m in this situation now, and it is not enjoyable in the least. Capture the requirements in a document, or series of documents and make sure they evolve with the application. Or better still, put them in a wiki.

• Gil Broza
  • Think-once-then-automate

Once again thanks to Dan Ariely’ blog we have the concept of ‘the identifiable victim effect’ (which amazingly doesn’t seem to have a wikipedia entry). He describes this as the effect of one individual, identifiable, victim who is known in full detail can evoke a much deeper feelings, emotions and sympathy than a large group of anonymous individuals.

This has potentially huge ramifications for when we write bug reports. I’ve been saying for awhile that a large part of what we do is marketing (and thus the increase of marketing-esque posts). And I’m not alone on this. The slides for the BBST Bug Advocacy section includes a section on ‘motivating the bug fix’. Here is a line from slide 21 of the course notes:

You may or may not have found the best example of a failure that can be caused by the underlying fault.

I’m pretty sure in this case they mean example to be ‘a reproducible thing you could call a bug’, but example could also be read as ‘a story which conveys the nature or intent of the bug report’. In other words, have you started writing a report which is written in very fuzzy, vague ways or one that is specific to a person (or an identifiable group of persons)?

More on the identifiable victim effect can be found in:

• Running Head: Learning about the Identifiable Victim Effect by Deborah A. Small, George Loewenstein and Paul Slovic
• Crisis Mentality – Why sudden emergencies attract more funds than do chronic conditions, and how nonprofits can change that by Keith Epstein

(both are big articles and I didn’t actually read them, but they were referenced a couple times when looking up the topic)

I first became aware of the guys behind Power of 2 Games from a post one of them made comparing the speed of various c++ compilers; speed being important in the feedback loop that CI provides you. Their blog is pretty low volume and is generally aimed at the c++ and game worlds but is generally pretty high quality when they get around to making a post.

One of them from this month was a reprint from a game industry magazine in which they talk about using a playback system to aid in their testing. The second part of the article deals mainly with the problems that can trip up your playback system and I think apply even to those who are not using playback techniques for games. Here are the choice bits (with my commentary):

• Middleware libraries will often have slightly different behaviors in debug and release mode – Lower environments almost always run with debug symbols, etc. installed. Production rarely does. Think about how, where and why your scripts will be run before trying to get value from them.
• As a general rule, treat debug and release builds as if they were different versions
• Cranking up the compiler warning level to the maximum will allow it to catch some of the most obvious cases of reading uninitialized variables. For extra checks, I recommend a static code analysis program such as PC Lint. That’s almost guaranteed to catch every use of uninitialized variables (along with ten thousand other things, some of them extremely useful, though most of them you probably won’t care about at all). – Be sure to spend the time to tune what the tool checks for to get rid of the ten thousand things fast or people will ignore the list quickly
• … it’s considered a good practice to avoid using pointer values for anything other than dereferencing them and accessing the data they’re pointing to
• Data received from the network most definitely influences the game itself, so it’s another input to the game. – I am reminded of a graphic in the BBST course (slide 17) which can be attributed back to Doug Hoffman.
• ReplayDIRECTOR – appears to be a record/playback tool on some serious steroids

I get a copy of Business Week from my father-in-law after he is done with it. The back page of each issue is by Jack and Suzy Welch where they answer a reader’s question. This week’s was what big business imperative doesn’t get enough attention. Their answer was stickiness.

• Quality doesn’t get talk about as much as it used to, but that’s because it is a given.
• What is surprising, however, is how exceptional and inventive customer service needs to be to stand out these days
• … but another approach to stickiness can pack an even bigger wallop. It requires moving … to a product-and-long-term-service-business by guaranteeing productivity gains to customers
• Almost all companies can create stickiness by sharing knowhow.
• … unbridled sharing of expertise creates a real affinity …
• … many businesses can create stickiness by building user communities. Every year … hosts conferences. These aren’t sales events per se, but you can be sure that attendees leave with a feeling of partnership …
• Organizers must come to see the world through customers’ eyes

The first point is a biggie. Now they were talking about manufacturing (I think) but it applies to software as well. Especially when there are viable competitors in the space. Quality should be a given.

The other points are also interesting from a larger, more strategic perspective. How can you share expertise and knowhow? Open up your bug tracker. Or have someone from the QA or test groups contribute to the corporate blog about how they work to protect the customer’s interests.

Which leads to the last point. Running out of ideas to test with? Stop looking at the application with all your insider knowledge. Look at it instead as a user would, without all the tricks and secret switches you have picked up along the way.

Here is a link to the full article to read it in the proper context.

I have been looking for a Ruby/Rails equivalent of checkstyle since I started working with it, but have only found a single orphaned project. Until yesterday that is. Yesterday’s Double Shot had a link to the Roodi project. Roodi stands for ‘Ruby Object Oriented Design Inferometer’, but essentially it is checkstyle for Ruby.

The Java community has a rich set of guidelines to build static checks from, but the Ruby community seems to lack that cohesion. That is going to make for an interesting collection of checks over time and no doubt some discussion over what should be and checked and how. Here are the checks it currently does:

• AssignmentInConditionalCheck – Check for an assignment inside a conditional. It‘s probably a mistaken equality comparison.
• CaseMissingElseCheck – Check that case statements have an else statement so that all cases are covered.
• ClassLineCountCheck – Check that the number of lines in a class is below the threshold.
• ClassNameCheck – Check that class names match convention.
• CyclomaticComplexityBlockCheck – Check that the cyclomatic complexity of all blocks is below the threshold.
• CyclomaticComplexityMethodCheck – Check that the cyclomatic complexity of all methods is below the threshold.
• EmptyRescueBodyCheck – Check that there are no empty rescue blocks.
• ForLoopCheck – Check that for loops aren‘t used (Use Enumerable.each instead)
• MethodLineCountCheck – Check that the number of lines in a method is below the threshold.
• MethodNameCheck – Check that method names match convention.
• ModuleLineCountCheck – Check that the number of lines in a module is below the threshold.
• ModuleNameCheck – Check that module names match convention.
• ParameterNumberCheck – Check that the number of parameters on a method is below the threshold.

Thats all well and good, but a tool that is not run is just a clever piece of code. And relying on people to remember to run a tool is not a plan I would bet the house on. But that is why we have things like CruiseControl.

Here is a pretty heavily edited version of our Rakefile.

desc "Cruise Control"
namespace :cruise do
  output_dir = ENV["CC_BUILD_ARTIFACTS"]

  task :build do
    ...
    CruiseControl::invoke_rake_task 'cruise:roodi'
  end
  
  desc "Run the roodi tests"
  task :roodi do
    `roodi -config="#{RAILS_ROOT}/config/roodi.yml" "#{RAILS_ROOT}/app/**/*.rb" > #{output_dir}/roodi.txt`
  end
end

There is not (yet) a roodi_on_rails plugin, though I expect it is only a matter of time, so I just run the tool on the commandline and put the output in the place CruiseControl needs it to display it on the report page. The output is, to say the least, un-pretty but if the community backs this tool I can see it growing nice rspec like reports with links to the actual offending line.

Oh, and here is config/roodi.yml, which is just the default configs for all the rules, but explicit is better than implicit.

AssignmentInConditionalCheck:    { }
CaseMissingElseCheck:            { }
ClassLineCountCheck:             { line_count: 300 }
ClassNameCheck:                  { pattern: !ruby/regexp /^[A-Z][a-zA-Z0-9]*$/ }
CyclomaticComplexityBlockCheck:  { complexity: 4 }
CyclomaticComplexityMethodCheck: { complexity: 8 }
EmptyRescueBodyCheck:            { }
ForLoopCheck:                    { }
MethodLineCountCheck:            { line_count: 20 }
MethodNameCheck:                 { pattern: !ruby/regexp /^[_a-z<>=\[\]|+-\/\*`]+[_a-z0-9_<>=~@\[\]]*[=!\?]?$/ }
ModuleLineCountCheck:            { line_count: 300 }
ModuleNameCheck:                 { pattern: !ruby/regexp /^[A-Z][a-zA-Z0-9]*$/ }
ParameterNumberCheck:            { parameter_count: 5 }

I have been churning on the idea of ‘tests as conversations’ for the last week or so. One part of this involved the skills necessary for the conversation to happen. Specifically whether knowing math and concluded that no, you do not needed to know math. Except when you do.

Huh? That’s pretty have-it-both-ways even for me.

Math is a ridiculously large field of study, so it is near impossible to say that you do not need any math, but the slice of math I think testers need to know is pretty slim.

• Basic statistics (mean, median, mode, standard deviation, sample size, etc.)
• Combinations
• Permutations

Yup. 3 things, though I’ll concede that the first one is a decent sized bucket. In over 10 years of testing I have not had to employ any math outside of those areas. Even when testing a banking application I did not have to flex my math brain as the bankers created oracles for us to verify against.

Of course, this is heavily dependent on my own experiences. I have never tested an application that is heavy in its math, like a missile guidance system for instance. I would likely argue that at that point what you want is not someone who is conversant in math, but someone who is deeply fluent in it.

I have been out of high school 12 (or 13 years; I can’t remember) which was the last time I had to do any serious math. I can still do long division, but I had to teach it to my daughter when she was in public school. Calculating the square root of something? Not a chance.

And that is ok. In no way does this lack of knowledge impact my ability to have the conversations I need to.