Posted on September 29, 2008 in Quality by adamNo Comments »

I’ve been neglecting my 5 questions with… folder recently. Here is the catch-up; with commentary. Of course.

Amol Kher

  • Models are what developers will hate to love or love to hate – Or both?

Ethan John

  • I had no idea that [testing] was something I could do for a living. – This is such an important problem for the testing community. One which I have no idea how to fix.

Michael Corning

  • The first thing a tester needs to know about testing is that you cannot test what you do not understand – Oh I so don’t agree. Maybe if you tacked on a ‘or does not want to understand’ I could go along with the statement. As it is now it is equivalent to saying that all programmers need to know Assembly. Heck, a lot of programmers nowadays don’t even know C and the ‘joy’ that is pointers and manual memory management.
  • In my mind, the tester is an auditor. The function of the auditor is to attest. In financial auditing, the auditor attests with his or her signature that the financial results fairly represent the financial results of business operations. From this attestation, other decision makers make their decisions. Both the auditor and the tester provide information to the eco-system. – Attestation implies a certain level of guarantee. Again, to cite the diagram in the BBST Foundations material, there are so many factors that attesting to the quality is a trap — at best. An opinion, sure. But attestation? Nope.
  • I do my best to use the machine to generate test cases for me. – All hail the machine. This to me is the point you want to evolve your automation towards

Keith Braithwaite

  • Write tests to capture requirements. Count tests written to measure scope. Count tests passing to measure delivery. Use tests to drive design. Use tests to communicate between teams, with customers. Use tests to meet regulatory loads and show compliance without mountains of paper. Oh, and to find defects too. But that last one is just the cherry on the cake. – Count tests to get a useless number; I can write a million tests that provide useless information but still shows 7 figures in the count. Use tests to confuse the customer; maybe he deals with different people in different organizations, but the people on conference calls I have been on would go glassy-eyed in under a minute if presented with tests. Which isn’t bad as they don’t need to know how to read the tests. This applies too to requirements. I’m in this situation now, and it is not enjoyable in the least. Capture the requirements in a document, or series of documents and make sure they evolve with the application. Or better still, put them in a wiki.

Posted on September 29, 2008 in marketing, Quality by adam1 Comment »

Once again thanks to Dan Ariely’ blog we have the concept of ‘the identifiable victim effect’ (which amazingly doesn’t seem to have a wikipedia entry). He describes this as the effect of one individual, identifiable, victim who is known in full detail can evoke a much deeper feelings, emotions and sympathy than a large group of anonymous individuals.

This has potentially huge ramifications for when we write bug reports. I’ve been saying for awhile that a large part of what we do is marketing (and thus the increase of marketing-esque posts). And I’m not alone on this. The slides for the BBST Bug Advocacy section includes a section on ‘motivating the bug fix’. Here is a line from slide 21 of the course notes:

You may or may not have found the best example of a failure that can be caused by the underlying fault.

I’m pretty sure in this case they mean example to be ‘a reproducible thing you could call a bug’, but example could also be read as ‘a story which conveys the nature or intent of the bug report’. In other words, have you started writing a report which is written in very fuzzy, vague ways or one that is specific to a person (or an identifiable group of persons)?

More on the identifiable victim effect can be found in:

(both are big articles and I didn’t actually read them, but they were referenced a couple times when looking up the topic)

Posted on September 27, 2008 in Quality by adamNo Comments »

I first became aware of the guys behind Power of 2 Games from a post one of them made comparing the speed of various c++ compilers; speed being important in the feedback loop that CI provides you. Their blog is pretty low volume and is generally aimed at the c++ and game worlds but is generally pretty high quality when they get around to making a post.

One of them from this month was a reprint from a game industry magazine in which they talk about using a playback system to aid in their testing. The second part of the article deals mainly with the problems that can trip up your playback system and I think apply even to those who are not using playback techniques for games. Here are the choice bits (with my commentary):

  • Middleware libraries will often have slightly different behaviors in debug and release mode – Lower environments almost always run with debug symbols, etc. installed. Production rarely does. Think about how, where and why your scripts will be run before trying to get value from them.
  • As a general rule, treat debug and release builds as if they were different versions
  • Cranking up the compiler warning level to the maximum will allow it to catch some of the most obvious cases of reading uninitialized variables. For extra checks, I recommend a static code analysis program such as PC Lint. That’s almost guaranteed to catch every use of uninitialized variables (along with ten thousand other things, some of them extremely useful, though most of them you probably won’t care about at all). – Be sure to spend the time to tune what the tool checks for to get rid of the ten thousand things fast or people will ignore the list quickly
  • … it’s considered a good practice to avoid using pointer values for anything other than dereferencing them and accessing the data they’re pointing to
  • Data received from the network most definitely influences the game itself, so it’s another input to the game. – I am reminded of a graphic in the BBST course (slide 17) which can be attributed back to Doug Hoffman.
  • ReplayDIRECTORappears to be a record/playback tool on some serious steroids
Posted on September 26, 2008 in Uncategorized by adamNo Comments »

Here are a bunch of quotes from articles originally found via slashdot claiming ‘Studies Say Ideology Trumps Facts’. I think they go a long way in explaining the various political battles that go on in the testing world and how people seem to argue themselves into a corner in the face of (what appears to be) overwhelming facts to the contrary. Likely also explains the eternal optimism of product management in the face of a major bug backlog, etc..

ars technica

  • Cognitive dissonance won’t help people make rational decisions, but it also suggests that there’s little point in arguing with someone who holds an opposing belief.
  • The research might also apply beyond the political to other attitudes—I’m thinking of the constant flame wars between fans of the PS3 and Xbox 360, or Mac and PC users.
  • It seems to suggest that this effect might lead to problems when it comes to efforts to educate people about controversial or politically charged topics

Washington Post

  • … misinformation can exercise a ghostly influence on people’s minds after it has been debunked — even among people who recognize it as misinformation. In some cases, correcting misinformation serves to increase the power of bad information.
  • “backfire effect”
  • Upon hearing a refutation, … might “argue back” against the refutation in their minds, thereby strengthening their belief in the misinformation.

On The Media

  • … when people are predisposed to buy a piece of bad information, the refutation ends up essentially not correcting the misinformation entirely. It corrects it only part way, leaving residual feelings of negativity toward whoever the target of the misinformation was.
Posted on September 25, 2008 in Quality by adamNo Comments »

I get a copy of Business Week from my father-in-law after he is done with it. The back page of each issue is by Jack and Suzy Welch where they answer a reader’s question. This week’s was what big business imperative doesn’t get enough attention. Their answer was stickiness.

  • Quality doesn’t get talk about as much as it used to, but that’s because it is a given.
  • What is surprising, however, is how exceptional and inventive customer service needs to be to stand out these days
  • … but another approach to stickiness can pack an even bigger wallop. It requires moving … to a product-and-long-term-service-business by guaranteeing productivity gains to customers
  • Almost all companies can create stickiness by sharing knowhow.
  • … unbridled sharing of expertise creates a real affinity …
  • … many businesses can create stickiness by building user communities. Every year … hosts conferences. These aren’t sales events per se, but you can be sure that attendees leave with a feeling of partnership …
  • Organizers must come to see the world through customers’ eyes

The first point is a biggie. Now they were talking about manufacturing (I think) but it applies to software as well. Especially when there are viable competitors in the space. Quality should be a given.

The other points are also interesting from a larger, more strategic perspective. How can you share expertise and knowhow? Open up your bug tracker. Or have someone from the QA or test groups contribute to the corporate blog about how they work to protect the customer’s interests.

Which leads to the last point. Running out of ideas to test with? Stop looking at the application with all your insider knowledge. Look at it instead as a user would, without all the tricks and secret switches you have picked up along the way.

Here is a link to the full article to read it in the proper context.

Posted on September 24, 2008 in Ruby by adamNo Comments »

We have the pretty standard server configuration of a load balancer (apache) in front of two machines which serve the actual content. Through the process of trying to upgrade a machine I realized they were not configured to easily bring up or down on their own. In commercial production (Websphere and WebLogic for instance) you just go to the admin console and take a node out of the pool. Not so with our rig. Here is the solution I came up with. I think it is pretty elegant (though I am sure it is just reinventing the wheel).

  1. Directory structure – I created 2 directories in the apache config directory (/etc/apache2 in this case): nodes-available and nodes-enabled
  2. Creating the nodes – We had all our load balanced ports in a single file. Those got broken in to separate files depending on which machine they were on. (Lets call them zfp1 and zfp2 for this example.) These files get placed in the nodes-available directory.

    $ cat nodes-available/zfp1 
    BalancerMember http://x.y.z.a:8000
    BalancerMember http://x.y.z.a:8019
  3. Enabling the nodes – The nodes are at this point both disabled which is not really an ideal situation. To enable them we create a symlink from the nodes-enabled directory into the nodes-available one for each node. For example, nodes-enabled/zfp1 links to nodes-available/zfp1.

    $ ls -l nodes-enabled/
    total 0
    lrwxrwxrwx 1 root root 33 Sep 16 14:12 zfp1 -> /etc/apache2/nodes-available/zfp1
    lrwxrwxrwx 1 root root 33 Sep 16 14:36 zfp2 -> /etc/apache2/nodes-available/zfp2
  4. Use the new system – Until this point we were just playing with files in a way that was completely transparent to apache. To start using this you need to make use of the Include functionality of apache’s configuration. I changed the reference to the single file (that we split in step 2) to be Include nodes-enabled/zfp*. By using a wildcard we can add more nodes without having to do anything to the actual server configuration. It also means we can use the load balancer for multiple clusters by just having differently named files.

    <Proxy balancer://mongrel_cluster>
      Include nodes-enabled/zfp*
  5. Reload the config – Apache will happily run forever without re-reading its config file, so once you are comfortable with your new configuration, so you have to remember to tell it to reload.

All the above is still really only half the solution though. Actually, it is what makes controlling individual nodes possible, but it is a pretty manual process still. For this particular property we are using Vlad the Deployer to manage things, though I suspect you could use Capistrano or Puppet just as easily.

desc 'add a node to the server'
remote_task :add_node, :roles => :load_balancer do
  if ENV['node']
    run %{[ -f #{ apache_root }/nodes-available/#{ ENV['node'] } ] && [ ! -f #{ apache_root }/nodes-enabled/#{ ENV['node'] } ] && sudo ln -s #{ apache_root }/nodes-available/#{ ENV['node'] } #{ apache_root }/nodes-enabled/#{ ENV['node'] } && sudo /etc/init.d/apache2 reload || echo "Node does not exist"}
    p 'You need to specify a node to be able to add it. node=foo'

desc 'remove a node from the server'
remote_task :remove_node, :roles => :load_balancer do
  if ENV['node']
    run %{[ -f #{ apache_root }/nodes-enabled/#{ ENV['node'] } ] && sudo rm #{ apache_root }/nodes-enabled/#{ ENV['node'] } && sudo /etc/init.d/apache2 reload || echo "Node does not exist"}
    p 'You need to specify a node to be able to remove it. node=foo'

(Yes, I know that the error messages are not very nice, or even accurate in a lot of situations, but it does the job which is all I require of it right now.)

To enable a node you simply do a ‘rake lb:add_node node=zfpN’ where lb is the namespace you created for your load balancer machine and N in this case is the node number. In this example there is only zfp1 and zfp2 but it should scale linearly.

Posted on September 23, 2008 in Quality, Ruby by adam2 Comments »

I have been looking for a Ruby/Rails equivalent of checkstyle since I started working with it, but have only found a single orphaned project. Until yesterday that is. Yesterday’s Double Shot had a link to the Roodi project. Roodi stands for ‘Ruby Object Oriented Design Inferometer’, but essentially it is checkstyle for Ruby.

The Java community has a rich set of guidelines to build static checks from, but the Ruby community seems to lack that cohesion. That is going to make for an interesting collection of checks over time and no doubt some discussion over what should be and checked and how. Here are the checks it currently does:

  • AssignmentInConditionalCheck – Check for an assignment inside a conditional. It‘s probably a mistaken equality comparison.
  • CaseMissingElseCheck – Check that case statements have an else statement so that all cases are covered.
  • ClassLineCountCheck – Check that the number of lines in a class is below the threshold.
  • ClassNameCheck – Check that class names match convention.
  • CyclomaticComplexityBlockCheck – Check that the cyclomatic complexity of all blocks is below the threshold.
  • CyclomaticComplexityMethodCheck – Check that the cyclomatic complexity of all methods is below the threshold.
  • EmptyRescueBodyCheck – Check that there are no empty rescue blocks.
  • ForLoopCheck – Check that for loops aren‘t used (Use Enumerable.each instead)
  • MethodLineCountCheck – Check that the number of lines in a method is below the threshold.
  • MethodNameCheck – Check that method names match convention.
  • ModuleLineCountCheck – Check that the number of lines in a module is below the threshold.
  • ModuleNameCheck – Check that module names match convention.
  • ParameterNumberCheck – Check that the number of parameters on a method is below the threshold.

Thats all well and good, but a tool that is not run is just a clever piece of code. And relying on people to remember to run a tool is not a plan I would bet the house on. But that is why we have things like CruiseControl.

Here is a pretty heavily edited version of our Rakefile.

desc "Cruise Control"
namespace :cruise do
  output_dir = ENV["CC_BUILD_ARTIFACTS"]

  task :build do
    CruiseControl::invoke_rake_task 'cruise:roodi'
  desc "Run the roodi tests"
  task :roodi do
    `roodi -config="#{RAILS_ROOT}/config/roodi.yml" "#{RAILS_ROOT}/app/**/*.rb" > #{output_dir}/roodi.txt`

There is not (yet) a roodi_on_rails plugin, though I expect it is only a matter of time, so I just run the tool on the commandline and put the output in the place CruiseControl needs it to display it on the report page. The output is, to say the least, un-pretty but if the community backs this tool I can see it growing nice rspec like reports with links to the actual offending line.

Oh, and here is config/roodi.yml, which is just the default configs for all the rules, but explicit is better than implicit.

AssignmentInConditionalCheck:    { }
CaseMissingElseCheck:            { }
ClassLineCountCheck:             { line_count: 300 }
ClassNameCheck:                  { pattern: !ruby/regexp /^[A-Z][a-zA-Z0-9]*$/ }
CyclomaticComplexityBlockCheck:  { complexity: 4 }
CyclomaticComplexityMethodCheck: { complexity: 8 }
EmptyRescueBodyCheck:            { }
ForLoopCheck:                    { }
MethodLineCountCheck:            { line_count: 20 }
MethodNameCheck:                 { pattern: !ruby/regexp /^[_a-z<>=\[\]|+-\/\*`]+[_a-z0-9_<>=~@\[\]]*[=!\?]?$/ }
ModuleLineCountCheck:            { line_count: 300 }
ModuleNameCheck:                 { pattern: !ruby/regexp /^[A-Z][a-zA-Z0-9]*$/ }
ParameterNumberCheck:            { parameter_count: 5 }
Posted on September 22, 2008 in Podcasts by adamNo Comments »

Michael has explored the usefulness of emotions as oracles when testing. I tend to explain this with a physical component as well; if your eyebrows move, it is a bug. Furrowed in frustration is a sign that we need to explain something better or set their expectations in a different way. Oppositely, when they go up in surprise it can mean the same thing for different reasons.

In this podcast, Dr. Marco Iacoboni discusses ‘mirror neurons’ which allows us to empathize with people. The more neurons, the more empathetic. It so happens that one of the traits of people with autism is that they lack (or disfunction) of these neurons. (The first half of the podcast is the most interesting as it talks about the neurons and how they test for the presence of them. After that, it goes off about the election and a few other things which wasn’t nearly as interesting.)

My first thought when listening was whether we want people in the testing craft to be those with more, or less mirror neurons. On further thought though I think it wouldn’t make much difference as the interaction in a pure testing context is between tester and machine; so no empathy necessary. One place I can think it would be very important is if their role is to debrief testing sessions. In that case you want someone who can pick up the nuances of what people are actually telling you through their voice, body language and inflection.

Walking further out on the limb, I think a lack of empathy might also be valuable when logging bugs. If a developer is having a bad week, a highly empathetic tester might hold back on logging a bug as a result. A person with low empathy would log it anyways. If our primary objective as testers is to identify issues in the code then clearly the low empathy person is the one actually fulfilling the mission.

To link this back to the shaky ground I’m thinking on here.

There are a number of people who are successful in technology who have the minor form of autism called Asperger Syndrome. BitTorrent creator Bram Cohen comes to mind immediately, but I recall reading an article which posited that Bill Gates has a number of the traits of this as well. Does anyone know of someone who has been successful as a tester that has been (clinically) identified as having AS or a more severe case of Autism? Although I have been testing for over 10 years now I have only worked closely enough with about a dozen testers to be able to make any sort of guess about them (rightly or wrongly). In my very limited sample set, I don’t think I have.

Posted on September 12, 2008 in Podcasts by adamNo Comments »

Every couple months on A&E or Discovery they have a show on ‘Inside the Casino’ where they showcase the technology at use inside the Las Vegas casinos. Jeff Jonas is someone who helped build that some of the tech. Specifically Non Obvious Relationship Analysis (NORA) which seems to be able to be simplified to ‘really cool data mining’.

Take for instance the casino problem: 2000 cameras on the floor, 50 monitors in the security room, and 3 people watching. How do you know which monitor to watch? And more daunting is which camera to show on the monitor in the first place?

This data overload contributes to what he calls ‘enterprise amnesia’ – data a is in one part of the organization and data b is in a different part, but they are not linked in useful ways preventing the interesting questions from being answered? Take for instance the addresses of your dealers and those of banned players. The casinos have that information already, but not necessarily correlating it. He also cites a (admittedly small) percentage of employees in retail have been previously been charged with theft from their now employer. The information is there, but not in a useful way.

Both those examples could be done through traditional data mining techniques. Where his stuff goes is one (or two) steps further. In the systems he works with the data is only part of what is stored in the database. The queries are too. This means

  • The data finds the data
  • The relevance finds the user
  • Queries find other queries (which leads to collaboration between the people who are interested in the same thing as they are searching for it in the data set)

These techniques seem to me that they would be well employed in some of the central QA systems, especially bug trackers.

Most systems these days lets you save queries / filters so you can get information you care about, but that is only the first part of the solution. The next part is to have those queries run all the time and have the data reach the user. RSS is ideal for this and products like FogBugz incorporate this pushing of result sets to users. The final part of this is to have the system let you know who else was interested in this new bug as a result of their filters.

This might seem overkill for small, collocated companies where everyone generally knows what the others care about, but I could see this being massively useful in the HPs or Motorolas or Microsofts of the world.

You can listen to the full podcast (only about 20 minutes) at it’s IT Conversations page.

Posted on September 11, 2008 in Podcasts by adamNo Comments »

I blogged about Dan Ariely back in May (article here) after reading an article in the paper. I still haven’t read the book but have now listened to his Arming the Donkeys podcasts. Every week he has another one dealing with how and why people think what they do.

Here are the notes of things I found interesting.

July 2, 2008Jack Soll

  • Take what you are worst at and improve that
  • Small changes at the low end of things can make a big difference

August 5, 2008John Payne

  • Unconscious thinking is the better way to make decisions (see Blink)
  • Or is it?
  • It is certainly possible to do bad conscious thinking
  • This is often because of overthinking because you start to pay attention to irrelevant information in addition to the good stuff
  • Decisions of magnitude are good places to think about

August 11, 2008Alison Adcock

  • In order to learn something, be happy about it

September 2, 2008Stacy Wood

  • We forget the pain of having a car scratch
  • But we also also forget the joy of receiving flowers
  • What we actually remember is what we think we would feel like in those situations
  • Long wedding videos for instance are not necessarily a good thing for memories as they let you remember more about the day. Instead, just keep the happy photos as they reinforce the norm of having an enjoyable wedding day.

September 8, 2008Brian Elbel

  • If you present someone a list of 8 – 16 options, tehy are more likely to take the default or defer a decision to later.
  • By putting irrelevant options on the list, it forces people to think about the presented choices
  • The order of the choices counts too. If the thing people are comparing against is at the bottom of the list then people will consider the other options in more details.
Next Page »