Posted on May 28, 2007 in Quality by adamNo Comments »

This article gives a list of “the 26 most studied and widely accepted cognitive biases.” If some of them sound familiar, they were mentioned in my post on “How Doctor’s Think”.

Posted on May 25, 2007 in Quality by adamNo Comments »

Thanks to an older post by Jerry Weinberg I realized another similarity between coaching lacrosse and teaching testing.

The Weinberg Target is as follows: After exposure to my work, does the audience care less about the subject than they did before? If the answer is Yes, I’ve failed.

This is very similar to what we were told in the introductory coaching clinic: The measure of whether you were a success as a coach is not the number in the win coloumn, but the number of kids from this year’s team that register to play next year.

Posted on May 24, 2007 in Quality, Video by adamNo Comments »

This video is one of those good, but annoying ones. In it Jeffrey Feldstein shows his, and apparently Cisco’s bias / conciet towards only hiring programmers as testers. Natually I think this is foolish, but I’m pretty sure that is also Google’s policy and mainly Microsoft’s as well so I could be wrong on this one. But I doubt it. Testing is a completely different mental process than development. Of course, I could just be jealous that I can never work for Google, Cisco or Microsoft. *shrug*

  • 3 Types of testers
    • Classic – experts in the product, very good at creating test cases, not a computer science / engineering major
    • Scripter – replicates the user’s interactions very simplistically using a tool or scripting language
    • Test Engineers – can design test systems that are at least as complicated as the system they are testing
  • 85 – 90% of test engineers at Cisco (and when I say Cisco I mean Jeff’s part of it) are programmers
  • An interesting cube layout is to have test in the middle with with development surrounding them
  • Developer’s perception of testers
    • Testers are dumb
    • Testing is boring, manual and repetitive
    • Testing is not creative
    • No innovation takes place in testing
    • Testing is a necessary evil standing between my brilliance and the user
    • Testing is not a career
    • Test engineers are pencil pushing quality (process) geeks
  • Organizational things that can be done to recruit testers
    • Testers get paid as equally as developers
    • Has to have a career track within the company
    • Test not treated as just a service for development
    • Development and Test seperate reporting units
  • Reasons Jeff uses to convince developers to work in test (he mentions that he only looks at development resumes when staffing his test group):
    • Create sophisticated software, but with a different purpose
    • You get to decide what to build, not the product manager
    • Test teams are (generally) smaller so you stand out more in a crowd
    • Get to be creative (and do say, Model Based Testing)
    • Get to break your buddy’s code
    • Get to be innovative – Jeff mentions having 4 patents; I couldn’t find them though, but I didn’t search too too hard
    • Get more interaction with customers (at least at Cisco)
  • If those don’t work, you can always try
    • Interns – they will work wherever you want to, and be happy about it
    • Promise the developer that they can move into development after a fixed stint in test – ummm, if you have to do that, then perhaps your hiring practices are busted
  • Good test engineers can out develop developers
  • They also understand the customer space better than the customer

Direct link is here.

Posted on May 24, 2007 in Quality, Video by adamNo Comments »

When I was at HP I built a UI to a commandline test framework I wrote to make it easier for the other testers (who didn’t know it inside-out since they didn’t write it) to run. Someone I was working with remarked that “it looked like an engineer developed it”. A backhanded compliment if ever there was one, but he was right. It had an enforced workflow, lots of buttons, menus, drop-downs, etc. When testing, I now try to make sure it doesn’t look like an engineer designed the UI (even if one did). Part of that is the “aesthetics” of the page, that is the gut reaction to seeing it. And wouldn’t you know it, google just posted a video on the very subject. I recommend anyone who has to test a UI look at it and keep the things mentioned in mind when doing their testing, or even better yet, when reviewing the mock-ups during the design phase of things.

  • Steve Palmer runs the Palmer Visual Perceptions Lab at UC Berkley
  • Aesthetics is one of those terms that is hard to define, but everyone knows it when they see it. “Ahh! Wow! That’s Great!” vs. “Yuk! Ugh! That’s Awful”
  • A new acronym – 2AFC: 2 Alternative, Force Choice
  • A couple people I know who come from science backgrounds lament the lack of empirical studies (with proper controls) as it relates to software development. While his experiments are not explicitly done with development in mind, they appear to be well thought out and executed
  • Center Bias – people like to see images / items in the center of the frame more than towards one side or another
  • Inward Bias – if an image / item is to either side of center, people prefer to see it facing the center
  • When the variable that changes between the frames is the vertical positioning, the one closer to the bottom is more pleasing
  • An unknown – does having training in asethetics affect one’s opinion of what is aesthetically pleasing? For instance, if the large part of the community has art training
  • An unknown – does person’s cultural background affect aesthetics? Cursory evidence suggests yes
  • A heuristic for aesthetics – Make the most important information most easily visible within the frame
  • Only intentionally violate aesthetic expectations when it fits the message
  • When positioning 2 items in a frame, the findings for 1 item can basically be disregarded
  • The four most aesthetically pleasing hues (in order, most to least): purple, cyan, blue, red. This is apparently the same order of preference as with babies
  • The most aesthetically pleasing colour combinations are those where the hue is the same, but the saturation is different
  • A cool colour is more harmonious with a warm colour
  • A cool colour is prefered if it is presented on a warm background. And vice-versa
  • Men tend to prefer saturated colours and women do, but their study ground was more female than male so this could be a false finding
  • Oddball fact – the study participants were paid in coupons for frozen food donated by a food company since they could not get grant funding

I have mentioned before that the best keynote I have seen at a conference had only a slight association with the conference topic. It would be nice to see one of the bigger testing conferences get Dr. Palmer in to present his research in a testing spin.

Direct link is here

Posted on May 24, 2007 in Quality, Selenium by adam4 Comments »

My brain hurts, in the peculiar way when you are trying to work out a problem which you know is going to bite you later if you get it wrong. Here is the problem I have been working on and what I think might be the solution.

Problem: I am working on a Selenium scripting framework which will run Java, Python and Ruby Selenium RC scripts as well as ones written in Selenese from the Selenium IDE. Making things dynamic in one of the RC scripts is “easy” as you have the full language behind them. The Selenese ones notso much.

  • The application these scripts is for has multiple environments to go through before getting into production, but I only have to care about the first 4.
  • The only difference between the environments from a scripting perspective is the login process: different protocol / host / port / username / password. Once you are into the applicaiton proper, things are the same.
  • In order to run these scripts (which are “discovered” by the framework), you have to start the Selenium Server with the -htmlSuite option, which means that for a script to be run, it has to be wrapped in a test suite. Not a problem, we can create one dynamically in the framework
  • Since we’re making the test suite in the framework, we can inject the correct login script as the first script

But how to pass in the pass in the the username and password values in a way which means the actual scripts are not tailered to one environment? (Hopefully the problem description makes sense).

Solution(?): This is what I have come up with, which unfortunatly is also the most complicated solution (but might buy me a performance improvement a bit down the road)

  1. Framework discovers a Selenese script
  2. Framework looks in the test for the “id” of the user which is going to be logged in
    <!--start config-->
    <tr>
     <td>store</td>
     <td>superuser</td>
     <td>userId</td>
    </tr>
    <!--end config-->

    I’m using ‘store’ because the IDE lets you easily add it which will ease training. I’ll check into CVS a read-only template script which has the config section already done.

  3. Framework gets the rest of the user’s information from the config file via the id
    <!-- users -->
    <users>
     <user id="superuser">
      <username>adam-super</username>
      <password>password5</password>
      <type>super</type>
     </user>
     <user id="english fs">
      <username>adam-jnh-ca-fs-0002</username>
      <password>password1</password>
      <type>fs</type>
     </user>
    </users>
  4. Framework creates the test suite with 2 scripts: the login script and the actual test. The parameters needed for logging in will be passed into the login script.
    <html>
     <table>
      <tr>
       <td>IP Filtering (Jonah)</td>
      </tr>
      <tr>
       <td><a target="testFrame" href="login_jonah.html?userName=adam-super&password=password5">Login to CAMS</a></td>
      </tr>
     </table>
    </html>
  5. Framework looks in it’s configuration to figure out which environment it is working in to get the protocol information
    <!-- environmental stuff -->
    <environment>
      <protocol>http</protocol>
      <host>jonah.r1dev.com</host>
      <port>9080</port>
    </environment>
  6. Framework launches the script
  7. The login script does something along the lines of
    <tr>
     <td>type</td>
     <td>Login.Token1</td>
     <td>${userName}</td>
    </tr>
  8. Then we are at the generic part of the whole mess where scripts “will just run”

Alternative Solutions I rejected:

  1. I had thought about using the include user extension to include the login script in each test, but rejected that because
    • It breaks the Selenium IDE
    • I still have to pass in the various values in the suite
    • Using the above solution I could later collect all the Selenese scripts relating to a certain user and run them as a batch instead of at once
  2. Embed the login values in the login script using ‘store’, but that means
    • I have to constantly modify files that are stored in CVS atrifically making it look like there have been revisions
    • Avoid point 1 by copying them to a temp dir, but what if the script crashes? They won’t get cleaned up properly
  3. Embed the login values in the login script in a comment
    • see above
    • The Selenium IDE overwrites the file when you save it with the new version so anything that would not have been nicely produced by the IDE is wiped out

Is there another option I haven’t thought of or great gaping flaw in what I’ve come up with? I’m coding this up tomorrow.

Posted on May 24, 2007 in Podcasts, Quality by adamNo Comments »

I’ve subscribed to IT Conversations as another source of useful background noise (see all my video posts for the most obvious example). Today they sent Feeding the Game: Online Game Security Issues which is an interview with Deb Radcliff who is Vice President of Publishing at The Security Consortium. I know a bunch of WoW addicts and it mentions Security in the title so I gave it a whirl.

  • Gary McGraw has a Short Cut called Cheating Online Games which could be considered a teaser trailer for his next book, Exploiting Online Games
  • How do I get a job playing WoW all day farming gear for various grey-market organizations? Apparently there are gaming sweat shops in Asia that do just this. Not surprising since you hear of these sorts of things in the context of click-fraud
  • Stealing whole accounts is apparently big business. I suspect the loss here is more emotional and time-spent than real monetary loss for people affected. Here is an example… Back in the day (13 years ago maybe?) I nursed a pretty big MUD habit (think, text-based MMRPG). Anyways, there was a class of characters who were vampires and of course, you know that if you stake a vampire they are dead dead. Well… one night I was feeling particularily dastardly and long story short, staked a very senior player who had spent a TONNE of time building up his character. *POOF* All his gear lost, hundreds of hours of leveling, gone. He was (rightfully) annoyed, but more at the time he had invested than anything else. Going out on a limb, I would say the same is true for today’s generation of gamers.
  • Keystroke loggers appear to be the favorite tool of the bad guys. That and misconfigured webservers such as the one at Guild Portal which has over 1.5 million users. By exploiting the ANI bug they insalled them all over the place. The Super Bowl’s website was also hacked to install key loggers looking specifically for WoW passwords
  • So what does this mean for the enterprise? Well, passwords are reused something like 45% of the time. So if I have your WoW password, do I also have your EQ one, your bank one, your VPN one? Also, a lot of the games these days have monitoring systems you have to install to see if you have any sort of cheat software running. But what if you have a work spreadsheet also open? Is it capturing that too? And sending it to Blizzard?
  • If gear which can be earned in the virtual word can be sold for real cash, how long until this million node bot nets start being employed to play WoW?
  • In the interview, Deb says that you should not use IE, but should instead use FireFox. Generally safe advice, but she said it in the context of the ANI bug. As mentioned in yesterday’s post the ANI bug affected FF as well because they use a common Windows dll for that bit of functionality.
  • Should this be something corporate risk managers be thinking about? Maybe. More than likely actually.
  • At the end they detour a bit and talk about how the kiosks at airports likely have keyloggers. One solution they mention is that they should all be using something like readonly, non-state-saving virtual sessions. Sounds like a decent idea, heck, we had a similar system in place at college 10 years ago to avoid this same thing.
Posted on May 23, 2007 in Housekeeping, Quality by adamNo Comments »

For the last 2 days, I have been having difficulty connecting to MSN or viewing feed contents in my RSS reader. I had initially blamed our flakey network at work (we’ve outgrown our firewall), but when it happened at home too I did some digging. It turns out that when I was developing some Selenium tests, I had some crashes (type-o’s, etc) which left IE with a proxy set to a custom profile configuration script. Of course, this setting is used by anything else that uses the IE connection settings to interact with remote servers.

One manifestation of this is MSN Live Messenger not connecting with an error 80048820 (extended error 80048439)

Posted on May 23, 2007 in Python, Quality by adamNo Comments »

In some qa organizations, the tester’s role is to not only discover bugs, but to track down their cause. In agile teams, testers are often also responsible for static or whitebox tests. In both these scenarios, being able to reverse engineer the product is a useful skill to have. It is also one that not many people have anymore as schools teach computers at a higher and higher level of abstraction these days. In this video, Alex Sotirov, a vulnerability engineer at Determina who found the first publically disclosed remote exploit in Windows Vista discusses how he reverse engineers Microsoft’s monthly patches and shows a demo of the exploit.

  • Metasploit – an exploit framework written in Ruby
  • IDA Pro – another reverse engineering tool
  • BinDiff – a plugin for IDA Pro which does what the name implies; diffs binaries
  • PAIMAI – a scriptable (in Python) debugger
  • Buffer overruns – The use the /GS flag on Microsoft compilers can greatly reduce your exposure to these kinds of errors by inserting code to check for overruns and hijacking of the return address (see for more information). However, the compiler uses a heuristic when putting in the cookies and remember that heuristics are by definition failable. In this case the original heuristic was if there was an array, then cookie is. The ANI bug however dealth with a fixed length structure. The heuristic has since been updated to include this scenario
  • ASLR (Address Space Layout Redundency) – Putting the same stuff in the same spot in memory every time is a baaaaaaad idea
  • Heap Spraying is responsible for most of the browser exploits in the last while, and is fighteningly reliable an attack vector
  • DEP (Data Execution Protection) – mark certain pages of memory as read-only. Of course, IE (even in Vista) does not run in DEP enabled mode by default
  • Even with IE running in ‘protected’ mode, things are not so good. Protected mode prevents the attacker from modifying the system’s files, but it can still read them. It can also install things into running proccesses (like a key logger into IE)
  • Even if you have ASLR and DEP implemented, you can still shoot yourself in the root. Windows has 8 bits of entroby for the ASLR algorithm, so it only takes 256 guesses to find the thing in memory you want to stomp on, and in the case of the ANI bug, there was exception handling in place which made it easier.
  • Always check the rest of your code for similar problem patterns. The ANI bug was fixed in 1 place, but was still an issue in 2 others with the exact same signature
  • “C++ is wrong in almost all but a limited set of circumstances” – instead use a higher level language and encapulate the performance critical parts in a small library that is called from that higher language
  • At the end there is a chart which shows which of the protection methods he described are implemented in which OS. Vista, Linux and OpenBSD are all green. Amazingly, Mac OS X is all red. Well, DEP is green, but only on Intel chips.

Direct link here

Posted on May 23, 2007 in Quality, Video by adamNo Comments »

Elisabeth Hendrickson is one of the names you come across a lot in the testing community and just so happened to be at Google talking about Agile Testing way back in December 2005. It’s a pretty good presentation overall and she certainly has more than enough enthusiasm for the topic. It would however be interesting to see if in the time since then if her thoughts about what agile testing is have evolved further as she mentions a couple times that this is what she thing now.

Notes:

  • Mary Poppendieck writes about applying lean techniques to software development
  • Test artifacts should capture the essence of their intent, and nothing more. 96 page test plans don’t impress me either
  • Customers need test documentation to be of value to them
  • Test documentation that is not going to be seen by a customer needs to have enough information ‘for us’
  • Agile teams often have the testers embedded right in the development team making a single ‘product’ team since everyone is responsible for quality. Be careful about being designated the ‘tester’ though. That is asking for technical debt.
  • Automated unit tests can be used as change detecters
  • If it hurts, do more of it. In theory you will fix the hurt so it does not hurt anymore (and not just get numb to the pain)
  • Testers are there to be
    • agents of product management
    • advocates for the customer
    • supporters of development
  • Shared code ownership assists in keeping up the quality as everyone has a stake in all parts of the app

Direct link is here.

Posted on May 23, 2007 in Quality by adamNo Comments »

The 2007 Google Test Automation Conference is now accepting applications to attend. Given the publicity that this is going to get, and the limit of 150 attendees you have to submit a little blurb about how you would contribute as an audience member. I originally was thinking I should go to this, but looking at the presentations I am a bit disappointed with the content. The only presentation I want to see is “Building a flexible and extensible automation framework around Selenium” which of course will be online shortly after the conference. Hopefully the sound is better than the ones from London though.

Next Page »