Posted on July 30, 2009 in Uncategorized by adam3 Comments »

I’m quite convinced that the path to startup success is to do one, and only one, thing. And do it well. (If not awesomely.) The same thing holds true for your automated tests.

Imagine you have an application that has the idea of a logged-in user, and a non-logged in user. Some of your scripts will be around the login process, and some, the vast majority in fact, will be for some activity once you are logged in. The trick is to completely separate the two.

The first bunch of tests you produce will be around the the login process proper. These will have method names like testGoodLogin, testBadPassword, testFiveFailedLogins, testAccountLocked, etc.. They all test individual slices of the login process and both can and should fail a test if something is amiss.

And then there is all the other tests. But they all require you to login first. Well, DRY says that you should just call the testGoodLogin with the desired credentials and be done with it. Except now you have created a situation where something external to what you wanted could fail a test. Which is undesirable. What you want to do is create a helper method which will do a login for you. It will still likely have asserts in the method for page element timing and such, but if something goes wrong it won’t fail your test. Don’t necessarily ignore failing asserts, but error the script rather than fail it. We can debug the why of the error using the proper login tests.

Now, this is somewhat anti-DRY since you have a login being done in a test as well as in a helper and when the login flow changes you have multiple place to change, but that DRY a heuristic and so can be broken when appropriate.

Posted on July 30, 2009 in Uncategorized by adam1 Comment »

I just listened to the most recent Stackoverflow podcast, and in it they mentioned that they have run out of audio questions. My great scheme is to have readers of the blog call in with their own testing questions and see if we can have Jeff and Joel talk testing for half and hour.

Here is how to submit a question:
If you’d like to submit a question to be answered in our next episode, record an audio file (90 seconds or less) and mail it to podcast@stackoverflow.com. You can record a question using nothing but a telephone and a web browser. We also have a dedicated phone number you can call to leave audio questions at 646-826-3879.

Here is the script of the question I just left.

Hi Guys. This is Adam Goucher and I’m a tester in Toronto. The FogCreek way of hiring programmers has been well documented both online and through this podcast. What I haven’t seen is any discussion of how testers are hired and how they fit into either of your views of how software should be built. Thanks.

So there is your mission. Call and leave them a testing related question.

Posted on July 28, 2009 in Uncategorized by adam2 Comments »

I would guess that it would be the rare situation in your career when you get to build your team out from scratch. What will likely happen is that you inherit and existing team either by being completely new, or being elevated.

In either case, the team, its success and its failures are now yours.

First thing I think you should do is meet with the team as a whole and explain who you are and how you think about testing. Those opinion, rightly or wrongly, will dictate how the team goes about its work.

Next thing is to meet with each team member. The goal here is to find out what makes them tick. Why are they doing what they are doing? What would they rather be doing? Where do they want to be in 1 year? 2 years? 5 years? What do they think needs to be changed?

Of course, during this little ‘interview’, you might find out that the person just isn’t interested in their job, has lost their passion or is just generally burnt out. Well, that is now your responsibility too. As I view the world, you have two choices.

The first is to help them find the fire again. Maybe they like to code, so get them automating something, or analyzing the developer’s unit tests to help improve their coverage, techniques or speed. Perhaps start having lunch-and-learns with Google videos of test techniques that are innovative to your group. (Just remember to bring the food.)

You could also remove them to the position. But with this option comes a loss of institutional knowledge that you likely don’t want.

One way or another, the ones that remain will be dedicated to where you are going and will be passionate about the journey. When I hire, I’ll take passion over (initial) clue any time. You can teach clue. You can’t teach passion.

Posted on July 27, 2009 in Uncategorized by adamNo Comments »

I should likely wait for the next issue to come out on Saturday before doing this, but the 4 issues I haven’t processed yet are bugging me…

April 2009

  • CERT C Secure Coding Standard – Never heard of this before. Warning flags waving at the notion of a ‘coding standard’ and then the inclusion of ‘Rules’. What happens if you are exploited, but followed the rules? Do you get to sue CERT? Oh, and of course they only apply to C. Would be much more useful if they made them generic for all languages. What about languages that aren’t C, but are implemented in C? Wow I’m a cynic.
  • Dynamic Taint Propogation – Dynamic solution that tags and tracks user input at runtime and prevents its improper use to maliciously affect the execution of the program of safe code are enforced by a combination of static (e.g. bytecode verification) and dynamic (e.g. array bounds checks). DTP does not rely on fault injection and does not disrupt the normal behavior of the application. For this reason, it does not require any effort beyond standard functional testing (Don’t think I had heard of this one either.)

May 2009

  • YUI Test – YUI Test is a testing framework for browser-based JavaScript solutions. Using YUI Test, you can easily add unit testing to your JavaScript solutions. While not a direct port from any specific xUnit framework, YUI Test does derive some characteristics from nUnit and JUnit.
  • Ross Collard has an excellent article (first in a series) on Test Patterns. Its available behind the ST&P pay/registration-wall (seriously, you give away full subscriptions, why hide individual articles?). I’d like to see all these somewhere on a wiki, but until then, I’ve tucked this list away for later. I might spend the hour or so to make them more consumable at a later point.

June 2009

  • ‘What I really wanted to do was to marshal the set of tools available to me to take the set of test cases, assign each test case to one of several queues, then to run each queue simultaneously on a number of virtual machines.’ – From Chris McMahon’s article on building out build systems. Build systems are one of my favorite rabbit holes.
  • Ross concludes his Pattern discussion with another big list of them. Seriously, these should be on a wiki somewhere.
  • Elfriede Dustin and Bernie Gauf talk about hiring our an automation team. It assumes a certain type of organization, which for me, wasn’t very useful, but their list of ‘other’ attributes for automation team members is food for thought.
    • Be quick learners who enjoy performing a variety of tasks, learning new things, and touching many different products
    • Have an aptitude for conceptualizing complex activities
    • Understand complex test requirements, be able to formulate test planning and design approaches to support requirements and be able to perform multiple tasks concurrently
    • Be able to develop work-around solutions to problems encountered during test development and execution
    • Be creative, with the ability to think of multiple ways to exercise a system or application to ensure that it works in all circumstances, and be able to identify all conditions under which the software or system could fail
    • Be diplomatic and discrete, especially when having to report major defects, and be able to work closely and effectively with a variety of stakeholders
    • Possess strong verbal, written, and grammatical skills, with the ability to communicate any issues and articulate ideas to the customer/stakeholders
    • Be detail-oriented and able to identify difficult-to-find glitches, and have a strong interest in improving the quality of a software product, including being self-motivated and trying to find all defects possible
    • Be process-oriented and skilled in the ability to understand inputs, logical sequences of steps, and expected outputs
  • The Perception vs. Reality article is about process improvement an seemed to be focused on big-P process improvement. The kind that come under the guise of BPM and other such buzzwords. But this paragraph was good.

    Although often not explicitly recognized, implicit in generic process improvement is the need to identify and improve the REAL process. Many (and probably most) generic and branded process improvement efforts fail because they don’t properly understand what a process is and therefore don’t identify the real process that needs improvement.

July 2009

  • The only thing that was new to me was the WS-I which gives testers a world of resources to help increase confidence in the interoperability of their Web services beyond the scope of commercial testing tools. Oh ya! Another WS* alphabet soup organization. Do we really need such a group? Web Services are client specific and even there, context specific. Guess what? You are going to have to do custom mapping between multiple ones. Would it be easier to do this mapping if they all adhered to a specific standard designed to address this? Sure. But by doing it, are you creating more work for the development team? And the test team? And is that work worth the Opportunity Cost that it represents?
Posted on July 26, 2009 in Uncategorized by adam2 Comments »

Unless you are a customer of Cast Iron, then this post will make very little sense to you. These are the notes I took over and above the book they provided. They are going up here in case I should find myself in the future without access to the corporate wiki.

  • All orchestrations need to begin with a ‘startup activity’ which are blue
  • There are two ports on an appliance
    • data (which is what the client uses)
    • management (which is how you control orchestrations, view logs, etc)
  • Don’t hardcode anything into any parameters, use configuration properties whenever possible (field has a green ball in it). That way you can set / change them on the server without having to re-save and upload your project. Handy for switching between environments
  • Good Practice – Have a HTTP Receive activity as a starting one to be able to trigger job on demand (use a Pick activity to combine it with a different activity, like Poll or Schedule)
  • Good Practice – Split projects by data type (account data, lead data, order data) so you can reload the project and only affect that particular flow
  • If you have HTTP Receive as the starter activity, you can chain orchestrations using HTTP Invoke
  • Enabling persistence on the server is slower, but lets you resume a failed orchestration at the point of failure
  • If you say ‘requires reply’ for an HTTP Receive, then you need to have an HTTP Send Response on the orchestration somewhere
  • Good Practice – Don’t upload directly from studio, instead, export your project and upload it. Big benefit is that you can give something a version number on upload this way.
  • Good Practice – Add comments to everything; it will auto-generate documentation for you
  • Projects are undeployed by default when published
  • Default log level is ‘error’
  • log synchronously removes the lag between something happening and it being recorded, but there is a performance penalty
  • Its a good idea to constrain the number of threads
  • But you have to undeploy to change settings
  • For some database actions, CI orchestrations certain tables and triggers to be created (it provides the SQL)
  • EVERYTHING IN CAST IRON IS XML
  • So almost anything non-standard requires an XSL to transform it
  • Activities are JS, so custom ones can be created
  • Filtering items in XML is done using XPath references
  • Constrain SQL when you want specific data
  • Filter output when you want to split specific data sets into components
  • XPath is case sensitive
  • Most activities have Delivery Rules
    • Exactly once – no duplicates, requires CI custom tables
    • At least once – could have duplicates, doesn’t’t require custom tables, faster
  • Good Practice – use batch delivery
  • The Database activities are scary as they let you dodge the business rules
  • In order to parse and incoming file (csv, xml, other), the schema needs to be known (again, everything is xml under the covers)
  • When you remove an activity, the variables in it are not removed from the project until you goto the variables panel and say ‘remove unused variables’
  • When doing REST style WS calls, expose the optional headers in the mapping panes and put your values in there
  • Good Practice – Create a global error handling orchestration which has a WS Receive starting activity. Centralized the error handling and reporting and makes your main orchestrations much cleaner.
  • When building flat file schemas, use the wizard then tune using the editor
Posted on July 26, 2009 in Uncategorized by adamNo Comments »

I was in Atlanta last week on some corporate training, and aside from learning the application I also learned how to become a better trainer. (Much as how you learn to be a better speaker by watching others.) This post outlines the things I will try to do more, and in some cases less, when teaching my courses.

  • There is no logical reason anymore to use a travel agent. The lowest price is available for N number of aggregators. The reason why you use an agent is for the extra knowledge and personal touch they bring to the experience. Same thing with in-person training. If someone is there for in-person training, it is imperative that you go off-script to bring your own experience to the material. If you don’t have it, you shouldn’t be teaching the content.
  • Look at the fury hurled at airlines, partly in part to their inability to deliver you to your destination on time. Yet it seems that trainers get off easy on this. If a class is 9 – 5, then start at when scheduled, and finish exactly at the finish time.
  • And don’t have 15 breaks all the time. If everyone is done the exercise, move onto the next. Yes, in a smart class the material will be done faster, but you should be able to supplement it. (see the first point)
  • If a single person is struggling with an exercise, have someone who is finished pair up to help them. Both will learn more.
  • I tend to operate under the assumption that everyone in the class can read. This means that you do not need to read your slides word-for-word, and move the mouse over each word as you do it
  • You probably shouldn’t show real customer stuff on the projector. Especially when it has fields like ‘password’ on it.
  • If you have printed materials, bring a couple extra just in case. And make sure the extras are the current revision.
  • If there are multiple ways to configure your application / environment, make sure that you are using the one that the class is using.
  • Turn off your IM client(s), and your bittorrent software, etc.
  • Speaking of which, clean up your desktop and have your system setup in such a manner that ‘Open File’ doesn’t go into your pirated music folder
  • Timebox labs and go over lab solutions on the projector at the end of the box regardless of whether people are all done.
Posted on July 26, 2009 in Uncategorized by adam1 Comment »

Flying home from Atlanta I listened to an Arming the Donkeys podcast with Dilip Soman who did some research on decision process injection. In one of the studies he gave movie goers 4 smaller bags of popcorn instead of a single glutton bucket. What he found was that they would consume less popcorn since there was a mental queue, or decision point, that the participants had to pass (do I start another bag?) in order to continue.

I think the obstacle of decision points is what makes small (2 – 3 week) iterations work.

While at HP we did 18 month, classic waterfall releases and I can’t think of a time when we make corrections mid-way through. The only time we could look at what we were doing and how was at the end.

That is 36 2-week iterations. If we had done iterative development with actual retrospectives at the end of each, who knows how much more successful the product would have been. (Or how much less frustrating working there might have been towards the end.)

Here is the link to the paper, The Effect of Partitions on Controlling Consumption (from co-author, Amar Cheema’s website).

Posted on July 24, 2009 in Uncategorized by adam2 Comments »

If one was to look at my Thunderbird client, they would see a very big number beside the agile-testing list. In fact, I haven’t really processed it since the third week of January it would seem. I was going to go to the baseball game tonight, but it ended up being an afternoon game so I didn’t go and since I’m in Atlanta with nothing better to do, a processing I go…

  • Four definitions of ‘designed for testability’:
    1. Designed for automated, unattended evaluation
    2. Breaking down the app so it can be tested as a series of logical chunks
    3. Scaffolding
    4. Places logs and write timing information to them
  • ‘Planning for testability’ is just as, if not more important, than ‘design for testability’
  • PNSQC paper by Bret Pettichord about designing for testability
  • If you need it and know how, do it. If you need it and don’t know how, learn how and do it. If you don’t need it, don’t do it. – Ron Jeffries
  • Panopticode – cool visualization tool for things like code coverage
  • Sonar – Sonar enables you to collect, analyze and report metrics on source code. Sonar not only offers consolidated reporting on and across projects throughout time, but it becomes the central place to manage code quality
  • For entry level automaters doing Siebel work, Mercury’s Quick Test Pro may be a good starting point. QTP uses VBScript, so you won’t be able to reap the benefits of an OO language as the team gets better (and they can be huge benefits), but QTP has a huge advantage in the market – they have a contract with Oracle to provide libraries that recognize the screen objects for Oracle’s applications. I’m fairly certain QTP has Siebel libraries, but you should doublecheck. But it may not be as friendly to the backend systems (how do you verify the input to Siebel propagated properly to the integrated systems and vice-versa? The question to ask is – Can QTP automate that?)
  • Use in-memory databases for unit tests
  • Understanding and Solving Internet Explorer Leak Patterns
  • One of my colleagues brought up the best definition for “seniority” I’ve ever heard. It’s not age, or tenure, or experience. It’s “When I don’t have to worry about you”. – Rachel Blum
  • Rumbster is a simple SMTP server that receives email sent from a SMTP client. Received emails are published to observers that have registered with Rumbster. There are currently two observers; FileMailObserver and MailMessageObserver.
  • Dumbster – the Java version of the above
  • and a .NET version
  • Methodologies fit the context
  • TOMOS – TOMOS improves communication and collaboration between business analysts, developers, testers and managers.
  • Break “large refactorings” down into a series of small ones, with the tests passing after each one. Not only is this to be safer, but also faster.

Hmmm. Thought that this would be much longer. Some useful things though.

Posted on July 23, 2009 in Uncategorized by adamNo Comments »

I’m going through a bunch of queued of podcasts, this post is about four episodes of The MDN Show.

Episode One

Starting at the 11 minute mark they talk about how to survive WWDC, but some apply to any conference.

  • Avoid the food – Sugar highs and/or carb induced comas don’t help you learn
  • Take a walk – Let your brain absorb what you just learned
  • Don’t plan on going to every session – If you try to make every session, you will fry your brain. (This of course works best if sessions are recorded.)
  • Plan ahead – Know which sessions you want to be in ahead of time. As well as a backup for each time block.
  • Take notes – The chances that you are going to remember something you heard on the first day is minimal. It best.
  • Don’t be afraid to leave a session – As someone who has started to speak more, it is not offensive. I would rather you be somewhere else and learning than in my session and bored
  • Arrive to sessions early – at least for the ‘hot’ sessions
  • Don’t do work at the conference – You paid to be there, do you work at work.
  • Socialize – With people you meet, or track down people you want to meet. Put your twitter name on your badge to help facilitate this.

Episode Two

After the opening preamble there is an interview with Dave Dribin about unit testing. Its kinda from the perspective of never having heard about how unit testing works, but here are the sound clips I liked.

  • It is part of a strategy
  • Answers the question ‘Does it do what I think it should do?’
  • Do it at the method / class level
  • OCUnit – xUnit framework for Objective C
  • XCode has support for OCUnit built in
  • If your unit tests take a long time to run, they aren’t defined as well as they should be
  • Have your main build target depend on the unit testing target. That way the build always have to pass
  • If you depend on the database or Core Data in your unit tests, then that will slow down your unit tests. Get our your test doubles.
  • The Three Rules of TDD
  • At around the 33 minute mark there is a discussion of dependent vs. independent test bundles. I don’t know enough about OC development and so it went right over my head; sounded important to know though.
  • Adding unit tests to an existing project is tricky
  • Side effects of unit testing
    • Forces you to design for testability
    • Better separation of concern
    • Smaller classes that do one thing, and do it well
    • Less singletons and global state
    • Cleaner code overall
  • CoverStory – UI for analyzing files using gcov on the Mac
  • OCMock
  • Use an in-memory store for your Core Data tests to avoid hitting the disk
  • If global variables are bad, well, so are singletons. Use the dependency injection pattern instead.
  • You really cannot refactor code unless you have unit tests. If you don’t have them you are redesigning

And then at the 1h8m point, the second interview starts in which Joachim Bondo talks about ‘simplicity’ in Mac applications.

  • Myth: The more features, the better the product
  • You can’t compare the value or enjoyment or productivity of an application, so they choose features and price
  • It is not easy to find the balance between the core features for the application to work, and for it to sell
  • If you want a simple interface, you have to work hard to put it. Putting everything in on-by-default toolbars is a cheat
  • Have a product statement. Everything else falls out from there.
  • Only expose things to users that are relevant right now
  • Don’t stick anything into anything unless you have gone through a total process of justification.
  • There are many different ways to measure success
  • Write applications you would want to use
  • You can’t separate design and implementation

Episode Three

The big draw of this episode is the interview with Wil Shipley.

  • Find a crowded product area, and then become the most outstanding, most user friendly, most approachable product in it
  • Two ways to make software friendly
    1. Make the software seem like it doesn’t do much
    2. Come up with a core innovation (or two) – and be really willing to work on it
  • ‘Just add a button’ is a synonym for ‘Just stab a kitten’
  • People use more features if you give them less features because they can actually use what you give them
  • Only add features to major releases. Not point releases. You can charge for major releases.
  • (Major) Releases should have a theme
  • You don’t want to be a pioneer [of a graphics library]
  • advertising idea – showcase the features that are not in the product

And then Matt Gemmell provided some little cliches

  • Simplicity is the ultimate sophistication
  • Designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away

And then he had some thoughts on how to achieve the concept

  • The overall set of features should be as small as possible
  • Design a 2.0 application, and then cut it down to 1.0
  • Do the most wanted thing; please the bulk of people
  • Options are a failure to simplify or focus your feature
  • Spend more time designing apps that do less

Episode Four

And finally, episode four.

It starts with an interview with the guy behind a Cocoa REST framework, but the 13 marketing tips was more interesting (start at 39m). Rather than write down notes, here are links to part 1, part 2 and part 3.

Posted on July 22, 2009 in Uncategorized by adam1 Comment »

I’m in Atlanta this week for some training and since I don’t know anyone in the area I sent out some inquiries and ended up having dinner last night with Eric Jacobson and Marlena Compton; both local testers. At the end of the evening I suggested they should make such an outing for testers a monthly occurrence, so I’m going to label this the first such event. If you would like to participate in the next one, contact one (or both) of them.

Here is a list of resources and such that came up over the course of the three hour conversation.

Books

Courses

Schools of testing

Other

All round, it was a good meal and good, geeky tester talk. Hopefully I didn’t bore them too bad. (Realized later that I talked a lot)

Next Page »