The only thing I can guarantee to you about your career in testing is that at some point (likely lots of points) you will miss something. Of course, you will learn about the omission once things hit production. Don’t worry about it. Accept it. It will happen.

So what do you do when it happens?

Someone in class tonight asked specifically who was to blame when a bug made its way into production. My answer was that no one specifically is to blame. If blame must be assigned, it should be done so collectively; the developer whose code introduced the problem, the tester who missed the problem, the customer who signed off that the code functions as requested, etc.

As soon as you start playing the Blame Game though you are in trouble. Suppose you did successfully pin the blame on the developer. Not only have you jeopardized your relationship with them but you have set yourself up as a target the next time something makes it out undetected. Have you ever tried to correct someone’s spelling or grammer on a mailing list? Usually the correcting response will then contain an error. Call it karma for trying to look like a jerk.

The topic was timely for me since I missed something last week that ended up in production. Yes, there are 3 teams between me and production, but I’m the first (and have high standards for myself). I could have deflected the blame by pointing out that they missed it too but I did what I think is the grown-up thing. I took the “blame” and said sorry. I missed it, it was an accident, I’m still human.

The last time I had lunch with Michael he (somewhat sarcastically) said that he has never made a mistake. He has however had a number of learning experiences. (or something in that spirit). To me, this is what this miss was. The root cause was that I tested the external behavior of the application and while I knew that we were pumping values into a stored procedure in the database I did not check what those values were supposed to be nor whether we were sending the right ones. In this case, we were not.

I’ve added it to the checklist though.

The planning for CAST2008 is well under way. In fact, registration has been open for a week but just not really advertised – yet. The keynotes and tutorials are already listed online and the talks will be in a week or so (I think) after those who had talks accepted confirm their attendence.

Speaking of which, my talk on how being a houseleague (lacrosse) coach can help you be a better team lead was accepted.

Go me!

The product I spend a most of my time has a number of components, each with a whole bunch of ‘projects’ in eclipse-speak. I spent yesterday and today setting up a development environment for one of them and in doing so uncovered another smell regarding the system as a combined entity.

More than one library for any one function is a symptom of much larger architectural problems.

In this case of this product I noticed there are 4 different ORM systems being employed

• Component A – this is likely about 85 – 90% of the customer facing product and uses Hibernate2 and TopLink
• Component B – this makes up the bulk of the remaining customer facing products and uses Hibernate3
• Component C – is used my internal support staff and uses iBatis

Not only does this fragment developer expertise across 4 different tools (hibernate2 and 3 might as well be separate entities) but it is going to make upgrading of the database even more painful that is already going to be.

What is interesting is that this smell doesn’t actually originate with the software directly instead is actually organizational. Enterprise Architects tend to get a bad rap, but once an application gets to be a certain size they become almost a necessity. An EA with sufficient organizational care and clout could have this unnecessary duplication cleared up in a couple months (wholesale, flip-a-switch conversion would be a bad idea) but I have in 14 months seen exactly one communication from this particular EA’s office and that was a security guideline document that very few developers actually seem to follow.

In the span of 3 weeks this month we got more snow than we did all last winter. This meant that the route I take from the train to the office was a maze of caution tape and ‘beware of snow and ice’ signs. These are necessary of course to place the fault on the pedestrian if a chunk of snow or ice fell off the building and hurt them. What amazes me the most however is that a number of the buildings seem to have been designed in such a way that they collect the most snow possible on their various ledges and jetties. Hellloooo. This is Canada. We get snow here. This is an established fact. You would think that architects would take this into account when designing their buildings. Instead it appears that when you spent many-millions on a building you also get a couple yellow standards with the title tenant’s name on it pointing out that while it might look nice, it is a bad design for a section of the year.

Aside from a bit of a rant, how does this fit in with testing? Enter rant part 2.

Much like buildings which should interact with their surroundings in a sane manner, so too much software. Rather than the environment being ‘downtown Toronto’, software’s environment is the OS. This means that software should know how to interact with the host OS. On windows this means being able to handle paths with spaces in them (care to guess how I spent part of my afternoon?) and no unix file and directory names that begin with a ‘.’. If your product cannot be

• built
• installed
• configured
• run
• removed

on a Windows platform with a space in a path somewhere then it is a bug. I would argue it is one of the highest priority as well since the default install location for user applications on Windows is ‘c:\program files’.

I’ve rattled off this list to a couple people before and the common belief seems to be that I am on the right track except for the built aspect. I’m keeping it on the list though. To me, a smell of whether a product is going to be fragile or not is how easily it can be built. If I have to reproduce the exact state and configuration of the build machine in order to make a working copy of the software there is something seriously amiss.

Billy Hoffman is a manager in HP’s Security Labs (via their SPI Dynamics purchase). He is also the co-author of AJAX Security which is how he ended up on Technometria. They ended up talking not only about AJAX but security in a rich environment in general covering Flash as well. I only paid about 60% attention, but here are my notes.

• Flare – a flash decompiler
• If you are doing crypto, don’t put the key in the flash object (see previous bullet for why)
• How many times do we have to mention don’t trust the client? Don’t people get this yet? Repeat after me: I will not put state information in the client nor will I make decisions based upon state in the client.
• Today’s tester needs to get familiar with tools like Wireshark and Ethereal to watch the protocol traffic as some errors are hidden by the browser.
• With the rush to SOA you have to make sure your architecture does not create a DoS vulnerability. For example, one service reserves a seat or ticket and another one releases it (if a payment failed). By calling the first service but not the other your inventory can disappear rapidly.
• JS comments are visible to the end user; make sure they are sanitized.
• <script> tags do not abide by same origin
• JSON hijacking
• There is a lot of uncertainty around the origin of a request; is it a browser, or a script? Right now you have to do a lot of log correlation but once that technique is commonplace the scripts will up their intelligence to outsmart that too.
• In CSS, the last definition of something is what wins. If you let users upload their own CSS they can do all sorts of nasty stuff

Direct link to MP3.

The big article in the February 15, 2008 issue of SDTimes (and the only one of note actually) is on software piracy. Not a great article and you have to be a complete lamb to be surprised by some of the statements. “Piracy groups have become their own ISVs,” said DeMarines. “They’ve got developers, testers and distribution.” for instance. Ummm, let’s see. I’m 32 this year so this statement was already true 17 years ago. (Trust me). However there were a couple good points:

• Some development shops are growing their own form of software protection, which could be a costly mistake. First, software protection is likely not the company’s core competency…
• Even if there is a commercial solution in place, software developers may not be … applying it correctly.
• Sebastian Holst, senior VP of PreEmptive Solutions – “How you build software translates to security and financial risks, and therefore you’re obligated to align software development practices with business stakeholders.”
• Software piracy means risk, and the best way to mitigate that is to integrate risk management into every phase of the business and software life cycles.

Every so often I find myself browsing a website with a more critical eye than I might normally. Usually it is because a friend has asked me my opinion on a beta or if I am considering a relationship with either the company itself or its product. After running through SL (Security, Languages) of SLIME (don’t have enough information to do the rest) I check out the site for staleness.

What determines whether something is stale? Typically two things

• A date range that should include the current date, but does not. Open a (new) browser and look at the footer of your corporate site. Does it say © 2005 – 2007? A shocking number do. 2007 ended 46 days ago.
• Company driven dynamic content of > 1 month. These are things like ‘News Releases.’ You mean to say that you as an organization have not done anything worth telling the world in the last month? No software update? No new customer? No new hire? Really?

I have turned down interviews from companies whose ‘Announcements’ section had not been updated in 10 months. If an organization cannot spend the time on their public image, what does that say about the company? This would actually be pretty easy to check in an automated fashion. In fact, here is some pseudo code.

if file.last_changed() > curr_date - 30 days then log bug in tracker

Yes, this doesn’t do the 1 month boundary perfectly, but in this situation the approximation is more than sufficient at achieving the goal.

A third place where staleness creeps in is harder to detect but is just as bad looking is in the actual site content. Take for instance Penguin’s bio of Ron Chernow. Now look at his Wikipedia entry. Notice anything different? How about his wife dying 2 years ago. You would think his publishing house would be aware of such a thing. It cannot be hard to push a bit of new content out onto their servers. As things stand now, it just looks sloppy. Unfortunately, this class of bug has no easy way of detecting in a non-human intensive manner. But I’m sure it is possible if an organization decides they are not going to accept having stale content on their site.

Johanna’s comment in the Are pre-defined release dates bad for Quality posts has had me thinking for the last little bit. Are pre-defined dates okay for Agile projects? I think they might still act as a hindrance to the overall project.

Let’s say you are working on a project that is using 2-week iterations. The number of iterations for the project is often a straight calculation from the pre-defined release date / 2. Eeach iteration is then mapped out with desired features. As I understand the theory, iterations content should be based upon the complexity of features coupled with the teams’s velocity.

Sure, not all Agile projects have pre-defined release dates. But I suspect a lot of so called Agile projects are really Wagile ones. Almost by definition, if there is a contract, you are Wagile which is dangerous. Sure, you might use iterations, and story cards, but if the contract says that you deliver at X, you better be sure to deliver on (or before) X. This is significantly different than the model where you control all the pieces and can release when things are actually ready for release.

This leads to the 3 things I think would help you overcome the danger of Wagile release planning.

• A really good build person – An intelligent, effective build is something that should not be taken for granted and might be the most important deliverable other than the actual code
• Everyone on the same team – I’m starting to think that a smell for whether a project has a greater chance of success is whether there is a single “team” or if there is “the dev group”, “the testing group”, “the database group”. Yes, in Agile projects there is the notion of shared ownership so everyone does everything else, but I’ve been in Waterfall organizations where that was true as well. The split between dev and qa, while not optimal is not (usually) fatal. I think though that a separate database group through is the kiss of death. Yes, a really clueful DBA is a must, but their role should be in making sure the database runs smoothly and point out which interactions are having negative impacts on that goal, not holding up development, forcing bad data models upon development or acting as a roadblock to innovation and refactoring.
• Architect for refresh – In today’s web application, there is almost no excuse to not design things in such a way that an update cannot be pushed out without minimal notice to the consumer. This often means building your application from the start to be deployed inside a cluster, a way to get new configuration information (either re-read files on disk or pulling from the database) and perhaps storing session information somewhere central to all parts of the application.

The common element is that these all help increase your Agility by upping your ability to Release Early / Release Often; another tenet of Agile. While perhaps pushing production code 37 times in one day (like Flickr is reported to have done once) is a bit excessive, being able to do so frees you the eventual release date and lets you concentrate on the now. When the end of the project rolls around, the release then becomes a non event as it is something that you have been doing all along.

Something to chew on.

I just finished testing part of an application which basically retrieves information from the database and displays it to the operator. The complexity in this situation comes from ensuring that the correct information set gets retrieved and displayed in the correct place.

At first I wanted to use the other portion of the application to create my own test data as I don’t know the origin of what is already in the database, but I couldn’t get it all wired up correctly. Plan B had me pulling my test data directly from the database (which sorta means Oracle was my oracle). Fortunately, or unfortunately this database hasn’t been refreshed in quite some time so there was a tonne of information to choose from.

Too much actually.

This got me thinking, how could I get the database to feed me a single data point at a time? A few minutes digging led me to SQL to Select a random row from a database table. A few minutes more and I had my search SQL tweaked to provide new test data on every refresh.

Very cool from a sitting-at-a-sql-prompt perspective. Crazy cool if you were to rig things up so your selenium scripts where being driven by this sort of thing.

SDTimes – January 15, 2008

• Another thing for the release checklists: deleted files from virtual hard disks are made non-recoverable at least one of the machine images available for download at TechNet did not have its free space wiped, and files thought deleted proved recoverable from an evaluation copy of the Internet Explorer Application Compatibility VPC Image.
• Eclipse Accessibility Tools Framework
• Some points from the Overcoming SOA Insecurity article
  • Identity Management (it’s not just people anymore)
  • XML Security (XML Injection, encryption)
  • Write rules for your SOA and apply them
  • Reuse (trusted) components
  • Clean seperation of services
  • Do your own penetration testing
  • Get proof from partners that their services are tested (for security) and bind it contractually
  • Do it all again; SOA is a moving target

ST&P – February 2008
Nothing much of interest this month except for Karen’s article on multi-user testing. I’ll be sending a pointer to this to at least one person.