Posted on April 30, 2009 in Uncategorized by adam5 Comments »

I’ve spent the last couple days trying to make a particular page (actually, a widget) of ours load in 2 seconds. It was taking upwards of 10. And now I’m happy to say it is actually less than 2 seconds consistently. Here is what I did to solve the problem.

  • Isolate – Part of a larger performance problem affecting our widget was the database getting smoked by something periodically as a result of a design decision in another product that shared the mysql instance. By recreating our product stack on a different machine, that issue cannot cause issue for the widget. At the expense of extra server maintenance and deployment headache, but that is an opportunity cost we’re willing to pay.
  • Compress – Since this is a widget, everything non-image related is either a css file or a javascript one. Because the browser is what is parsing and interpreting these files and not humans, they do not need to be nicely human readable. Enter minification. I’m using the YUI Compressor to compress both css and js files upon deploy. The timing of this is key as minified files suck when you are developing and you don’t want to leave the mandatory step of minification before someone updates a file; that’s just asking for missed revisions and confusion.

    Here is the relevant capistrano tasks to do this minification

    task :default do
      update
      minify
    end
    
    task :minify do
      run "cd #{current_path} && find . \\( -name '*.js' \\) -and \\! \\( -name '*-min.js' \\) -print -exec bash -c '/usr/local/java/bin/java -jar jars/yuicompressor-2.4.2.jar --type js -o {} {}' \\;"
      run "cd #{current_path} && find . \\( -name '*.css' \\) -and \\! \\( -name '*-min.css' \\) -print -exec bash -c '/usr/local/java/bin/java -jar jars/yuicompressor-2.4.2.jar --type css -o {} {}' \\;"
    end

    Those paying close attention will see that I don’t try to compress if the filename ends in -min. These files are already minified and are usually 3rd party libraries we are using.

  • Compress some more – Through minification the files being send down the pipe are smaller (which means faster to complete, all things being equal), but we can make them even smaller. All the major, modern browsers and web servers support transmission of compressed data. Here is how you do it for Apache and IIS.
  • Cache – Since client side issues are generally considered to have the biggest impact, any speed improvements were now going to involve code changes. The first of which was to see what the various calls were doing and see if we could prevent them from hitting anything. In this case we were generating an xml file our of an activerecord object and then converting that to json. And the result was always the same. Always the same is a big flashing red sign that says ‘cache here!’ so both the xml and json were put into memcached. This was actually the 2nd largest performance improvement and will pay off in huge dividends in he future. The trick here was to actually cache both items. Originally we planned on only doing the xml, but after walking through the problem on paper the json became an obvious candidate as well. I suspect there is some sort of ‘when dealing with caching, think in layers’ heuristic that I’m backing into realizing.
  • Thinking – This step I didn’t do as I’m not a developer; I just play one on teevee (as the expression goes). We had a call into the system which was taking 4 – 6 seconds on load and was about 80% of the load time after all the other tricks were done. Someone (a real developer) had to actually look at the code, figure out the problem and create a solution. In this particular case it was as simple as merging a change from a different branch over to the one in production, but she spent a couple days originally to address this particular problem. (And got the call into the 400ms range)

Aside from the first and last one, these are quick and easy wins for performance and have now been added to my initial testing checklists. I’m almost at the point of thinking that not implementing these is a pretty severe bug.

And of course, the big lesson in all this is Server tweaks and client delivery good practices will get you a fair distance, but ultimately pale in the face of a real human using their brain to tackle a problem.

Posted on April 29, 2009 in Uncategorized by adam1 Comment »

I finally got around to watching James’ Competitive Swashbooking video. Aside from being envious of the view from Pirate HQ, one of the things I picked up on was the repeat mention of anchors. I could have sworn it was used more often, but upon re-viewing it gets used only twice.

Jon uses the term in two ways: anchoring mission and anchoring paradigm.

So what is an anchor? I’m not sure if the Bach brothers have an ‘official’ definition, but I suspect it is a macro variation of the consistency heuristic we use in testing. Specifically, it is a thing or idea which all other things or ideas swing around.

When I was in highschool I took all the fine arts credits I could (something like 20% of my diploma consists of them), and part of the final year class all your work is towards creating a portfolio that had a theme. For example, the mini-portfolio that was done the first couple weeks of class was ‘trees’.

Like James in the competition, I just jumped in and worried about the theme later. Eventually I turned in my portfolio with the theme of ‘Things Adam thinks are cool’. Nice and broad, but also tied things together.

To keep with the nautical theme, you don’t need to have an anchor when embarking on a journey, but you need one by the time you need to stop the boat. James had his anchor airlifted to him (or perhaps he pillaged it from a passing ship), whereas Jon had acquired his before even leaving the wharf.

Some unanswered questions remain though after watching the video.

  • What was Oliver’s anchor for his judging? There certainly appeared to be some logic and pattern to his questions
  • Did Lenore have the same assessment of the presentations? Since some of the books came from her library, she is married to one of the contestants and is mother of the judge, she is not an un-invested stakeholder.
  • Did either James or Jon re-evaluate their anchors, and how their books were related to it after having to discuss and explain their findings?

So what is the take away from this other than an interesting way to spend a day if you have a huge amount of books available to you? I think it is this:

When doing any testing, you need to know your mission (why are you doing this testing) and your oracles (how do you recognize a problem). But before you can determine that, you need to know your anchor(s). And these are by necessity big, as in organization level. Google’s would likely be ‘organize the world’s information’ and ‘do no evil’. Everything falls out of there. But your individual anchors also have sway too, these are often called ethics.

Do you know the anchors that affect you and your testing?

Oh, and as an aside, when I started thinking about this I went along the lines of guidelines for number of anchors on a ship and their placement and deployment. I didn’t find that specifically, but the wikipedia article on anchors was pretty interesting nonetheless.

Posted on April 28, 2009 in Uncategorized by adamNo Comments »

I would guess that a large number of posts by testers (or geeks in general) about Twitter and Quality tear a strip out of them for their dubious availability and the now infamous Fail Whale. This is not most posts though.

Twitter is rapidly becoming a medium of choice of marketing. It’s cheap (free), has a low technological barrier to entry and has a wide (and ever widening) audience. I, like a number of people I know, have very broad mandates about where we are supposed to stick our noses. I’m not the Manager of Specific Product Quality, or even Manager of Software Quality. I need to worry about everything the company does and the image it projects in the marketplace. That is my preferred method of operation but also means that the Tweets from the corporate account also fall under the umbrella.

What concerns me about corporate use of Twitter is the complete fixation on the number of Followers the account has.

Bigger is better! Just look at the race between Mr. Demi Moore and CNN to a million followers earlier this month. But c’mon, is bigger really better? If you are a spam-bot, then absolutely; in a corporate account setting the answer is clearly no. We want Quality over Quantity.

I think mandatory reading for all corporate Twitter holders should be the 1000 True Fans essay. The main thrust of the article is the following:

A creator, such as an artist, musician, photographer, craftsperson, performer, animator, designer, videomaker, or author – in other words, anyone producing works of art – needs to acquire only 1,000 True Fans to make a living.

For the remainder of this post when you see ‘fan’ swap in ‘follower’.

True fans are the ones you want to nurture as they are the ones that are most likely to use your service or purchase your good.

Twitter has joined the rest of the internet and is full of spam, hucksters and auto-follow bots which will inflate your Follower count without you doing anything. These are not real fans, and certainly not true fans.

Fans you acquire from giveaways (‘Follow me by May 3rd and be entered in a draw’) are not true fans, likely not even fans and could be anti-fans if they don’t win or is a hint of impropriety in your contest.

Schemes which offer followers through a simple program are preying on the weak and narrow minded. I saw ‘new FREE tool to massively increase the number of Twitter followers you have!‘ today for instance. They might as well be selling email lists.

How do you get more followers? Perhaps not surprisingly, the process is very similar to normal SEO; participate in the conversation through appropriate and topical replies and re-tweets and generally provide compelling content. People will follow you and they have a much greater chance of being true fans.

You will also have high quality followers, if perhaps less in quantity than someone with thousands of people trying to sell you the newest natural ‘enhancement’ or swampland in Florida.

Posted on April 28, 2009 in Uncategorized by adamNo Comments »

On April 22, 2009 I gave an 80 minute talk at the KW SQA conference on Scripting Recipes for Testers. This was a shorter version of the talk I gave last fall at GLSEC. I was far more prepared this time round and think it was more successful in this format.

The big change I made in the process was actually writing up a paper then building the slides from that which is something I am going to try and continue to do. The paper can read here and here are the slides.

And here are direct links to posts I wrote about the individual scripts.

Posted on April 28, 2009 in Uncategorized by adam1 Comment »

David Leadbetter, golf instructor to the stars, was a guest on the radio on April 24th. He was there to talk about Tiger’s knee and such, but the conversation started with a question about whether he teaches a Method or Philosophy to his students.

Methods have come and gone and I feel like my thought process is I have a philosophy rather than a method. A method will work for certain players and certain people but we’re all different. No two of us are built the same, we don’t think the same, we don’t walk the same, our movements are different. So it’s very difficult to say ‘This is the method’. Just go look at every great player through history. Every player swung differently. Sure there is some similarities as far as the physics of the golf swing, and the geometry of the plane. We all look different. There is no way that two people could swing the same way. I remember, all the yars I was working with [Nick] Faldo, there were so many Faldo lookalikes, trying to mimic and copy his swing. It doesn’t work. So I’ve always felt that one has to tailor instruction to the individual. That’s why its difficult writing books and videos. Yes, people will get something out of them, but the fact remains that instruction has to be geared to each individual.

What a great blast of wisdom. Let’s apply it to testing.

  • A lot of the Great Debates that rage are largely around methods. And of course these methods are all used by most organizations but given greater weight or prominence depending on who is doing the arguing. The Green Bar is a method which helps embody the philosophy of having a set of constantly passing unit tests.
  • There is not a ‘the method’ for anything. There are a number of methods that may or may not be appropriate.
  • When someone has success, especially in a public forum, imitators appear out of the woodwork. What’s the old tagline? Often imitated, never duplicated?
  • Tailoring instruction is something I’ve been doing for awhile. I have a bunch of content I want to get through, and I recycle stories all the time between classes, but the overall delivery alters depending on the make-up of the class. Tech vs. non-tech, skilled vs. novice, etc..
  • Wrap what you want to communicate in a story. I’ve been told that the best part of my classes are the stories I use to illustrate what I want them to learn. Stories that involved me directly help establish my credibility with the audience and make the idea ‘real’ rather than ‘philosophical’. There is lots of information out there on using story telling; here is a recent interview with Karen Johnson about it and Brian Marick made a comment about it on Twitter a couple weeks ago as well.

After actually talking about Tiger and how his swing has evolved David brought it back to the original topic by saying

He [Tiger Woods] could probably make any method work if he believed strongly enough in it.

This brings up the notion of expert-ism and natural ability. Swap in any number of the rockstar testers for Tiger in the statement and it still rings true.

I don’t golf (time and money being the constraints rather than lack of desire), but I’ve already learned something from a golf pro. Not a bad trick.

Oh, and if you want to hear the whole interview, it is in the middle third of this mp3.

Posted on April 27, 2009 in Uncategorized by adamNo Comments »

I’ve used WordPress for this blog for over 3 years and have recommended it to countless people. It’s interface is quite usable and its 3rd party infrastructure is such that I’ve never really had to extend it more than just tweaking a template here and there. In short, I was quite happy with it.

But in the last month, that charm has faded. A lot. What happened? I had to make it jump through a hoop. In theory it should have been a nice, pliable hula-hoop, but instead it turned out to be an electric hoop, with spikes, and on fire.

WordPress can import posts from a number of different blogging platforms, one of them being another WordPress installation. I needed to take content from an internally developed cms-ish platform, store it in the WordPress format and then import it into a real WordPress installation. This should have been as easy as:

  1. Figure out how to extract information from current system
  2. Lookup the WordPress format in the Codex (the WordPress documentation)
  3. Import

What it ended up being was a lot of red herrings and gnashing of teeth.

And a re-enforcement of some ideas I’ve been carrying around…

  • Document, document, document – Documenting how to use code is a lot less fun than creating the code in the first place. The WordPress eXtended RSS (WXR) appears to be a giant black hole of information. What little there is exists solely in the blogs of people who have had to do a similar exercise as I have just done. I don’t care if it is generated by WordPress and so ‘no one will need to write it by hand’. If it is publicly accessible, document it. Accurately.
  • XML describes your data – The big idea to wrap your head around with dealing with XML is that it describes your data versus HTML which formats you data. What this means is that your XML can be presented as one giant blob or with nicely structured and it will be considered by the application as the same thing. This magic is possible because you parse an XML file, you do not regex each line as an individual line. But that is exactly what WordPress does. This means that you have to have, for instance, each category element on a single line, otherwise it doesn’t get imported. It also means that the actual order of things inside the file matters. PHP has an XML parser in the base language; use it. Heck, once you have XML being treated as XML you can do things like validate against a schema which goes a long way to help Document the format.
  • Consistency Counts – The consistency heuristic is one of the more powerful ones we use to find bugs. It is also one of the more important ones when you are looking at extending your system. There are 14 different systems WordPress can import from, and there seems to be at least 3 or 4 different ways of doing it. I understand that these were user contributed, but a number of communities dictate what patches need to look like before acceptance. The creation, announcement and promotion of consistency hoops is not only a good idea, it is a fundamental one.
  • Provide actionable error messages – Thank you for telling me that there was an error do to a configuration setting, but unless you tell me which error message and where I can find it, you might as well be silently failing. I lost a couple hours tracking down 5 or 6 settings and then figuring out where they were being overwritten so as to not take my changes.
  • Do not munge – Unless I explicitly say ‘mess around with my data’, do not change it. Turning string data to a consistent, known case (upper or lower) is a strategy I teach in my scripting classes to find values without having to worry about case. But when you use the data you found, use the original data. WordPress however takes your remote attachments (such as images) and lowercases them on import, regardless of how it specified in the import file. This in innocuous enough on Windows and Mac, but on enterprise unixes where case matters, you get a tonne of 404s.

Now, in the took-far-too-long run, I was able to succeed by reading the import and export code of WordPress which illustrates the utility of not only having access to the code, but knowing how to read it. But by having to dig into the guts of WordPress, its sheen has taken quite a tarnish.

Posted on April 17, 2009 in Uncategorized by adam1 Comment »

Once upon a time there was three friends; the Zealot, the Anarchist and the Robot. They were best friends and all worked together to produce high quality products. Until their differences got in the way.

The Zealot said he was the best, but the Anarchist disagreed. So the Robot designed the Documentation challenge. They worked the whole night through and in the morning the Zealot produced a wonderful, standards compliant, audit-able document with lots of citations and references. The Anarchist had produced a couple scribbles on scraps of paper he had in his pockets in a few short minutes then went to bed. The Zealot was declared the winner.

“I’m the Best! I’m the Best”, shouted the Zealot. But the Robot disagreed.

So the Anarchist designed the Coverage challenge. They worked through the night and in the end the Zealot had barely scratch the surface as coverage was not in their Book(s). The Robot create a large volume of work which covered a large part of the product in a repeatable, robotic way (as only a robot can do). Ignoring his true nature, the Anarchist declared the Robot the winner as it will, robotically, cover the product. (The Anarchist then spent the next hour or so trying to convince the Robot to forsake his Robotic ways and join the Anarchist cause).

“I’m the Best! I’m the Best”, danced the Robot. But the Anarchist disagreed.

So the Zealot designed the Exploration challenge. They working through the night and in the end the Anarchist had bounced around and followed hunches and had explored throughly. The Robot had reproduced their work of the Coverage challenge adding some extra hooks for dynamic inputs. The Zealot begrudging declared the Anarchist the winner as they did explore the most even through their process could not be encoded and etching in Holy Writ.

“I’m the Best! I’m the Best”, gloated the Anarchist in self congratulating glee.

Having reached a stalemate, the three friends went their separate direction in a huff.

And they thought.

And they thought.

The Zealot realized that while he was great at producing structure documentation, he wasn’t as good exploration or generating coverage. The Robot realized that he could generate lots quite, repeatable tasks he wasn’t as good at exploration or documentation (after all, the tasks are all the documentation is all that is needed). And the Anarchist also realized that their knack for exploration was countered by their lack of repeatability and documentation.

Having realized that they each have both their own strengths and weakness, the three friends decided that fighting was silly, acknowledges their differences, set them aside and went back helping each other produce high quality products.

The End.

Of course, this is a fable. The one thing I have learned from the Zealots, Robots and Anarchists in the testing community is that they are good at acknowledging differences but seem incapable of setting them aside for any great length of time. While is a shame.

It should also be pointed out that this is completely stolen from The Legend of Ninja Cowboy Bear.

Posted on April 14, 2009 in Uncategorized by adam1 Comment »

As I see it, there are (at least) 2 different types of belief that people have about their job. These beliefs directly affect how successful you will be as a tester, if not consciously, then certainly subconsciously.

  • Belief in Product – The core of this belief is rooted in the actual product or service you are testing. If you believe in its worth and value then you will want it to be the best it can be to achieve its full potential. This doesn’t mean that you must believe in a product to do a good job. What really drives that is your level of professional commitment and personal ethics.
  • Belief in Leadership – This is abstracted up a couple levels from the product itself. Do you belief in the leader (or general leadership) of the organization. We might all get scurvy, but the captain hasn’t let us down yet. This type of belief manifests itself when people join companies based on the founder(s) or leadership team.

To different people, the amount of one outweighing the other differs wildly. Do you have equity? Are you a contractor? Have you been burned by failed leadership? Are you battling burn-out?

For me, I’ve found that leadership trumps product consistently. But I know at least one person who believes the opposite. It is an interesting exercise though, to figure out why you are where you are; the product or the people steering it.

Posted on April 13, 2009 in Uncategorized by adam3 Comments »

I’ve been aware of the term ‘smell’ as related to software for (at least) over a year now. A smell in testing terms is an oracle; something fails the sniff test. We may not know exactly what, or why, but something just doesn’t smell right when you open the fridge editor.

Smells are useful.

Smells lead you to bugs.

But I realized there is a (near) fatal flaw to smells. The problem with smells is that they are trailing indicators. That is, they confirm something we already thought. Unit tests, green bar and high coverage are all trailing indicators.

What we need is a set leading indicators to help us treat software quality holistically rather than just a phase in a larger process. I don’t have a list to provide, but my gut makes me think that these are less about the code and more about the people that produce the code and the culture they operate.

Posted on April 9, 2009 in Uncategorized by adamNo Comments »

Illusions

We’ve got a new product coming out sometime in the next month and the head of a different part of the company gave it a whirl recently. In talking to him about his comments he got up on his soap box and started to yammer on about quality. What struck me was his firm commitment to the belief that he is the gatekeeper for the product. Here are two quotes to illustrate:

The <department> is implementing a series of Quality Control measures…

If my department says that it is not of sufficient Quality then it isn’t being released.

This is a common stance for new testers, so it is not surprising that someone who has a decent profile within a company might feel this way. I used to, but came to see the folly of that way of thinking.

While this conversation was happening it occurred to me that he is falling for the illusion that he is a gatekeeper. And then it occurred to me that you can replace the word quality with illusion in almost any text or discussion without it really affecting the context too greatly:

  • Quality Assurance == Illusion Assurance
  • Building Quality in == Building Illusion in
  • Quality through innovation == Illusion through innovation
  • Our 100% code coverage gives us high quality == Our 100% code coverage gives high illusion (ok, this is not really clean grammatically)

It is an interesting trick that people, including myself sometimes, need to be mindful of. Are you really talking about quality, or are you talking about the illusion of it.

Sheriffs

When later mentioning the conversation with my boss I used a sheriff metaphor to describe the illusion this person was falling for (in reference to him trying to police the product — QC is (almost) always a policing action). The sheriff is ultimately the person who can green or red light something.

I don’t debate the presence of a sheriff as there most certainly is one. Except he sits in the big office down in the corner and not in the pit with everyone else. I also don’t debate that the person I was talking to doesn’t have a badge as he most certainly does. Knowing the politics I would guess his is even shinier than mine, which again, is ok. What I do debate however is that the badge is stamped ‘sheriff’. It is actually stamped ‘deputy’. (Credit to Michael Glenn for the metaphor extension.)

And there is nothing wrong with being a deputy. That is our role. We provide information about the state of the world to the sheriff who then decides whether to act on that information or not. They could always take our suggestions / opinions or they could take it as a tiny piece of the decision making process. But it is ultimately they who hold the power.

Grassroots

Lets focus in on the power that the Sheriff wields for a second. Ultimately, they steer the ship (yes, mixing metaphors all over the place) and the deputies and regular citizens have to go where they lead (or we could jump off the ship, but in this economy its more difficult than usual)

It is a commonly held belief that the quality of a product can be dramatically improved if actual developers and testers working on the product care hard enough. And by care I mean doing things like TDD, code coverage, static and dynamic automation, manual exploratory testing. To some extent I think this is true. Teams that care do tend to produce better code and as a result better applications.

But.

These are all grassroots activities and they can all be undermined entirely by the Sheriff. If the Sheriff is not absolutely committed to having Quality products then nothing the developer team can do will be enough to produce Quality products. They might have bits that are of high Quality, or being Quality-ish, but not taken as a whole. Its impossible. Here are some examples:

  • Product to developer ratios of 5:1. Yes, that is product to developer, not the classic developer to tester ratio
  • Having all the developers working on a new product while ignoring the others. Yes, developers is plural there, which coupled with the first example means a lot of code is going unmaintained and customers unsupported
  • Contracting development of products which it has no programmer skill to support / maintain; even if there was capacity

Those are just the ones that are really bugging me (this week) and so come to mind. All of them dramatically affect the Quality of all the products across the company, and none of them are something a grassroots effort to improve would be able to touch. Unless the Sheriff is 100% dedicated to Quality and is willing to act (they are able to because they are the Sheriff) on it, then Quality is really just an illusion.

Next Page »