Posted on October 22, 2007 in Quality by adam1 Comment »

This is a Canadian specific post, just a warning.

One thing that irks me is when people call themselves ‘Test Engineer’ or something similar. The key thing that bugs me is the whole engineer part. In Canada, only those with a P.Eng as licensed by their provincial body can legally call themselves an engineer.

Here is the appropriate quote from Engineers Canada‘s P.Eng site:

The licensing bodies have also mandated CCPE to develop other national guidelines on their behalf. Those guidelines promote greater consistency in the regulatory practices of the licensing bodies, which has led to full national mobility for licensed engineers within Canada. Engineering’s licensing bodies protect Canadians against the misuse of the title engineer by unlicensed individuals through a variety of means. In addition to initiating public awareness activities to inform engineering students, employers and the public about the requirements for licensure, and responding to complaints, through CCPE the licensing bodies hold Official Marks on the terms engineer, engineering, professional engineer, P.Eng., and consulting engineer, as well as their French-language equivalents. This gives the profession an extra legal tool in its efforts to protect the public

If you are a P.Eng, by all means, call yourself the title you posses. If you are not however, perhaps you should point out to HR that your title needs some work to be actually legal in Canada.

Posted on October 22, 2007 in Quality by adamNo Comments »

I’m pretty sure that at CAST Kieth Stobie let it slip during his keynote that there was some sort of testing community initiative in the works at Microsoft which would launch later in the year. I’m guessing that what he was talking about was the new Test Center in their MSDN site. The blurb Alan Page sent me says that it in conjunction with StarWest though I’m not sure why they would partner with them.

So my initial thoughts based upon an hour of poking around pre-launch?

  • I realize that the D in MSDN is Developer, but it irks me that yet again ‘tester’ is being lumped with ‘developer.’ Perhaps we should start a movement to change Developer to Development as we are part of the development process even if we are not developers.
  • I like the idea of trying to bring MS’s internal testing activities to a more public spotlight. Hopefully too we’ll see some of their internal test training videos appearing. Or maybe their tester development criteria.
  • There is already an article posted which is dated next month. Who knew that in addition to their OSes and Office suite, MS has a time machine division.
  • The page which has the Community blog posts is showing   instead of a non breaking white space. Oh the irony, a bug on the testing site.
  • The whiteboard video is a link to ‘get silverlight’ — how is that relevant?
  • shudder a ‘Best Practice’ article
  • Videos which are streaming only do not work without an internet connection (like on the train)
  • Hopefully there will be more people posting blogs that are linked to the Test Center as I already subscribe to the ones linked already
  • Changing my language / locale (in the top-right) should not take me to the main MSDN page, but just change the page content (if it has been translated) — well, if I was testing this for MS.

But the one comment I have about the site that I would like the folks at MS to pay attention to (thus being outside the bullets for attention getting) is the lack of RSS to pump changes to me via the fire hose. I don’t have the time (or patience these days) to hunt for updates I am interested in; just send them to me.

I’m not sure how often I’ll be poking around there (heck, I didn’t even know about developer centers until yesterday), but it is bookmarked. Get me RSS and I’ll be more involved.

Posted on October 22, 2007 in Quality by adamNo Comments »

Much like the art world has their isms (fauvism, impressionism, pointalism, etc), the testing world has their ilities (usability, accessability, maintainability, etc). Anne-Marie Charrett‘s blog (which has some very broken links, so if you get a 404 or similar, remove the www from the beginning of the url) introduces the notion of sustainability. Here is what she groups under this umbrella, but I’m sure there are others.

  • Sustainabile – how energy efficient is the application under development?
  • Peripherals – how well does the application perform using energy efficient peripherals
  • Monitor – Does the application function properly under “minimal power configuration”
  • Switching Off – How well does the application function when peripherals are switched off
  • Virtualization – How well does the software function in a virtual environment
  • Printing – What alternatives does the application provide to printing (PDF?)
Posted on October 19, 2007 in Quality by adamNo Comments »

Your product’s version control system can provide a wealth of information about the state of your product. I recommend that you nag your admin to add RSS notification for changes so they just stream out the firehose at you. In checking out a ‘fix’ just now, I discovered a new heuristic to apply to the information that comes out.

+2, 0

For those unfamiliar with CVS, this is how it reports that 2 lines were added, and none were removed. This is important information because that is how many lines it takes to add /* and */ around a block of code.

If a block of code is causing problems, there are two ways to fix it. Either fix it, or comment it out and hope it goes away. Now, you could say that there is a 3rd possability, which is to delete it, but that at least implies a fix. Someone has to conciously look at the code and determine that it is not worth the hassle to debug and nukes it. But a comment implies that they think it is valuable, but they do not know how to solve the problem (right now).

Of course, in a crisis situation it might make sense to comment out the troublesome part and come back to it later, but if that was to happen I would expect there to be a bug logged and the bug number placed in the comment.

Posted on October 17, 2007 in Quality by adamNo Comments »

It is not uncommon for many e-commerce type sites to store customer credit card data for ease of transaction history and auditing purposes. This is unfortunately the exact type of information a thief is looking for. It is equally unfortunate that it is yet another thing we testers have to keep in mind when verifying these systems.

It is not completely hopeless though. The Payment Card Industry (PCI) (Visa, Mastercard, American Express and JCB) has what it calls the Data Security Standards (DSS). Consider this to by your oracle when testing card data. Lifted from the Wikipedia PCI DSS page, the high-level objectives and requirements are:

  1. Build and Maintain a Secure Network
    • Requirement 1: Install and maintain a firewall configuration to protect cardholder data
    • Requirement 2: Do not use vendor-supplied defaults for system passwords and other security parameters
  2. Protect Cardholder Data
    • Requirement 3: Protect stored cardholder data
    • Requirement 4: Encrypt transmission of cardholder data across open, public networks
  3. Maintain a Vulnerability Management Program
    • Requirement 5: Use and regularly update anti-virus software
    • Requirement 6: Develop and maintain secure systems and applications
  4. Implement Strong Access Control Measures
    • Requirement 7: Restrict access to cardholder data by business need-to-know
    • Requirement 8: Assign a unique ID to each person with computer access
    • Requirement 9: Restrict physical access to cardholder data
  5. Regularly Monitor and Test Networks
    • Requirement 10: Track and monitor all access to network resources and cardholder data
    • Requirement 11: Regularly test security systems and processes
  6. Maintain an Information Security Policy
    • Requirement 12: Maintain a policy that addresses information security

Each of the requirements are flushed out in the Official PCI DSS Standard which is a free pdf.

So does this apply to you? If you store, process or transmit the PAN (Primary Account Number) then it certainly does. If you use a branded payment gateway you might be off the hook. If the PCI finds that this does apply to you and you are out of compliance, you might be in for some surcharges, fines and excess liability in the event of a breach.

Heck, if they do not apply to you now, they might later so you might as well start implementing parts of it. You could do far worse from a security perspective.

Posted on October 16, 2007 in Quality by adamNo Comments »

One of the problems you face when writing a metaframework is getting configuration information to the various appendages that it grows. In my case, I have got Selenium (Selenese), and Selenium RC (Python, Java, Ruby) to deal with. While this might seem easy since they are all Selenium based, their structure is quite different. Yesterday afternoon I stopped working around the problem and managed to get all my tests reading information from the same file.

Like most programming activities, my framework grew pretty organically. I knew eventually I would want some sort of config file when I started and when I needed it I just plunked it into the main script. As the processing got larger I encapsulated some stuff in function and eventually into it’s own module. (The framework is in Jython BTW) Yesterday those modules get wrapped in a class and turned into a Singleton.

In Python there appears to be two ways of more or less solving the single instance problem; the Singleton Pattern and the Borg Pattern. I initially played with the Borg one because it has a great name, but decided it was not the proper solution because as once the __new__() method completes it would execute the __init__ which is not something I wanted to do. (At the time I was overriding __init__ to read my config file, etc. but I’ve re-engineering things slightly so I think I could now use this pattern. Oh well, not going to change working code now.)

So here is my config file class which is stored in the module fmwrk_config. I mention the module here explicitly because I initially had it as config, but that clashed with a log4j class.

import xml.dom

class config(object):
    # singleton
    _instance = None
    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = super(config, cls).__new__(cls, *args, **kwargs)
        return cls._instance
    
    def configure(self, config_file):
        # parse config
        config_dom = xml.dom.minidom.parse(config_file)

        # environment
        self.environment = {}
        environment_nodes = config_dom.getElementsByTagName("environment")[0]
        for n in environment_nodes.childNodes:
            if n.nodeType == xml.dom.Node.ELEMENT_NODE:
                self.environment[n.tagName] = n.firstChild.data

        #users
        self.users = {}
        user_nodes = config_dom.getElementsByTagName("user")
        for user_node in user_nodes:
            for el in user_node.childNodes:
                if el.nodeType == xml.dom.Node.ELEMENT_NODE:
                    if user_node.getAttribute("id") not in self.users:
                        self.users[user_node.getAttribute("id")] = {}
                    self.users[user_node.getAttribute("id")][el.tagName] = el.firstChild.data

Inside the main framework, I read in the config file path from the argument list and create the file

# argument handling
try:
    opts, args = getopt.getopt(sys.argv[1:], "c:", ["config="])
except:
    usage()
    sys.exit(1)
# make sure we have our config file
config_file = ""
for o, a in opts:
    if o in ("-c", "--config"):
        config_file = a
if config_file == "":
    usage()
    sys.exit(1)
else:
    if not os.path.exists(os.path.abspath(config_file)):
        print "config file (%s) does not exist" % os.path.abspath(config_file)
        sys.exit(1)

cf = fmwrk_config.config()
cf.configure(config_file)

All tests are currently launched from within the main script (a later refactoring will make script ‘types’ pluggable) with the Selenese ones building a test_suite.html dynamically based upon a regex directory crawl. Python based Selenium RC scripts are done through the standard unittest module where a suite object is populated in a similar way as the Selenese tests.

This is what initially caused this problem since the Selenese tests used the config file for runtime information (like usernames and credentials) but the Selenium RC ones could not get access to them. But now I have the following in each test classes’ setUp method.

self.cf = fmwrk_config.config()

Which gives me all my config information so I can put things in tests like

self.selenium.select_frame("RouteOneFrame")
self.selenium.type("Login.Token1", self.cf.users["id"]["username"])
self.selenium.type("Login.Token2", self.cf.users["id"]["password"])
self.selenium.click("realLogon")

So what are the advantages of all this?

  • It was fun; the advantage of which cannot be overstated
  • I can store all configuration information in one spot instead of embedding it in different places for different types of tests
  • And what has the most potential upside is that tests can now be grouped by function rather than by user as they currently are. Having tests grouped by user made them easy to debug, but spread the work out in a manner that didn’t always make sense
Posted on October 13, 2007 in Quality by adamNo Comments »

For the last couple weekends I’ve been building a garden shed in the back corner of my yard to move ‘backyard things’ from the garage and into the backyard where they belong. (To make room for the yet to be purchased dirt bike :)) The hitch in the plan is that the corner where the shed is going is quite sloped so I’ve had to build a platform for it to sit on.

Building things that are attached to the group in places where the ground freezes involves one of two processes. The first is getting a hole digger and some concrete and putting it below the frost line. That works well if you have a hitch on your car and don’t have firewood stacked up in the walkway from the driveway to the yard. The other way is to use 4′ metal stakes which you bash into the ground. I chose the bashing route.

There are two styles of stake you can get. The first is a single piece, and the other is 2 pieces and allows the part that holds the 4 x 4 to pivot side-to-side and around 360 degrees. I chose the pivoting model because I didn’t trust my ability to bash them in perfect alignment. Seemed a reasonable choice at the time.

While this is all interesting, what does it do to testing?

Because I had an option between something that allowed a bit more configuration over something that did one specific thing in a specific way, I went for configuration. The problem is, that this extra bit of configuration weakened the structure of the stake and through the course of bashing it into the ground, the stake bent and flattened around the pivot bolt. This means that not only does it not pivot anymore, but it is stuck at whatever angle it ended up at. Oh, and I paid extra for this functionality.

Still not seeing where this is going?

If your application has extra configurability (sometimes called ‘advanced features’) which allow your users to shoot themselves in the foot or makes it less stable, you should question whether if it should actually be there. I would so so far as to say that extra functionality which makes something less usable, stable, valuable should be labeled as a bug and removed.

Posted on October 12, 2007 in Quality by adamNo Comments »

Most of the presentation support materials (aka slides) have been posted to the files section of the GTAC google group. If you don’t have that bookmarked, here is the direct link.

Posted on October 12, 2007 in Quality, Video by adam1 Comment »

All the videos from this year’s DefCon (what is DefCon?) can be found on YouTube through this search. In order to test the way the bad guys are going to use your application, you have to think like them.

And yes, I know that it is not just black hats in attendence. But everything the white hats present in the name of further expanding the field of computer security is going to be taken and further by the black hats. Thats just the nature of the game.

Posted on October 11, 2007 in Quality by adamNo Comments »

This is just hitting every funny receptor I have.

« Previous PageNext Page »