I dove back into Selenium on a week or so ago, and part of that meant getting the nightly build. And much to my dismay, the Ruby client is not packaged as part of it.

Bug 1: The gem binary is not part of the nightly build

So I had to check the tree out of subversion to make it myself.

Bug 2: The repository url is wrong

On the OpenQA site, the url specified is for openqa.org, but seems like things are migrating to a new domain, so you need to use

https://svn.seleniumhq.org/svn/selenium-rc/trunk/

instead.

Once I got the code I wandered over to the ruby client directory and tried to figure out which Rake task to run. rake -T is the standard way to have rake parse it’s Rakefile and tell you the tasks. Except, you need to have built the system before it would tell you that.

Bug 3: rake -T doesn’t work

Okay. Fine. Build the whole thing (mvn install) then go back to the ruby client directory and build it (rake gem). Finally you need to install the gem. Just make sure that you specify the correct version of your gem in the script

This is more of a feature request, but if you are building a gem over-and-over I can see your gem store getting pretty cluttered. To partially address this versioning issue, I propose

Bug 4: version of gem should be timestamped in addition to version number in non-release builds

So I admit to not being too active on the forums so I may be blowing smoke here, but Selenium seems to be losing traction in the market right now. I don’t think we’re at the turn-off-the-lights-when-you-leave phase, nor do I think it will ever really come to that given the number of existing users, but from a going forward perspective I think there might be some pain.

I use Selenium as part of larger tools that I build. I might buy a tool if it did everything I wanted, but like to build my lightsabers. Let’s follow the the Star Wars analogy for a bit.

• Leadership – At the top of the Jedi heap is the Jedi Council who lead and guide the rest of the herd and in that group there is a single, identifiable, recognized leader. Watir has Bret Pettichord, but Selenium has …? We need someone to come out and take ownership of Selenium, coordinate the parts and releases and plan out the roadmap.
• In-person gathering – The Jedi Council would often get together to work on problems and assign tasks either in person or using remote presence technology. Skype, IRC, forums, etc. are all great, but nothing beats in-person communication. I seem to think that the Selenium developers got together at Google once, but that sort of even needs to happen more often; twice a year maybe? The Watir community is having an event January which is not necessarily just the developers, but I would bet that the core group will be there.
• Corporate Support – All member planets / systems in the (Old) Republic contributed to the expenses of the Jedi. Watir now has the backing of WatirCraft. Selenium has…? Well, ThoughtWorks is where it came out of, but are they actively promoting it or employing core members of the team? Google poached Jason Huggins from ThoughtWorks and seemed like it could be the corporate sponsor, but he left. So who is there?
• Outsourcing – When things got tough and the Republic needed a lot of (relatively) cheap bodies to do the grunt work, they called in the clones. (Well, sorta. But it could be argued that the outsourcing much like the clone army often backfire). But what offshore outsourcing firms have Selenium expertise? Most I have seen / talked with are HP/Mercury shops that if pressed will tell you they have a couple people on staff with some clue. But clue to what level? I think this is a potentially untapped market just ripe for exploitation
• Consulting / Training – Aside from ThoughtWorks, I couldn’t come up anyone who is billing themselves as a consultant or offering training on selenium. (PushToTest does to some degree, but that is in the context of their TestMaker product). Look at the industry around Mercury toolset. Training and value-add is often cited as the path to success with open source, so where is it with Selenium? (Sorry, no Star Wars metaphor on this)
• Books – There is a Short Cut on using Selenium, but that seems to be about the extent of commercial documentation. A consultant who has written the book on Selenium could make a decent living I think.

So how do we address these? Or do we want to? (And now I have to go re-subscribe to all the Selenium mailing lists since I seem to be jumping back in.)

For awhile now I have been advocating that building an automated test is a four phase process. The phases are:

1. Record
2. Add checks
3. Data drive
4. Make it smart

In this post I’ll illustrate these steps in automating our One Minute Calculators using Selenium. Because this is tutorial in nature, not all aspects are automated; just enough to illustrate. It is also a huge post, so it is behind the break rather than make the main page huge.

(more…)

It seems that the posts that get the most attention these days are those that deal with Selenium. I’m not sure how I feel about that, but here is another one. Here is an ant file which will check a bit of environment stuff, launch the selenium server, run my tests (a custom metaframework), and then stop the server. Previously I was just running this whenever someone asked me to, but it was decided that it should be run as part of the nightly build which is controlled through ant.

<project name="Run Test" default="run_test" basedir=".">

    <!-- sensible defaults; override as necessary -->
    <property name="config_file" value="c:\path\to\framework\config.xml" />
    <available file="${config_file}" property="config.ok" />

    <!-- see http://adam.goucher.ca/?p=21 for an explanation -->
    <property name="test_filter" value="ro_cas" />

    <target name="run_test" depends="check_jvm, check_config" description="Start Proxy ; Run Tests ; stop Proxy">
        <antcall target="start-server"></antcall>

        <exec executable="jams.bat" >
            <arg value="-c ${config_file}"/>
            <arg value="${test_filter}" />
        </exec>

        <antcall target="stop-server"></antcall>
    </target>

    <target name="check_jvm" depends="get-jvm" unless="jvm.ok">
        <fail message="You need to use at least Java 1.5 for the selenium tests" />
    </target>

    <target name="get-jvm">
        <condition property="jvm.ok">
            <not>
                <or>
                     <equals arg1="${ant.java.version}" arg2="1.4"/>
                </or>
            </not>
        </condition>
    </target>

    <target name="check_config" unless="config.ok">
        <fail message="The config file you specified does not exist" />
    </target>

    <target name="start-server">
        <!-- this is a 'known location' for this -->
        <java jar="../server/selenium-server-1.0-SNAPSHOT-standalone.jar"
              fork="true"
              spawn="true">

            <arg line="-proxyInjectionMode"/>
        </java>

        <waitfor maxwait="30" maxwaitunit="second">
            <and>
                 <socket server="localhost" port="4444"/>
                 <!-- this url will 403, so we say that it should start counting errors at 404 to skip -->
                 <http url="http://localhost:4444/selenium-server/core/index.html" errorsBeginAt="404"/>
            </and>
        </waitfor>
    </target>

    <target name="stop-server">
        <get taskname="selenium-shutdown" 
            src="http://localhost:4444/selenium-server/driver/?cmd=shutDown"	
            dest="result.txt" ignoreerrors="true" />
        <echo taskname="selenium-shutdown" message="DGF Errors during shutdown are expected" />
    </target>
</project>

Like most things these days, I can’t claim the original work for everything involved; I just the following together:

• Selenium RC and CI for the base outline
• Testing for the JVM version to make sure that a JVM that will run the server is in use (the product this is for is still using 1.4)
• The ant docs for all the niggly little bits

File this one under ‘I wish someone had written this post already.’

Today I started exploring why my Selenium tests were not working in IE. Until now I have been able to get away with using Firefox and still be able to execute scripts even though we technically only support IE but the only feature that is a good candidate for automation in this last release doesn’t work in FF due to heavy javascript. Switching the browser from *chrome to *iehta should just work but nothing is ever that simple.

Right away I encountered an issue where selenium claims that it could not find my user-extensions.js even though the path I handed it was correct. A bit of searching showed that this has been a problem for at least 11 months…

Anyhow, 3 hours later I have produced a patch which solves the problem. It is in the but report and below the cut.

Some lessons learned:

• Open-source codebases are an absolute mismash of styles – The chrome and iehta launchers are significantly different in style both at a code level and philosophical level. In production software I would log this as a bug and hopefully would be picked up by static analysis tools and code reviews
• Knowing how to code is increasingly important for testers – Okay, I knew this one, but it reinforced it. I could have waited forever for someone to fix this, but I did it myself. Just imagine how cool the tools would be that we could be using if everyone who used them could fix problems and add features.
• Being able to Build your own lightsaber is critical
• Firefox, being the darling of geeks, gets all the lovin’ when it comes to feature completeness – I’m amazed that passing in a custom user-extensions file to iehta wasn’t a huge priority for people. Again, in production software this would have been run up the priority list quite quickly.

(more…)

I just finished testing part of an application which basically retrieves information from the database and displays it to the operator. The complexity in this situation comes from ensuring that the correct information set gets retrieved and displayed in the correct place.

At first I wanted to use the other portion of the application to create my own test data as I don’t know the origin of what is already in the database, but I couldn’t get it all wired up correctly. Plan B had me pulling my test data directly from the database (which sorta means Oracle was my oracle). Fortunately, or unfortunately this database hasn’t been refreshed in quite some time so there was a tonne of information to choose from.

Too much actually.

This got me thinking, how could I get the database to feed me a single data point at a time? A few minutes digging led me to SQL to Select a random row from a database table. A few minutes more and I had my search SQL tweaked to provide new test data on every refresh.

Very cool from a sitting-at-a-sql-prompt perspective. Crazy cool if you were to rig things up so your selenium scripts where being driven by this sort of thing.

Chalk it up to diving into the deep end without a life ring, but it took a bit more work to get Java to use Python objects than it did the other way. I did eventually cobble together a solution based on various fragments dribbled around the interweb. There is a tonne of examples on how to use Java in Python, but there seems to be a hole for the inverse. Sure, there are Simple and Efficient Jython Object Factories, but to me they are neither simple, nor efficient. Simple and efficient would be importing a module and things Just Work&tm;.

Here is a simple JUnit test which checks whether a link is available to a certain user. Note how the url the Selenium server connects to is build from information in the python cf.environment dictionary. Likewise, the username and password come from cf.users.

import org.python.util.PythonInterpreter;
import org.python.core.*;
import junit.framework.*;
import com.thoughtworks.selenium.*;
import java.lang.System;
import java.lang.String;

public class ro_tmp_dsp_mnge_0002 extends TestCase {
    private Selenium browser;

    public void setUp() throws Exception {
        // get ourselves the config_file object
        PythonInterpreter interpreter = new PythonInterpreter();
        interpreter.exec("import ro.config_file");
        interpreter.exec("cf = ro.config_file.config_file()");

        // connect to the selenium proxy
        interpreter.exec("env = cf.environment");
        interpreter.exec("url = '%s://%s:%s' % (env['protocol'], env['host'], env['port'])");
        PyObject pyUrl = interpreter.get("url");
        String url = (String)pyUrl.__tojava__(String.class);
        browser = new DefaultSelenium("localhost", 4444, "*chrome", url);
        browser.start();

        // get our user info
        interpreter.exec("user = cf.users['setupuser']['username']");
        PyObject pyUser = interpreter.get("user");
        String username = (String)pyUser.__tojava__(String.class);
        interpreter.exec("password = cf.users['setupuser']['password']");
        PyObject pyPassword = interpreter.get("password");
        String password = (String)pyPassword.__tojava__(String.class);

        // login
        browser.open("/Web/myProduct");
        browser.selectFrame("LoginFrame");
        browser.type("Login.Token1", username);
        browser.type("Login.Token2", password);
        browser.click("realLogon");
    }

    protected void tearDown() throws Exception {
        browser.stop();
    }

    public void testTabNotAvailableToSetupUser() throws Throwable {
        assertFalse(browser.isElementPresent("link=Manage"));
    }

}

Now a Java programmer can add selenium based tests to the metaframework without having to know python (other than to parse and process the config file — which is painfully easy).

If I come close to be a ‘programmer’ in any language, it is in Python. Actually, I teach Jython which has all the simplicity and power of Python but lets me reuse 12 years worth of Java code when I don’t feel like re-inventing the wheel.

So, by choosing Jython I get: Python and Java in one bundle.

Part of my job is to automate testing where I think it will be of future benefit (or where it lets be be lazy at some point in the future :)). This automation often takes the form of Selenium scripts. But because I do not like working within the limitations of Selenium Core, I jump straight to Selenium Remote Control so I can make better use of oracles, logging, temp files, etc. The common paradigm for writing Selenium RC scripts is through the languages *unit framework.

So, by choosing Jython I get: Selenium scripts using Python’s unittest module and Java’s JUnit in one bundle.

Jython needs at least version 1.4 of Java in order to function, but works iwth 1.5 and 1.6. At one point I had myself convinced that I needed to run the same version of Java in my metaframework as our application was running, but I have overcome this bit of nonesense. By doing that I allowed myself to start running with 1.6 which means (a lot of things, but most importantly) I can make use of JSR-233: Scripting for the Java Platform.

JSR-233 provides hooks for (currently) 25 scripting engines to be accessed from within your native Java (Jython) code. This is incredibly trick. Want Groovy? Sure. How about Awk? It’s there too. In my context however, what I want is Ruby which I can do via JRuby.

Let’s catch up again. By choosing Jython I get: Selenium scripts using Python’s unittest module, Java’s JUnit and Ruby’s unit::test module in one bundle.

In order to get unit::test based scripts to behave, you used to have to jump through some hoops. Those hoops have in the last month been removed, so you will want to get the JSR-233 code from at least 12/12/2007. Many thanks to Yoko Harada on the JRuby mailing list for not only identifying, but providing fixes for the hiccups I experienced trying this.

Now we most certainly have Selenium scripts using Python’s unittest module, Java’s JUnit and Ruby’s unit::test module in one bundle. Which means there is now very little reason for developers to not only produce unit tests for there application / business logic, but Selenium ones for the web / presentation tier as well.

For those wanting the Jython code which will initialize and execute a JRuby script, here it is.

import javax.script.ScriptContext
import javax.script.ScriptEngine
import javax.script.ScriptEngineManager
import javax.script.ScriptException

m = javax.script.ScriptEngineManager()
r = m.getEngineByName("jruby")
rs = open("c:\\\\temp\\\\my_selenium_file.rb", "r")
y = r.eval(rs.read())
rs.close()

This is of course, a very simple example, but it works. Oh the power of the letter J. Python code executed in a Java interpretor which then creates an instance of the Ruby interpretor and runs Selenium commands written in Ruby.

Update 12/13/2007: Ruby unit::test scripts should just work with the current jsr-233 code

This post is more of a heads-up and spider bait for anyone who has mysteriously had their Selenium RC scripts start crashing. It appears that the Skype Firefox extension (which is installed by default with a current install) will cause lauched Firefox or Chome instances to crash before even getting to the TestRunner page. No idea why, but uninstalling the extension seems to have fixed the problem.

My brain hurts, in the peculiar way when you are trying to work out a problem which you know is going to bite you later if you get it wrong. Here is the problem I have been working on and what I think might be the solution.

Problem: I am working on a Selenium scripting framework which will run Java, Python and Ruby Selenium RC scripts as well as ones written in Selenese from the Selenium IDE. Making things dynamic in one of the RC scripts is “easy” as you have the full language behind them. The Selenese ones notso much.

• The application these scripts is for has multiple environments to go through before getting into production, but I only have to care about the first 4.
• The only difference between the environments from a scripting perspective is the login process: different protocol / host / port / username / password. Once you are into the applicaiton proper, things are the same.
• In order to run these scripts (which are “discovered” by the framework), you have to start the Selenium Server with the -htmlSuite option, which means that for a script to be run, it has to be wrapped in a test suite. Not a problem, we can create one dynamically in the framework
• Since we’re making the test suite in the framework, we can inject the correct login script as the first script

But how to pass in the pass in the the username and password values in a way which means the actual scripts are not tailered to one environment? (Hopefully the problem description makes sense).

Solution(?): This is what I have come up with, which unfortunatly is also the most complicated solution (but might buy me a performance improvement a bit down the road)

1. Framework discovers a Selenese script
2. Framework looks in the test for the “id” of the user which is going to be logged in

   <!--start config-->
   <tr>
    <td>store</td>
    <td>superuser</td>
    <td>userId</td>
   </tr>
   <!--end config-->

   I’m using ‘store’ because the IDE lets you easily add it which will ease training. I’ll check into CVS a read-only template script which has the config section already done.

3. Framework gets the rest of the user’s information from the config file via the id

   <!-- users -->
   <users>
    <user id="superuser">
     <username>adam-super</username>
     <password>password5</password>
     <type>super</type>
    </user>
    <user id="english fs">
     <username>adam-jnh-ca-fs-0002</username>
     <password>password1</password>
     <type>fs</type>
    </user>
   </users>

4. Framework creates the test suite with 2 scripts: the login script and the actual test. The parameters needed for logging in will be passed into the login script.

   <html>
    <table>
     <tr>
      <td>IP Filtering (Jonah)</td>
     </tr>
     <tr>
      <td><a target="testFrame" href="login_jonah.html?userName=adam-super&password=password5">Login to CAMS</a></td>
     </tr>
    </table>
   </html>

5. Framework looks in it’s configuration to figure out which environment it is working in to get the protocol information

   <!-- environmental stuff -->
   <environment>
     <protocol>http</protocol>
     <host>jonah.r1dev.com</host>
     <port>9080</port>
   </environment>

6. Framework launches the script
7. The login script does something along the lines of

   <tr>
    <td>type</td>
    <td>Login.Token1</td>
    <td>${userName}</td>
   </tr>

8. Then we are at the generic part of the whole mess where scripts “will just run”

Alternative Solutions I rejected:

1. I had thought about using the include user extension to include the login script in each test, but rejected that because
   • It breaks the Selenium IDE
   • I still have to pass in the various values in the suite
   • Using the above solution I could later collect all the Selenese scripts relating to a certain user and run them as a batch instead of at once
2. Embed the login values in the login script using ‘store’, but that means
   • I have to constantly modify files that are stored in CVS atrifically making it look like there have been revisions
   • Avoid point 1 by copying them to a temp dir, but what if the script crashes? They won’t get cleaned up properly
3. Embed the login values in the login script in a comment
   • see above
   • The Selenium IDE overwrites the file when you save it with the new version so anything that would not have been nicely produced by the IDE is wiped out

Is there another option I haven’t thought of or great gaping flaw in what I’ve come up with? I’m coding this up tomorrow.