I’ve come to the realization recently that Selenium has a bit of a marketing problem. But its not the marketing problem that you might think. Getting mindshare is not a problem. What is a problem is communicating what I’ve been calling the ‘Selenium Value Chain’.
To the credit of the gang who put together the Selenium site, it does have all the components of the suite listed with numbers beside them, but it doesn’t seem to be working. My email, traffic to Se-IDE posts,talking to people at Agile the other week and the questions in #selenium imply that a lot of testers stick with the Se-IDE, often far longer than they should.
Here then is an explanation of the Selenium Value Chain, complete with illustration.
Before going into each part in more value, some quick notes. First, notice that there is a split now between what anyone can get for free (Se-IDE, Se-RC and Se-Grid) and commercial offerings. This has happened in the last year and is an important step in the maturation of Selenium as a solution. Also notice that this is a series of components. You could jump in anywhere in it, but the easiest route is to follow it in order.
I have no idea the download or subsequent usage numbers, but my impression is Se-IDE is hands down the most used component. And I begrudgingly admit that it likely should be so. Within limits. As Se-IDE stands right now, it is useful for
- Quick ‘smoke’ tests
- Easy bug replication
- Automation proof of concept
- Recording outlines for scripts to be completed in Se-RC
Those are all pretty useful things. But Se-IDE also has some pretty major downsides which limit it.
- Lack of logical decision syntax (the ‘no if’ argument) – Yes, you can have it eval() a block js and hide the logic in there, but even that is limited in application
- Lack of looping
- Cannot data-drive tests
- Cannot access the filesystem
- Cannot access databases/other datastores
For anything other than the most trivial ‘does it work end-to-end’ those are killer discrepancies.
I’m sure you noticed that ‘Recording outlines for scripts to be complete in Se-RC’ was bolded. That is where the true value of Se-IDE lays. Unfortunately people tend to stop there. As someone who has been using Se for ~ 4 years now I rarely actually save a script I record in Se-IDE. Instead, I export it to be used by Se-RC.
Again, repeat after me. Automation is Programming. Regardless of how slick your vendor’s tool is, if you are not programming while doing your automation efforts, then you are getting nowhere near the value you should / could be getting. Once you have exported your script from Se-IDE, you are programming.
Why is programming a Good Thing?
Put simply, you have the full language (Java, C#, Python, Perl, PHP, Ruby, Erlang) available to you. This means you get if’s, loops, access to the filesystem (for data-driving) and database access (for further validation). It also lets you abstract out common command sequences for better test organization and less fragile tests. These then form the beginning of a DSL for your application.
Great! Let’s go! But…
You have to know how to program.
You don’t need to be a guru. While automation code needs to be treated like production code, it is not production code. Also, Se automation requires only a small portion of what languages have to offer. That means you could take a 2 or 3 days course on Python and have enough to be an effective Se automation programmer.
As you get more scripts in your suite and the more configurations of OS/browser you need to test, eventually you start to run into scaling problems. At this point you should be move to the next step in the chain which is Se-Grid.
It seems pretty logical that if you have 10 tests that each take 2 minutes and you run them in sequence then your testing will take about 20 minutes. That is an easily swallowable time amount, but its even nicer if you could run them in parallel. Then it is only 2 minutes for a complete run, but that would require 10 machines though and controlling those is a pain.
Unless you are using dedicated software to control them; like Se-Grid.
What Se-Grid allows you to do is farm out your tests to a number of machines, often with the goal of decreased goal of execution time. While that is nice, I think the real value is in having a pool of test machines with all different configurations and then farm the tests out there to increase the breadth of testing rather than just the speed of testing.
The hitch is that suddenly you are dealing with the configuration and maintenance of potentially dozens of machines (virtual or physical) which is no fun at all. Fortunately, that is a problem that people have started offering as a commercial service in the cloud.
Commercial ventures like Sauce Labs and BrowserMob have built ways in which you can massively scale your testing and yet not deal with the headaches of maintaining a giant farm of machines.
BrowserMob started out in the niche of performance testing which is often only done a few times per release (at most) so it would be wasteful to maintain that infrastructure in-house. They have also moved into the monitoring space. Both are driven through Selenium.
Sauce Labs’ OnDemand service could be considered Se-Grid on Steroids and addresses both the issue of parallelized execution and configurations. According to their site, they support IE6, IE7, IE8, Firefox 2, 3.0, and 3.5, Safari 3 and 4, Opera 9.96, and Google Chrome on Windows, and Firefox 3.0 on Linux.. Even if you multi-purposed machines that at least 4 that you would need and even then would only be running at ~ 25% parallel efficiency.
Managed, out-sourced scalability is a big market that is ripe for exploitation. I expect there will be a few more vendors offering added-value services for Selenium over the next year as well.
That’s a quick run through of Selenium suite of tools, why you would use each, and in the order which you get the most value. The thing that too often gets missed is that it is an order, a progression, that delivers the most value. Don’t just stop at Se-IDE. Or Se-RC. Or even Se-Grid if that is the scale you are operating at.