I tend to sit in #selenium on the freenet irc network during the day and almost all the questions that get asked in there seem to involve the IDE component of Selenium. Either they are trying to make it just through a hoop that it can’t jump through, or it could go though, but the hole is really small and its on fire. Inevitably, both of those problems would can be solved by using Selenium Remote Control (RC).
Source of pain is a useful heuristic in sniffing out problems, and the amount of pain people (appear to) go through with the IDE would seem to imply a problem.
So what to do about the Selenium-IDE?
This morning I quipped on Twitter that the Selenium gang should reduce the functionality of the IDE. I think that the IDE is an absolutely necessary component of the Selenium suite, but only as an initial skeleton producer that would be consumed by RC. The first thing that could get removed to this end is the notion of Test Suites. The only time you should be running a test inside Se-IDE is to debug the flow of things initially; then you export into language of choice. Next up would Custom JS Extensions. Use modules / libraries / whatever-your-RC-language-calls-them instead. Finally, I would take a hard look at all the Commands available to you when inside Se-IDE. addScript and its ilk are all suspect. I’m not suggesting they be removed from Se entirely, just the IDE.
Patrick Lightbody (one of the core Se folk) countered instead that the solution might be to increase the power of the IDE. For example, to convert it from a (pure) Firefox extension to a Firebug plug-in which would make its default language Javascript which would add powerful scripting features like conditions and loops. That is interesting, but only solves 60% of the problem. The other 40% I don’t think can be solved by having script execution live entirely in the browser.
The real value proposition of RC is not that you get conditionals and loops out of the box (as compared to custom JS extensions), but that you can access things that a browser cannot. The database for instance. It’s not enough to just validate a form for the string ‘Update Successful’ when testing. You have to go to the database to verify the update. There are any number of failure modes otherwise. And you can’t do that from the browser. You also cannot write a file out on disk to archive your results. Or data-drive your tests. Or wire them up to other domains (due to same origin). Or run them (from IDE) in non-Firefox browsers. Or. Or. Or.
The big argument for Se-IDE is that you don’t have to be technical (pronounced ‘know how to code’) in order to get value from Selenium. I think this might be a bit of a stretch. Yes, you can get a browser to do stuff, but is it high-value testing? I’d argue not.
Repeat after me: Automated testing is a programming project. You will write code.
And that scares a lot of people in test roles and is vendors rarely tell that in their slick pitches to customers.
I suppose I should amend the previous statement a bit.
Automated testing is a programming project. Someone will write code.
The path to success with automation systems like Selenium appears to be to write a DSL (Domain Specific Language) that abstracts the technical parts away from the non-technical users. Fitnesse is likely a classic example of this I think, but if you watch the various GTAC videos, this is a common theme. But to get to the point where you have a nice non-technical DSL takes a lot of technical coding (and thought). And is likely harder than just writing against the Se API directly in Python or Ruby.
So what to do about the Selenium-IDE? There are three options: nothing, add functionality, or remove functionality. I think doing nothing leaves us in the weird limbo we are in currently, adding functionality addresses some problems but not all of them, and removing functionality pushes people up the value chain earlier but also forces a higher degree of programming skill. There are trade-offs in each, but I think the value Se get from being on RC (and later Grid) means greater long-term success in using Selenium even if it means a bit of short-term pain as the user base adapts / evolves.
Categories
- Android (2)
- BITW (5)
- Books (21)
- CAST2008 (2)
- GLSEC2008 (14)
- Housekeeping (22)
- Interviews (1)
- javascript (1)
- marketing (1)
- Podcasts (28)
- Process (23)
- Python (19)
- Quality (400)
- Ruby (7)
- s3cr3t (1)
- Selenium (10)
- Star West 2010 (8)
- subversion (1)
- Uncategorized (408)
- Video (54)
Find things by
- Browsing...
- Searching...
Miscellaneous
Books
-
Software Security: Building Security In by Gary McGraw -
Real Process Improvement Using the CMMI by West West
I agree that SIDE is allowing non-technical people to start automating and that those tests can be of little value. However, having those people generating a basic template test, since they are normally subject-matter-experts, and passing them over to technical, spelt can-code, and allowing them to beef it up, apply Data-driven aspects or reporting to disk/database and going through those things add a lot more value and will get a lot more people using Selenium.
I would go for the option of adding some of the missing functionality and after that have notices saying “Time to lose the training wheels” as I said on Twitter (http://twitter.com/AutomatedTester/status/3178734869). I think that dropping items from the IDE will give the same value as dropping intellisense from Visual Studio. You can get around it but you lose the newbies because it appears the learning curve is just too great.
I do understand all the issues that Selenium can cause since I get emails about it, but normally just coaxing those people to Selenium RC with a decent example gets them onto RC a lot quicker than just scaring them into doing it.
And finally, I think the idea of writing custom DSL is the right way to be going!
All in all a good article!
As usual, wonderful spot-on points, Adam.
However, I’ll side with Patrick here. I’ve been threatening for years to deprecate (or at least “un-default”) the current table-syntax of Selenium Core/IDE in favor of “Plain Old JavaScript” for the similar reasons you mention — getting people to realize they’re *programming* sooner. Regarding your “40% comment” though, yes, you can do more in Perl/Python/Ruby/Java/C#/PHP today, however, JavaScript in the browser — running as trusted browser extension– is getting more and more capable over time. And I don’t see that trend stopping. (Heck, look at the CrunchPad or ChromeOS — the main UI of the OS will be just a browser and JavaScript.)
One last note, if you want to see what life is like with something that uses Plain Old JavaScript to script the browser, check out ChickenFoot. http://groups.csail.mit.edu/uid/chickenfoot/quickstart.html
100% true, you really nailed the topic. SE-IDE is good for learning Selenium and getting yourself familiarize with the Selenium and also for Google Search examples.. 🙂
It is certainly a misconception that SE-IDE can be used for Production ready test code in enterprise projects, the maintenance cost is too high to implement SE-IDE for large scale projects.
All in all a very good article.
Cheers
Haroon
Very interesting article, especially as I’m a Selenium newbie. I fall into the “can’t write code” group so for me IDE is fantastic. To me it has a lot of functionality that I wouldn’t use. But it also has limitations which is why I installed RC and start learning how to write proper code, but only up to a certain level, after that I’ll hand it over to someone in the team who knows programming a lot better and can get results quicker.
Without IDE I wouldn’t even have looked at RC as the step from being non-technical to use something that you can only use with programming skills would have been far to big.
We use IDE for smoke testing – it doesn’t have to be too fancy but we do want to know when some of the major functions are impacted by a new build as has happened in the past. So going from a half an hour smoke test to Selenium which takes a minute to run is good enough for us. The indepth testing then comes later, which is currently a manual exercise. Maybe with more Selenium knowledge that will also change over time, who knows.
As a newbie I didn’t get the impression on the various sites that RC is basically the only way forward if you want to use Selenium seriously so maybe with very little change on the websites that could do more than removing or adding functionality.
Thanks for the article.
Thomas