One place I see Selenium users stubbing their toes on time and again is around automating HTTPS connections. I’m about to tell you that you should save yourself a lot of pain and just not do it.
Having a secure connection in production is an extremely good idea. Especially at places like checkout and authorization. But in testing? Not so much.
There is exactly one situation where you need to be running HTTPS in testing, and that is if you are using certificates as an authentication means. For every other test scenario, in test, you can run things just as HTTP. Not only can you watch the network traffic in the clear over HTTP but you don’t have to worry about the trust issues certificates introduce.
And there is almost always trust issues.
Why?
Because ‘real’ certificates cost money. The whole notion of security is based around trust, and the way to establish that trust is through money and verification. If you look in your browser, it will list somewhere all the CA (Certificate Authority) root certificates that it trusts by default. Those vendors pay non-trivial amounts of money to get in there. And then they charge non-trivial amounts of money end consumers to sign a certificate with their root certificate. Production sites will (almost) always go that route since your customers are likely to be a little concerned about all the browser warnings about things being wrong with the certificate.
In test though, machines are often rebuild and used for multiple purposes so companies are loath to spend the money on real certificates. When manual testing, this isn’t anything more than a nuisance dialog to dismiss, but when automating, it is much more annoying.
The problem is that the error box is not part of the application, but is part of the browser. In Selenium 1.x there are some workarounds, and I believe Selenium 2.x and WATiR solve this by low-level control of the browser.
The easier solution though, is to just not use HTTPS in the testing environment. Sure, your testing machines won’t be as ‘production-like’ as they might otherwise be, but you have to ask yourself, does that materially affect the quality of your testing?. In most situations, it won’t. In fact, I would argue that it increases the breadth of testing available to you; both manually and in an automated fashion.
edit: To be clear, the workaround mentioned are just some of the possible solutions in Selenium 1.x for dealing with HTTPS pages. There are a lot of situations where it will Just Work. Especially in RC where the server is a proxy and certificates get injected on the fly. I would still say in those cases to turn off HTTPS though; it is really hard to figure out what is going on at a network level when it is encrypted.
Categories
- Android (2)
- BITW (5)
- Books (21)
- CAST2008 (2)
- GLSEC2008 (14)
- Housekeeping (22)
- Interviews (1)
- javascript (1)
- marketing (1)
- Podcasts (28)
- Process (23)
- Python (19)
- Quality (400)
- Ruby (7)
- s3cr3t (1)
- Selenium (10)
- Star West 2010 (8)
- subversion (1)
- Uncategorized (413)
- Video (54)
- What I Learned Yesterday (3)
Find things by
- Browsing...
- Searching...
Miscellaneous
Books
-
Systematic Process Improvement Using ISO 9001: 2000 and CMMI by Harvey Stromberg -
Best Practices in Software Measurement: How to Use Metrics to Improve Project and Process Performance by Manfred Bundschuh
Great advice, Adam!
[…] written better ourselves about the first and most important advice regarding HTTPS and testing: Do yourself a favour and don’t test using HTTPS Share and […]
Load testing is one case where it is worth taking the step of setting up https in a test environment. There is a CPU cost to https, so if you want an accurate test, you should set it up. At load test time, you can set up a self-signed certificate for free, you just need to make an exception for that cert, or have a good tool that handles it with no problem. Or, you should be able to use your cert from the production site for test purposes, actually, as long as the exception gets handled.
Self-signed certificates:
http://www.tc.umn.edu/~brams006/selfsign.html
Here is a link to an older paper doing this comparison:
http://iweb.tntech.edu/hexb/publications/https-STAR-03122003.pdf
Summary: “Once the server is saturated, the system performance of HTTPS achieves around 67% of HTTP in terms of throughput.”
Re load testing: Quite true! There are also likely a number of other scenarios just like that, but in the general case, having the test environment ‘like’ production (with https), is not worth the pain it can cause.
Hi Adam, I agree with what you are suggesting, but sometimes the test environment is out of our (tester’s) hands. For instance, when I used to test a Financial web app, our “staging” (test) environment was set up to mirror the production environment – right down to the https. There was no way we (the test team) could change that to http. We just couldn’t.
Thinking back, there were a few bugs that we specifically found in the test environment that the developers didn’t catch because they used http on their local (i.e. desktop) environments. Off the top of my head, it was probably 3-5 out of 10,000 or so bugs found over 5+ years. Not a terrifically high ratio that would justify using https. Oh well. Like I said – out of our hands.
Cheers! Paul.
Adam,
Great practical advice. Thanks for sharing. I’m retweeting it. Please keep these pragmatic bite-sized pearls of wisdom coming!
[…] on 2010/01/23 Adam Goucher (whom some of you met at the code sprint) has a new post up about the pain of testing with Selenium and HTTPS. I’m sure he’d be happy to answer […]
All our development slices have valid certificates, valid wild-card certificates, and this has improved our development and testing significantly. Testers get a slice of their own with source access and, again, valid certs. I can’t imagine working at a company that did things another way. The value for cost spent is well worth it.
Ian
PBworks Head of Quality
The only thing that costs money is to create a certificate that all browsers will trust by default. However this is not required in most cases.
You can create your own certificate authority for free using openssl. Then, as long as you can pre-install that CA in your test browser(s), all your servers whose certs are signed by that CA will work just fine.
In the case of firefox/selenium, you just have to install the CA on one browser, and copy the profile to whereever it’s needed.
In many cases you will find bugs that do not occur with http. Each app is different and testers should weigh the risk with their app, instead of taking blanket advice to skip SSL testing.
It’s not that much money though. And it pales in comparison to the amount of money spent in terms of hours-spent on the automation so I label that excuse as a complete cop-out.
And yes, creating your own CA is possible, but the time spend managing profiles, etc. quickly overtakes the cost of just getting a real certificate.