I’ve known Chris McMahon through the Internets for a number of years, but only met him in person at Agile 2009. His talk was immediately after mine and seemed to be one of only a few purely experiential sessions: this is what the situation was, this is what we did and this is what we learned in the process. I like this session a lot more than pure workshop ones, though I’m beginning to see their value too. So even though I know a bit of inside-baseball into the Socialtext automation rig, I still got over two pages of notes. Not to mention the paper that he mentioned exists somewhere that I’m sure he’ll add a link to in the comments. (Won’t ya, Chris…)
- Manual test cases looked the same-ish as their automated counterparts
- Tests were named after features
- Manual tests were note there for regression purposes, but as placeholders for future automation
- Poverty of Oracle problem – automation reports only one problem
- Tip – Break out of the frame to increase testability
- Once the manual test is automated, throw out the manual one
- There is a need for both canned data and data created through the tests. Canned data is for when you have a large required data set for a test
- Lewis Carol poems make great (string) seed data (especially for search)
- If you are creating data, make sure to make it unique through something like Julian timestamps
- Weave a ‘web’ of tests through your application
- 90/9/1 Rule – 90% of people read content, 9% comment on it, 1% create new content
- Smell – All automated tests share the same initial path through the system (a branching strategy)
- Smell – At a certain number of steps (~200 in Socialtext’s case) the script is too large to (easily) diagnose script failure causes and needs to be refactored
- Ran scripts end-to-end, continuously
- If a script fails, run it again immediately and only on second failure mark it as failed. (But don’t lose the fact that you ran it the second time)
- Scripts can have manual prompts in them for human inspection points. Speeds up screen traversal and environment handling
- Four points for efficient, long-term automation
- Ability to abstract into fixtures / dsl
- Strive for feature coverage
- Build a web of tests
- Have fast, routine reporting of failures
- The only reason to automate is to give expert testers time to manually test
Hi Adam – there’s a topic for an entire blog post in each of your bullets!
You state: “There is a need for both canned data and data created through the tests. Canned data is for when you have a large required data set for a test”
This is spot on. I’d add that the level of functional complexity under test also dictates whether you should use canned data or test-generated data:
Deciding whether to separate tests and data for automated testing purposes depends on the complexity of your functionality. If you have simple functionality (for example, Google search) then it makes sense to separate out the tests from the data: a series of simple tests can iterate over vast data sets.
If you have complex functionality (For example, complex entities within your system which go through state transitions and can perform actions on other entities in your system), separating tests and data is not recommended.
Some of the ‘data’ are entities which must be created in the system. Even if you separate the data from the tests, you would have to create these entities in the system anyway. So ultimately you have to do the same amount of work but in two different places. It’s difficult to keep these coordinated.
For example, if a test needs a suspended user, you have to add code to the ‘test’ module to create a suspended user for this test. Then the other properties of the suspended user have to be listed in a ‘data’ file, which will be read by the test. This doesn’t really make sense! It is much better to create the user and set their state and other properties as and when the test requires it.
This is mentioned in a blog post I’ve written: (shameless plug) http://www.fluidlogic.org/post/2009/03/30/Writing-solid-test-plans
(Kudos to Chaminda Peiris for helping me to clarify my thoughts on canned-versus-generated data.)