One of the traditions in my wacky subset of the testing world is not only building mnemonics, but adding / remixing them. And so, I introduce COP FLUNG A GUN. Which is COP FLUNG GUN plus Automation.
The primary way to test mobile apps remains to be, literally, by hand. But we’re starting to figure out how to do it though automation. (No thanks to the OS vendors…) As automation frameworks get more mature we should check whether automation hooks are provided by the developers. Unique ids or other identifiers for all interaction bits, synchronization points exposed, etc.
Also included in this is integration into your CI system, since that is the bit that controls the execution and reporting of the automation.
Find things by
One of the traditions in my wacky subset of the testing world is not only building mnemonics, but adding / remixing them. And so, I introduce COP FLUNG A GUN. Which is COP FLUNG GUN plus Automation.
There are certain things that I can say have been with me most of my life. The Dark Tower series is one of them. (The Wheel of Time series, Star Wars, comicbooks have as well. Likely baseball too now that I think about it.)
So last night I’m paying ish attention to twitter while watching the world series and see this flit across my stream.
Wow! Finally finished Stephen King’s Dark Tower series. Those are long books. Enjoyed it, and liked the ending.
— Corey Haines (@coreyhaines) October 28, 2012
Which got me thinking about the whole series. Yes, the whole series. This is the point where my world-builder’s-disease kicks in. See, the Dark Tower is not just the ‘core’ seven books. It is in fact pretty much all his books. If it had a wizard, it is a Dark Tower book. If it had a lost puppy sign, it is a Dark Tower book. If it happened in Derry, it is a Dark Tower book. And yes, The Stand is absolutely a Dark Tower book.
But does the Dark Tower deserve all the praise it gets? I dunno.
For pure scope and intent, absolutely. But that almost feels like it was added after the fact. Or at lease communicated afterwards. Anyone reading King long enough understood his ‘universe’ held Derry near its center, but then suddenly it was part of something else. And his books would have asterisks next to books that were part of the Dark Tower. If it was always his intention to have things part of a larger epic, how come they were not there in the books that came out in the 80s and early 90s?
As for the individual books, on the whole I didn’t really enjoy them. The Gunslinger is a decent read, the next two feel like they are just moving the plot along and the flurry of the last three after his accident feel like a person’s sudden response to facing their mortality and fearing their epic won’t get completed. His inclusion of the accident in the final book seemed heavy handed and too breaking-the-fourth-wall ish. I literally stared at the page in disbelief when that happened. It is an interesting thought experiment to see how it all would have turned out had it not been for that fateful day…
‘Wizard and Glass’ however is an outstanding book and likely my favourite King book (with The Stand (unabridged) being a close second). Its enough of a standalone story that even if you have no want to read the whole series, it is worth a read. ‘The Wind Through the Keyhole’ is much the same tone and feel as ‘Wizard and Glass’ though outside of the main series. Chronologically it takes place before ‘Wizard and Glass’ but is another standalone book.
The more I think about it, the more I feel like the Dark Tower series proper has failed yet the world it has setup has succeeded greatly. ‘Wizard and Glass’ is character backstory set in the world. ‘Keyhole’ is character backstory about the world. ‘Hearts in Atlantis’ is secondary character backstory, etc. Similar to how the Terminator and Star Wars franchises were flushed out through Dark Horse Comics in the 90s, the Dark Tower is being flushed out by Marvel Comics.
What I think I would like to see however is more ancillary stories come out set in the Dark Tower world. Maybe not even by King, kinda like the Star Wars universe. (See The Thrawn Trilogy for instance to see how other can expand a universe while still staying true to it.) But now that Roland has been eight years since Roland reached the top of the tower and ‘Keyhole’ only came out this year gives me great hope that the world will continue to produce things to consume my money.
Other random bits in my head around the Dark Tower…
- ‘The Stand’ is a pretty ambitious book to choose for your first highschool book study
- I read ‘The Drawing of the Three’ (or started at any rate) during a school trip to Stratford to see ‘As You Like It’ and was listening to The Cure’s ‘Mixed Up’ so certain songs are forever associated with the book (such as ‘The Forest’)
- After reading the last page of ‘The Dark Tower’, I immediately went to this page
I’m starting to get back into comicbooks thanks largely to the reboot of the Valiant Universe. I have almost all the original Valiant books in boxes in the basement and I think the reason I was drawn to them was its tight continuity within the universe. Events in one book directly impacted, and did not contradict events in another.
The ‘big’ universes of Marvel and DC have long had issues with continuity due to their sheer size and age. I suspect managing this problem is part of the reason why DC rebooted their universe last year (dumb, dumb move if you ask me…) and Marvel is apparently thinking about it as well. That is not to say that its not possible to have working continuity in a universe that large, and with more than one title per character (Batman has 8 titles right now). My example here is the classic Knightfall crossover (which Batman Returns is loosely based). It was across all the then Batman books and, well, was excellent. And in tone for each of the books. There wasn’t much bleed over into the other books in the universe though… but oh, did Batman bleed.
(What I need right now is a word that means something like hindsight, but for recognizing the choices that might have led the the current now.)
While reading Cem’s The Oracle Problem and the Teaching of Software Testing it dawned on me that in my brain, ‘Continuity’ and ‘Consistency’ are synonyms.
Wiktionary provides a narrative device in episodic fiction where previous and/or future events in a story series are accounted for in present stories as one definition of continuity and Freedom from contradiction for continuity.
So how do I think Valiant did originally, and should now, keep their continuity in tact? Glad you asked! Or didn’t and I’ll say it anyways.
- Keep the number of titles manageable. And my manageable I mean in the area of 16. Now I have no idea the economics of the publishing world are like, but that could give a couple ‘mini worlds’ that could tightly collaborate sacrificing the whole. (Harbinger, Shadowman, X-O, A&A).
- Don’t let the groups of books dictate direction. Where the universe is heading should be a Publisher level decision. The v1 universe had that direction available for anyone to see in big-scope-terms in Rai 0 (best cover of all time)
That’s it. Only two things.
Tying this back to testing/agile, I am starting to notice [now that I am looking for it…] this sort of failings in teams. Teams trying to tackle waaay too much work in a given iteration and when features hit production they don’t work as well as they should have when get into their hands. Happens. All. The. Time. I’ve also seen when agile teams or test groups wag the product dog based on their own whims and biases. Management is supposed to do that. Not you.
(Unless the name card on your desk says ‘publisher’.)
Beyond the Matrix is an excellent article for a number of reasons. Not only does it give a peek to how a major movie came together, but it also gives insight into the difference between writing a book and making a movie.
Here is a paragraph from page 7.
The set was rudimentary: the control room of the satellite-communication center would be completed with computer-generated imagery, imagined by the Wachowskis down to the minutest detail. The scene in the control room, for example, features an â€śorison,â€ť a kind of super-smart egg-shaped phone capable of producing 3-D projections, which Mitchell had dreamed up for the futuristic chapters. The Wachowskis, however, had to avoid the cumbersome reality of having characters running around with egg-shaped objects in their pockets; it had never crossed Mitchellâ€™s mind that that could be a problem. â€śDetail in the novel is dead wood. Excessive detail is your enemy,â€ť Mitchell told me, squeezing the imaginary enemy between his thumb and index finger. â€śIn film, if you want to show something, it has to be designed.â€ť The Wachowskisâ€™ solution: the orison is as flat as a wallet and acquires a third dimension only when spun. Mitchell, who had been kept in the loop throughout the process (and has a cameo in the film), was boyishly excited by the filmmakersâ€™ â€śgroping toward exactitude.â€ť â€śI was like Augustus Gloop in the Wonka factory,â€ť he told me. â€śIâ€™ve witnessed a long sequence of decisions, which I never had to make while writing a book. Intellectually, I know itâ€™s a replacement, but I donâ€™t feel a loss at all.â€ť
It is this sort of detail that appeal’s to my tester’s mind. And one which I have been noticing since hearing an interview with L. E. Modesitt, Jr. on Writing Excuses (I think it is this episode, but didn’t double check) where he discusses things like the support logistics of great fantasy battles that authors seem to forget [ignore].
And now, you will notice them too.
My son is presently baseball-mad and so went to training day with some Blue Jays last January as part of their Winter Tour. I have no idea how to coach baseball to kids, so I took some notes. They’ve been floating around the basement since then so I’m de-paper-ing them by putting them here.
- butt kicks
- side strides
- long steps
- frankensteins (high kicks)
- flamingos (balance on one foot, lean forward)
- circles with arms
- fall, then sprint
- cross legs, then up without using arms then sprint
- watch the pitcher, then sprint
- Use a tee
- line up with the front of the box so if the ball drops straight down it doesn’t hit the plate; if it does its a strike
- pinch the bat
- rotate feet (into ‘athletic position’)
- Base Running / Paths
- touch at the front of the bag
- run straight!
- lean when crossing the plate to make it seem like you are further ahead than you actually are
- game: relay where half run from home and half run from second
- the high-5 at the bag is important
- when going multiple bags, start running in the shape of a banana halfway to first
- take two steps from the bag, square them, then two side steps to be in optimal position
- Infield Fielding Drills
- direct grounder
- glove side
- throwing side
- random side
- Outfield Fielding Drills
- routes to ball
- glove positions (above or below hips)
Twitter is excellent for planting troll-bait one-offs, but the blog is better for planting long-tail ones. So this afternoon I tweeted something along the lines of this.
Unless your are interviewing someone for a job that requires [stupid] logic problems, then they don’t have any place in an interview.
And I stand by that. Though a lot of twitter seems to disagree, to one degree or another.
Context. I was at a client today and the two people I am working with came back from an interview…
Them: Do you like logic problems?
Me: Not really.
Them: Want to try the one we just gave in the interview?
Them: How long should we give him? 8 seconds? 5?
Them: Design an algorithm that solves
Them: Give up?
Me: Nope. I didn’t come up with something immediately and don’t care enough to focus any thought on it
This of course led to a much larger conversation about the value using these sorts of things in an interview has. My mentioned above, I don’t think they have any. (Or at least very tiny minuscule amounts to avoid the use of an absolute…).
More context. The interview is for a position that will primarily be webdriver automation using python with some manual testing thrown in there as well (and will be taking over the stuff I put in place.)
So. How does the ability to answer a logic problem help determine the candidate’s ability to do the job? Very. Little. Here is a secret; web automation is not a ‘logic’ problem. It’s not even that hard. It is way easier to be an automator than a tester.
What should you spend the time in an interview for a web automator instead of wasting everyone’s time with the logic problems? Ummmm, how about you have them automate your app. Right there. For real. Projected on the screen.
- What oracles do they establish?
- Which patterns do they use?
- How ‘robust’ are their locators?
- How is their synchronization?
These are all critical things to understand about an automator. Especially the one that will take the lead on the project. Notice a distinct lack of explanation why manholes are round or how to determine the odd billiard ball out in a minimum number of steps.
(Oh, and looking up syntax is absolutely ok. It shows they know where to look. We have this internet thing — we don’t need to memorize everything anymore. Now, reliance on an IDE to program for you is not ok.)
I see logic problems in an interview almost as a crutch that the interviewer is using to hide that they don’t really know what they are looking for. And as an interviewee, if a logic puzzle rears its head in an interview the bozo bit gets partially flipped on the whole situation. Do I want to work at an organization that values people who can solve logic problems? Or that are awesome automators and/or testers. (This isn’t to say there isn’t a possible intersection between the two groups, but it is not guaranteed.)
What are a few arguments I have received today in favour of logic puzzles; none of which sway me…
- It shows a high IQ and high IQ is needed for a testing role – People play lawyers on the internet, so I’m going to play neuroscientist. So first, all being good on logic problems means is … you are good at logic problems. A lot of people spend their free time doing these things so they train their brain to spot patterns and build their own solving heuristics for them. Same for crossword-ers or word search-ers. Heck. Look at the whole ‘tester game’ thing. (Of course, the tester game phenomenon has a large degree of arrogance associated with. And no, I won’t play.) Back on topic. No, you also do not need a high IQ to be an excellent tester. Again, I could be conflating IQ with other things (only playing neuroscientist remember…) but there are more firms hiring Autistic people as testers (article).
- It lets us see how they work through a problem(1) – Sure, but is it a class of problem they will experience at the job you are hiring for? Those are the problems you are hiring for.
- It lets us see how they work through a problem(2) – As usual, Michael Bolton provided a Feynman reference (even if it is fictional); Round Manhole Covers, or: If Richard Feynman applied for a job at Microsoft. Its a fun read and on target in capturing what I have seen of Feynman. But he gets a job offer for marketing. Not in test. Not in web automation. His answer would tell me nothing about his ability to do those jobs, though would tell me not to engage him in an argument about manhole covers.
A slight variation on this is of course, FizzBuzz. FizzBuzz is loved online and is somehow seen as the bar that should be met to be called a programmer. I have not once had to use anything FizzBuzz like in automating apps since I started in early 2000. (And that is of course the only experience I can have.) If you are hiring someone whose responsibility includes programming, absolutely you should have them write code. But have it be relevant to the job! As mentioned above, for a web automation job, have them write web automation code! And if you must use FizzBuzz, then you should write a web app that does it and have them automate it.
Way at the beginning I mentioned that the job that originated this discussion is mainly automation and partly testing. The automation side I think I’ve beat to death. Interviewing for testing is, perhaps unsurprisingly, very simple. Have them test! Logic problems will not help you determine if they have a set of heuristics and philosophy for mobile, or payment gateways, or video, or, or, or. Unless your product is vapourware (or super crazy top secret — at which point, have them test something else), have them test your real product! Again, live! And not just silently do it pointing out [possible] bugs but out loud explaining their thoughts. (I wouldn’t have them write bug reports as an interview is a fake scenario and you’re not going to get a ‘real’ bug report.) Flashing back to Feynman, this is the useful part of the article. The thinking and rationality behind the thoughts. As an interviewer, that’s what I care about. Does it clash with mine, does it create overlaps, does it fill capability holes, etc.
But you can’t get that from the creative process solving a logic problem! As a community we claim to care about context, but interviewing outside of context seems like a waste of everyone’s time.
I have come to realize in the last two weeks that there are two words in the Agile lexicon that really bug me. Not because they are bad per se, but that they are so commonly misused and abused. Not to mention being insanely context sensitive.
I’m doing what might be considered an agility assessment for a company and am entering the report generation phase after a number days of observation and data gathering. There is nothing too ‘OMG! MY PANTS ARE ON FIRE!!!’ but one thing that sticks out is their working definition of ‘Done’.
I don’t think anyone who has played the role of Agile coach or consultant has not written a blog or paper at some point on ‘Done vs Done Done vs Done Done Done’.
This company has four different teams which are pretty autonomous and effective and one of the things each have done is define their own understanding of Done. While this seems like a good idea on the surface it is really quite dangerous. With each team having their own definition of Done there can be no common understanding when something is Done and that information rolled up with other non-contextual information to management.
Having the development teams generate the definition of Done also means it is limited to their scope of influence, care and interaction. The net result is a very naive scope of Done. Stories that are ‘done’ frequently will go out into production without adequate analytics hooks or means of monitoring the health of the component being created.
This nicely dovetails to the other most dangerous term; The Whole Team. I was at an automation workshop this past weekend and the people there spent almost 45 minutes talking around this subject. In retrospect no one asked what everyone meant by ‘the whole team’. This is another area I suspect Agile teams need to work better on. I would bet that most people’s working definition of the whole team includes the developers, the testers and someone in some sort of product ownership role (either directly or in proxy). But what about the analytics team, or the sales folks who use the output from the analytics team? Or the Ops team, including the DBAs that need to push the code out onto the iron? How about the customer support people who deal with the brunt of the code team’s mistakes?
The list is not endless, but it is large. One thing I like from the Lean Startup [Cult|Movement] is the the story isn’t done until it in production getting validated by real users. The user is on the team whether they realize it or not. We’ll just gloss over the lack of humans in the release process…
I think there is actually some sort of circular feedback loop at play between Done and Whole Team and you cannot address just one in isolation. Instead, really figure out who your team’s stakeholders are and what skin they have in the game. Only then do you have a chance, and only a chance, of coming up with an appropriate, inclusive, accurate definition of Done.
I’m most of the way through a client on-site engagement to do an ‘assessment’ of where their testing processes are and to suggest tweaks to how they do things. I have some broad recommendations that will morph into blog posts over the next month or so, but right now I am challenged by coming up with something that doesn’t sound too much like a consultant-y dodge around ‘When will our [functional] automation remove the need for our extensive manual regression testing?’.
This is of course a trap.
There is no way to determine this number with any degree of certainty.
But what if you really needed to provide an argument around the inability to predict this? My current, best argument around this is that we, as humans, we are really bad at predicting the future.
- Just because you are having a certain degree of automation success now, does not mean that the rate of success will continue at all let alone along the same trajectory.
- Is the competency of the developers going to change? For better? For worse?
- Is the competency of the testers going to change? For better? For worse?
- Will the market fundamentally change?
- Will the types of stories that come into the development groups remains such that there is a high degree of [perceived] value from automation?
All of these are completely out of control of, well, everyone. Yes, even the ones about the competency of staff. Sure, you can change the way you hire and the way you train, but you can’t change the fact they are people and people are messy. There are however things that we can at least fool ourselves into thinking we have some degree of control over. Though not accuracy…
- This particular client is migrating two older application stacks to a new one, so the source of some of the regression risk (the older stacks) will be removed from the equation (replaces, of course, with a new set of risks but the plan is they will be less).
- With [better] exploratory testing will regressions be caught during ‘feature’ testing?
- If the developers adopt a TDD approach to code generation rather than ad-hoc post-coding security blanket unit testing, will that decrease regressions?
- Will the internal focus on a single browser allow for more focused testing (with breadth covered though a testing partner)?
- Will proper management of environments lead to less false positives, less re-testing and/or idle testers?
Because we cannot predict the output of any let alone all of these things, which are all inputs into the equation that pops out a date for that magic point where we could be a half-step away from Continuous Deployment (no humans) then ‘tomorrow’, ‘July 23’, ‘Q3 2013’ are all correct. And all wrong.
But if we think of each of these are part of the overall system that results in regression duration that is deemed unacceptable and/or too long then we can address them each individually there is likely to be improvement in the overall system. To what degree and by which contributing measures is however anyone guess. And only a guess.
So this graphic is making the rounds today.
And while it is interesting (at time of posting, Selenium has crossed QTP and the trends are going the ‘right’ way), it is important to also remember this:
Donâ€™t confuse slamming your competitors with improving your own product.
— Mike Monteiro (@Mike_FTW) April 1, 2012
The Un-Conf is a common event type these days; no scheduled speakers, just a bunch of smart people and a space to facilitate conversation. The success of these is entirely dependent on who is there. And there is the classic Testing Conf where speakers are selected, assigned slots and talk about things directly related to Testing.
But what I think I want a Un-Testing Conf. This would would be the same format of a classic conference but without topics about Testing.
Obviously about testing.
I am sick of going to Testing conferences and hearing the same sort of stuff over and over. Or something that I could quickly pick up from any number of industry related books or blogs. I want to learn new, mind blowing things. And do you know where those things hide out? Hint: its not in the testing world!
Tangent. Have a single track conference; 5 or 6 concurrent tracks is too much.
So what would my Un-Testing Conf look like? Well, it would likely be expensive since you can’t just comp the speakers a ticket and expect them to graciously accept the invitation. Currently the line-up would be something like…
- Mike Monteiro – Design Is A Job
- Hugh MacLeod – Social Objects
- Austin Kleon – Steal Like an Artist
- Dirk Hayhurst – Something baseball-y
- Mary Robinette Kowal – Puppetry
- Someone who has been in the International Space Station
- Thomas Mahon – Customer Service (or something)
- Sian Beilock – Choking (or how not to)
- Mark Cuban – (Don’t know what I’d like to hear him talk about, but he is usually pretty fun and has interesting stuff to say)
- Chris Blais – Dakar, and now
- Roz Savage – Ocean Rowing
I can, without even trying, explain how each and every one of the above people have important things to say that would make testers better at their job.
That is a full two days of speakers already. Who else would you like to see? The only ‘rule’ is they cannot self-identify as a Tester.