Twitter is excellent for planting troll-bait one-offs, but the blog is better for planting long-tail ones. So this afternoon I tweeted something along the lines of this.
Unless your are interviewing someone for a job that requires [stupid] logic problems, then they don’t have any place in an interview.
And I stand by that. Though a lot of twitter seems to disagree, to one degree or another.
Context. I was at a client today and the two people I am working with came back from an interview…
Them: Do you like logic problems?
Me: Not really.
Them: Want to try the one we just gave in the interview?
Me: Sure…
Them: How long should we give him? 8 seconds? 5?
Me:
Them: Design an algorithm that solves
Them: Give up?
Me: Nope. I didn’t come up with something immediately and don’t care enough to focus any thought on it
Them: …
This of course led to a much larger conversation about the value using these sorts of things in an interview has. My mentioned above, I don’t think they have any. (Or at least very tiny minuscule amounts to avoid the use of an absolute…).
More context. The interview is for a position that will primarily be webdriver automation using python with some manual testing thrown in there as well (and will be taking over the stuff I put in place.)
So. How does the ability to answer a logic problem help determine the candidate’s ability to do the job? Very. Little. Here is a secret; web automation is not a ‘logic’ problem. It’s not even that hard. It is way easier to be an automator than a tester.
What should you spend the time in an interview for a web automator instead of wasting everyone’s time with the logic problems? Ummmm, how about you have them automate your app. Right there. For real. Projected on the screen.
- What oracles do they establish?
- Which patterns do they use?
- How ‘robust’ are their locators?
- How is their synchronization?
- etc.
These are all critical things to understand about an automator. Especially the one that will take the lead on the project. Notice a distinct lack of explanation why manholes are round or how to determine the odd billiard ball out in a minimum number of steps.
(Oh, and looking up syntax is absolutely ok. It shows they know where to look. We have this internet thing — we don’t need to memorize everything anymore. Now, reliance on an IDE to program for you is not ok.)
I see logic problems in an interview almost as a crutch that the interviewer is using to hide that they don’t really know what they are looking for. And as an interviewee, if a logic puzzle rears its head in an interview the bozo bit gets partially flipped on the whole situation. Do I want to work at an organization that values people who can solve logic problems? Or that are awesome automators and/or testers. (This isn’t to say there isn’t a possible intersection between the two groups, but it is not guaranteed.)
What are a few arguments I have received today in favour of logic puzzles; none of which sway me…
- It shows a high IQ and high IQ is needed for a testing role – People play lawyers on the internet, so I’m going to play neuroscientist. So first, all being good on logic problems means is … you are good at logic problems. A lot of people spend their free time doing these things so they train their brain to spot patterns and build their own solving heuristics for them. Same for crossword-ers or word search-ers. Heck. Look at the whole ‘tester game’ thing. (Of course, the tester game phenomenon has a large degree of arrogance associated with. And no, I won’t play.) Back on topic. No, you also do not need a high IQ to be an excellent tester. Again, I could be conflating IQ with other things (only playing neuroscientist remember…) but there are more firms hiring Autistic people as testers (article).
- It lets us see how they work through a problem(1) – Sure, but is it a class of problem they will experience at the job you are hiring for? Those are the problems you are hiring for.
- It lets us see how they work through a problem(2) – As usual, Michael Bolton provided a Feynman reference (even if it is fictional); Round Manhole Covers, or: If Richard Feynman applied for a job at Microsoft. Its a fun read and on target in capturing what I have seen of Feynman. But he gets a job offer for marketing. Not in test. Not in web automation. His answer would tell me nothing about his ability to do those jobs, though would tell me not to engage him in an argument about manhole covers.
A slight variation on this is of course, FizzBuzz. FizzBuzz is loved online and is somehow seen as the bar that should be met to be called a programmer. I have not once had to use anything FizzBuzz like in automating apps since I started in early 2000. (And that is of course the only experience I can have.) If you are hiring someone whose responsibility includes programming, absolutely you should have them write code. But have it be relevant to the job! As mentioned above, for a web automation job, have them write web automation code! And if you must use FizzBuzz, then you should write a web app that does it and have them automate it.
Way at the beginning I mentioned that the job that originated this discussion is mainly automation and partly testing. The automation side I think I’ve beat to death. Interviewing for testing is, perhaps unsurprisingly, very simple. Have them test! Logic problems will not help you determine if they have a set of heuristics and philosophy for mobile, or payment gateways, or video, or, or, or. Unless your product is vapourware (or super crazy top secret — at which point, have them test something else), have them test your real product! Again, live! And not just silently do it pointing out [possible] bugs but out loud explaining their thoughts. (I wouldn’t have them write bug reports as an interview is a fake scenario and you’re not going to get a ‘real’ bug report.) Flashing back to Feynman, this is the useful part of the article. The thinking and rationality behind the thoughts. As an interviewer, that’s what I care about. Does it clash with mine, does it create overlaps, does it fill capability holes, etc.
But you can’t get that from the creative process solving a logic problem! As a community we claim to care about context, but interviewing outside of context seems like a waste of everyone’s time.
Yes, it is a joke, but the Feynman story follows McLuhan’s thesis that all jokes are rooted in a complaint. The “fact” that “Feynman” gets offered a job in marketing supports your points about the ineptitude of the interviewer rather nicely, I believe. (There: a McLuhan reference AND a Feynman reference in the same post.)
I’ve had some amazing success using Jon Bach’s audition technique for “interviewing” candidates. I try and get a bead on their personality and communication style during the first phone interview. Then I send them a coding audition if the position requires coding. If I like what they give me, I have them come in and test the code they themselves wrote.
The surprising part of this equation to me is how many people with lots of letters behind their names and experience with some big companies can’t work backward and write test cases for their own stuff.
I was going to get a bit upset when you mention that automation isn’t that hard and testing is much more difficult. I then remembered how very hard good testing is, and realized I agree with you.
Overall this is a great post and I agree with more or less all of it. Companies always want to hire “the best” people for roles, but what about looking for great fits, regardless of whether they’re the “best” or not? Your comments on IQ and hiring nail this completely. Look for people that can solve the problems you have, not some nebulous concept of what you think should be done.
Wonderful, Adam. Thanks for posting.
FWIW, I too don’t like (or use) the logic puzzles either. I know some companies still use them, but I don’t think they know *why* they use them, or what they expect to learn.
There is (or may be) another aspect to consider. In my job, I interview for two completely different types of roles. Sometimes, I interview for a position – can the candidate be successful in *this* job or perform this series of tasks (which may include testing, automation, etc.). Mostly, however, I interview someone for the company – i.e. does this candiate have what it takes to be successful in the long term (definitely beyond whatever project they will initiall work on).
For the first role, I’ll focus on skills, attitude and team fit. For the second role, those attributes are also important, but I also need to know how they learn, how they adapt, how they figure stuff out, and all of the other things people need to be successful beyond the immediate project.
But I’ve rambled…
Bottom line is that I think interviewers often ask what they *think* they should ask candidates instead of forming questions based on what they really need to know about candidates. The gap between the two is larger than many people realize.
Some thoughts:
The sidebar about FizzBuzz really mystifies me. It is not a logic puzzle, it is a phone screening question. It is the programming equivalent of a tester coming up with a basic test case. Any candidate who could not complete it in less than three minutes is unlikely to be able produce usable automation in any reasonable amount of time.
While I agree that Aha! type logic puzzles are a poor choice of interview tools, some of the open ended ones (“How would you move Mt. Fuji?” perhaps being the most well known) can serve as an audition type question to evaluate creativity for people who have little prior experience in testing. Since the problem is non-technical in nature, there is much more common context to work with.
It’s not clear that the link on autistic tests you cite supports your thesis that IQ is unimportant. From the MIT article linked there, they report 7-8x accuracy for data entry tasks, +50% accuracy for following pre-written test cases, and +20% accuracy in ad hoc/systems integration testing. Not really consistent with what context driven testers do, to my mind.
Oh, to pick on Michael Bolton’s essay briefly, regarding “The people authorized to open manhole covers could easily be trained to do it safely.”, I’d think Feynman would be well aware of the propensity for people shortcut proper procedure; it was one of his defining characteristics after all.
I really enjoyed reading this article. I agree with the entirety of it. There are always trends in recruitment, be it having a certain style of CV, the interviewer asking certain questions in the interview (“Where do you see yourself in 5 years?”), and of course the numerous ways of making the interviewee jump through hoops while the interviewers clap and hoot for more!
If we turn it on its head, all a logic problem task in an interview shows is that the interviewer may not know exactly what they want and is just going with the current trend, as found on google.
Finally, @Curtis I really liked your idea of taking the interviewee and getting them to write tests on their own code in the interview.
[...] Logic Problems & Programming in Interviews Written by: Adam Goucher [...]
Still can’t agree with you.
I see how posing a logic problem (of which there are several sorts of course) can help you assess a person’s skills, provided that:
1. There is more to the interview than just the logic problem. Can’t you have a practical (code/test) something “Right now!” AND a logic problem?
2. The person attempting to solve the logic problem is required to explain his ideas and thoughts on how to solve the problem, what the variables are etc. WHILE he is solving it.
The reason for the logic puzzle – for me anyways – would be to assess the person’s ability to communicate/describe his work. What if you get a candidate who can solve any puzzle or practical exercise with ease, but can’t ask questions or comment on his work and reasoning?