Designing a great technical test experience
Your goal, as an interviewer, is to find out whether a candidate will provide more value to your company than what you will pay them. This “value” is not only their current skills, but among other things, how well they complement your team culture and how well they can learn new skills.
In this article, I hope to convince you that your current technical test process is likely too narrow-focused, and you’re missing crucial information about your candidates because of it. I lay out some ideas to redesign your tests so you’re more successful at finding the people who will add real value.
The problem: there’s not enough time to do a perfect job
In an ideal world, you’d have a team of professionals, able to spend a few weeks with each candidate, working on real work, but not interrupting your normal business operations. Then you could easily tell if they’ll be a good hire.
But you don’t have a team of hundreds available, nor are candidates willing to work for weeks for free. So we are forced to use relatively blunt heuristics to predict whether a candidate is going to work out.
Among other things, we want to know: does this candidate have the right technical skills to join our team? And one way we do this is the technical test.
The wrong solution #1: fizzbuzz
You’ve probably heard of fizzbuzz, but I’m referring to a whole class of tests designed to answer whether the candidate can program at all. Sadly, there are a lot of candidates that can’t, and they’re time consuming to process.
Fizzbuzz-style tests give you very few data points to work with. All you find out is whether they can program (or use a search engine). It’s also a turn-off to good developers, who rightly see your interview as a reflection of your internal culture. If you’d hire them off a fizzbuzz test result, they’re worried about the skill level of your other employees.
The wrong solution #2: computer science questions
There are some tech companies where a deep knowledge of computer science is required for some roles, but chances are that it’s not your organisation.
Your candidate will never, ever need to write a sort algorithm from scratch. They need to have an intuitive feel about the complexity of an algorithm, but it’s not important they know whether it’s O(n) or O(n log n).
Remember, you’re trying to work out whether the candidate is going to add value to your organisation. All you find out from these questions is whether they can remember their CS course, and you’re unnecessarily selecting against people who didn’t go through a university education. If you think CS knowledge and being a good programmer is exclusively correlated, you haven’t worked with enough people from non-CS backgrounds.
The wrong solution #3: passing an automated test suite
There are a myriad of online services, such as HackerRank, that will offer a generic problem for your candidates to solve and score them based upon a hidden test suite.
While this solution is superior to fizzbuzz or CS questions, it’s missing the collection of two crucial data points:
- Does this candidate care about code quality and elegant solutions?
- Is the candidate going to be able to work with the technical challenges we have in our organisation?
You can write code that passes a test suite, and it can be horrible code, but it’ll pass just as well as an elegant solution. However, a coder who isn’t concerned about their code quality will destroy a project, both in terms of its commercial value and any happiness you may have built up in your developer team.
Additionally, you’re still stuck with the problem that writing a tic-tac-toe solver doesn’t really determine whether they’re going to be able to add features to your enterprise .NET application.
The wrong solution #4: a take-home test
You’ve come up with a test that will demonstrate the skills that the candidate will need to use as part of their job. Excellent! We’re definitely getting closer.
But you’re asking your candidates to spend a considerable amount of their own time, unpaid, to work on your test. Even if you don’t have ethical problems with this, not everyone has the luxury of unlimited amounts of time and energy after their current day job to do the best work they can. You’re selecting for younger people who have fewer commitments, and that can affect diversity and experience in your team.
The wrong solution #5: pair programming, in house, as part of the interview
You’ve got a test that’s relevant to your business, and you’re pairing with the candidate in house to see how they react to feedback and how quickly they pick up suggestions and new concepts. Spectacular work!
Only that some people find the pressure or strangeness of an in-house test unbearable.
Or maybe they’ll be superb at working in your business domain, but they need a few weeks of mentoring before they’re up to speed.
And now you need to invest 2+ hours of a senior developer’s time to test every candidate that looks promising…
OK, I get it. Everything is the wrong solution. So what do you suggest?
I admit I’ve led you down the garden path a bit.
The reality is that all the approaches above can work fine.
The problem isn’t the method of testing, but the information you’re collecting and the conclusions you’re drawing.
The secret is that it doesn’t matter how well the candidate does in the test. As with the entire recruitment process, everything you do is to provide opportunities for information about the candidate to leak out. With that information you’re building a picture to answer the question: will they add value?
What should I be looking for, then?
A technical test provides a framework for the following information to leak out:
- How do they handle feedback, both positive and negative?
- How fast do they pick up on new concepts?
- How much have they been exposed to the concept of elegant code?
- What do they do when they don’t know something?
- What are they fast at? What are they slow at? What are they sloppy at?
- Where are the gaps in their knowledge? How far does their knowledge extend? How aware are they of this?
- If given an opportunity to, do they cheat by going outside of the rules of the test? Do they admit it when challenged?
- How do they justify the decisions they made in their code? How defensive are they?
- What are their values when it comes to development? How flexible are they with those values?
If you’re doing it right, it should be possible for your candidate to completely “flunk” the test, but for you still to hire them because you see the value they’ll bring over the next year. Likewise, they might ace the test, but you discover components of their personality which will mean they’ll be a negative influence on your team or codebase.
Designing a great technical test experience
Step 1: design your challenges to leak information
You need at least two challenges: one to use as a screening test, and another to use as a final stage test.
Do your screening test, preferably remotely, before doing an in-person interview. You’re trying to minimise the amount of time you and your peers spend on candidates that won’t be hired.
Both challenges should be designed to give up the most information possible about the candidate; see the list of questions above for inspiration. Include the kind of logic, factoring and problem-solving tasks that they’ll face on a day-to-day basis at work. If you choose to impose a time limit, be generous; people work at different speeds, and the fast developer is often the scrappy one.
I have had great success designing screening challenges around simple console-based games. They contain just enough logic to give an interesting result, and most developers find them enjoyable to code up.
Step 2: cater the challenge to the candidate
This is not practical for the screening test in most cases, but for a final stage test, there’s every reason to allow the candidate to take the test in the way they feel most comfortable.
Allow them to use the language of their preference; you’ll learn a lot more about them than forcing them to use a language they swotted up on over the last week. You may even want to let them choose between a take-home or in-house test, although talking through their submission in person afterwards is mandatory. Some people only code using TDD; allow them to do that. Use their preferences to build a picture of who this person is.
Step 3: use the challenge submission to start a conversation
Discuss your coding standards and preferences, give them feedback on their submission, and enter into a dialogue. In a final stage test, ask the candidate to refactor their code based on feedback. Discuss the pros and cons of approaches. Talk about how you could extend the application. Ask what they’d do next if they had more time. All these conversations are opportunities for you to observe their reactions. Observe, observe, observe.
Step 4: ignore the code and base your decision on what you’ve learned about them
If you take one thing away from this article, it should be this!
Step 5: provide feedback to everyone
Part of a great experience is the candidate feeling like they’ve got something out of the process, even if it wasn’t a job. Let them know how they’ve gone, and provide them with honest feedback based on your knowledge and expertise. A rejected candidate who has a good experience with your recruitment process will be far more likely to recommend you to their friends and colleagues.
I’m currently looking for a new challenge in Wellington, NZ. If your company needs a CTO, Head of Development, or Software Development Manager, please get in touch with me at roger@operator.nz.
