Should candidate assessment exercises test candidates on their own domain of expertise or on the company's own product/problem?

Sri Batchu · Jul 26, 2023

The case study should be set where neither side holds asymmetric information — a third company, or a genuinely unsolved new problem at your own company

Asymmetric information clouds your judgment of the candidate's performance; an unsolved problem lets you jam on it together and see how they think

21:10 Use a neutral third party or genuinely unsolved problem to avoid asymmetric information

David Meyer · Nov 1, 2023

The best way to assess a candidate is a low-stakes simulation of working together, and asking them to advise on your own product is the wrong version of it

Advising on your product is a big lift and they lack context, so they may say something stupid and you'll judge them for a failure you caused; better to take a problem from their own domain, change the context or a constraint, and work through it with them

44:26 Use candidates own domain problem not companys product to avoid false negatives

Harry Stebbings · Nov 1, 2023

Testing a candidate on a problem from their own domain is unreliable because the interviewer lacks context and will be impressed by anything from someone who has worked on it for three years

44:54 Testing in candidates own domain is unreliable since interviewer lacks context to judge

David Meyer · Nov 1, 2023

Knowing nothing about a candidate's domain is an advantage, and you can detect bullshit in any domain by asking naive questions and digging until you understand

Dumb-seeming questions aren't the ones they've prepared for, which throws them off, and what you're really testing is the intellectual coherence of the story and whether they take credit without explaining the how — a made-up world can feel real but consistency is checkable

45:06 Naive outsider questions expose bullshit and test intellectual coherence better than domain expertise