Thread by @asanwal

Reference checks are not taken seriously enough

A couple of years ago, I got religion on them

Here are 8 reference call questions that get past BS and really help understand a candidate

🧵

2/ I used to treat ref calls as this perfunctory thing that didn't really influence my decision

Bad move

They're an opportunity to learn more about a teammate and how to make them successful

Or sometimes, they might change you mind and save you both lots of future grief

3/ Q1: I’d love to start by getting an overview of when you and XX worked together and in what capacity? And then I’d love to hear about the single biggest accomplishment you had together?

4/ Why?

I don’t love the start of this question, but it helps to set the stage and get the conversation rolling. The 2nd part is more interesting. By asking for 1 accomplishment, you reduce cognitive load on the reference.

That’s important.

5/ You might hear “Just 1? We really did a bunch of great work together. But let me see, what 1 would I pick?”

That is a great sign.

That means this was a very effective relationship

6/ If they dig into 1 accomplishment, I like to hear that it was specific, measurable and ambitious

Those are the best examples.

If the accomplishment doesn’t really feel like an accomplishment, that’s a challenge

7/ Sometimes, you’ll hear things that make you wonder “isn’t that just the job?” and so why is that an accomplishment.

That's another bad sign and something you want to dig into

8/ If they struggle to identify 1 impactful thing, then 🚨

Or if the answers are vague “We just had a great partnership and I know I could always call him/her when I needed guidance?”

Generic blows

You want folks who’ve made a clear discernible impact where they were

9/ Q2: What's their 1 superpower / exceptional skill?

It’s less open than “what are they good at?” which again puts a lot of cognitive load on the reference

Asking for 1 is easier

If they can’t identify 1 thing or if it's vague and vanilla, 🚨

10/ It’s also good to see and map what the person says they’re exceptional at versus what you believe the role you’re recruiting for needs.

If the thing they’re exceptional at doesn’t map to your needs, that might be worth digging into.

If it maps, this could be magical

11/ Q3: Name a time that you both disagreed. What was the topic and the root cause of the difference and how did you come to a resolution?

This is about conflict resolution

I hate hearing “we never really disagreed on anything”

12/ I like hearing someone has an opinion and that they bring diff perspectives which need to be worked through

If someone is a manager of others and doesn’t have different opinions on some topics over time, this is often someone optimizing for relationships over results 🚨

13/ If they have had something they’ve been on different pages about, I want to understand what the disagreement was about and how far apart they were and why.

And I mostly want to understand how they resolved it.

The key here is getting a sense for 2 things

14/ Thing 1 - Did they get to the ‘right’ answer or did they split the difference?

Splitting the difference is not a great way IMO to resolve conflicts (most of the time)

15/ I want to know if the person either changed their mind in the face of new info or helped create alignment and convince others of their perspective.

Or in some cases, they made a hard call in the face of resistance and took a call

16/ Thing 2- How did the reference and others feel about the resolution?

Trying to understand even if the reference/others didn’t agree, did they feel like they understood the reason for the decision and that there was a ‘disagree and commit’ culture the candidate created?

17/ Q4: How many other YY have you worked with in the past? Where does XX person rank among them?

I want to know they’re the top 1 or 2 of the people they’ve worked with in a similar role

If they’re not, that's worth digging into

18/ This can be adapted to each person, i.e. vs other managers, vs other people in that function, etc. It’s a pretty flexible question.

Follow-up question(s) I might ask

Q4b: Since you didn’t rank them 1, what is it about the person you ranked #1 that set them apart?

19/ This will help indirectly ID dev areas the reference sees

It may also highlight gaps that are actually not meaningful to you

They might say something like “They're really good with investors” and you might know that that’s not that important for what you need

20/ Q5: If we invested in a coach for XX, what do you think the coach and XX should work on? This could be anything from specific functional skills to leadership/management skills.

IMO, this is a better way to ask the “What is the 1 thing they need to work on?”

21/ It uncovers what areas they might need to work on but in a less judgmental way and so tends to get more candid responses in my experience

22/ Q6: What advice would you give me to help XX be successful?

This is another way to understand how to motivate, lead, etc this person.

It may also reveal any opportunities/challenges

23/ Q7: As a [manager/leader or head of ABC function], on a scale of 1-10, how would you rate them? You can’t say 7.

The aim with this question is to get their views of them as a manager/leader or their functional expertise (engr, sales, finance, CS, marketing, etc etc).

24/ If they say 6, problem

If they say 8 or 9, ask them them what would make them a 10.

8 in particular dig into as it's kinda the vanilla good rating :)

25/ Q8: This may seem like an odd question. What’s something ambitious you worked on together that didn’t work out. Said another way, what’s your favorite failure?

This is the lead to understand if they set ambitious goals and sometimes didn’t hit those goals.

26/ Once someone shares their favorite failure, it’s about understanding why

Was it they couldn’t convince others on the team, poor execution, lacking clarity of thinking?

I like to lead with “this is an odd question” as it helps put the person at ease

27/ If I hear they never failed in doing something, is that a function of risk appetite of the individual or the org?

I like to know folks are pushing for ambitious goals and sometimes fail

Not failing might mean playing it too safe

28/ Other random things I find useful

If you hear something you want to dig into, do that in subsequent ref calls by asking

“In earlier calls, I’ve heard _______. Can you give me an example of when you’ve seen them be ______ and generally, how do they respond to ________”

29/ For example, I hear on a call that the person might get frazzled at times, I'll say

"In a prior call, I heard that XX gets frazzled in high stress situations. Can you give me an example of a high stress situation that frazzled XX. Generally, how do they respond to stress?”

30/ When a reference hears someone else has said this, it generally makes them feel more willing to be open as someone else has.

31/ A variation on this can also be effective if there were things you had questions about in the interview process.

“One of the concerns a member of the interview panel had was around _____. Have you experienced them having challenges with ______?”

32/ Some of the ^ questions may seem duplicative or even a bit too difficult

A lot of this is in the delivery and overall, the call should be oriented around positivity and trying to get to know a potentially future teammate better through those that already know them well

33/ That's all. Any questions you've found to be particularly enlightening in reference calls.

If nothing else, I hope you'll avoid the mistake I made and take references very seriously

They matter

34/ The other side of recruiting (the top of funnel) covered a bit here with resume red flags

https://twitter.com/asanwal/status/1063884274565345281

35/ answers to some DMs

- I use these for senior hires
- rarely ask all 8 questions as generally unnecessary. Pick & choose based on how convo is going
- generally 15-20 mins per call