Principles for hiring engineers

This week we’re joined by Jacob Kaplan-Moss and we’re talking about his extensive writing on work sample tests. These tests are an exercise, a simulation, or a small slice of real day-to-day work that candidates will perform as part of their job. Over the years, as an engineering leader, Jacob has become a practicing expert in effectively hiring engineers — today he shares a wealth of knowledge on the subject.

Discussion

2022-02-17T20:25:07Z ago

Hi Jacob

Thanks a lot for your blog post series and all your thoughts in this podcast show.

I am a senior engineer and I have been involved in hiring in the past myself. Some months ago I discussed an idea for work sample tests with my colleagues at work and I would like to know your thoughts about that idea.

The basic idea is the following:
Like lots of other companies, the company I currently work for, relies on open source software for lots of things. As we all know, this software is not perfect and therefore has bugs or lacks some features. Some of these bugs/missing features are important or urgent and we will dig into the problem our self. But in other cases, we “only” open an issue on e.g. Github to let the author know and monitor that issue loosely. We now started to collect these issues and the idea is to provide this list to the candidates to select one (or more) of these issues and open a PR. This PR would then be the foundation for the follow up call.

The advantages we see for this approach are:

In contrast to most work sample tests, where the result is simply thrown away, after it has served it purpose during the hiring process, the contribution to an open source project might serve the project it self, us as the company (because we are waiting for the fix/feature) as well as the candidates (by making their Github profile more interesting).
Every issue is unique, so there is less risk of the candidate finding existing solutions from previous candidates.
The candidates needs to familiarize them self with an existing code base, that in almost all cases will be new/unknown for them. This is also the case when the candidate joins the company, because all the existing code of the company is unknown for them.
During the following interview, one may talk about why the candidate picket a certain issue, the existing open source code base (what is good what is maybe not so good), the process of interacting with the author/maintainers of the open source project, etc.

Of course, not every issue is suitable, some of them might be to big to serve as a work sample tests, in other cases, the existing code base might be just to big to get into in reasonable time.

I hope, I could explain the overall idea and I look forward to your feedback.

Cheers,
Lucas

2022-02-18T18:04:20Z ago

Hey Lucas, thanks reading/listening.

It’s an interesting idea, and I can see why you find it attractive. There’s a lot of ways that it fits the framework I wrote about. I also like that it indicates something important about your company: that working on and participating in open source is part of the job.

However, I’d have a few concerns about using this as a work sample test:

Time: as you say, you typically need a good understanding of the underlying codebase to submit a good PR. I don’t know that someone new to that library could learn it well enough to submit a a PR, and write that PR, all in under three hours. That seems unrealistic.
Using this code at work: so, if this is a library you use at work, and the candidate writes a PR and gets merged, now you’re using someone’s code they wrote during an interview, for work, without compensating them. That’s unethical (and I believe illegal, but IANAL). The fact that it’s open source complicates the issue – it’s a lot more clear if they’re writing code for your proprietary codebase during an interview, which you later use – but it’s still the same. I’d want to be 100% certain that the code is never merged. Or that the candidate is compensated.
Open source community relationship: does the open source project know you’re using them as part of a interview process? Are they OK with it? Issues and PR generate sometimes non-trivial amounts of work for open source maintainers (see, for example, the Hacktoberfest debacle). If you did this against one of my open source libraries without getting clear permission ahead of time, I’d be pretty annoyed.

I think if you can navigate these issues it can work, but it seems hard. I probably wouldn’t use this format myself. If I wanted to capture the same kinds of signal, I’d use something more like the reverse code review I wrote about.

Best of luck,

Jacob

2022-02-28T11:44:01Z ago

Hi Jacob,

Thank you so much for this episode. I really appreciate the well put thoughts and the rules you shared with us. I am currently conducting hiring interviews at my company, especially the “soft skills” that can help determine the personality of a candidate and if he is a good fit for our team. I’m wondering if you have some tips for this kind of interviews or any good resources you have found helpful?

Thank you and thank you to the Changelog team for the great podcast!

2022-02-28T14:26:33Z ago

Hey Jeremie -

Yes, I do have some material that I think can help. I wrote a series on unpacking interview questions that I use, which includes a couple of questions that measure so-called “soft” skills: leadership, and conflict resolution. I’ve also written about how I develop interview questions around company values. The process is the same for “soft” skills.

[I’m air-quoting “soft” skills because I don’t love calling them “soft” – it implies these things are easy when it’s not! I usually prefer the term “professional skills”.]

Thanks for the listen and your kind words!

Jacob

2022-02-28T14:34:10Z ago

Wonderful! I totally agree about the professional skills! That’s why I quoted them as well.

Thank you so much for sharing!

2022-03-05T11:03:18Z ago

This podcast is gold ! I’m so glad I found this ! People should listen to this ! I’m totally agree with Jacob especially this quote from your website “Interviewing fundamentally measures a candidate’s skill at interviewing!” this quote hit me so hard …

2022-04-20T02:25:30Z ago

I must have previously read some of your content because i landed on, and managed to convince my company to accept the same three options for work sample tests.

What’s unfortunate is that most candidates come through recruiters, and they seem universally unequipped to communicate the options to candidates; ditto hr, honestly.

And beyond that, probably 95% of candidates have empty or pretty-much-empty public code profiles (which at least for myself, would be where a code sample would come from) such that i don’t think we’ve ever had anyone take me up on the idea.

Based on that, I’m not especially surprised (but disappointed) that code samples haven’t caught on more. I bet many companies or people in hiring decision positions would think that loss of standardization probably isn’t worth the extra complexity, given the small proportion which would benefit from it, so you end up with a process catered in the other direction instead

2022-04-20T14:38:14Z ago

Indeed. There’s a whole other point here about managers who let recruiters and HR define their process. Good recruiters/HR are partners, and work with the hiring manager to make sure everyone meets their goals. But lots don’t; they don’t really pay much attention to a hiring manager’s needs, and just want a one-size-fits-all process. Managers don’t realize that they can usually push back – and should. Managers need to realize that they have to own the hiring process, because they’re sure as heck gonna have to own the poor results if they hire the wrong person.