Principles for hiring engineers

We are joined by Jacob Kaplan-Moss, who’s a software developer, one of the co-creators of Django, and an engineering leader. Welcome to the show.

Hey, thanks for having me.

Happy to have you. I didn’t expect that when we had you on The Changelog, we would have you not to talk about Django, but that’s the way it turned out. We have listeners who have been asking for another Django episode, and to those listeners I say, we’re working on something. We have an idea, and stay tuned for that. But this is not that episode, because you’ve been focused on other things as of late, and you’ve been writing quite a bit, 10 posts, in fact, a large mini-series on your blog, all about work sample tests. So that’s what we’re here to talk about today. Jacob, maybe share your role, what you’re up to, your background, why the hiring process is something that you’ve been involved in, and why these work sample tests. You’ve been doing a lot of deep thinking on this topic.

Sure. Yeah. So these days I work at a security consultancy called Latacora. We mostly work with startups, well under 100 employees, usually more like 25 to 50, helping them get their security house in order at that early stage. So as you can imagine, hiring there is something we help out with a fair amount. And the reason that I’ve been writing about and thinking about hiring a lot lately is that it’s been a running theme of my career. Maybe for at least the last 10 years I’ve been in some sort of hiring position, at times doing quite a bit of it.

[04:14] At a previous job, I was working at 18F, which is a part of the federal government, and I rebuilt our entire hiring process there and was probably responsible, in some sense, for hiring dozens, if not over 100 engineers. The hiring manuals I wrote there are still being used, even several years later after I’ve left.

So when I made a goal of trying to write more, starting about a year ago, I knew I wanted to focus on engineering management in general, and fairly quickly, a theme of hiring emerged. There’s just a lot to say and a lot of bad practices and mistakes throughout the industry, most of which I’ve made myself. So every time I think, “Okay, I’m done with this series on hiring. I’m not going to write about hiring again for a while”, there’s another thread to pull that leads to another area to think about. There’s just a lot there.

The other thing I’ll say about hiring and why I find it so important from a management context is it’s probably the highest leverage activity that a manager will engage in. I mean, if you hire well, that person could be on your team for two years, four years, ten years, and could do a huge amount of work for the company. And so the investment in time in making sure you make a good hire pays huge dividends. On the other hand, if you screw it up and you hire someone who is incompetent or outright toxic, you can just absolutely destroy a team. So when I think about being an effective manager, if all you’re good at is hiring well, you’re probably better than 80% of managers in the field.

Yeah. Picking winners and losers, for basic terms, right?

Yeah, I guess so. Yeah.

The hiring in the tech industry is somewhat notoriously bad, or the process is fraught. There’s so many Twitter threads of people who’ve gone through bad hiring processes, or things that have failed, or why this company’s bad, why that company is bad. It’s obviously a difficult thing to do well, as you’ve just stated. But why do you think it’s so hard? Where do you think the core of the problem is? Is it software companies specifically that struggle, or is this like a system-wide thing? What are your thoughts on, if you could whittle it down to what’s wrong with it?

I don’t think it’s just software engineers. I don’t really know, because this is my industry and I don’t know other ones as well.

Sure.

But I do think there are some things that are distinctive to our industry that make it more difficult. The first part - if you think about it, interviewing is weird, right? You want to hire someone to write software, so you get on a Zoom call and ask them some questions about writing software; but that’s not actually writing software. And we all know people who can talk a great game, but can’t actually deliver. We don’t really have good mechanisms for doing, I don’t know, tryouts. Can you imagine if – I’m a basketball fan. I’m gonna talk about basketball. Can you imagine if the Warriors, instead of having someone come work out with the team, they called them over Zoom and said, “Hey, how good are you at shooting free throws? Oh, you’re good? Awesome. You should join our team.” But that’s kind of what we have to do.

The real problem is that when you join a new company to write software, it can take weeks or even months to become fully productive with the tooling. You’ve got to learn all the existing code, there’s probably some tool you haven’t experienced yet, the CI system is all weird and you got to figure out how to interact with that. It might even take you a week just to get credentials for your Git repository. So we can’t just say, “Hey, come shoot some free throws for us. Hey, come write some code for us.” At least not directly. And that’s the whole theme of the work sample test thing. And I think we’ll get back to that.

[08:06] The other thing I think we have going on within tech particularly is as an industry, we love to think that we’re special somehow, and we so rarely look at practices from outside the industry. We just sort of go like, “Oh, no, no, no, software is different.” We ignore hundreds of years of management theory, because I don’t know, we get computers, and that makes us different. I can’t even finish that sentence. I don’t know what the theory of the difference is.

Right.

Yeah.

Why is that the case.

There’s a real disdain for looking outside the tech industry for patterns or common solutions, rather.

That may have to do with our ethos around disruption, innovation, creation. These are things that we strive for, right? We’re trying to change and do things differently and think different, as they say at Apple back in the day. And so maybe we just apply that to too many domains and say, “No, we’re going to reinvent this, because we reinvent things”, and maybe that’s part of it at least.

Totally. I think a lot in terms of the concept of innovation tokens… I’m blanking on who came up with this term, but he gave a talk titled Choose Boring Software, which is all about choosing tried and true and boring technology to base your products on. And he makes the argument that you only have a certain number of “innovation tokens”, of areas where you can be disruptive and innovative. And if you try to be disruptive everywhere, you’re just going to be like a chicken with its head cut off. And I think that’s true. I think that is totally true. There is a real upside, and there’s a good side of this “we’re different” ethos, which is just what you described, of questioning the status quo and being willing to be disruptive. And absolutely, we have technology now that enables us to do things that were just wild a few years ago, but we’re bad at recognizing, okay, hiring is not actually one of those places. Why are we trying to be– this is not an area to be special. This is an area to do the boring thing.

You’d mentioned your history at 18F, and then obviously now where you’re at now, and gravitating towards writing these hiring manuals and the documentation. Obviously, from the Django project, you’ve got lots of experience writing the documentation around that as well, I’m sure, so it’s sort of maybe in your blood. But what attracted you to solve this problem, or at least make the mistakes and document how you would not make the mistakes and again, and potentially share that with others? What drew you to, I guess, care so deeply?

Yeah. I mean, it’s just what you said, this is the manual to hiring that I wish I had, that would’ve prevented me from making a lot of the mistakes that I look back on and cringe. I hired someone once who was so bad that the rest of my team almost quit over it. And luckily, I was able to do something about it before I lost five other people because of this bad hire. But that one hurts, and I’m responsible for a period of time when everyone on my team hated coming to work and hated their jobs. And that sucks. And I don’t want to go through that again, and I certainly don’t want anyone else to go through it again.

And so, yeah, I write. When I care about something, I write about it. It’s the format I’m most comfortable expressing myself in. It’s the way that I know how to make an argument, and lay something out. And yeah, so those things came together into the work I’ve been doing over the last year.

So let’s get a definition here on the table for everybody, work sample tests. This was a term that was new to me. And in fact, at first, I thought it was about sampling a part of your software and testing that. So I thought it was a testing series, but it’s a hiring series. And so you talk about work sample tests… Here’s a bit from your intro post. You say that it’s clear that actually working together is the gold standard when you’re hiring somebody. What if we found a way to do that in a compact and timely fashion? That’s exactly the line of thinking that leads to work sample tests. So what are they? They are an exercise, a simulation, a small slice of real day-to-day work that we ask candidates to perform. They’re practical, hands-on, and very close or even identical to actual tasks the person would perform if hired. They’re also small, constrained, and simplified enough to be fair, to include in a job selection process.

[12:39] You say work sample tests are a critical factor in effective hiring. Interviews are not enough. Hiring without work sample tests risks selecting people who excel at interviewing, but can’t actually perform the job. So in there is your opinion in about interviewing not being enough. And it seems like the reason is what you state there, is that, well, now you’re hiring for who’s good at interviewing, which is not actually the point. Is that what you’re trying to say with interviews not being enough?

Yeah, exactly. Real quick, the term “work sample test” - you’re right, it’s not a super common one within the software industry. And I cast around for the right term to use when I was writing the series. It is common within HR and more old-school management stuff. So I chose it over practical exercise or coding homework or other terms that I considered… Because if you Google, for any of those things, you find a bunch of random stuff. Coding homework really doesn’t work to find discussions of this sort of thing. Whereas if you Google for work sample tests, you do find a lot of discussion about this from HR and a management point of view. So it’s not the best term, but it’s the best one I could come up with to get me moving on this post.

So yeah, to your question, yeah, the thing is that interview processes are always, to some degree, a proxy, right? We’re doing this exercise where we want to know if someone is going to be good at, I don’t know, building a Django app, right? We have a job requirement. We know that we need someone to come in and write the backend to our web application, write Arrest API. We know what tools we want them to use, and we know what professional skills they need. We know they need to be able to be compassionate in code reviews, and we know that they need to– there’s going to be a mentorship aspect to the job. They’re a senior person… We have this list of things, so job duties we know someone needs to perform.

So we are trying to design a selection process where we measure something that we hope is correlated to those workplace behaviors. We can’t just measure the thing we want directly. Some of the reasons I’ve already talked about: it’s too long to get up to speed. It’s too long to measure those behaviors. It would be unreasonable to ask someone to really contribute to our code in a substantial way, potentially even illegal to have them do work for us for free…

So whatever we do, any interview process, any selection process we design is we’re not measuring the things we care about. We’re measuring something else and we’re trying to correlate it to the thing we care about. And interviews can do some of that. I think I’m pretty good at using interview questions to suss out professional skills, like communication and conflict resolution, and that sort of stuff. But conversational interviews really just can’t measure technical skills. They don’t measure software development. They don’t measure ability to produce code. They don’t measure familiarity with a key piece of technology that’s super-important. I can’t think of any other way to tell if someone’s a good Python developer than to ask them to write Python and look at their code. Nothing else correlates better than the real thing.

Yeah. There’s so much that goes into being good, in particular at software development. Some would say that you do most of your programming before you even code, where you think about the system. And you have some sort of domain knowledge, which can be play into being better at, which comes from experience within actual programming or within a specific team, or a new domain. If you’ve never– sure, maybe you’ve played with Python, but maybe you haven’t really done a web app with Django before. Maybe you’ve done some CLI tools or some scripts or something like that, but you haven’t really done things with maybe an API or something like that. So you’re dancing in new waters and it’s super challenging to, I would say, test the skill set

[16:42] Maybe what you’re really testing for is the potential of a skill set, right? Because you don’t actually want them to necessarily have the exact skills. You want them to be able to gain them, alongside with the rest of the team because not all the team began with domain knowledge, began with all the skills. They had to start somewhere. So it’s the potential of having the skills.

Yeah, totally. And it’s going to depend on the job, of course. There are going to be– some roles will have non-negotiable experience that needs to be brought to the table.

Sure.

If I’m a non-technical founder and I’m looking for my technical co-founder, I need someone who can build an app. I mean, I care about the language, but I need someone who can do something.

Right. In those cases, it’s paramount for the skills to be there.

Yeah. Or, I don’t know, if I’m hiring an infrastructure engineer and we use AWS and I need them to be productive reasonably quickly, preexisting experience with AWS might be non-negotiable. But for most positions - yeah, I mean, a number of times I’ve hired people for Python development roles who have backgrounds in Ruby, Go, PHP, Perl, and are just willing to switch to Python. And in those cases, a work sample test is going to tell me how big the gap is. They’re going to give me a realistic picture. Are we looking at a month for this person to get to where I need them to be? Are we looking at six months? And the answer might be that six months is fine. Hiring is long game. But maybe it isn’t. And knowing the difference between someone who is competent, but rusty and someone who really has never done this before is super important and really, really difficult to suss out without looking at an actual code sample or something similar.

So I’m hearing echoes of my conversation with Paul Orlando. We had him on the show a few episodes back. He writes about unintended consequences. We are discussing Goodhart’s law, which is, he summarized, if a measure becomes a target, it ceases to be a good measure. And he breaks that into two thoughts. And the second thought that he has about it is really what you’re bringing up with proxies. So we are trying to decide if something that’s very difficult and almost qualitative to a degree or heuristic is like, “Is this person a good hire?” It’s like, well, define that. And then you create, especially - let’s focus on the software side, are they a good software developer according to the needs of the role? And it’s like, well, how do we actually know that? And like you’re saying, interviews aren’t enough. I don’t think you’re advocating for no interviews. Interview plus work sample tests, right?

Exactly.

Yeah. But the work sample tests are the proxies, because you’re not making them do the job, but you’re making them do things that are like the job, close enough, and you’re trying to get it close enough that it actually works. And that’s really the hard part, right? In that conversation I use the obvious example of like, if you measure lines of code in terms of productivity, that’s like a terrible proxy, and so we all laugh at it. But people who aren’t in the know have done that in the past. The closer that those proxies get to approximating the truth or the reality of what you’re hiring for, it seems like the better they are. And we’re going to go through – you have eight rules for doing these well. So it seems like the nut of the problem is like, okay, we need these work sample tests, but still, even with that knowledge, it’s like, okay, how do you do that well? Because it sounds like that’s really– the most important thing is not knowing “Well, we need to have some work samples, but how do we go about that?” Because there’s whiteboards, there’s puzzles, there’s all sorts of these, which are like work sample tests, but they’re terrible in practice, for many reasons. So what you’re trying to put out there is what you think are good work sample tests. Is that right?

[20:29] Exactly. Goodhart’s Law is such a great law to bring up here because one of the really common and I think really broken practices is to use algorithms exercises, leet code style exercises, again, as proxies for programming skill. And there’s this whole history here, which is hilarious, where back in the day, Microsoft had this famous interview process where they asked these brain teaser questions. How many ping pong balls can you fit in a 747? Why are manhole covers round? I’m trying to remember all–

That one is famous. Yeah.

Right. So they ask these questions and the idea was this is a proxy for critical thinking, but what happened was people just studied Microsoft interview questions and memorized the answers.

Memorized the answers. Yeah.

And so then Google comes along and Google goes, “Well, this is a bad practice. People just memorize the answers. We should ask algorithms and data structures questions. We should make write code.” Okay, fine. There’s a theory – I guess you can make the argument that that’s correlated to job performance… Although I have a BA in English and my lack of CS 101 knowledge hasn’t been an issue for me in 20 years in the industry; but fine, maybe I’m weird. That’s no big deal. Whatever. Google comes up with this idea, they think that it’s a proxy, but then what happens is Google forgets that it’s a proxy. And now if you want to get a job at Google or Facebook or Microsoft, or a lot of the big tech companies, literally, if you have the time to just grind on leaked code for six months, you’ll probably make it through the job interview. They’re not measuring whether you’re going to be good at the job anymore. They’re measuring how much time you’ve spent grinding leak code. And that’s pretty good for a new college grad who wants to get a high-paying job, but it’s not great for the company or the teams. And it’s super not great for anyone who’s not someone with a ton of spare time to sit in their room and grind out questions about link lists or whatever. Someone who has a kid or family or parents to take care of and is trying to switch careers is just not going to be able to compete in a job interview, despite the fact that, who knows, they might be better at that job.

So yeah, a big goal of me writing the series was to try to layout, not just, “Leet code is bad. This idea is good.” That would’ve been easy. But I wanted to explain why leet code is bad and what’s better and why it’s better, so that people can develop their own work sample tests. I think the big meta argument is nothing off the shelf is probably going to work so super well for you because–

It’s bespoke.

You mentioned these eight rules. The first one is simulate real work as closely as possible; the work sample test should simulate what the job is like. And every job is different. So the idea of being able to pull something off the shelf that will automatically work for any job on the planet is probably a fool’s errand. You’ve got to consider, what do I need this person to do? And then how can I construct something that is as close a simulation to that real work as you possibly can? And if you do that, you’re almost certainly not going to come up with writing on a whiteboard, unless the job involves literally writing on a whiteboard. If you’re hiring someone to teach a class - sure, make them write on a whiteboard; it’s probably part of teaching, and it might be fine for that. But if you’re hiring someone to write code and you have this concept in mind of simulating real work, it’s really easy to see why a whiteboard isn’t going to work.

Break: [24:12]

It’s almost like you’ll need a glimpse of their future, right? That’s what you’re trying to do. But then there’s also this aspect that if you’re simulating real work as closely as possible, there’s a lot of upfront investment and effort into every hire, which kind of comes back to your earlier point, which is they can be so critical, so crucial. You’ve had people who wanted to quit their job and hate their job because you hired the wrong person at one point in time in your career. So it places a lot of emphasis, obviously, on the process, but there’s this intentionality that’s required to deliver on 0.1 of this framework of yours. 0.1 requires a lot.

I mean, it’s not easy, right? And I think that’s probably why some of these practices persist. It’s a lot easier just to be like, “What does Google do? Cool. I’ll just copy it.”

Right. What’s the easy button?

Yeah. There’s no dancing around it. Hiring is hard. My rule for myself is when I have an open position, from the moment I start writing the JD until that person starts, that’s like half my time gone. That’s how much work it is to design a process, to recruit, to screen, to interview, to select, to keep in touch with them before they onboard, to onboard them. And so I think a lot of people, a lot of first-time hiring managers don’t know this and they think that they can just squeeze hiring in around the edges of what the rest they’re doing. They don’t realize it’s probably going to be the biggest task they have. And then also, their managers probably don’t know that. So if I go to my manager and say, “Cool, I know you want me to hire for this position. I’m going to need half my calendar back”, they’re going to tell you no. So yeah, it’s hard, but I don’t know that that’s– it’s also worth it, right? It’s also, what else are you going to do during those 20 hours a week that will have tangible, positive impact on your team and on your company’s bottom line for years? If you have other activities that are that impactful, absolutely go do them. But I think most people don’t.

[28:24] I got in a bind when I was at a nonprofit once, and I was younger and I would just say more naive to, I guess, just process. And I was telling the founder of the company what we needed, because I was feeling the pain as the job doer, the person delivering the work and executing, and I needed support to help me do my job. And I’ll get less specific just to keep it short, but I was just like, “We need more people.” And I didn’t feel his pain. I didn’t understand the financial constraints, the ability to make money, the profitability of the company, things that this person was super-aware of, that I was just very less aware of. And one thing he said to me was just like, “If you hire somebody, you have to be willing to fire them, too. So not that I don’t want to hire somebody, but I want to understand the role that we need to fill before we’re just so anxious and fast to hire. I want to make sure I understand what we’re hiring them for, because they’re going to be here for years, hopefully. I don’t want to hire somebody just to fire them because we didn’t understand.”

Yeah.

And that to me just rung true for so long, because I was like– even to this day, our company is rather small. Jerod and I are the only full-time people. We just hired somebody for the first time after like basically 13 years in business; I mean, seven or so really making money. But we’ve been really slow to hire, because I think we don’t know how to hire really. We have desires to grow and do different things, but how do you slow down enough when we have jobs and roles to fill as a smaller company? It only exacerbates whenever you have a much higher scale. In a company where you’ve got 10 or 12 or 50 people, and maybe even more when you come to enterprise-level of just how much time it takes to understand the roles you’re hiring for and how important it is to really understand that, so that you don’t hire just to fire.

That is such good advice, yeah. I mean, and that’s the other thing, is firing people sucks.

It does. It’s the worst.

I mean, it mostly sucks for them, but it also sucks for everyone else.

I’ve had to fire a couple times and it sucks for me, too. I got PTSD from it. It’s not fun.

So yeah. I think maybe you can also think of getting hiring right as a way to avoid putting yourself in that position, or limiting the risk of being put in that position, for sure.

So as we walk through some of these eight rules or principles, I should state, as I stated at the top, you’ve written a lot on this topic. This conversation will not cover the breadth or the depth of what you have out there. So link is in the show notes. If this is fascinating, if you’re in similar positions to what Jacob is, then definitely go read the whole thing. This is like pretty much a book. Maybe you’ll turn it into a book at some point, but we’re just going to keep it focused on these eight rules because we feel like those are digestible and conversible. You mentioned the first one, simulate real work as closely as possible. And I think we’ve touched on that enough. If we look at the second one now, limit work sample tests to less than three hours. Do you want to unpack that for us? Why is that a rule?

Yeah. So if you think about it a little bit, these first two are a bit intention, right? Simulate real work as closely as possible. If you follow that as far as you go, you would say, well, we should ask people to do a lot. If their job involves– if we’re hiring a full-stack developer, we should ask them to develop a front end and a backend and deploy it to AWS and set up monitoring, and suddenly you’re asking someone to spend like a week. The real challenge - and I wrote a whole article about this - in designing good work sample tests is balancing between predictive value and inclusivity. We want to design a test that is as predictive as possible, that is going to as closely correlate performance on the test with job performance, eventual job performance, your predictive job performance. But you also want to design a test that is not going to accidentally select out people who have children, or limit you to candidates who are willing to spend a week on a work sample test.

[32:15] Right.

So three hours is the practical limit that I came up with. And briefly, the way that I get there is common practice in interviewing pre-COVID is for a candidate to fly out and spend a day on-site with the company. Most people agree that that’s reasonable. There will still be some people who can’t afford that amount of time, but there’s always going to be a certain – it’s reasonable for a company to ask for some of your time when you’re interviewing with them. And we can debate around the specifics, but I just started with eight hours. It seems like that’s a common enough practice. It’s going to be hard to find a job that requires less time investment from a candidate, and I think more is starting to get pretty unreasonable. And so when you subtract out all the other stuff that a candidate is going to need to do for you, you end up being left with about three hours for a work sample test.

I’ve also done a ton of these and I know that you can create a good work sample test within that time window. And so it feels like a really practical time window. And I wanted it to be very specific. It’s not a target. Less is better. If you can design a work sample test that only takes someone an hour, that’s awesome. Your interview process will be better for it. But three hours is, in my opinion, the limit. If a company is asking you to spend more than three hours, say, writing code for them, they’re being unfair to you, and you can probably do better.

So is that time period – and maybe this leads us into number three, but is that time period part of the test? Because people work at different speeds. Your three hours might be my six, or maybe it’s Adam’s seven minutes, because he’s amazing. But is that part of it? Is it like the three hours is a limit, or is it a constraint, or is that just like, “Well, we think this generally takes three hours. However long it takes you, fine”?

Yeah. So in terms of when they can spend that time, unless there’s a reason why time-boxing is super-important to your work sample test - and this is rare - letting people do that work whenever they have time; I almost never set any sort of deadline. When I assign a work sample test, I say, “Here’s what we want you to do. Absolutely no pressure. Just let me know when you think it’ll be so I have an idea of what to expect. But if that changes, no big deal.” We’re interviewing for a role right now and one of our candidates was planning some travel, and then it got canceled because of COVID, and this whole situation ensued, and it’s going to be a month between when we ask them to give the work sample test and when they completed it, and that’s totally fine. That’s not going to influence my decision in any direction. It’s simply like a fact that has no bearing on how I’m going to make a decision.

And this is a role that can wait that long. It doesn’t need to feel that bad.

It is. It is. It might be different if I had some pressure. I don’t. But for this role, it’s fine. In terms of people working at different speeds - so I really do consider three hours to be a time limit. The way I communicate this to candidates is by saying, “Here’s the assignment. I expect this will take less than three hours. So if you’re getting anywhere close to that, I think something’s gone wrong. Please reach out and tell us where you are and what’s going on.”

Internally, when I develop these tests, I usually do them myself, if it’s at all possible. And I usually ask people in that job or a similar job internally to try them out, so that I can validate that this time is appropriate. And actually, I usually aim for like half that, so like 90 minutes, when people internally are doing it, because they’ll be more familiar with the problem space and the types of tools, again, because it’s going to be close to the work.

[35:56] So most of the time I would say candidates actually complete these in something like an hour or two. Three hours is more like an upper limit. I really don’t want people spending longer than that. I think it’s unfair, and it starts to get into, again, what are we measuring here? Are we measuring something that correlates with job performance, or are we measuring something that correlates with free time? Because I don’t want to hire someone who has more free time. I don’t want the candidate with the least responsibilities outside of work.

Right.

I want the candidate who will be best at the job, and I don’t really care what happens after they go home.

Or stay home.

Stay home. Right. Exactly.

Go home or stay home.

Go downstairs, get off the couch, whatever it is. [laughs]

End their Zoom call. Yeah.

I noticed that in the definition for this– I’m not sure, I guess, what would you call this… The written version of what it means to use a strict time box, that there’s nothing that says about compensation. Did that come into play? Does that come into play at all with any of these sample tests, the compensation of time? Or do you feel like both the company and the candidate has to invest in the possibility of a relationship?

So I think if it’s going to be longer than eight hours total, I think compensation is appropriate. I wouldn’t consider asking for more than that without compensation. However, I’m starting to see a couple of positions that are offering compensation for anyone taking a work sample test, or sometimes finalists, or something like that as a way of indicating respect for their time and indicating some company values. The example that I can think of most clearly is the Software Freedom Law Center, which is a nonprofit involved in open source and free software law. It was conducting a search, I believe, for an executive director, and they compensated everyone who they selected to go to the interview stage. I believe they gave them $500 each.

This is an organization that is keenly aware of the difference between paid and volunteer labor. They’re in the open source world, where this is like a massive area of conversation right now… And I think - putting words in their mouth, to some degree; I don’t know why they did this, but I think this was a way of living their values of, “This work is important. The way we represent that in our society is we give people money for it.”

I would like to experiment with compensating everyone who gets to a work sample test phase. It’s not going to be entirely my decision, because I’m part of a company, and I can totally understand. I think reasonable people will differ on this. I don’t think it’s unreasonable to say, yeah, both candidates and companies should invest some time. And the norm is that interviews are not compensated. And I don’t know that it’s unfair, but I would like to try and see what impact that has on the candidate pool. And if that moves the candidate pool in a positive direction, I’m inclined to say that the cost would be minimal compared to potential rewards there.

So this is outside your framework, but while we’re on the topic of compensation, I wonder your thoughts on the idea of process, of doing extended paid trial runs. Maybe it’s just with your finalists, or maybe you’re down to a couple of options and you’re like, “How about you work for us for two weeks at this set rate, and then we’ll make a decision at the end?” Is that something that is fruitful, or are there problems with that idea?

I talked about this a bit in– I have an article that I titled something like What Doesn’t Work, where I talked about some of the stuff that I discarded, and this is an idea that I don’t think it works most of the time; not because it’s a bad idea. I think it’s actually probably the best from a correlation to job performance, right? If you want to know if someone’s good at the job, give them the job for a month. You’ll find out. The problem is that if you flip it around and think about it from a candidate standpoint, I’m not going to quit my job that I have right now to go work for you for a month and then maybe get a full-time offer. That’s just too big a risk.

Yeah.

Maybe if we had universal healthcare, I might feel differently, but we don’t. And so that’s a pretty big deal for a lot of people.

Yeah. It’s selective towards those who are currently not employed. If you’re currently employed, then it pushes you out of the process.

[40:04] So I’ve seen situations where it does work, right? An example is I’ve been in situations where there’s someone who’s a consultant, who we initially talked to about doing something on a contract basis, and over the course of that conversation, they go, “Huh. Your company’s really cool. Maybe I don’t want to be a consultant anymore. Maybe I’d like to be a full-time employee.” And in those cases, where they’re already set up to do consulting, maybe you’re already working together, it makes a ton of sense to say, “Awesome. Let’s fill it out for the next three months. Let’s get you on that contract. Let’s work on this project together. And if we’re both still really happy three months from now, then you’ll just keep doing the same thing and you get a W-2 now.” I think that it can be really great. I just think that situations where it’s fair to both parties, especially to the candidate, is rare. So as much as I’d like to use it more frequently, I’ve only seen that happen a couple three times.

It’s a case-by-case basis. It’s never–

It has to be a pretty specific situation.

Yeah. I think this really leads to why the process of hiring is so challenging, because there’s just so many different cases where this could happen. It’s like the worst if else statement ever. It’s just so many different paths you could take when it comes down to how you respond; like in this case, “Let’s hire them for two weeks.” Well, that’s terrible for somebody who currently has a job, but great if it’s an existing consultant and it’s not much of a change for them. It’s really just trying it on even further, with a different possibility. It certainly speaks to why hiring is so hard.

And you also have to figure out not just every time you have one of those differences from one candidate to another. You have to decide whether that difference matters. Is this signal? Is this something that I should include in my hiring decision? This person I mentioned who needed to take a bunch of extra time on their work sample test, is this something I should be considering in my hiring, or is this something I shouldn’t be considering? And every single piece of information you get from a candidate, you kind of have to decide, is this signal or is it noise? And sometimes it can be really, really difficult to tell.

If someone calls into a video interview and their audio and video really suck - is that, “Whatever, fine. Working from home is hard”? Probably. But what if they’re in a position that’s going to be primarily client-facing, where they’re going to be spending six to eight hours a day on video calls with clients? Maybe you need to ask them about their video setup and whether they would be open to some feedback and some improvement there, or just ask them what’s going on. “Do you live in a place with really bad internet access?” Maybe that’s a deal-breaker for this job, right? But you have to be really intentional about all of that. It’s really easy to get someone on an interview and be really frustrated by a bad internet connection, and make a decision not to hire them or to score them lower because of that internet connection, without actually thinking, does this matter? Does this just mean if we hired them, we would need to pay for them to get a better internet connection, or do they live in a cabin in the woods and literally can’t get a better internet connection? Every single door has this, like, “Is this core related or is this not?” And it takes a lot of thinking and intentionality to ask that question over and over and over again.

So let’s hop into rule number four. We talked about making the work as simulated, as real as possible. We talked about limiting it to three hours. We talked about being flexible when they do that, avoid deadlines if possible. Now, number four is provide as much choice to candidates as possible. Give them the choice of several kinds of work sample tests languages, environments, et cetera. That sounds like more work for you, the hiring manager, to just come up with as many tests as you can.

Yeah, it does.

Yeah, it is. And probably if there’s one “rule” that has more flexibility than others or that I would forgive people for not investing as much time in, it’s this one… Because yeah, I think in an ideal world, you’d offer people a few different choices that allow people either to play to their strengths or that fit better to their personal schedule.

So to get concrete, the ones that I usually offer people are coding homework, like “Here’s an assignment. Go home and write this.” I offer to look at code they’ve written previously, like open source code or a previous job, or I offer to do a pair programming exercise. So same as the first one, but instead of like, “Go off and do it on your own, let’s get on a call together and we’ll pair and write it together.” That’s only two exercises I need to develop, and then kind of two modalities for that exercise. And the reason for that is I want to give people the ability to get as close as they can to what working in their optimal environment might be, while fitting within what I need to measure and see from them.

So let’s take at open source/previous work situation. Obviously, that’s the one that people would choose if they have that as something that they can fall back upon, right?

Right. But most people don’t, which is why it’s not the default. It’s pretty rare for people to have code that they can actually share with me. Most people, the code they’ve written has been for a previous employer and they legally couldn’t show it to me. Or even if they could, it would be a bad idea ethically.

It’s kind of a nice get-out-of-jail free card though, or get-out-of-your-work sample test free card, because you’re like, “Here, I’ve already written some stuff. We can just take a look at it.” So in the cases that they do, let’s say I have an open source Python library that you and I could sit down, then what do you do from there? Are you with them, poking through the code, asking questions, or you have them describe to you what it does and they can make excuses for why that function’s not as pretty as it could be, because it was under constraints? How does that actually go in practice?

[48:19] Yeah. So that’s a great lead into principle or five, which is that the code is not just like a simple pass/fail. It’s the beginning of a conversation about that code. So regardless of whether someone writes code to my specifications or gives me something that they’ve written previously, we’re going to then get on a call and discuss it, and I’m going to ask them, “What did you do here? Why did you choose this dependency? I see you use mocks for testing here. Why did you choose mocks? Are there other approaches you thought about? Did you consider a different testing framework?” “Oh, I use PiTest.” “You like PiTest? What do you like about it? What don’t you like about it?” Whatever. I’ll just dig into the code and ask a bunch of follow-up questions.

I think a mistake that a lot of people make is they take the code sample, they run it, it produces the output they have expected and they say, “Cool. Pass.” Or they take it, they run it, it doesn’t they say, “Nope. Fail”, reject the candidate. And I have rejected people who have submitted working code, because their code was really poorly written, or worked accidentally, or they couldn’t explain what they did, or how it worked…

Worked most of the time…

Worked most of the time… Right? And I’ve totally hired people who have given me code that didn’t work, and were able to explain, “Oh, yeah. This was the part that I didn’t know how to do”, or “Oh, I misunderstood that part of the assignment”, or “Oh, I really wanted to focus on testing and I got distracted, so I didn’t finish the rest of it.”

Just for having the guts to submit non-working code, like “Look, here it is. In all its glory.”

Yeah. I mean, I told them three hours. And so if they hit three hours and weren’t done, I would expect them to give me not-done code. The analogy that I like to think about is I’m not looking for production-quality code. I’m looking for ready to open the pull request. So I may not even be– at the point of opening the pull request, it doesn’t necessarily work. I mean, how many times have we gotten like 90% there and then gotten stuck, and we open a pull request and we say, “I think something’s wrong. I don’t know what’s going on. Can someone else take a look and help me figure it out?” And as long as that code is mostly good, it’ll eventually hit production. We’ll just get some feedback, right? So I kind of think of any sort of work product that I see as part of the work sample test as a draft or as a pull request. It’s the first version you might share with colleagues, because the part that the work sample test doesn’t include is the feedback that you get that makes whatever your work product is really good.

Another good example, the position I have open right now, the work sample test involves writing in English this time. And if you look at the writing on my site or the writing that I do for work, it’s well organized, it’s mostly free of typos, it’s fairly clear, it’s cogent… And you might think that it springs from my keyboard looking this way, but you’d be totally wrong. My rough drafts are full of typos, confusing, hard to follow. In almost every case, there’s an editing step; always me doing a few passes of editing, but often someone out else, especially for my work product. I’ve got a whole team of colleagues that I share drafts with that give me feedback on.

So when I’m assigning this work sample test that involves writing, I’m telling people, “I’m not looking for something that would be done here. I’m looking for ready for feedback from my colleagues level.” And it’s the same thing with code. I’m not looking for ships to production. I’m looking for ready to open a pull request. So the bar is much lower, but I need to know that people can understand and talk about and have a conversation about that code, because that’s exactly what’s going to happen at work. I’m going to open a pull request and someone’s going to say, “Oh, you’re using mocks here. I thought we usually use stubs. Why don’t we change this?” And you have to be able to have that. I need someone who can both write the code and also then have that conversation with their colleague and get to the right point.

[52:06] So in a sense, it’s kind of like an interview sandwich with a work sample in between, because you’re talking to them first and then giving them this exercise, and then it seems like afterwards, whether it’s the code review session or whatever it’s called, the way they handle themselves afterwards, maybe the way they receive criticism or the way they explain even, “Here’s where I got stuck and why” - it seems like that has a large bearing on the decision-making process as well. Is that fair to say?

Absolutely. Yeah, especially the getting stuck thing. We all get stuck. Gosh, someone who is really self-aware and can tell me, “This is how I got stuck, and this is why, and if this was real work, here’s what I would need to get unstuck”, that is so valuable… Versus someone who I don’t know that about; I’m going to have to learn over time what situations they get stuck in. Someone who has that level of self-awareness is really, really valuable. So just as I said, interviews aren’t enough. I could also say work sample tests aren’t enough. You need the interview part two. They go hand in hand. They teach you different things.

If it’s not a pass/fail, like you said, it’s got to be a conversation, but it’s got to be some sort of awareness of how this person works, and an idea if they can do it; not that they are doing it, but can they do the tasks that you need them to? Obviously, there are certain criteria that you need to edge upon, but if it’s not pass or fail, it’s just got to be more information on their ability show up and do and to merge well.

I mean, ultimately you are going to make a binary decision. You’re either going to hire someone or you’re not. You can’t 80% offer them a job. There’s no “kind of hire.” There’s either a yes hire or a no hire. So at the very bottom level, there is a binary decision. But the way that I get to that decision or the way that I recommend other people who get to that decision is by considering the totality of the evidence in front of them.

It is rare for one poor behavior during an interview to be the reason why you reject someone. It’s not unheard of. There are red flags, there are things that someone can do during an interview that’s so egregious that it doesn’t matter how good the rest of their behavior is… But usually, by the end of an interview process, you’ve got a big old list of strengths and weaknesses. And if the strengths mostly outnumber the weaknesses, and the weaknesses are stuff that you feel pretty confident you can help them with, you can manage around, won’t be an issue, then you make the offer. And if you look through that list of weaknesses and you go, “You know what, just not going to happen. This person won’t be happy here. This person won’t be able to contribute. This person will be a bad teammate”, then you don’t make them the offer. But you kind of can’t know that.

To bring it back to the work sample test, the situations where I’ve made people offers despite submitting bad code have been where everything else was so good that I’m like, “You know what, they’re bad at Python”, and first thing is going to be a bunch of classes, a bunch of online stuff, some pair programming. We’ll send them to PyCon and pay for a bunch of tutorials. And I just know that the first six months with this person is going to be leveling them up in the language. But I’ve seen that they’re a great teammate, I’ve seen that they have tons of experience writing software in other languages, I’ve seen that they’ve demonstrated an ability to learn… And based on the totality of that, I’m more comfortable hiring this person who is going to fit in with the team super-well and bring all sorts of other skills to the table, beyond the Python experience, versus a candidate who’s maybe going to be able to write Python on day one, but is less of a good colleague, worse at communicating etc.

[56:02] I don’t have it readily available, but there’s a– Jerod, I have to mention TikTok, I’m sorry. I’m beginning to apologize, I do it a lot… But there was a TikTok I found. It was of Simon Sinek, who is well known for the book Know Your Why. And he’s got a lot of things. And I really respect a lot of the stuff that that man has to say. He’s really got a way of just looking at the world and making wise decisions, I would say, and just good at mindset… And I don’t have it on hand, but I’m going to put it in the show notes. But it’s something where he talked to how the Navy SEALs evaluate adding teammates.

I’m going to paraphrase what I think it says, and we’ll link it out and people can check it out themselves. But essentially, it wasn’t about their skill level, necessarily, as a Navy SEAL, it was about their ability to be trusted. Can you trust this person? Do you want to work with this person? And maybe that’s, like you had said, some parts of what you hire upon. But if all things being fair and equal - capable, willing, all the things, if you can’t trust them, they’re not going to be a good teammate. And he’s like, it begins at the trust level and the character level. If they have good character, good, sound judgment and you can trust them to show up and be where they need to be and to have your back when you need to have their back– now, obviously, Navy SEALs can be in dire situations, and it’s a different circumstance… But I think potentially as a coder, you might be too, because I mean, you might put a bug out there that does some serious damage to the world in serious ways. So that was something I thought was really interesting, which is like the barometer being around trust as a person.

Sure. Yeah. I mean, there have been a number of situations in my career where I’ve taken a flyer on someone who– essentially, where I’ve hired on potential; where I’ve said, “I love this person. I think they have a ton of the professional skills that will make them a great teammate. I think they bring a bunch of things that aren’t necessarily in the job description to the table. They’re kind of weak on some of the core things that I need for the job, but you know what - everything else about them is so great.” I’ve never regretted making a decision like that.

I have regretted making a decision where I look at someone and I go, “Well, they tick all the boxes on my job description, but I think they kind of might be a jerk”, and I ignore that and I hire them anyway, and it turns out they are a jerk. And it doesn’t matter how good someone is at software, if they’re not nice… Life’s too short, man. [laughs]

Love it. Don’t be a jerk. You’ll be infinitely more hirable than if you are a jerk. Seems obvious, but we have to state the obvious sometimes. Speaking of obvious, number six is all about being obvious. Don’t surprise them. So tell candidates ahead of time about the work sample test. Give them clear instructions when assigning the test. How early do you tell them and how do you tell them? Is it on the job description? Is it the first thing you say when they sit down for their interview? How do you go ahead and lay that all out there?

I like to put it on the job description. “Here’s what the selection process looks like. You’re going to have a phone screen and then you’re going to do a work sample test. And here’s a just brief description of what it’ll be like. And then you’ll have an interview with this person and then that person.” I put this all in the job description. Or that could be something that comes up and they’re like – and the first time you talk to them, you have that initial phone screen, you decide, “Yeah, this is worth pursuing. Okay, let me tell you about every step in the process.”

I’ve seen so many times people, they interview for like a week, they get all excited, and then they get this email from the recruiter that’s like, “Alright, as the last step, we’d like you to spend the weekend writing this web application”, and they’re like, “Well, God damn it, if I had known you were going to ask me of that, I wouldn’t have even bothered with the previous interview steps. I don’t have time for that.” And I don’t understand why companies do that. Why would you want to invest the time in someone just to–

Hey, you’re wasting your own time.

Yeah. I don’t understand why this is a thing that I need to tell people, but it is a thing that I see all the time. Just lay it out. Don’t be cute. Don’t be sneaky. Just tell people what it’s going to be like. And if it’s something that they don’t have the time for, that they can’t invest the time in, let them make that decision up front.

[01:00:04.13] Yeah. I couldn’t agree more. I mean, just put it out there. And then if they’re still surprised at the point that you ask for the work sample test, well that might be an indicator that they don’t read details, which is obviously something that you have to do in most jobs.

I’m afraid that people think I’m – I don’t know, that I think they’re dense or something like that, because I repeat it like three times. I have it in the job description, and then we have one when we have the phone screen, I tell them what to expect, and then when I send them the work sample test, I reiterate what the expectation is… I say it over and over and over again, because it really is frustrating when there’s a misunderstanding and someone does the wrong thing and it’s like, “Oh, man, I just wasted everyone’s time.” So I repeat it so many times. I think I sound like a broken record sometimes.

That’s a good thing though, honestly. I mean, setting clear expectation is such a good thing, especially in this context, because there’s already so much stress on both sides. As an organization, you want to get the person on board, there’s some sort of urgency level. I’m sure there’s always an urgency for hiring, especially in software development, and you obviously want to get the best candidate as soon as you possibly can and start making progress on your goals, and your mission. And as a candidate, you don’t want to be strung along. And how many times have we all been strung along with unclear, ambiguous process that just has nothing in these principles at all in terms of respect? To me, these principles scream “Respect on both sides. Respect your candidate and respect your own organization well enough to prepare, and have a process that is clear to you and to them so that there is no missteps.” I mean, applaud you for that. That’s super good. And if this becomes the rule book, or “the way”, then at least everyone’s leaning on the clarity.

Yeah. And if you decide that you really need a three-day work sample test, and it’s just not negotiable, it’s what you have to have for this position, and whatever, it may be unfair, but you don’t care and you’ve got to do it, just be upfront. Let me make that decision. Like you said, have enough respect for your candidates to tell them what’s going on and let them choose whether to engage with it or not.

So number seven, we touched on briefly already. We can probably just list it here: test your tests internally before giving them to candidates. Also, it’s pretty self-explanatory. Eight, kind of attached to the sixth to a certain degree, so let’s hop to that one - offer exercises late in the hiring process. So while you tell them about them upfront, you say, “That should be late.” And you say, “I recommend they be the penultimate step”, which is probably just an excuse to use the word penultimate. Well played.

Absolutely.

Yes. Which is the second to last step, for those who are not familiar with the word, before your final wrap-up interview with the hiring manager. So why so late? Why not just kick off with it?

Yeah, this is one of the places where there’s – first of all, ‘as late as possible’ is doing a lot of heavy lifting in that sentence. Sometimes as late as possible is first. [laughter]

Okay.

The reason being that it’s likely the most time-consuming part of the exercise. So if there are going to be a situation where you’re going to discover an obvious mismatch, again, it’s not respectful of everyone’s time… It’s a waste of everyone’s time to put it later in the process– or sorry, to put it earlier in the process. I’ve seen some people, for example, that the work sample test is actually the gate to an interview. You don’t submit your resume, you just send a code sample, and if that’s good, then you get an interview… And I don’t think that’s a great idea because there’s going to be a lot of people who you would otherwise screen out much earlier in the process. You would have a phone screen with them and you would see that they’re just obviously not a fit. I had a phone screen with a candidate recently whose salary expectations were just wildly higher than we could offer; more than double the top of our range. And this person may very well be worth that, they probably are, but it’s simply not in our budget. And so I wished them well and offered to help with the rest of their job search… But if I had asked them to do a work sample test and then discovered we couldn’t afford them, I think everyone would’ve been a little annoyed. So I just want to make sure that people aren’t asking for a bunch of time from someone that’s going to be, in the end, totally meaningless.

[01:04:32.00] Because it takes so much time. It’s the time aspect, right? So that’s not so much that it’s the three hours, it’s the expectation of go away, come back… Some sort of investment of our most valuable resources as humans, which is our time; time away from our families, time away from our existing job, our hobbies, our wellness, our self-care, our sleep, whatever it might be. You want to make sure that the candidates are whittled down to the people that you would want to offer a position for, if everything checks out, as you said, the totality of all the things, not just this pass/fail, because it’s not a pass/fail.

Yeah. Another way to look at this is – resumes are dumb and resume screens are not great, but the one advantage resumes do have is you do it once. You make a resume for your spring 2022 job search, right? You invest that time once, and then you send it out to 30 places. Work sample tests aren’t like that. If every single one of those 30 places asked you for a three-hour time investment before you could even talk to a hiring manager there, you’re looking at 90 hours of work, right? Are you really going to do that? I wouldn’t.

Unless they were all the same test. I’m just kidding. [laughs]

Right, exactly. So that’s the thing. As much as I think resumes suck, I think they are the best balance of respectful enough to candidates’ time and giving us as hiring managers just a little bit of information, enough to look at a resume and be like, “No, you’re wildly off base”, or “Okay. Yeah, worth talking.” And it takes very little effort from the candidate’s side to send that 19th copy of the resume, that 50th copy of the resume, right?

Well, while we’re talking about saving time, I should have brought this up during the work sample types section, but I forgot. Now I want to bring it up, make sure we don’t miss it. You have this other type of work sample test that you called the Reverse Code Review. I thought it was kind of interesting. Do you want to tell us what that is and maybe if it works well? You say it’s kind of like an edge case kind of thing, but this is where they’re reviewing your code, right?

Yeah. This is one of my favorites, but it doesn’t apply super-often. But for some roles, instead of saying, “Hey, you write some code and I’ll take a look at it”, you can turn it around and say, “Here’s some code I wrote. Take a look at it. What do you think?”

Yeah.

Weirdly, this works really well at either end of the seniority spectrum, it works really well for very junior engineers who asking them to do a lot of code writing might be a little fraught, because they don’t know very much yet… And absolutely, when you’re hiring junior developers or entry-level particularly, you’re definitely hiring on potential, right? You’re absolutely hiring based on where you think they’re going to be in a couple of years. And so sitting down with some Python code you’ve written and saying, “What do you recognize? What don’t you recognize? Oh, yeah. That’s what this is. These are optional static typing. Did you study static typing in school? What do you think about it?”, can be a really good approach.

And similarly, this can be a really good approach for very senior people, where code review and feedback becomes an increasingly major part of that job. As you get to that level of seniority, the code you write is maybe even less important than the feedback you give the rest of your team. And so sending someone your code, some code you’ve written and saying, “What would your feedback be?”, can really help you measure both technical ability, because you’re measuring how much they know about the ecosystem to give you that feedback, and also you can measure how they give that feedback, how compassionate they are, how they talk about the parts that are bad. That sort of stuff.

[01:08:09.01] In many ways, it’s like a mini pull request, but in reverse. It’s like they’re viewing your pull request because it’s already code written.

Yeah. I’ve actually done it exactly that way. When I did this in the past, we set up a dummy repo with a web app in it, and I opened a pull request and invited him to the repo and said, “Review my pull request.” And that was how we did this exercise.

And that’s good too when it fits, because it’s just so close to the day-to-day practice as a software developer. You’re going to review pull requests, regardless which code you’re writing, which language you’re choosing, which framework you’re choosing, or whatnot. It’s the part of the flow that everyone in your team is going to be a part of. So it’s going to be a natural next step as a team member.

Absolutely. Yeah. My friend, Sumina, has a really funny story about a time when she was hiring and they had an exercise like this where they gave someone buggy code and asked them to enumerate the bugs in the code. And they knew of 12 bugs in the code, so they expected that a perfect score would be 12 out of 12 if they’d find all of them. And the person they end up hiring sent them a list of 13 bugs. There was actually one in the code that they didn’t know about, and so they were like, “Oh my goodness, this person is even better than we expected anyone to be.”

Insta-hire.

I agree with you. I’m liking that one a lot, because when it would apply, it really leads to a lot of investigation. It doesn’t even have to be criticizing the code, it could be just pointing out what they see, particularly like you had said. I love that example a lot, because it really just gives that person a lot of free-range to just ask questions… In a lot of ways too – Jerod and I get this, because we get to ask a lot of questions doing our jobs. That’s a fun thing to do, I guess, to people like us, but it’s not natural to everybody. How to ask good questions to get to that next step, or why did you do this, or did you know about this module or framework instead, or whatever it might be, instead of doing it the way you did it? And that’s a cool way to discover somebody’s skill set.

Why is it the edge case? I understand why it would be great for seniors or entry levels, and maybe better for those groups, but is there a reason why it’s not more commonly practiced? Or maybe you’re just used to the typical way?

I certainly haven’t tried it that much for the bulk of roles, because I find the default of writing code or submitting something you’ve written previously to work so well.

Yeah.

I think the drawback would be that for most mid career-ish roles, I don’t care as much about their peer review ability. I need them, again, not to be jerks, but I don’t need them to be able to catch 13 out of 12 bugs, right? I’m less interested in the type of feedback that someone more senior would give. I just want them to be reasonably competent there. But on the flip side, I am interested in their ability to solve problems with software. So I think for most roles, if the job is mostly to write code, then the work sample test should be mostly involving writing code.

I could also see the argument that maybe it’s slightly easier to BS your way through than producing code is critiquing code that exists. So maybe you have a little bit less – maybe just a slightly less approximate proxy then. Like you said, especially if the job is writing code, let’s test them writing code, versus reading it. But it’s definitely a cool option to have in your toolbox for times where it makes sense.

Yeah. I love it. I just don’t use it that often, because I think you’re right on there. I think you’re right that as you’re developing as an engineer, your ability to recognize problems outstrips your ability to solve them. We learn what’s bad more quickly than we learn what to do about it. And there’s another thing that happens when you get more senior in career. There’s this galaxy brain moment where you realize that the things you used to think of were always bad sometimes aren’t, and you realize that all of the best practices you’ve learned have–

Caveats?

[01:12:12.22] …have situations where they’re – caveats, yeah, exactly. And so you’re much more likely to find someone who can tell you that X is bad when it is in that case, but hasn’t yet realized that it’s more contextual and not some sort of rule of the universe or something.

Well, you’ve got, I think it’s nine total posts in this series?

Plus the Q&A.

9 or 10, something like that? Yeah.

I just counted them and I forgot my count, so I apologize. I literally just counted them.

Off by one. We’re all programmers here. [laughs]

There you go. Yeah. You’re fired. Or hired.

I estimated 10,000 to 15,000 words written. I’d love to know the actual words written, but definitely a culmination of written work towards the subject matter. Obviously, we only really dove deeply into the eight rules of fair tests, which is one of ten… And I really also appreciated your wrap-up in Q&A to summarize it. And I’m just so thankful that you care enough to write it all down, to maybe correct your own mistakes and maybe give future Jacob a path forward in the correct manner, but also to share that with the rest of the world.

I know that we always appreciate when people like you do this, because it helps us I guess in the future hire better and to treat people more fairly; our organization, our teammates, as well as the future candidates of our future endeavor. So anything else in closing, Jacob? Anything else you want to share about the process? Anything left unsaid?

No, I mean… Thanks for the kind words. I do hope it helps. I do hope it helps you and other people hire more effectively and more humanely. That’s the goal. It’s been really fun getting to put this stuff down on virtual paper, but the thing that– I can measure words written and feel good about the amount of content I’ve produced, but the thing I can’t measure is the impact that this may or may not have. And my real hope is that, yeah, this will nudge hiring practices in our industry even just a little bit towards something a little friendlier, a little more humane, and a little more effective.

A little editor’s note here. I put my foot in my mouth here in a second or two, trying to describe to you which episode this is. Lo and behold, I got it off by an entire episode. So it’s not actually episode 478 it’s episode 479. So if you have any thoughts, any questions, anything to discuss, head to changelog.fm/479.

So in the note of feedback loop for Jacob - hey, if you listened to this show, we do have comments on our episode. So you now know the number of this episode, 478. So go there.

Link is in your show notes.

It’s in the show notes. Yeah. Thank you, Jerod. There is a link in the show notes to discuss it on Changelog News, that links to the comments. Jacob will get an email when that happens. And maybe it’s a question, maybe it’s a feedback to say, “You know what, I’ve read this” or, “I’m reading this. I’m so thankful.” Or maybe it’s a year from now or two years from now when this episode is out and it’s further in the future, and you’ve implemented it and you’ve seen it in practice in your organization; whatever it is, may this be one area of feedback loop for you, Jacob, to rest upon.

I’m sure that your email’s pretty public out there or something like that, Twitter I’m sure… But hey, in the future, if you’ve listened to this, if you’ve read all the culmination of this work and you’ve put it to practice and you’ve got anything to say back to Jacob, I know we here at Changelog appreciate the feedback loop. It happens so not often, but when it does, it’s usually kind words, and we appreciate those kind words.

Yeah. I’ll definitely be paying attention to that. So if anyone in your audience has questions, or comments, or hate mail they want to send me through the Changelog entry, I’m happy to read it. [laughs]

Good deal. Good deal. Alright, Jacob. Hey, thank you so much for your work. Thank you so much for your time today. We appreciate you.

Yeah. Thanks, guys. I really appreciate it.