Presenting a Pragmatic Perspective

Alright, welcome back everybody to another episode of GoTime. Today’s episode is number 57. We’ve been gone for a couple of weeks… Our production studio was actually in Houston, but good news is that Adam and family are all doing well, so we are back on.

Today on the show, myself - Erik St. Martin. Brian Ketelsen’s here… I think he’s still here. Brian?

Muted?

Alright, maybe muted, but still, I’m here. The dog was barking.

And Carlisia?

Hi! I’m here, definitely.

And our special guest today is Cindy, also known on Twitter as copyconstruct, and I’m sure many of you have probably read a lot of her operations and devops posts that have been gaining some popularity recently. Welcome to the show, Cindy.

Thanks for having me.

See, she’s not on mute, Brian.

[laughs] Thanks, Erik.

So for anybody who’s not familiar with you or some of your work, do you wanna give just a little rundown, your history?

Sure. It’s kind of a little strange for me, because I have been programming for over half of my life. I started back in 2003-2004 when I was 13 years old, and it started with me programming for school, because we had computer science classes. I remember being really, really bad at it. I wasn’t very good, and what that made me do was spend almost an entire summer doing nothing but programming just to get better at this thing which I didn’t consider myself to be very good at.

What happened was I spent 6-8 weeks doing quite literally nothing else but programming. At the end of it it wasn’t really a struggle anymore, it just became extremely enjoyable and it just became really fun. 13, 14 years later it still remains the same. That’s really how I got my start.

So here’s a question, because a lot of us talk about this type of stuff, too. So you said when you got started you didn’t think you were very good at it… How do you feel about your work now? Because I know personally I went through this cliff, like – you know, through teenage years trying to learn programming, I thought I was terrible, and then I went through this inflated ego phase in my late teens and early 20s where I thought I was awesome at it, and now way later in my career I think I’m terrible at it again, like I don’t know enough and there’s too much to know.

I still don’t think I’m particularly good at it… I just like to think that I’m improving; I’ve definitely improved a lot over the years. Back when I started, I wasn’t really building stuff out, it was just programming. I think there’s a huge chasm between just writing some code and actually doing it professionally for a living. Software engineering as such is just so much more than just writing code.

[04:09] I don’t particularly think I am the best programmer right now, and I also don’t think I’m necessarily the best at software engineering, but what I strive to do is try to get better, because just like you said, I think there are just so many people we can learn from and just so much that pretty much anyone still needs to learn about so many various things. I don’t really think it’s ever gonna be possible for one person to completely understand everything that has to do with software development; there’s just way, way too much. But what we can do - or at least what I strive to do - is to continually get better at it. If I’m doing that, I think I’m doing okay. At the end of the day, that’s kind of what matters to me.

I think that’s why so many people call it a practice instead of something else. You have to continually practice and improve.

Pretty much. But I also think it’s plausible for some – and some people actually do this, where they go very deep into one particular field. It can either be security, or it could be web development, or it could be systems programming, or very low-level programming, or even for instance JavaScript programming, or very high-level… That’s very possible, and I see a lot of people doing that. For me, at least for now, I would consider myself a very generalized software engineer. That’s kind of also what I really want to be doing for the next few years - just sort of have a baseline understanding of a lot of different things, and keep doing that.

Yeah, I think having a broad view of the landscape is kind of fun. It allows you to not get bored. You can kind of switch over to stuff. But then you wish you were really super knowledgeable, like more knowledgeable in a particular area, especially when it starts gaining popularity or people are starting to use it for really interesting things and you’re like “Wow, I wish I’d spent more time going deeper down that path.” But I think the flipside of that is had you done that, there’s other paths that you’d be like “Wow, now I know nothing about those things, and I wish I had spent time on them.”

I think people beat themselves up a little bit, too… Like, you become the world’s best brain surgeon, but then you’re upset that you’re not the world’s greatest heart surgeon. It’s impossible to be both, right?

Cindy, tell us what it is that you’re doing now. What are you working with?

I’m a part of a startup for an image-processing company, but I don’t really work on any image processing software. Like I said, I’m a fairly generalist software engineer, which means I do a whole bunch of different things, including API development, infrastructure development, and also a lot of operations, because I work for a really small company, and what that means is that as a software engineer I am expected to also be running all the services that I write. That is pretty much what I do in a nutshell.

I write about some of the things that I work on or some of the things that I learn about. Primarily, it’s more for my own benefit than to actually write. I really just started writing maybe a few months ago this year, because for the longest time I didn’t believe that I had anything important to say, or that I was even doing anything particularly interesting that other people might want to read about, about my opinions or my thoughts.

[07:50] So it really began with me starting to write more for my own understanding. Writing isn’t something new… I have an extensive series of notebooks and just things that I’ve jotted down on the side all the years as I have been learning things, but it just wasn’t something I ever thought was even worth making public, because I just couldn’t really see who else would be interested in these things.

But I kind of started writing, again, more for my own benefit. I think with platforms like Medium, they just make it really – they just lower the bar to just sort of publish it, because you’re just writing something for yourself, literally copy/pasting to something else and hit Publish, and a whole bunch of people now can read it.

So that’s kind of how I really even started writing, and I still don’t really consider myself a blogger. It’s just writing about things that I learned, or writing about things that seem interesting to me.

Currently, I’m going through what I call an operations phase. It’s kind of weird, because I really am not an operations engineer. I do operations for the code that I write, but it is, I would say, about – not more than 15% of my time is spent working on things like monitoring, or deployments, or any of the things that I actually write about… Yet, that sort of is the thing that I have been writing about, mainly because that’s what I don’t know. Building software, or rather writing programs is something I’ve been doing for a long time now, but things like actually deploying my own code is surprisingly not something I have been doing for very long, so that’s what I don’t think I know very well… Which gives me more of a reason to actually write about it. Being able to just formalize my thoughts, being able to put it all in one place - just being able to write helps me understand a lot of these things better. That’s what really got me into writing, more than anything else.

Now, do you think that’s the reason why a lot of your posts - like Julia Evans’ posts and stuff like that - resonate with people, because it’s really this developer transitioning into the operations space, which is what a lot of what the devops stuff is doing? You’re sharing your journey with people who are also trying to make the same leap and learn more about operations that came from traditional programming backgrounds…

I’ve actually never thought about it that way. I’m even completely surprised that some of the posts that I’ve written have sort of gotten the attention that they have. It’s very new to me, it’s also very surprising, because like I said, a lot of those things were more for my own understanding, and it’s just very strange to me that something that I personally thought would only ever make sense to me is something that other people find interesting, as well.

I’m not sure why that happens or what it is that resonates… I mean, I’m happy to hear that it resonates with a lot of people, but I don’t know. For me, a lot of things – I’m not the first person to say this, but what I believe is happening right now is a lot of conversation, especially about a lot of tools and a lot of new technologies that are emerging, a lot of the conversation is being monopolized by specific people or specific groups of people, who present a very tailored narrative, or a very tailored perspective. What I feel is lacking is just a different story, a different perspective, or just a different side of the same story.

[11:53] I think probably - I’m guessing - the reason why some people think that some of the things that I say make sense or have merit is because it just presents this alternative viewpoint that probably isn’t being talked about a lot otherwise. I think that’s very true with – let’s for instance take the post that I wrote on schedulers. The main reason I wrote that was because it was meant to be an internal doc; it was never meant to be a public blog post. I didn’t start out writing that with that intention. We had just introduced a scheduler, and my service was the first one to be picked to be moved over to that paradigm of deploying applications.

One of the reasons why my service was picked was because it was like a fresh greenfield project that I was working on, so it sort of made sense to pick that - it’s brand new; if it fails, it’s probably not the worst thing, because it gives us a lot of time to test. We weren’t really changing anything that’s currently in production. So my service was the first one that was picked, which also meant that I could no longer just be on the sidelines, just sort of be hearing what people are saying about it and just reading these blog posts and these tweets about everything, but it really meant that I had to actually understand what it was and why we were doing this.

Like I said, I work at a startup, which means that not a lot of things are well-documented. I didn’t really have our SREs sit me down and explain everything from the beginning to the end, saying “Okay, this is how we are doing it, this is why things are not working, this is what needs to change and this is why we’re using a scheduler.” A lot of the information in that blog post was just me trying to understand how things are even set up for our work, and why things are set up that way, and why they’re even moving.

I think understanding that, that that kernel was very important to even understand why we need a scheduler, and then to sort of understand how things are gonna be changing. So the whole post sort of originated as me trying to just sit down, understand all these different moving parts that no one had really documented or written down any of that or properly explained, and to sort of demystify all of that for once and for all. Then once that post was done – it was an internal doc, it went into a wiki. Then I was like, “You know, there’s actually nothing here that’s super secretive, or that cannot actually be shared with other people.” It was extremely long - it was embarrassingly long, so what I thought to myself was that “Jesus, this is just so long… I just can’t possibly imagine who would wanna read it.” But then I thought there were some interesting ideas there, or rather, I wanted to know if what I was thinking even really made sense. I mean, it made sense internally to us, but a lot of times I think in software development it becomes very easy to validate your own viewpoint or your own biases, because you talk to other people - in my case it was my co-workers - who also all believe the same.

So I thought it would make sense to make it public, because that way I could get other people’s opinions. Maybe even if there was just one other person who read it, maybe I’d find out just what they think, and even see if some of our assumptions and some of the way in which we approach these problems really made sense.

Then I published it and then it sort of took a life of its own, which is still extremly surprising to me. So that’s how that came about, and the reason I believe those who liked it did so is again because it was just a different perspective, it was just a different voice. It was more about actually solving problems, as opposed to just using technology. That is something that I think about a lot, and that is actually something that I spend most of my time doing, solving problems, and technology is just like a tool that I use to solve different problems. But at the end of the day, it’s more about problem-solving.

[16:09] I think that a lot of tools, especially the standard opinions and the standard narrative that you get about a lot of these technologies is about, you know, “Hey, here’s this cool mutex. This is what it does, this is how it does it, this is how it’s super cool and this is why you should be using it.” And understandably, because every single organization and every single person is gonna have a different problem, and it’s probably not gonna be possible for someone who’s creating a tool to go around saying “This is gonna solve all of these problems in all of these different ways”, so at the end of the day it becomes the users of these tools; people like me, people like my co-workers, people at my company who actually use these tools to solve our problems, to tell our side of the story, to explain how it really makes sense and what are the challenges and what are the tradeoffs, and how things are gonna fit together.

Yes, I think that’s pretty much what I do. If all these things make any sense to other people, I’m guessing it’s because this kind of from the trenches story isn’t very widely told.

That is a very good point, I love it… Talking about the actual problems and solutions that you are working with, and using these tools to help solve is completely different than just talking about the tools in a vacuum. I agree with you, I’m sure that that’s the main reason why people are so drawn to your writing.

I also wanted to say that your posts are extremely well-written. It’s really rare to find blog posts that are so well-written… And not even to judge people’s intelligence - I think people write posts in a hurry, just because they have something they wanna put out there, and everybody has a job, so… I don’t know how much time you take to craft your post, but it looks like it’s really well crafted and really well thought out… Plus the writing skills really show through.

Oh, thank you… It really depends. The post on clusters took a couple weeks to write, mainly because, like I said, it was just meant to be an internal post and I was just understanding how these things even worked as I was sort of working with it… And I wasn’t even full-time working on schedulers; it was probably - like I said, I spent probably 15% of my time doing operations work. The vast majority of time during these weeks when I was writing that post was me just building this new service that we eventually ended up deploying with the scheduler. But that’s what I was doing for the vast majority of the time.

There was this other 15%-20% of the time where I was sort of learning these things, because it was very new. As I was learning, I was sort of writing things down. So that post took several weeks to write, actually, because writing that post wasn’t the only thing that I was doing. I was learning things, and as I was learning, I’d jot down this one thing or this one part that came to me, or this one idea, and at the end of all of it it became a blog post.

Some other posts that I’ve written probably have been cracked out in a matter hours. I would say that’s the case for most of them. Especially that was the case for a post that I wrote on function length… Because I was having a Twitter conversation with Dave Cheney and Sam Boyer and a couple other folks, and I went back home that evening and I just started writing… That was written probably in about two hours, and then I posted it. Immediately it made it to Hacker News, and I couldn’t understand what the reaction was like…

[20:07] Pretty much everyone calling me an idiot for writing that and for thinking that way. But it was fun. It was probably the most read post of mine, and it’s also the most polarizing, because I’ve got a whole bunch of people agreeing with me, but Jesus, a whole bunch of people also completely disagreed with me. So that was new, and that was fun, and that took like two hours to write.

Maybe had I spent more time writing that post, it could have been more – I certainly think it could have been a little shorter, and that’s true with all of my posts; they’re probably way too long. I mean, I’m a professional software engineer, I’m not a writer, so this is part of what I do in my free time. Again, I’m not writing professionally, so I don’t really bother with things like editing and probably making sure I’m not repeating myself… It’s kind of weird, it’s kind of ironic that I repeat myself a lot in my blog posts and I also wrote a blog post about how repeating yourself is not a really bad thing. But yeah, most of them take a couple hours to write. The clusters one took much longer, and probably a lot of these could a shit ton of editing. I don’t do much of that. It’s more like write something, post, tweet, go to sleep, wake up next morning and see what people are saying.

That’s very impressive; I’m impressed. I’m impressed both because you’re saying you don’t really pay attention to editing and yet, to me, they come out really well written, even the posts you’re saying you didn’t spend more than two hours on. And also because you’re saying two hours, I’m thinking – I don’t write blog posts, but I feel like writing posts every once in a while, and if I think it’s gonna take two hours, I’m like “Oh my god, I don’t have two hours.” It has to take five minutes, but of course, you can’t write anything in five minutes, so it doesn’t get done because I don’t want to allocate two hours to write a post. But it’s a good reminder that these things take time, but it’s also worth it, because you get to have a conversation about it, or you get to put your thoughts out there, or you get to just write for your own benefit, like you were saying, for your own edification.

Another thing that I wanted to say to you is that I personally love the different takes that you have on things. I think it’s welcome just for the sake of the opinion being different or contradictory, but you made me think in different ways from reading your posts. I think it’s beneficial to me, definitely, I appreciate it, but I think it’s beneficial for the tech community in general to be exposed to that… Even if at the end they don’t agree – you know, herd mentality is horrible; we need to hear different voices, different opinions.

Yeah, I think that’s true. A lot of people follow the dogma, and I think we need people to challenge that sometimes, and for us to at least question… Even a difference of opinion - it can do one of two things. It may make you more empathetic to why other people choose different tooling… Even though you may still believe that something is more superior, you may dismiss them as being completely wrong, and you’re right. It just makes you more set in your ways. Or it may bring you over to the other side, like you haven’t considered it that way.

[23:57] There’s a lot of stuff, especially as trends – like Kubernetes… I love Kubernetes, but if you’re running three Digital Ocean boxes or something like that, it may be a bit of overkill. In Cindy’s example - you’re working for a company that has a small enough team already and your developers are also the operations people… You’ve added overhead in supporting the cluster, time that you don’t necessarily have.

Yeah, that’s very true. That brings me back to my original point of actually solving problems… Because that’s what I do, and that’s what most professional software engineers are doing - they’re solving business problems for their employers, using tools, some of which they may build, and some of which they may repurpose, and some of which they may buy.

At the end of the day, it really boils down to solving problems, and I think a lot of people tend to forget that, because it’s very easy - and I’ve been guilty of this myself - to just get swept over by hype, or just how cool something is, or just how amazing a piece of technology is from a technical perspective, and you just sort of wanna use that or sort of play with that. I think that’s a perfectly legitimate instinct. I feel that all the time; I really want something that’s super cool, and like “Oh, that’s so amazing. I wanna work with that.” But I think it also becomes extremely important not to treat your employer as some sort of playground where you can just go and play with whatever tool you want, especially when it comes to tools like Kubernetes, which is both amazing and it’s also incredibly complex.

At the end of the day, it’s about making decisions as to “Is this complexity even warranted? What’s it gonna buy us, and what happens if we don’t do this? What really is the opportunity cost here? Are we willing to make this investment, and what are its biggest benefits?” And even if you decide to do this, how will it fit in with what you already have? How is all this gonna work out? Because I think that’s less spoken about, as opposed to what Kubernetes is and what it does, what it doesn’t do… Kubernetes is just an example, but this is true for any technology, really.

This isn’t new, right? We’ve been adopting bleeding edge software for ages and putting it in production… Especially Brian and I - we are terrible about it.

Mostly me, though. I usually just talk Erik into it.

Remember my addiction with the fact that we needed the GPU database, even if we had to build one?

Oh my god, okay… Yeah, guilty. [laughter]

And these are amazing things… Honestly, I think people should be sort of like going completely wild in their free time with whatever crazy technology they wanna work with or they wanna build or they wanna play with. But at the end of the day, most of us - at least most of the people I know - are working for employers and we are solving business problems, and it becomes extremely important to balance the technologist’s instinct to adopt cool new things, balance the instinct to use new things and at the same time really solve business problems; solve problems that the company is actually having and continuously build product that’s gonna make the company more money.

[28:06] I’ve most certainly been guilty of prioritizing one thing over the other. That’s a lesson I’ve learned, and it becomes really important to be cognizant of that, even when we find something super interesting or super cool.

I think it’s easy to feel like you’re being left behind. When it feels like the whole industry is going containers and Kubernetes, and you’re like “Oh, we’re still using Ansible or Puppet, or something like that…” It’s easy to feel that way, but at the end of the day if you’re building quality software that’s highly available, then you’re doing your job.

I worked for a manager one time that had a cool rule. For greenfield projects where you’re building something new for the company, you’ve got a credit - you could pick one bleeding edge technology, but everything else had to be well proven already. I thought that was a pretty good rule, because you always end up down these rabbit holes where it’s significantly more complex, or you run into weird bugs that nobody’s had before, or there’s just not a lot of content, there’s not a lot of people to tap to help you when you run into weird things that nobody’s ever seen before. So building your whole stack on new technologies gets extremely difficult.

That’s true. Another downside to new technology is the operational burden of it. I’m a software engineer and it is one thing for me to say “I wanna use this cool new language, or this cool new framework”, but when I’m also wearing the ops hat, even if it’s just for 15% of my time, actually working with it, actually being responsible for what it does when it’s in production, it means that I kind of completely see the other side, the flip side to a lot of this new and unproven technology, and that it’s operationally incredibly difficult to reason about… Especially it’s incredibly difficult to reason about failure modes that aren’t well advertised or well publicized, which again brings back the importance of things like observability and monitoring, and why it becomes important for observability not to really be an afterthought, but to be a part of system design, to be something that one ideally proactively thinks about.

The good thing with old, boring technology is that being able to proactively predict failure modes, or being able to understand how it needs to be instrumented or how it needs to be monitored is really easy, as compared to something that’s bleeding edge… Because you know, all these blog posts where people are explaining their story about how they use it isn’t quite written as yet. These are things that people have to figure out themselves.

My philosophy is that when I’m writing a new service or when I’m operating a service, the fewer things that I have to actually figure out for myself, the better. It’s not gonna be possible to completely have everything figured out for you already. If that were the case, none of us would have jobs. It becomes really beneficial if the number of failure modes that you run into or the number of surprises that it could potentially have are very few.

Very good point.

So you had started talking about the one post that was one of the most popular that hit Hacker News and you kind of got some slack about, which was your short methods post. Did you wanna give a little bit of insight into that? What was the negative reaction to that? Was it over-read into kind of they assumed you were taking a polarized view where it was like never a small method ever?

[32:05] It was more about people genuinely believing that a lot of the things that I said weren’t something that they agreed with, and that is totally fine. It wasn’t particularly negative, the fact that the vast majority of people who commented were disagreeing with me. There were a handful of pretty negative folks, but that’s fine… The vast majority, at least of people who disagreed, were just disagreeing with the content.

Some people complained that it lacked examples, but again, I cracked it out in two hours; it wasn’t like something I spent two days writing, so that probably explains why I didn’t feel the need to inject a whole bunch of examples.

I also think in a lot of cases examples can be very contrived. It’s kind of hard to translate a real problem that one sees in a real codebase that has been developed over several years by several different people with several different styles, and a codebase that’s been built to satisfy increasingly different and varying business requirements and constraints.

Technically, you can write an example, but it’s just very hard to really capture what you’re saying - at least in my case, to capture the experience that I’ve had… Like saying “Hey, here’s an example, here’s the function”, split it into two things and sort of make sense.

That probably explains one of the reasons why there weren’t too many examples, and a lot of people felt that a lot of the content of that post was very abstract. And I understand that, it can come across as abstract to someone who hasn’t really felt the same pain. That’s also the flipside of writing posts - it’s probably not gonna strike the same chord with a lot of people. But the good side is that it also had a lot of people writing to me and just saying how much they agreed with it. That’s pretty cool as well, because it’s not just me feeling that way. It wasn’t particularly negative, it’s just that a lot of people disagreed with me, and that’s fine.

I wasn’t really trying to be contrary for the sake of being contrary. It was more about something that I generally thought didn’t make sense. Because I grew up learning, reading all these books; I grew up reading clean code and refactoring, and book on design patterns of the Gang of Four and for a long time I sort of internalized all of that myself. I’ve written a lot of code (a whole lot of bad code) that was just me trying to completely adopt these ideas and shoehorn whatever code that I was writing to adhere to these principles. What I’ve learned is that doing that doesn’t necessarily lead to better code; it doesn’t even lead to better user experiences, and that’s something that I’m very cognizant of these days. The code we write is actually a user experience to another developer who’s gonna be also working on it. And taking some ideas that you might find in a book or a blog post and just blindly applying that to something that you’re building could actually prove to end up creating a really bad user experience to other people.

Something that might seem very obvious to you, something that might seem very simple might actually not be the case for someone else, especially for someone completely new to the codebase or the technology or the tool. The way they think about it might be – they might be missing a lot of context, a lot of the assumptions that went into some of the decisions you made. I think making these things explicit is probably the most helpful thing that one can do to provide a good user experience to a person, or make it at least very intuitive.

[36:00] When it comes to code, I think the best way to make something intuitive is just to be explicit, and that is I think one of the most amazing features of Go - what you see is what you get. There is not magic, there is no hidden abstractions or any talk of zero cost abstractions or any of this; it’s just dead simple. It’s verbose, it’s not everyone’s cup of tea, but it is dead simple. You look at it and you can understand what it does, and that’s amazing, and that’s extremely valuable for providing good user experiences to other developers.

I couldn’t agree more with what you said - everything, and especially what you said last, that Go is very different, and it strives to make everything explicit. I absolutely love that about Go, too.

As far as your blog post on small functions, for me personally, when I started programming, I was more struggling with knowing how to do it right - I still struggle, it was just that I struggled a lot more - than learning how to do it perfect. But I always kept reading books like Clean Code and Pragmatic Programmer, and Martin Fowler books, and learning the best practices. When I started doing Ruby, – I was at a point where I had enough experience with software development, but I was learning Ruby as well and watched a bunch of conference talks online and read a lot of books. It was so much about Don’t repeat yourself and Write small functions, Refactor, refactor, refactor, so I came to Go from that perspective. Then I noticed the files were so long, and the functions were longer than I was used to, but it didn’t phase me because I quickly saw it just worked. Everything’s there, I’m loving this; I didn’t question it. I didn’t strive to apply the Ruby on Rails dogma to Go, because I just took it as it was; I didn’t try to change what I was seeing.

When I read your post, it resonated with me, and what caught my attention was the combination of having had experience of doing Go for long enough to feel comfortable with a different way of doing things, which is longer functions, making things very explicit, repeating yourself… Like Dave Cheney says all the time, it’s much better to repeat yourself than to just abstract things away, especially if you’re doing it too soon. And a lot of people say that, as well.

So a combination of that, and also the fact that your post was so well written… Because if it was a blog post writing about this but the post had been written in a so-so way, I wouldn’t have given it much thought. But your post is really well written, and it really caught my attention. I was thinking “This absolutely makes sense.” And it’s not to say that it has to be that all the time, everywhere, every language, but it resonates with me, and the work I do works out much better this way.

Thank you.

One of the things that you mentioned, Cindy, that resonated with me was the idea of building examples that don’t necessarily make sense just because it’s hard to build examples. That was something that always gave me the hardest time building training materials, because you’re trying to exercise a particular point or a topic and you don’t want to build an entire application to prove that point, but sometimes making those examples that teach a thing are really difficult. Students would always say “What does this have to do with anything?” Well, I’m trying to show you interfaces, but I can’t really show you interfaces without building some sort of app.

[40:24] Right. I think it’s a lot easier for authors of books though, because they can start with one example and keep building on top of that. That totally would not work for the blog post format, because it’s just super hard to… Unless you’re doing a series of posts and you repurpose the same example. If it’s just one isolated thing, it’s just super hard.

I’m sure there are ways, but it’s something that I find extremely hard, to sort of capture without really making examples seem very contrived. And I had a lot of people actually write to me saying that “Here’s this simple…” – I mean, I actually tried doing that a little bit, where I was like, you know… Well, imagine if you have, in a function, which is all about creating a user in a database, right? I think that’s a super common example. Any application or any service that supports users logging in has that feature.

I sort of did that perhaps I was like, “Okay, so let’s think of this example where I have to create a user.” The point that I was trying to get across is that when you say a small function should do one thing, that one thing can be really hard to define. Creating a user, from a logical perspective, is one thing, but it actually involves several things. If we have the user in the database, then probably send them an email; you write an event tool, a message with like Kafka, so that all the analytics tools can pick that up. That was a really contrived example to say doing one thing necessarily just mean doing one thing, it could mean several things.

A lot of people wrote to me saying “Of course those things have to be separate functions. Why would you want to put all of that in one function?”, where that really wasn’t my point, that those all have to be one function. The idea I was trying to convey there is that one logical thing maps to more than one programmatic thing. A lot of times these boundaries get very blurred, because not everything that can be sort of programmatically isolated in doing potentially doing unit-testable things necessarily should be, and when it actually does make sense to do actually do that, which in my opinion is when you have network calls involved, or you’re writing to disk or something, or you’re just doing anything that’s just not completely just very programmatic.

If I could go back, I’d actually go back and edit that post a lot, because I just kind of feel that a lot of ideas could be expressed both more concisely and more reusably. But frankly, I don’t think I’m gonna be bothered; it’s done, it’s over. I’ll probably write a different one, that’s all.

I find myself actually doing the opposite of refactoring into small functions with my code in Go. Sometimes I see small functions that I just wipe out and put the code back inside the function, or I refactor things out and then I change my mind and put it back in. I find myself doing that a lot more than refactoring things down to tiny functions.

I think the one thing people underestimate is just how hard it can be when you technically just have to move around, even when you’re reading code. One of my co-workers, who sort of started working on a Go project and he started thinking about what packages should be, what the API should look like, and he probably gave way too much thought to that way too soon.

[44:09] Two weeks into the project he told me that “I’m actually giving up. I’ve put everything in a package called main, and actually a function called that.” He was like, “I was so fed up with just making all these decisions; I just wanted to get this thing working.” So it’s now one package main. It really was just one file, one function called main, and he kind of got that working. Then he went back and sort of refactored things a little bit. Or maybe he didn’t, I don’t know… I should ask him if it’s still just one big main function. But I definitely remember him trying to decide on what the API should look like, what the package boundary should look like.

The first thing that he did when he started this project was do that, and then a couple of weeks later he was like “Geez, I’ve complicated this just way too much. I’m just gonna put everything in one main function and get it actually working.”

I think it’s a lot easier to abstract later in the process than to do it upfront. In this case it’s just a really extreme example of someone just writing that whole application in one function, especially the function main, but I had definitely seen a lot of projects that actually have just one package called main, where a lot of things that could have been smaller packages were just all sort of put in one main thing, one main package.

At the end of the day, I think it comes down to how easy (or not) something is to maintain and how easy (or not) something is for someone new to the codebase to understand. Optimizing for these two things I think should be the goal, which could a lot of times mean going against the grain or doing things that may not seem very intuitive to purists. But at the end of the day it’s all about tradeoffs, and there’s never – and I think this is the point I was really trying to get across in that blog post… It really is never about perfection, because what seems perfect today could just be completely invalidated tomorrow when you have a bug report or a new requirement.

As such, the goal is to just build something that is good enough, that can be extended and that can be modified without requiring a lot of cognitive overhead, or without requiring a fully-fledged refactor. Sort of making it good enough, and not perfect, probably makes a lot of sense to me. That helps in writing maintainable code and it also helps in writing code that’s not perfect… That means a lot of people are gonna look at that code and say “Hey, all these things could be done in such a different way and could be so much better.”

And the thing is it could be a lot better for a short period of time. I think that is what a lot of people fail to understand - perfection is sort of short-lived, especially in software development. You can achieve it, but it really is gonna be short-lived. Versus being good enough, which can actually get you a very long way… Especially for those of us writing software and writing services that are designed to have some amount of longevity.

The most successful projects are projects that evolve and live on. It’s not like something that you’re just going to put into production and two weeks later just get rid of. For a lot of these extremely long-running – and potentially any project that one wishes to be successful, the goal there should be to just make it good enough, not really perfect. Perfection is, like I said, short-lived.

[47:53] I agree with absolutely every single thing you said. In the end it is about tradeoffs, and it really sucks when you start making the tradeoffs too soon. The example you cited of your co-worker putting everything in the main package - it’s hard to say that there’s only one right way to do things, but for me that is absolutely the right way to do it. Do it like that, and then think about spreading things up later… Because depending on what the end product ends up being, you’re going to use different decisions to work that out. At least that’s how it happens to me - if I try to make those decisions up front, I go through the painful process of making that decision and figuring it out, and then at the end I see that I didn’t make the right call. It’s really hard to make the right call up front, before you have the finished product, or sort of what the finished product should be. I don’t have to go through the same decision process again.

So the more I work, the more I aim for the things that you were saying - readability and easy to understand code, versus optimizing for other things. And even when I start writing a function and I don’t know what the function exactly is going to be or what the scope of that function is going to be once it’s done, I just type like “asdf” as the name of the function; whatever I type on the keyboard, I put the brackets and I write the function. And then I’ll name it after it’s done. Otherwise I spend so much time trying to figure out “What is the shortest best name for this function?” and I haven’t even written the function yet. I sort of know what it’s gonna do… Once I’m done, it might be a bit different than what I was thinking. I might split things up… So what’s the point of going through that decision-making process up front? It’s a waste of time and effort.

Anyway, I think we should move on to projects and news…

I think we should. I think we’ve got five or six minutes left of this show. So interesting projects and news - we’ve been gone for about two weeks, so probably the first out is Go 1.9 is out of RC, so it is officially released… Please download it.

Yay! Go get some! And Erik has a new job - I think that’s probably the more exciting news than any of this. Why don’t you tell us about that, Erik?

I do have a new job. I announced last week we didn’t have a show, but I am joining Brian and everybody on Microsoft Azure. [laughter] It’s almost literally everybody.

Brian and everybody.

We’ve gotta get Carlisia, and then it will be the whole show.

Oh, I also have an announcement… [laughs] I don’t have a new job. [laughter]

That’s awesome.

For the record, I’m very happy where I am… So Microsoft hasn’t asked, but if Microsoft asks - you don’t even need to ask; I’m very happy where I am. Not looking to move.

I was really happy where I was too, but the opportunity to work on Kubernetes and Docker and all of that stuff, all of the things that I love to do in my spare time, during my business hours, is really awesome.

That’s incredible for you guys.

So what else have we got? Community Outreach Working Group - that was a couple days ago.

Yeah, we don’t even know how to pronounce it.

Cow G?

Yeah, who knows… But we have a Community Outreach Working Group, and our goal is obviously to spread the love of Go throughout lots of communities, but more importantly, help people help others learn Go. You can read about that at blog.golang.org. There’s lots of us members.

[52:09] And feel free to jump in too, Cindy, if you can think of anything that’s come up in the past week or so that anybody should know about.

Actually, I read a really good blog post yesterday. It doesn’t really pertain to Go so much, but it really is about how to optimize services for both low latency and high throughput. I saw it on the blog of Dropbox; it was very dense and extremely interesting.

Just tweeted that link out, too. Yes, very good blog post. My favorite kind.

I think I saw you tweet that out, Brian. I haven’t got a chance to read it yet. On a similar note, Samsara - they do a IoT device for cars - they’ve got a blog post (we’ll link it in the show notes). They’re running Go on low memory devices, like 170 MB (something along those lines) of memory for this kind of in-vehicle device that does some camera stuff and telemetry that it reports back. I thought that was cool. If you read through, basically they end up kind of tweaking the garbage collector. It’s an interesting read.

If you like low-level stuff, davidwong.fr/goasm has a really cool walkthrough of some example Go code and how it translates to Go’s internal Assembly language that it uses. So if you’d like to learn about that stuff, that’s really interesting.

Yeah, you can keep the Assembly code. That’s all you, buddy.

[laughs] I don’t know, I like that stuff. I mean, I don’t wanna sit here and write applications in Assembly, but I like being able to troubleshoot stuff and look through the actual assembly language that gets generated.

I’ve got one more news item - there’s a good blog post at blog.minio.io about a new standard for data at rest encryption. The blog post summarized said basically that we do lots about encryption and transit, but there’s no standard for encryption at rest. So they propose a standard and a Go implementation of that standard. It sounds really interesting. I like their reasoning for the whole thing. They intend to use that DARE, which if you’re an ’80s kid like me, it just makes me laugh. But they intend to use DARE for their client and server versions of Minio.

That is fantastic. Actually, I was thinking about mentioning a blog post, but it might not have anything to do with Go. Does Signal use Go? The Signal app? I seem to recall that they do, I don’t know why.

Yeah, I don’t know.

But really quick - Signal, the messaging app, they have a blog post explaining the encrypted profiles that they have now on public data… And I haven’t finished reading the whole thing, but it’s just things that you don’t even think about that should be encrypted (or could be encrypted) and how they’re being so careful about really providing privacy for people who are using messaging - encrypting images… This stuff is so important, people don’t realize it.

Good stuff.

[55:45] And real quick, I wanted to go back to the Working Group news announcement, and just mention that it’s for everybody. People frequently ask how they can get involved in the community - this is perfect, because they have a list of issues… First of all, you can open a new issue; anybody can do that, obviously - it’s open source, it’s for the community. But they already have a list of issues they can comment on, you can volunteer to help out, to take the lead… It’s fantastic. A bunch of different things, all kinds of things that need to be done. There’s something for everybody.

Nice. How about Free Software Friday? Cindy, this is a segment of the show where we like to give a shoutout to an open source project or a maintainer or a group, or pretty much anybody that’s doing open source that we happen to appreciate, and it doesn’t even have to be Go related.

My Free Software Friday shoutout this week is to Minio, because I love them a lot, and they’re doing awesome stuff in S3-compatible file storage, and releasing good tools just all around the board. They’re great corporate citizens, and they’re kind of awesome.

Yeah, Minio is a super cool company. I really wanna thank Fabian. I don’t know if you guys know him… Fabian Reinartz He’s a Prometheus core maintainer. That dude single-handedly rewrote the entire Prometheus storage engine for Prometheus 2.0. It was just one guy doing all of that.

I don’t know if you’ve been following some of the blog posts and some of the new performance improvements, especially in the storage engine of Prometheus; it’s coming up in the new 2.0 release. It’s just super cool, so I really wanna thank Fabian for that.

Oh, that’s awesome.

Yeah, sounds like beast mode.

Right? How about you, Carlisia?

I don’t have a thing.

I’m gonna – I know it’s Free SOFTWARE Friday, but I’m gonna break that due to everything that’s been going on… So with everything going on in Houston, the whole cajun army thing.

Yes.

No.

So there is a group of people kind of across the country that after Katrina kind of put themselves together… Volunteers, they call themselves the cajun army. When Houston was hit, they were driving from everywhere, bringing boats and everything like that; they had radios going back and forth, some people were out in boats and trucks, some people were at home just playing dispatch, basically working with people who were stuck in their houses and giving them advice and prioritizing calls and dispatching them out to volunteer rescuers on boats and stuff like that.

It’s not software, but I think they deserve a huge shoutout, everybody who participated in any manner for that, because seeing people help people is just awesome.

Yeah, right on! And they’re acting like the - I don’t wanna use the word unprofessional, but unauthorized national guard; they’re just stepping in where they need to help, and it’s been amazing watching what they’re doing. Unofficial, not unauthorized.

[laughs] Yeah, one sounds criminal… It’s like, “Way to go, I just hyped them up, Brian”, and then you called them criminals.

Yeah, they’re criminals. It’s fake news.

Is it okay to thank another project, would it be fine?

Sure.

The other project that sort of came out about a year ago but I think has gotten a lot of traction since has been Envoy. I think Lyft open sourced it, and it’s in the service mesh category. Everything that has to do with microservices and service-oriented architectures and why a service mesh necessarily makes sense there.

[01:00:13.11] What’s been interesting in the recent few days, or rather in the recent few weeks is that… First things first, I think Envoy has – I don’t think it is a part of the Cloud Native Computing Foundation as yet, but I believe they are voting on that, so I think that would be super cool. And more importantly, I have seen discussions about potentially building it into Kubernetes itself, so it becomes the official sort of mesh for all Kubernetes applications.

I think the main person driving Envoy development is Matt Klein at Lyft. I think he was primarily responsible for open sourcing it and for shepherding it ever since. I think he’s just been doing some great work on this. More importantly, it kind of makes sense… And what’s also really interesting about potentially building Envoy into Kubernetes is because I think Envoy is very similar to an internal Google system that they have. I don’t know if it has an official name, but talking to a couple of Google engineers, what I heard is that pretty much every Google service has a sidecar proxy, which does pretty much everything that Envoy does.

As we know, Kubernetes itself is based on the Borg scheduler that was developed at Google, and what I’m finding incredibly interesting is that a lot of the auxiliary tooling and a lot of the surrounding infrastructure is now being available for everyone to use, and that is actually pretty awesome. And it’s also pretty awesome that there are some great people who are very committed to bringing these pretty advanced tools to the rest of us.

I think Prometheus as well is an example of a tool that is built based on an internal Google tool, but then which fits in really well with the whole Kubernetes ecosystem, because it all just sort of plays really well with one another, and I think Envoy is another such tool.

I think it’s gonna be really interesting to see some of the developments in the infrastructure space in the next few years, because with Kubernetes rapidly approaching to being the standard way in which people are going to deploy applications in the future, it’s going to be really interesting to see how other supporting tools are going to be built in. And more importantly, at least for me, it’s about how seamless these tools are gonna be to adopt and to gain the most benefit from.

[01:03:04.01] And I wanna point out that Cindy has a blog post talking about Envoy and comparing it to HAproxy and Nginx. It’s a pretty cool post.

Yeah, I’ve been following Envoy for a few months. I haven’t got to play with it yet, and I’ve been itching too, and I’m hoping soon… It looks ridiculously cool and it’s like wire-compatible with Mongo - a couple of databases - gRPC it supports natively, and some things like that. It’s ridiculously cool.

Yeah, MongoDB and Dynamo. I believe they’re adding Redis support as well. I think primarily as of now – I think Google now have commit access to the repo. So it’s just not Lyft’s effort, I think at Google they have a dedicated team working just on Envoy, which is incredible that you can actually have this project and get a company like Google devote some of their engineers to actually help improve your product. But currently it’s just MongoDB, DynamoDB, and I think Redis support is being added… Though I can imagine why support for things like the MySQL wire protocol or the Kafka protocol would be just incredibly cool, and it’s just gonna help increase adoption.

Yeah, it will be really interesting to see how that comes along, especially if it becomes a CNCF project.

Right.

Alright, so I think we are a little overtime, but we’re good.

What’s new…?

We’re always overtime. Didn’t we start this out at like 20 minutes? Wasn’t that gonna be the original goal when we started the podcast?

It was. It was gonna be a short little podcast… [laughter] That’s okay.

It’s perfect the way it is, I think.

Now we’re at an hour and I don’t even think we keep that… [laughs] With that, thanks everybody, and especially thank you Cindy for coming on and talking with us; I wish we had more time, but we’ve gotta stop the show somewhere. Huge thank you to all of the listeners. Definitely share the show with friends and co-workers.

You can find us on Twitter. If you wanna be on the show, have suggestions for guests or topics, find us on ping. With that, bye everybody! We’ll see you next week… Although I won’t see everybody next week - I’ll be gone for two weeks - but everybody else will see you next week.

Somebody will see somebody next week. This show is great! [laughs]

Somebody will be here, we promise!

We’ll have our people call your people. [laughter]

I’ll be here.

Alright, take care, everybody!

Bye, everybody.

Bye!