Questions from a new Go developer

Well, hello, hello, hello, and welcome to this episode of Go Time. Today we are going back - well, at least Ian and I, we’re going back to what it felt like to be a new Go developer… Because chances are we’ve probably forgotten that feeling. Yeah, and we have a special guest coming on the show to talk about their experience of being a new gopher, and they actually suggested an episode, which you can do too, listener; you can suggest episodes, and who knows, maybe we’ll even invite you on a show and come talk about the stuff you’re proposing. But yeah, I propose that we answer some questions that a new Go developer would have… And we thought “You know what? Why don’t we bring him on the show and have him tell us what his experience has been and ask some questions?” So we treat this as an office hours style episode, and hopefully we’ll do more of those in the future. But yeah, this is – we’re very glad to have you on the show, Duarte Carmo… Hopefully I’m not butchering your name… Welcome to the show.

Good to have you. Good to have you. And co-hosting with me, as you might infer from my previous comments, is Ian Lopshire. Hello, Ian. How are you?

I’m doing well.

Good, good. You’re ready to feel like a newbie again?

I mean, I just felt like a newbie yesterday, making the same mistakes… So yeah, I think I’m prepared.

Okay. Alright, alright. So Duarte, why don’t we start with a quick intro? Give us a little bit of background, and we’ll get into also how you got into Go.

Cool, cool. And how did you stumble onto Go?

Nice.

[00:06:00.08] Okay. Well, let me just start by saying that getting 16 days into Advent of Code is no small feat. So don’t sell yourself short there. A lot of people I know, they’ll start, they’ll be all gung ho and excited, and then like day five, it’s like “Ah, I give up.”

Exactly, especially during the holidays. Yeah. To me, that has always been as much of a lesson in time management as it is about getting through all the exercises. But awesome. Well, glad to have you in the Go community; welcome, on behalf of the Go community, if I may. Definitely, we enjoy having one new developer in the community.

It’s interesting hearing you describe what led you to try Go. All of us, we hear about different programming languages all the time, and we don’t always necessarily run towards them or gravitate towards them for one reason or another. The fact that you came to Go looking for something perhaps not as – I guess if you consider a spectrum where you have Python, where you can develop quickly and get something out the door quickly, because it’s what you know and have been using for a long time, and then having Rust on the other end of that, which has all these wonderful ways of ensuring that the stuff you have in memory is the stuff that you actually want to have in memory, where you have to be very strict about what you’re doing and know what you’re doing… Go kind of gives you that sweet spot in between, where you can get good performance, where you can get something out the door, and if you follow all the idioms of the language, you will get something that looks like production-grade Go code, without having to be a master of the language.

And that’s okay. That’s okay. That’s why we’re gonna get into it shortly here.

Break: [00:07:55.22]

Let’s start with – I think one of the things you listed in some of the questions that you had was the idea of dealing with a binary; you like the idea of a binary. So what are you juxtaposing that to?

To somebody. [laughs]

You’re kind of discovering naturally, organically, the virtues of the language. Literally, part of the language’s aims, one of it was to literally allow for somebody who’s new to Go, or a junior developer, or somebody who is perhaps experienced, but still needed a language to basically write Go code that looks almost as good as a pro Go developer. It’s the first language that I’ve personally come across that really guides you towards that. I’ve been amazed to see new Go code being created by new Go developers, that is not too far off from what I would expect from an experienced developer. And I think that’s by design. The language kind of guides you; it kind of puts you on this path, it puts some guardrails around you, and your code ends up looking very much like what a professional would write.

Well, let’s start peeling back some layers.

Maybe we should also bring you back to do our roast, our Project Roast. We’ve been thinking about an episode, a kind of episode where we would roast projects… So we’ll bring you back for those. [laughs]

In Go.

That’s the thing, when you’re playing around with lots of languages, sometimes you end up –

[00:14:06.29] That’s a worthwhile endeavor as well. But yeah, you have a list of questions here… And I think I’ll let you guide the conversation as to kind of what do you want to tackle first.

Well, Ian, why don’t you get us started? Because I have thoughts, and opinions…

I think not only is it normal - and it can be jarring at first, because like you said, it does occupy a lot of space… But it’s also kind of my favorite part. So it’s hard to not like it, I don’t know. But yes, it’s completely normal and expected. And at least the if err == nil part, what you do after that kind of changes, but… I think you’re always gonna see that. There’s been so many proposals to try to change that, and all of them have gotten shut down, and I like that they have all gotten shut down.

Yes, but not because the creators of the language wanted to be anti, right? I can’t remember the name of the paper, but there was a paper once upon a time that I read, and then I immediately understood why the creators of the language chose to go that route… The paper talked about explicit handling of error values, right? As opposed to some languages where you’re gonna see some exception handling. [unintelligible 00:16:19.00] you see a try, a catch, finally, or something like that, and then basically they’re just bubbling up the error all the way up the stack, and hopefully something will catch it and handle it and do something with it. So Go doesn’t have exception handling. Go has error handling. And the error values need to be treated and handle then and there, when they happen. There’s no traditional approach of the try, catch, and finally. That style of handling exceptions is not present.

So when you see the typical multivalue return where the last values type satisfies the error interface, that’s basically the idiom within the Go community; it basically indicates that “Okay, you’re gonna make some sort of function call, and if something goes wrong, if there’s a possibility of something going wrong within that function call, expect that–” say you’re returning two values, right? Maybe the first value is what you really want, should everything go well, and the second one is something that satisfies the error interface, you should check to see if something went wrong and decide what you’re going to do right then and there.

Maybe in some cases maybe you want to call a different function, and maybe you want to retry. Maybe you want to log something out, or capture some sort of metrics, maybe you’ve got some open telemetry tracing in there, maybe you want to send a trace somewhere… However you want to handle that error, the idiom is to handle your error right then and there, as opposed to sort of allowing it to bubble up the stack and hopefully something handles it.

[00:18:21.13] Well, careful with the nomenclature. There’s no raising of anything in Go. And even panic means something very specific for us. So you don’t want to use that word when you mean an error. So to use your example, depending on how you want to handle your error, you could simply call a function; maybe you’re on the third of four functions. Maybe the third one calls a fourth, and if something goes wrong in the fourth and an error gets returned, in the third function you could decide to handle the error right then and there. Maybe you try calling again, maybe you do some other thing, maybe if you want to log it out, maybe you log it out, and then call it a day… Or more than likely you’d probably return that error back to whatever called this particular function.

What you typically want to avoid is – maybe you have four functions deep, and all they do is they all log out something. So in your log files you end up having the same error, or a similar error, a similar root cause all over your logs, with no additional value. Yeah, you could do it that way. You can have each function log something out, just so you can see “Well, which function errored out?” and make notice of it… But typically, you just want to allow – maybe it’s your top-level function that basically does the logging out or the tracing or whatever it is.

Personally, the way I approach it - and maybe Ian has a different approach. Personally, the way I approach it is deeply nested function calls. I don’t log anything out. I basically – within my context that I’m passing down to each of those functions I’m doing open telemetry tracing. So I’m literally tracing each function call. And then I’m not logging anything out; if there’s an error to be recorded, I can add that to the span that I’m tracing, that I’m adding to my trace. And then I allow basically the caller to do something with the error if they want to do something other than just make a note of it in the trace somewhere.

And then I’ll add to my top level - usually it’s my main function somewhere, or the main handler of some kind… I allow it to decide “Okay, do I want to – I know I’ve already been onto some tracing. Do I want to then send this particular error to an external bug tracker? Or do I want to simply log it out if I want?” But I’m localizing sort of beyond just the traces, I’m localizing the logging of that error in just one place, or communicating that error to an external bug tracker to just one place. But that’s because I know I have tracing throughout the application, so I can rely on the trace to provide me the breadcrumbs, without having to rely on my deeply-nested functions to be emitting log. Because that’s effectively what you’re doing. You’re trying to say “Hey, this function was running when this error occurred”, and you can go in your logs and kind of see what was going on… But traces give you the same benefit without you having to pepper your log file with the same kind of error. Ian, do you have a different approach?

I would agree with everything you said. I guess my main philosophy is either handle it or add context and pass it up. So you’ll see a lot of fmt.Errorf() with a w for wrap. And then they’ll say “Hey, there was an issue with this user.” So you can add context all the way up. So even if you don’t have tracing, a lot of the time you can see exactly the path that the error took all the way back up to where it’s finally logged out or handled, or…

There’s quite a few ways, but one of the more common ones you’ll see is it’s literally just like a format string, where you pass the error as one place, and then you can add whatever content you want.

[00:22:07.29] You also have to understand, an error – I mentioned the term interface earlier on very deliberately. So you can create your own custom errors. It doesn’t just have to be like a string, a value that you’re just bubbling up; you can have contextual, sort of a composite that it represents. As long as it satisfies the interface, meaning that it has a method on there called error, that returns a string, that’s all it needs. So you can create a struct that basically has fields that provide context. Maybe you might have a custom error that tracks a particular endpoint value, or a particular number of retries, or whatever it is. That way when you when you’re passing this error value right back to the caller, you can provide additional context beyond just doing the fmt.Errorf() and just embedding the existing error; you can use a customized error value that provides way more context than the simple error value.

Exactly?

Exactly. Exactly.

Yes.

Yeah, it’s all about the scope. So if I’m returning two values, if I’m trying to limit the scope of those two values to just my if block… Basically, I can do if, like, some variable for the value that I’m trying to capture, comma, error, and then my colon equal, and then call my function. And then within that same if block, I can then say if err != nil, then do something with the variable that I captured. So it’s all about the scoping. Basically, whatever value that gets returned, if you want that to be available outside of the context of your if, then you pre-declare your variable, and then you can call the function and then basically you’re gonna get back – you can predeclare your error variable, you can predeclare whatever value you want to capture… And then what gets returned from the call, those things will get assigned to those predeclared variables, and then you can do the if error nil check on its own line… But then what you can do if you detect an error, you can return early, and then if you decide to – basically, if the error check passes, and there’s no error, then you now have that variable that you’ve predeclared, and it’s now available to the rest of whatever scope you’re in. So to me it’s all about the scoping of the return values.

My general heuristic there is do it in one line, if you can.

If you can get away with it.

Yeah, if you can get away with it, do it. If not…

I mean, the multistage build with tiny images is really nice, just to be clear. But we actually don’t dockerizing our production stuff. We run it on bare metal, and it’s super-easy to ship a binary from like GitHub Actions straight to bare metal, and it’s still great. So I think it just more depends on your tooling, versus…

So there’s the tooling, there’s the developer side of things. When you were talking about that, my mind quickly went to the operational side of things. There are some advantages you’re going to have when you create an image of your executable for certain deployment environments. So if you wanted to have orchestration tooling - the Kubernetes of the world, the Amazon ECS’es, the Fargates, and all these services, these are container orchestration technologies. And you can manage availability, and having multiple instances running, and you can manage resources allocated to those programs, you can handle the networking between components, you can – there’s a lot of benefits to dockerizing, to having images that can be deployed when you’re using container orchestration tooling.

In the case of Ian, perhaps they don’t have those requirements, so they can deploy the binary directly. And I’m curious to hear how you manage your updates. Maybe you use like a package manager or something for the particular OS. That’s what I would probably think to do. But yeah, to me it’s all about the operational context for your program, and whether dockerizing, making images of your program makes it easier to scale up or down, and things of that nature.

Yeah, so I mean, we use Elastic Beanstalk through Amazon, which is just dead simple and easy. So that’s why we use it. And it can still do all the autoscaling stuff and all of that.

They just abstract that from you.

Exactly. It’s fully managed. Exactly. And yeah, for our small team, it works very well. I would recommend. A little bit weird, but we really haven’t seen the need for the big container orchestration type stuff.

Yeah. When you were talking earlier about how do you share a Python script, my mind immediately went to some – man, I have some battle scars from trying to share Ruby scripts back in the day with teammates and whatnot, variations, and which version they had, or little quirks in how their gem package management was configured, and weird issues… It just made it impractical to try and share tools between each other. And then you try to basically solve the people problem aspect, and communicate “Hey, this is the version of the things that we’re using. You have to be on Ruby this; you can’t be using non-standard VM there.” You have to end up basically creating all these rules around – sort of trying to standardize so that everybody’s on the same thing… But it’s way easier to just say “Hey, my coworker here uses Windows. Let me just create a binary. Let me just create an .exe for them. Just build an .exe for them.” “Oh, my friend over there uses this particular flavor of Linux? Okay, let me build something for you. I’m on Darwin, I’m on a Mac. Let me build something for me and my other Mac-using teammates.”

[00:30:28.25] And the build process in Go - man, I can’t tell you how amazing it is. Just by specifying a couple of environment variables, you’re just specifying what operating system and what CPU architecture am I building for; it takes care of the rest. It’s magical. Something like that is incredible if you come from build farms; if you have any experience with build farms or anything like that, you will love the experience for building different binaries, different executables for different platforms. That’s one of the strengths of Go.

The new framework?

I don’t even know where to start with that…

[laughs]

I mean, I think the main thing was when Go came out, it didn’t have a package manager. You had a single gopath, and you had to put everything in that gopath…

Everything in there.

…or you couldn’t find it. So there’s the necessity of the namespace, and then the convenience of being like “Oh, this is just the name is where it exists. So as I download code, it’s all in the place.” I guess I don’t know beyond that, if there’s some history there.

Well, we’re not gonna drag Duarte through our interesting package management history in the Go community. It’s safe to say that the module system that we have now - it’s not everyone’s favorite, but just like gofmt, everybody pretty much loves it at this point, only because of the standardized nature of it. Well, maybe “love” is a strong word, but it’s good enough, it’s good enough. And it gives you sort of a reproducibility and a lot of other benefits.

But to your question, having a github.com base domain name with your GitHub handle - and it’s not just GitHub. It’s GitLab, it’s BitBucket, whatever your code repository tool of choice is… Basically, it makes it easy for the go get command to know exactly “Where do I go and grab this particular package?”, to pull it down and store it in a local module cache. That way I don’t have to keep pulling it down every time I do a build, basically.

[00:34:03.04] And when you are looking at code, literally you just copy and paste this package path into your browser, and to the repo you go. So it makes it easy – there’s so many benefits to having the path. It provides namespacing, there’s not going to be any conflicts, as to naming conflicts… Maybe you have two packages named foo; if one’s under a particular github.com/username/foo, that’s gonna be very different. Just by looking at it you know it’s going to be unique, because there’s no two packages that are gonna have to have the same naming, and you don’t have to jump through hoops to do any funky aliasing or anything like that to know which one you’re dealing with.

So there’s a lot of benefits to doing it that way, and one of the things I try to teach developers is from the get go, when you first install Go, if the install doesn’t create it for you, you should create it yourself - there’s gonna be a Go directory that gets created under your root username account. So if you’re on a Linux machine or something like that, under your home there’s gonna be a Go folder that gets created. And this is where I typically tell folks “Look, when you download any packages that you do a go get, there’s going to be a PKG folder that gets created under this go directory for you”, and this is where all the module cache lives. This is where anything that you download that you don’t vendor - and we can talk about vendoring if you’d like - is going to go into that particular module cache.

And then from there, you can create an SRC directory where you put your code. So I create an SRC directory, I have a github.com subdirectory, followed by my jboursiquot GitHub handle… So I have this the hierarchy that allows me to basically say okay, anytime I’m looking for code, or anytime I need to check out code, or fork a repository, whatever it is, I need to bring it locally, I just go into my go src github.com slash whatever username slash project name. So it simplifies everything; I know exactly where to go look for the particular source code for a project that I’m working with directly, and I know exactly where to go in a module cache to find the module code I’m looking for. And even when I vendor my dependencies in a particular project, in my vendor directory, there’s a hierarchy in there; there’s a github.com, followed by user name, followed by whatever project that I’m relying on as a dependency. So it just keeps that consistency across the board. And I can even use multiple versions of the same dependency, because the module system follows a semantic versioning mechanism.

So I know it might feel a little odd, it might feel weird at first, but trust me, we in the community have been burned… Not burned, but we’ve gone through some hardships, and we understand the value of what we have now. It’s not perfect - obviously; nothing is - but it works well enough that it creates that predictability and that reproducibility for our builds.

Good. Glad you like it.

Yeah, I do think how we do the package naming really just fits well with like the Go ethos of just like stupid simple.

Keep it simple. Yeah.

I can’t think of a simpler way to do it than just where it exists.

What else you got?

I’m excited for Johnny’s answer on this one. I still struggle at this. I’ve still made mistakes that haunt me. A new thing I wrote last year, I’m like “Oh, that’s wrong. I would not do that again.” So yeah, I’m also interested.

Well, I wouldn’t go that far. Like you, I’ve come across many different approaches. And I think we even had – we’ve had attempts in the Go community at specifying sort of a standard structure for Go projects etc, etc. And even the Go team themselves have shied away from saying “This is how you should structure a project.” At first, I thought it was not great that there was no standard approach. And then the more I use Go, the more I realize “Okay, I have flexibility.” I’d rather idioms – if you’ve ever heard sort of… What’s the term, convention over configuration? …from my time in the Ruby on Rails world; that’s like a mantra. And I think we’ve carried that into – or I carry that mentality into my Go projects. But I will say that at a very minimum, once I know what kind of project I’m building, I follow a couple of different patterns. If I’m building an executable - ultimately, the project may produce executables, but also itself be a package. The first thing I do the moment I create my repository, I’m going to create a CMD folder. Inside of that CMD folder, I’m going to have another folder representing the name or names of whatever executables I intend to build. So if I want to build two executables, one named foo, one named bar, in my CMD folder I have a foo folder and I also have a bar folder. And in those folders, I have a main.go for each.

So now I know this project – the moment I go into any repository, if I see a CMD folder at the root of that project, I know this project at least is going to build one or more executables. That project itself naturally will be a module in and of itself, and I know that there’s going to be Go files that are at the root of the project, that are going to have a package declaration, and I know if this project can be imported in some way, somebody can do a go get, github.com/username slash whatever project is… So that means even if that project has some executables in there, as long as there are things that can be used at the package level, as a library within a project, somebody can still go get that repository, but they can also do a go install to point to the executables in the CMD folders.

So you notice that that’s the only set of folders that I start with. I don’t try to create a bunch of hierarchies, I don’t have model folders, I don’t have a types folder… Not that you can’t have those; these things are – I find myself nine out of ten times I find myself starting to introduce that hierarchy and that division once the project is mature enough for me to know that –

Once I need it, exactly. I don’t prematurely start to create a structure. This is not like a framework where you initialize a framework, you say “Create this project”, and it spits out a bunch of different folders, and files, and everything else. I’m not using a framework. So I create these hierarchies when I need them, and I go out of my way to not create unnecessary hierarchy, because the more folders I create, the more packages - and I’m gonna use the term subpackage, but there’s no such thing in Go. There’s only packages. The more of those nested packages that I create, the higher the likelihood that I might have some sort of cyclical import kind of going on there.

[00:42:16.16] If I need to have one package, use something from another package… It creates all this sort of extra stuff that I have to keep in my mind now. At most, if I’m building a server, a web server, for example, I might have a Handlers folder at the top level, where I can keep my handlers in there, that way I know where all my handlers are, and my handlers are probably going to share some code between each other, but that’s going to be relegated to the handlers package only. And then everything else at the root has to do with – think of it as a core business logic behavior that exists at the root of my project. And then I have a CMD folder with those executable folders in there.

That is 80% of all the projects, that structure, that approach of simplicity for those projects, that’s what they are. Again, the bigger the project - you don’t want to go into a Go repository and you’re just scrolling down for days, because all the files are all in one at the root of the project. I mean, maybe that’s suitable if that’s truly the case, if everything exists at the top level package… But you don’t want to necessarily from the get go try to create all these hierarchies, and folders, and folders inside of folders, because that’s just going to create a mess for you.

Good, good.

Yeah. So think of it this way. The folder hierarchy is more so for you and I to communicate as developers than it is truly for Go. Because at the end of the day, the Go tooling grabs all of your files, and the only thing it cares about is a package declaration at the top of each file. That’s the only thing it cares about.

Yeah, indeed.

[laughs]

I don’t think this will ever be like a solved problem, because I do think Go forces you to actually think about your architecture and your packages. And you’re just gonna have to figure it out for every project, in a way that kind of isn’t fun, but I think it makes it better at the end… Assuming you don’t really mess it up.

Oh, yeah.

[00:46:10.21] This one’s hard for me… Because I see the benefits of the ORM. The main monolith I work on uses an ORM extensively, and I hate it. But… But…

I will say, as far as like developer velocity goes, for 90% of the time it’s way faster. But I still don’t know if that 10% edge case makes up for it. When you’re digging through Gorm code, trying to figure out why I’m inserting rows twice. Like, that kind of thing. I don’t know, I can’t think of a good example, but… Yeah, that 10% edge case is the worst part, and I don’t know if it makes it worth it. I don’t know, that was a rambling answer… Johnny?

I can add my two cents… The ORMs and frameworks and all these things, that class of developer tools - it’s all about total velocity. It’s all about how quickly can we move to get this out the door, and we don’t have to reinvent patterns for data access, and everything else. An ORM helps alleviate that. If everybody knows – to use Ian’s example, if everybody on the team knows Gorm, then we know what Gorm code looks like, and we can intuit basically just by looking at a line of code exactly what data we’re looking for, where conditions are etc, etc. It’s easy, because everybody knows what the API looks like.

That said – and in most cases, if you’re just doing some CRUD work, it’s perfectly fine. It’s fine. Everybody knows how to use it. We can get moving quickly, and nobody has to spend time coming up with yet another way of doing the data access.

In some cases where you need tight control over performance, or the kind of exactly how you want your query to run, exactly what you want to join on, all these kinds of things in the case of relational databases, you have the ability to execute raw SQL statements using these ORMs. I think most of them, if not all, have the ability to run an exact – if you don’t want any output back, or run some sort of API call you can make to execute the raw SQL string that you’re going to provide.

So you have the benefit of both approaches with these ORMs. And if you really want to, and if you want to execute your own query against a raw database connection - well, you can just basically say “Hey, give me the actual SQL connection and I’m going to actually do my own stuff against there.” If you want to see a case of the ORM adding extra stuff, or is it just the SQL that we’re sending that’s maybe taking too long to return, or whatever the case may be. So ORMs are good for that particular use case.

And the ORMs will tend to sort of dynamically, based on the types you declare, based on the structure you declare, and struct tags that you add to fields and whatnot, they’ll try to do the best they can to infer a lot of things for you and manage creating tables for you, if you want them to, do migrations, and all kinds of crazy things. But in a case where you don’t want an ORM to actively be in the path of every query you make, or trying to infer columns and things based on the fields that you have in a struct, you can actually pregenerate things ahead of time. You can use – I think we mentioned Gorm on one end of the extreme. On the other end of the extreme you have things like sqlc, or even sqlx as packages within the Go community that have been used extensively. And a lot of folks have preference for those, as opposed to using an ORM.

So these things are more lightweight approaches to doing database access, over a full-blown ORM, which might have a little bit more bloat than you care for. So again, as with all things, it depends… But I think in my projects I find a happy medium between using an ORMs to move fast, and basically dropping down to writing raw SQL. Oh, by the way, you should always learn SQL. That’s a skill that you’ll never gonna regret. Regardless of language, or framework, or whatever, 10 years from now, whatever, you’re not gonna regret learning SQL. So learn SQL.

[00:50:13.04] And when I need to actually drop down to actually writing my own very SQL – especially trying to get things like window functions through an ORM, incredibly hard. Incredibly hard. I just drop down to raw SQL whenever I need to do things of that nature.

Yeah, some ORMs will come with that built in. I also know of specially dedicated projects that do migration; that’s all they do. So again, it depends on your use case. But try different things, and don’t rush, don’t commit too early to any one thing. Same approach with the packaging, how you structure your project. If you know you’re going to use an ORM to get off the ground quickly, you just do it, because that’s what everybody on the team knows - it’s fine. If you then have specialized cases where you need to do raw SQL, you can do that, too.

Yup.

Yeah, you’re not wrong… I mean, SQLite - SQLite is and has always been one of the favorite developer tools out there, regardless of language. It’s small, fast… You can go far with SQLite. Some people even ship with a SQLite database if they need to package a database with their projects and whatnot… It’s just a great all around project. It’s been around for years, and it’s just a great all around project.

The Go community liking SQLite - I think that’s something that predates Go. People just like SQLlite, and just say “Hey, why not put both things that I really like - why don’t I put them together and use them?” So I think that’s what you’re seeing; the love for both of those technologies together kind of makes it like a pleasant workflow.

Yeah, we’re a lot of things, but extreme doesn’t tend to be one of the things that we ascribe to Go developers. Yeah.

Generally, the approach is if you know how many goroutines you’re gonna need, if you need to manage the number of goroutines, and you know how many you’re going to have for any given task, a wait group is usually a good starting point. And then using your example, if you know you’re gonna hit five different websites, and have five different workflows running - you call the website, you get some stuff, you parse the RSS, and you start feeding that into an LLM, each one of those things could be a separate goroutine, a separate thread of execution. Because these things don’t depend on each other. One website doesn’t depend on the next, so you can simply have those tasks happen in their own goroutines. In this case you know you have five websites; you could trigger five different goroutines. You just have to wait for them, hence the name, wait group. So you initialize your wait group, you say “I’m going to add a count of five”, and then for each goroutine you launch, you’re gonna want to decrement a counter, and there’s a process for this, what we’re calling a done function, a done method in your wait group, from inside of the goroutine you launch. And then every time a goroutine returns, it decrements the counter automatically. So whenever you know exact how many goroutines you’re gonna need to launch, a wait group is usually a great starting point.

Channels are meant for communication between goroutines. So if you need maybe in your main thread, your main goroutine, if you need some information to be passed back and forth between your main goroutine and whatever other goroutines you’ve launched, channels are great for that. Channels have directionality. You can have send only channels, bidirectional channels… Again, it’s really used for communication purposes. And then there’s other ways you can use channels, almost like a queue kind of behavior, but that might be a bit more than we want to get into now. But really, the channel is all about the communication. And obviously, Go comes with things like mutex and whatnot for doing mutual exclusion, and that same notion exists in other programming languages, so I don’t have to explain it here… But for most people, I usually say “Hey, can wait groups help you accomplish this task if you know how many things you need to do ahead of time?” I usually say start there.

[00:57:58.06] No, not exactly the same thing, and not exactly in the same way. You’d be adding some overhead if you use channels. In the example that we just walked through, you’d be adding some overhead to do it via channels. But it’s doable, it’s just you’d be adding a bit of overhead.

I would definitely mirror… Like, kind of reach for channels last. I know it’s Go’s flashy feature… But I’ll often grab for a mutex way before I grab four channels. I think it’s easy to reason about… And sure, there are good reasons to use channels, but almost always I’ll avoid it, if I can.

I also wanted to add, there’s a [unintelligible 00:58:32.21] which is very similar to a wait group, but basically, instead of directly launching goroutines, you give it a function that returns an error. And so then you can wait until everything’s done, everything’s returned nil, or you see the first error. And also when you create this group, you can give it a context, so that it cancels everything after the first error. And I find myself reaching for that even more than wait group.

Yeah, that’s a great package.

Maybe my single favorite Go package.

Errgroup.

It’s in the experimental sync package.

Right. The last thing I’ll mention before we move on to unpopular opinions… When you’re doing concurrent work - and one thing I wanted to bring to your awareness, Duarte, when you mentioned it… Go is all about concurrency. You can enable your code to run in parallel. But concurrency and parallelism are not the same thing. You can have concurrency on a single CPU, for example. But to enable parallelism, having different things, multiple things happening at the same time, you’re gonna need multiple cores for that, and that’s where the parallel comes in. Go allows you to write code that can be made parallel, but out of the box it’s all about concurrency.

The other thing [unintelligible 00:59:50.02] concurrency is that you have to know which types are thread-safe. Maps, for example, are not thread-safe. So whether you serialize access to those via a channel, or whether you use a mutex, like Ian’s saying, you’re gonna want to protect that shared resource. Same thing for slices, they’re not thread-safe. Or counters, and things of that nature. If you need to increment a count, and maybe you want to increment a count from within a goroutine, you don’t want to use the regular, basic integer types, for example. You’re gonna want to use an atomic version of that instead. Or, again, use some lock around whatever the resource you’re trying to access. So that’s the other thing I’ll just throw in there… Whenever you have concurrency or multi-threading in your programs, you want to protect shared resources.

I always forget about the atomic packages. I need to remember those more.

Yeah.

Hey, you know, we’ve all been there. I think we all –

[laughs] Trust me, I did the same thing. I was one of those people like “Oh, channels everywhere” when I first came to Go. I was like “Yeah, this is amazing.” And then you realize “Oh, man, I’m totally overdoing it, and shooting myself in the foot.”

Yeah, you’re good. You’re good.

Yeah, we like to have patterns, those idioms. Yeah, indeed, indeed. Alright. Well, let’s switch it up to Unpopular Opinions, shall we?

Jingle: [01:01:34.18]

Duarte, I know you came with one in the holster. Lay it on us.

Yeah, I don’t disagree. I think most of the time it’s a matter of prioritization. And the other nuance I’ll add to that is if after you’ve done prioritization, which is the first step - if after you prioritize, you still don’t have enough time to get to everything you want to get to, that’s a signal that you have too many things on your plate, and that you need to give up some things. You may have a desire to do all the things, but there’s still 24 hours in the day. Ideally, you still need to sleep eight of those, and you still have to go to work, like most of us, eight plus hours a day… Whatever’s left, if you don’t have enough time to get to it, even after you having done your prioritization, if you still can’t get to everything, that just means you have too much on your plate. And I say that as sort of living proof in my household of that particular problem. At the end of the day, I’m like “Man, there’s so much I’d like to get to.” And I go to bed sort of kicking myself, “Man, I couldn’t get to this man. Man, I couldn’t get to that.” This is sort of the universe telling me that “Hey, you just simply have too many things on your plate.” Because I’m one of the people who tracks my time. My calendar literally has a block for every – I can tell you four weeks ago what I was doing at this hour, because it’s on my calendar. I track my time to that degree. I know, I’m a little bit anal about it. But I want to know that at the end of the day I maximized my time. Because time is the one thing I can’t make more of.

But at the end of the day, I’m going to bed thinking “Oh, man, I couldn’t get to this. I couldn’t get to that.” And if I do that enough times, I’m defeating my own self. I’m beating myself up, because I couldn’t get to everything. That just means that I need to give up some things. I just really don’t have the cycles in my day for it. So…

Of course. Of course. Yeah, of course. It’s a natural human thing to do, I think. I’m like “Man, I don’t have time to do anything.”

When I find myself thinking that, I try to change it to “I’m not motivated to do X.”

That’s interesting. Yeah. Good stuff. Well, I don’t have an unpop. Ian, I’m not sure if you do…

Maybe I go back to the channels vs. mutexes. I think in almost every case mutex is probably the right tool for the job. Not every case. I don’t know, maybe skip.

You’re going back on it. Okay. Well, maybe this will be my unpop - I think you should be very careful with your use of mutex, especially if you have a high throughput situation, because mutex will have a tendency to slow things down. Because once you lock a particular resource, everything is paused. Nothing else can access it other than the goroutine for which you open up the lock. So if you have a high throughput situation, whatever amount of time a particular goroutine spends updating a particular shared value, nothing else can work. So there may be more efficient ways of doing what you’re trying to do without a mutex. Again, in some simple cases, mutex is a go to, but you have to definitely consider the cost of using the mutex… Because the world stops until the goroutine is done for that particular shared resource.

I’m just trying to wrack my brain for one of those high throughput situations where even like a read/write mutex wouldn’t solve a lot of that… I don’t know.

Yeah, again, it’s one of those where you have to look at your use case, and make the appropriate decision.

I mean, when channels are the right thing, I’ll use a channel. But I just don’t think that’s very often.

I think Johnny said it really well earlier, when it’s about like communicating.

I don’t know… It’s also like one of those things where you just know it when you see it. Like, I don’t know, I can’t put words to –

Well, are you gonna tell a new gopher “You’ll just know it when you see it…”? [laughter]

Yeah, all you have to do, Duarte, is to spend the next couple of years hacking away at Go, and then you’ll know it when you see it… Says Ian. [laughs]

I know it’s not helpful…

Awesome. Thanks, Duarte, for coming on the show, and definitely for suggesting the episode. Dear listener, if you too want to come on Go Time and talk Go, whether you have a bone to pick with the language and you want some ears to hear you, or you have some questions along the lines of Duarte’s set of questions today, definitely reach out to us and we’d love to have you.

Alright, let’s wrap this up.