Structuring your Go apps

Hello everybody, welcome to Go Time! This week we’re gonna be talking about structuring Go programs. I think this is a question that pretty much everybody asks at some point, and we all probably make that mistake, or we do it what we consider the wrong way later, and yell at ourselves a little bit.

Joining me today I have Johnny Boursiquot…

Oh, yeah.

Did I say it right this time?

You did say it right this time.

Alright! I’m learning. I even had my recording from last week to say for sure. We have a guest, Cory LaNou [LaNew], or LaNou [LaNoy]?

LaNou [LaNew].

LaNou. I didn’t get a chance to ask you beforehand, in that 20 minutes of downtime.

[laughs]

And we have Mat Ryer, who wanted to take the backseat today, and just pretend like he was a guest.

Thanks for having me.

Alright. So if Mat sounds better today, it’s because he’s at his company’s HQ, in their podcasting lab, or whatever you wanna call it…

Yeah, I’m in Costa Mesa, which is South of L.A. So it’s California weather. I’ve actually come here to cool off, given the heatwave that’s happening in Europe at the moment… And they wanted me to mention that they are hiring. So if there’s gophers out there that are looking for work, careers.veritone.com… And yeah, do some computers in that.

And because I’m in the Go Jobs channel all the time, the very first question you’re gonna get is “Is that remote?”

That is a great question, and you have to ask through the process. I’m encouraging that, but currently that’s not the way that they do it… But I think if there’s enough people that ask that question, yeah.

Okay, awesome. So I wanna start off by I guess just discussing why is app structure something that we all care about so much. Why is this a question that everybody asks, why is it something that anybody who starts writing in Go is gonna be like “How do I structure my code, where do I put my files?”, why is this something that matters?

Yeah, I think that’s a great question. Essentially, when we’re learning Go, we tend to do it just in a single file, or sometimes we even use the playground that’s hosted online… We think of it at a very small scale. And in practice, when you’re gonna build something, the first thing you have to start doing is making files, and folders, and things. So it’s the first thing you do, but it’s the last thing we ever talk about. I feel like this podcast is gonna hopefully shed some light on this.

Essentially, for me it’s about maintainability. The structure of the program is really all about “How can you quickly go to the place you need in order to work on the thing you need to work on?” Good structures, good file names, good folder names - this stuff all helps with that process.

[04:21] To add to that, I think there’s a certain level of expectation, especially before you start to navigate a project, or if you’re in a team with multiple projects, multiple repositories. You expect that when you’re going from project to project, you’re gonna have a similar structure, or whatever it is that your team has standardized on. If you’re building web applications, you’re gonna expect the same structure from project to project. If you’re building an API, you might expect something different. If you’re in back-end services and that type of applications expect a certain shape for the repository and where things go. This definitely helps with readability, and knowing where to go, how to navigate repositories and projects.

I think what we will definitely touch on is what is the expected - or what should be the expected organizational structure of certain kinds of programs. So if I’m writing an application that is based on a framework, there’s gonna be certain expectations that the framework itself is imposing. If I’m writing some sort of gRPC service, there’s some things that you’re gonna do that you won’t do with a traditional web application. So again, it’s all about the expectations that you’re setting by adopting a particular layout structure.

Yeah, and Go itself puts some constraints on, too… Because obviously, everything inside a folder is grouped logically as a package. That’s significant. We have to know about that.

One of the interesting things that’s different for Go to other languages is nested folders don’t really do anything special. They don’t represent nested packages, or anything. So there’s no special privileges by having folders nested. Sometimes, in other languages, that is the case. So there are some rules that are worth learning, and necessary to learn. But of course, within that, there’s still lots of different ways to do things… Like, do you just bundle everything into one package, do you break everything out into tiny packages, or is it somewhere in between? And when do you do these things? All that is kind of almost an art form on its own, don’t you think?

Yes. Let me ask this question, because we’ve all been doing Go for a while… When you start a project - and let’s say it’s gonna be a medium-sized project - what’s the first thing you do? Do you just create your main file? Do you create 20 different folders? What is it that you do?

My guess is there’s not a right way to do this… But I’m very much the type of person who just starts with one flat structure, and then later, once I’ve got some more code there, then I’ll start trusting myself to separate… Because I’ve found that everytime I’ve tried to separate ahead of time, it’s – you can plan for a lot of things, but I feel like you miss something. You miss that one edge case or something that throws your whole thing, you have to throw it out the door, throw it out the window, whatever you wanna say… And I’ve just found that I do a little bit better if I get something down that I can work from, and then I come back and refactor.

Yeah, so you sort of let the structure emerge from the work you’re doing, rather than trying to imagine it upfront… And I think that as a principle is great; that’s actually how I do it, too. I tell people – because when you go and look at an open source project, it’s got its own structure, and it feels important, and it feels significant, so it’s very natural for people (especially junior devs) to think “Well, I need to learn how they made those decisions.” But really, they probably evolved, too. They probably started with something different, probably flat, and they started to pull things out as and when they needed to. And doing that in response to real pain is the right time to do it, because it’s very clear the problem you’re trying to solve for yourself.

[08:19] If you try and imagine upfront, like you say, Jon, you might get it right, but maybe you won’t. As we actually do the coding, we learn so much. So yeah, it’s something that I think people should be a bit more relaxed about than they are. Don’t worry about it. You can worry about it later.

I say worry about it a little bit, just the right amount, and whatever that is for you; that’s gonna be a sweet spot that you have to find. For example, my sweet spot is I define – now I’m working on a project and I have some sort of a binary as a work product. So the first thing I do is I open up the project, and then I go and create a cmd folder, followed by the name of the project; that way when I do my builds I get the right name for the executable… And I create the main.go. My main.go - my objective is to keep it as thin and light as possible.

So at that point I’m like “Okay, from this main.go, what am I gonna call in? What [unintelligible 00:09:20.18] the rest of the application which is basically providing the package naming structure. What am I gonna be pulling in? What do I need to actually start doing work from this main.go?” And obviously, in main.go, this is where I’m sort of reading options, arguments being passed in, if I have to do initialization or configuration reading from the environment, wherever they’re coming from - I’m doing all that in there, and basically trying very hard to keep any sort of business logic out of that main.go. Then from there, once I’ve got a starting point there, then I basically come out, if you will, of that main.go, and install the rest of the package.

In the past, I used to follow an approach whereby I’d create a pkg folder at the root level of the project, and then start to create things in there as well. I’ve sort of gotten away from that model. Basically, I try to keep Go files at the top level now… But I’ve gone back and forth, and it’s not a wrong way to do it; it’s just a slightly different way to do it - you package up all your go files in their own directory. This is usually useful if what you’re building has a bunch of different assets in there. Maybe you need to produce some HTML, and CSS, and JavaScript. I don’t like all that stuff being right next to my Go code, so I might create a pkg folder for that. But other than those slightly nuanced things, basically I start with that main.go for the executable, and I work my way up from there.

I used to do a thing where my main program, the command, the program itself, the package main thing - that would always just call out to another package. That helped me with testability, and it helped me with the fact that this package could be used now in more places… But I kind of stopped doing it, because I realized I never used that package in any other place; I only ever ended up using it in the main program. So now I sort of bundle things – I still won’t put much in the main.go file, because I feel like there’s more storytelling opportunities with having different files alongside that.

Johnny, do you tend to do that, where you call out to another package, or do you have the actual logic inside that main package?

[11:44] I usually call out. It’s really hard for me to – I’m usually really trying to keep package main as small as possible. I almost immediately call in a different package… Because to me, having that entry point - that’s the only purpose it should solve; so I’m not defining domain models in there, I’m not creating the experimental, sort of, you know, when you start out, everything’s inside main.go. I find that to be a little messy… But that’s the way I think; it’s not right or wrong, that’s just the way I think. So 99% of the time I immediately pull in a package.

One of the things I do really like about Go is the ability to take something that’s in main and pull it out and throw it somewhere else; it’s so easy to do, that I think that’s why when you say there is no right or wrong, I completely agree. There’s some languages where I feel like you have to define all the stuff upfront, and that’s not the case in Go at all. And because of that, you can kind of get away with just moving things around.

I will say that I would imagine that part of our differences in how we design stuff also depends on the context of how we’re doing it, whether or not we’re a big team, what we’re building, all sorts of things like that. I know I personally tend to work on projects that are very small; at most 2-3 developers, that’s very common for me. And as a result, what I have to worry about is very different from what somebody with a bigger team is gonna have to worry about.

Yeah, that’s very interesting. I think context probably does matter. And also, if you’ve already got an existing codebase that has some patterns, it’s probably worth, for the sake of it, just following those patterns, just being consistent… Even if you don’t love it, or there’s some trade-offs you’re not happy with. Consistency is probably important, if you already have that.

You always know there’s gonna be a 2.0, right? So you can push it down the road… But I agree, I think productivity is really important. One of the things that I really try to focus on when I talk to people about package design is I try to get them to understand “You need to ship the product. You need to get it across the line, and we can always refactor later.” And again, I’m not also arguing for “Oh, we can always push everything down the road, and someday we’ll have this massive refactor.” Clearly, there’s gotta be a boundary and there’s gotta be some give and take. But primarily, I find that - just write the code, get it to work. Things kind of follow it on their own after that anyway, and Go tends to be really fast to refactor.

So it’s not like other languages, where you know you’re writing all this huge technical debt. I think that’s the one thing in Go – yes, we have technical debt, but I’ve never felt technical debt in Go like I have in any other language; I think that’s pretty refreshing. It allows me to kind of just throw caution to the wind a little bit, get that code written, and then refactor it later, and not really worry about missing my deadlines.

I think the lack of OO in class hierarchies really helps with how good it is at refactoring in Go. We tend not to build these big, complicated structures that are then difficult to pull apart later. Everything tends to be quite flat, and that gives you more power.

But what about final names? Do you have any – now again, there are some constraints and some conventions in Go. If it’s _test.go, that’s a test file. So are there any other things like that? I don’t think there are… But do you have other patterns for file names, and things?

I think one of the patterns - and this isn’t my pattern, by any means - is if you have a package called foo, your entry point is foo.go. Because a lot of times people always ask me “What’s the first file?” So I find if you follow that pattern of naming your file after your directory name, you know that when you open up that directory, the first thing you should go to is there. That’s kind of where the meat of it is, the description of it… Things are gonna fan out from there. That helps a lot. That’s one of the bigger ones that I’ve adopted, and it’s worked well. It feels really strange at first, because it feels like an echo, and you’re always told in programming “Don’t echo anything!”, and here I’m gonna echo this naming pattern… But for me it works out fine.

[16:09] I think even having main.go – you don’t have to name it main.go.

No.

So that’s just kind of a…

Convention.

Yeah. We’ve even said main.go here several times inferring that we assume the main function is in there, and if somebody wrote something that didn’t have it, it’d be really confusing.

Yeah, that’s true.

I don’t know beside that… I’m trying to think if I remember any projects that did anything with OS-specific build files, but I don’t remember exactly what I saw.

Yeah, there’s another convention if you have _linux, and _darwin, and things. The tooling does take that into account when building. So there are some rules that are worth learning, but what about – do you tend to structure your packages by responsibility? If I’ve got a concept of comments in a program, I’ll have a comments.go, and then I’ll try and have everything to do with comments in that file. That’s different to other languages that I’ve worked with in the past, where the comments model is amongst other models inside that. How do you do it?

I personally find that if I’m not working on a framework where somebody’s conventions are being enforced, where there’s a configuration by convention kind of thing, things that are sort of expected to be named a certain way, I tend to sort of run away from that model quite quickly. I follow the same domain-based logic. To use your example, comments - I might have everything having to do with comments in there. So I wouldn’t have a models folder, or a models.go with a comment type in there. I might go with the name of the thing; that way it not only keeps the file smaller, because it only has to do with things related to comments, but it’s also good for navigability.

I can jump to what I need to, I know exactly where to go. And if I have a comments_tests.go, I know all my tests for my comments are in there. Or I have a models.go, or types.go, I tend to see some of these things in projects here and there, or a models folder with the different models in there - that means I still have to jump to some… Maybe there’s a controls folder, maybe there’s a utils folder (God forbid). I’m organizing by the domain I’m working with, not by some predefined, preset way… Unless I’m working in a framework where everybody understands what the expectations of the framework are. Then we can all be productive, because we know all the models are in the models folder, we know the controls are in the controls folder. So it’s not wrong by any means, it’s just what is expected on your team.

Another thing I find, especially because I teach a lot of corporate groups, is that when you try to approach them and tell them things like “Comments go in its own package”, the first thing you get back from them is “Well, how do I save it? I have to have this database layer, and I’m used to everything being in Models, and I pass a model to my data layer and it just saves it…”, and it’s really confusing, because you’re trying to explain to them, and of course, they’re new to Go, and you’re trying to show them “Well, no, you can use dependency injection. When you create that new comment, you can give it its method that saves it.”

What that allows you to do is really to create all your testing, like you were saying, around comments, in testing; all the functionality around comments in that package, into whatever you want to. Saving it has actually nothing to do with comments. That’s actually not a responsibility you care about; and that is probably the single hardest concept that you try to impart on people that are new to Go… And that really does come back into package and layout, because they immediately, instinctually, coming from these other backgrounds, are like “Well, no, it’s gotta be a model. It’s gotta have the data layer, and all these things”, and they just don’t understand…

And then, once you show them how to break it apart and you just inject it with something as simple as a one-line interface that has a Save, or a load - this interface is just itty-bitty, and now you don’t care about it at all. It’s not even something you test anymore. You’re just done with it. That is liberating, but it’s hard; that is a hard paradigm to understand.

It’s funny, because I feel like Cory and I deal with completely different audiences, in the sense that his audience is probably corporate people bringing a trainer in, and a lot of times I’m dealing with people who have not the most programming experience in the world, especially web development, and stuff like that. So I’ve actually taken the exact opposite approach of if you’ve seen something like Rails or any of those others, you’ve probably seen MVC, where it’s thrown into Models folder and a Views folder, and I’m like “It’s just gonna be easier to start with that. Let me just show you a way that you can get stuff going with that”, and then I very much do encourage people to spend some time refactoring.

Try a different layout, try reorganizing your stuff, try doing something different… But I want them to see that you can get productive very quickly in Go, even if you don’t do things the exact perfect way, or maybe it’s not the way you’re gonna do it ten years from now; but it’s good to have something to start with.

So I think that just knowing your audience and knowing what all they have to learn… Sometimes when you’re just learning about all these things, you have no idea how to encrypt passwords, you have no idea what CSRF is, you have no idea about all these different things… Trying to tell them “Okay, now we need to learn about context. We need to learn about what’s the context of this comment, where it could change from time to time” - that’s a lot to wrap your head around, so I think just having a way to skip over that is very valuable for some projects.

I think sometimes you can get caught up and think “Well, what if we have to refactor this later?”, but like Cory said, there’s always a chance for a v2… But also, it’s not that hard to start pulling things out and redoing it if you need to. I’ve done this to my own projects several times, and I actually do this pretty frequently as a way to – like, if I have an idea like “I wanna see what if I design my code like this, what does it turn out like, what are the flaws”, I will take an existing database, an existing app, and I’ll be like “Can I rewrite this in this other way?”, and see how it does, see what I think of it.

I think you should be very careful with the assumption… Unless you know for sure that you are gonna have a chance to do a v2… You should be very careful with the assumption that you’re gonna get a chance to refactor that code, because – I mean, under the right pressure… I mean, that thing is shipping as you built it. [laughter]

Yeah, I guess it depends on how you mean… I guess I say that in the sense that the vast majority of people who are learning, at least with me, are building something that is realistically not gonna be – like, they’re building the same thing I’m building, so… We don’t need a million of the same app, so they’re gonna probably build something new at some point. And even if you don’t get a full v2, you can go back and refactor some stuff. It’s not like you have to completely rewrite from scratch.

Sometimes when I say v2 I don’t necessarily mean “We’re gonna throw this whole thing out the door”, because I do agree with you, that’s way more rare in real businesses. I’d say that probably doesn’t happen a lot in real businesses.

I think this is actually a lesson for those real businesses that we’re talking about. They need to make the time and give to developers the space, so that they can do refactoring. It’s vital. And I say this a lot, but we’re obsessed with “How long is this gonna take to get done the first time?” We feel like we’re gonna create a product, and then it’s done, and then we deploy. And it’s a little bit like that, but in a lot of ways it’s really not like that much at all. If the project is long and it’s gonna be successful, then it’s gonna have a big maintenance cost to it, and part of that is refactoring to make your future maintenance easier.

Some small investment early can really pay dividends later. So we need to empower developers to do that, and engineers out there need to learn why this important and make the case for it as well. Because you can’t expect necessarily – a manager doesn’t know; they think of it as a “We do the dev, we do this and sometimes that’s literally how they organize it as well.

[26:35] So yeah, there needs to be more of a conversation around the value of refactoring, and you need to let a team know that they can make mistakes. You can structure the application however you like; you can get it wrong, because hopefully you’ve got a culture that lets you then fix that. But that culture is a privilege at the moment for sure, and it’s quite rare, in my experience.

Yeah, one of the projects that I worked on - I worked with some really bright engineers; I’m not sure why I was there, but they were pretty smart… And what they did is they really kind of instilled in me that you take your first pass, and then before you push it up, you refactor, and then you push that up, but before you actually start your pull request, you refactor again. And it got to the point where, you know, when I did any feature work, it went through two or three fairly significant refactors from myself… Because you get your rough draft, you get it working, and then you refactor, and then you refactor again, and you really kind of add that last polish.

At first, I thought – you know, because I’m an entrepreneur, I like to ship code, I like to get it out there and just be done… And I thought that it was just a lot of wasted time with that refactoring. And I’ve found in the long run it made you think a lot more about it, and then it also started making me think – next time I did a feature, that refactoring really lent kind of like a muscle exercise; I just learned to do things better upfront, and I didn’t leave as much technical debt upfront every time either. Those things make a big difference.

So that refactoring isn’t just coming back three months later. A lot of my refactoring happens before I ever actually ask for the code review.

Yeah, that’s really interesting. The point you made there is a great one. The refactoring is not just about fixing the code, it’s a learning exercise. And actually, if you do the refactor with another person, or even as a team… Sometimes it’s quite fun to do mob programming sessions; I don’t know if you’ve ever tried that. The learning you get - you’re right - informs the next time you do it, and that’s really how you build experience. That’s why sometimes we will jump over and create certain package structures just because of our experience. We might start doing that the first time, and that’s another thing I think junior developers see that and think “Well, I feel like I don’t know enough here.” So yes, it is a journey, and refactoring is a part of that learning process.

Yeah, I think this is very important, what you’ve just said there, Mat… My habits stem from having done these kinds of things over and over and over again, so I already have a well-worn path in my mind about what the final state should look like. So what is at low cost to me - early on I preset some things, because I don’t have to think about them that hard, because I know I’m gonna get there anyway, even before the first PR. So that big jump, that seemingly big jump for a junior developer may be sort of a barrier. They start stressing themselves out and thinking “Ugh, why can’t I do it that way? What am I missing? Why am I doing this wrong? Why can’t I get this?”

So if you’re listening and you’re a junior developer, it’s not you, it’s just experience. The experienced folks basically have seen enough patterns, they’ve seen enough things that they can start to anticipate certain problems and certain ways they need to structure their code, and get a leg up on that. But still, at the end, like Cory was saying, you’re still doing a continuous refactoring, even before you get to the first PR; even before people look basically at the code, you’re doing that refactoring, because “Oh, okay, I’ve set up these things here… I’ve thought about this domain a little more.” Maybe you got some information externally, and that informs your thinking, it informs your decision-making, and you go back and you start moving things around a little bit.

[30:29] But it’s an experience that’s at play there, not some pre-formulated way that you don’t know about; some secret that developers are hiding from you, that you don’t know about. It’s really experience, and less anything else.

I think what that generally means is as a junior developer, or as somebody new to Go, if you’re trying to figure out the right way to structure your stuff, your Go code, one of the great ways is if you join a team with experienced developers; you can submit PRs, and even if they’re not perfect, your team can walk you through the process of “Here’s why we’re gonna change these things.” It’s a really good learning experience.

So that works really well for a team that has some experienced developers… Do you have any advice for people who are either kind of on their own, or situations where maybe the entire team is new to Go?

I would always say open source is a nice place to go… And if you don’t have a team, the open source community can be that team, to some extent. Generally speaking, in Go I like to think that we are very friendly and welcoming to new people. Sometimes we don’t always get it perfect; you can say things and they can come across a bit harsh sometimes, but generally speaking, I like to think of Go as being quite welcoming… So that’s a nice way to do it. If you go and look at an existing project and contribute to a project, then that’s a way to get a bit more of that experience.

I think one of the other things, too - and I get this question a lot; it’s kind of related to what you’re asking - is when I’m training new corporate developers… Because you’re right, that is mostly who I’m usually training… They always ask me this question - “Can you point me to the best practices for package design, for package layout?” And the interesting thing about it is Go is still a relatively young language, and what we’re still finding is even - we’ll call it experienced Go developers that you want to be - that we’re still learning. If I look back at my code every six months, since I’ve written code starting in 2012, every six months looks like a completely different developer stepped in. Not even kind of the same person. And it’s shocking.

I can almost tell you what year I wrote the code based on the style it was written in, from my own code. It’s shocking. So it’s a really hard one to answer. So when I get the question, “What’s that best practice?”, it’s kind of like what we started the show out with - we put out five or six links and say “Well, here’s a whole bunch of ideas. They all have pros and cons, but there is no one single winner, and there is no best practice. The only best practice I will tell you is to write the code. That’s what you have to do. And then you have to refactor. But you can’t not write the code.”

I also tell people, too - and I think this is really good, like from what Jon was saying - don’t jump into the deep end. If you’re new to Go and you’ve seen this really cool talk on how to structure these Go projects, and you don’t even understand the basics of Go yet, you’re going to regret that decision, because it’s not going to work out for you, and you’re gonna create more technical debt that way than if you’d just done it some way that you can understand to refactor later.

Right. That can be a form of premature optimization.

Yeah. That would be my advice to somebody just starting out - if it’s a package, create that file with the same name as the package, like you say, Cory… Because by the way, you may not need more files as well, and then you just have that file. Then you also have the test file, always, next to it… And go from there, and set off on the journey, and write the code, and see. And ask for help, too.

[34:10] I’d be happy if someone tweeted me and said “Could you check out this package and give me your thoughts on it?” I quite like doing that. I always am happy to receive those kinds of requests. But like you say, we can’t just write a list and say “Follow these rules and you’ll be fine.” I don’t think we can ever really do that, if I’m honest. I think it’s all about trade-offs.

I wonder though… There’s something that I always do, and I always advocate for, and that is the monorepo of having everything for a whole company in one GitHub repo. I’ve done this, and I love it. I love the fact that a single PR can contain some documentation changes, the API tweaks, some of the UI to make those changes happen, maybe some database things as well… I love that that can all go as one PR, that gets merged in at the same time. You don’t ever have to worry in that case too much about components being out of sync. But like Jon, I work on tiny teams generally, so that’s much easier to do. But there are big companies that have monorepos.

Can I just say one thing for context? For everybody who – you know, since you’re all just listening… I get to watch one of the panelists shake his head no, while another one shakes his head yes while I’m saying that… So there’s definitely a difference of opinions here with some people, so don’t feel like you have to agree with Mat.

Oh, you have to. Mat’s always right.

[laughs]

One of the things about monorepos, I guess, to piggy-back on, is that dependency management is a pain. End of story. It’s always been very painful in Go, and we’re getting better at it; we’re not gonna go into that whole conversation, but what it does solve as a monorepo is it solves all of your local dependency management… And that’s a real kicker, because I’ve worked on projects where we had a monorepo, and then we split to a non-monorepo, and I think I spent more time by the end of that project getting everything in sync, because I would have five commits lined up. It’s one for this repo, one this repo, one this repo, one this repo. The one commit was the change I made, the other four commits were getting all the other packages to use that commit. And then you had to have all your testing frameworks set up to be able to use those right commits. It was a lot of work, and it was a real pain, and I really did miss when we had a monorepo… So I’m a big monorepo fan.

Earlier, Mat was talking about getting the freedom to refactor when you need to, and not being locked into one version of it, and how sometimes with management that’s hard. I do sometimes wonder if part of the reason microservices are so popular is because you know it’s such a small unit that even if that one thing is locked into some design you don’t like, it’s not the whole thing; it’s that one small thing. The next microservice we start from scratch, and we can learn from our mistakes.

So I will say that there are even – not just the cost of using it and implementing it and you’re making changes, like you said… There are other costs to having one big monorepo, where if somebody starts off with a bad pattern, and you kind of wanna just keep using it for consistency’s sake, a monorepo can get really big at that point.

Yeah, but it’s worth saying that just because you have a monorepo, that doesn’t say anything about the deployment or the architecture of your application. So it doesn’t mean you have a monolith because you have a monorepo. You can still have microservices in that.

At Machine Box, since the very beginning, for the whole life span, we had one repository, and there was a folder in there called boxes, and then subfolders for each of the different capabilities that we had; they were our products. Then we had our website in there too, we had some legal things in there… It was very nice and very simple, but each of those things were their own tiny, little component that we deployed in sometimes interesting and different ways.

[38:10] The thing with monorepos is that you’re gonna need tooling around how you do deployments, how you do CI and CD, and how you manage and orchestrate things. You usually need tooling with that… Whereby with a single project - I kind of understand what Jon was saying; if you have a small project, it can even be a microservice, but it doesn’t have to be small. I’ve seen very big microservices. [laughs] But for the sake of argument, if you have a small (relatively speaking) project that’s in its own repo, you can try certain things in that project, that perhaps is prohibitive in the monorepo. You can try a different deployment model, you can try a different approach of doing your CI. So there’s some flexibility; like everything else in engineering, it’s a trade-off.

I’ve seen the monorepo work well when you have enough engineers around that you can sort of peel off one or two of them to go build the tooling necessary to make the whole team productive. If it’s just you and maybe a couple of other folks - again, you’re gonna have to sort of experiment and see where the sweet spot is. But generally speaking, with monorepos, you need tooling and you need people that are gonna take care of that tooling in the long-term.

Yeah, I must admit that continuous integration is more difficult in a monorepo. For those who don’t know, continuous integration - you can get it so that when you create a pull request, it automatically runs a set of tests, and do some other activities for you before you then merge into master. So of course, if you have a big repo, you have to do extra work to figure out what’s changed, like “I don’t need to run all the tests, I only need to run these few that are touching what’s changed.”

At Machine Box we run every test, and that just meant we made sure that the tests ran extremely quickly… But yes, it is more difficult when you do that. And of course, the other thing is for open source projects if you’ve got a package that you’re gonna open source, then that should just be its own repo, because that’s just how we do packages in Go. But for company work, I must admit I’m in love with the monorepo at the moment.

I have a different topic real quick, while we’re talking about structuring Go projects… One of the questions I get that’s related to structuring is actually how many lines should be in a file, and how many files should you have per package?

Hm… See, people want to know what they should be doing, don’t they? They want to be told these answers to things, and I just don’t think there’s an answer to that. I’m quite happy with quite long files, as long as it all makes sense; and the way I structure Go files, I tend to have – I do it by importance. So if it’s a comments.go file, I’ll have the comments struct at the top, because that kind of sets the tone for the rest of the file. It tells you “This is the sort of data structure that we’re working on.”

And then you might have constructors, and important functions, and then you might have methods and things, and all the way down to maybe some little helper functions that are pulled out, just because I can unit-test them easier; they’ll tend to be down in the bottom of it. I don’t know, I don’t think there’s a maximum limit, but I think just naturally they haven’t grown out too wild.

[42:04] One thing that’s interesting to me about that is I feel like we instinctively just wanna split things up all the time. We don’t like big files… We don’t like opening a folder and seeing 50 Go files in there. For whatever reason, we just do not like it. It doesn’t matter if it is really easy to navigate, we’re still gonna think “Something’s wrong here”, for whatever reason. I guess it’s just weird to me at times, because you almost have to make the mistake of splitting something up too much before you finally take a step back and realize – it’s to the point where you’re making a package with one function, then you make another package with one function, and finally you’re like “Maybe I’m going a little bit extreme here.” But you have to almost do it before you believe it and before you catch on that that might not be the best approach.

That’s because we’re visual creatures, so we tend to look at – you sort of make decisions, whether you realize it or not; you’re already making decisions simply by navigating into the folder structure of a project. Again, if you saw Models, Controllers, Views, you might say “Oh, this is an MVC kind of thing.” If you saw a bunch of files into the root of the project, you might be like “Oh, maybe this is a library, something that’s meant to be imported.” If you saw a cmd folder, you’re like “Oh yeah, this thing’s gonna build and execute at some point.”

That’s part of the “idiomatic Go”, right? There’s some expectations that are set, by both a community and perhaps within your own team, that helps you sort of navigate and understand. This is part of the readability thing. But there’s no hard and fast rules… Like Mat was saying, there’s no hard and fast rule on how many lines you should have in a Go file, or how many Go files you should have in a folder… It’s all gonna depend on how you reason about the code. And the funny thing is some people reason naturally, they reason differently, so for some people the same project could be organized in 12 different ways, and it would still make sense when they come back to it. So if you’re not the originator of a project, when you start navigating one, you’re gonna have to put yourself in the shoes of whoever created it, if you can. It helps you think the way they might have perhaps thought, in order to assemble the project structure you’re looking at.

One of the other patterns I picked up too when it comes to “How do I know when to break a package up?” or “Where does it belong?” is that – like, let’s take your comments package that you had; all of a sudden, it starts to evolve, and I can feel like there’s two concepts in here… Suddenly, Comments is bigger than Comments, so you know it’s time to refactor this package. And let’s just take the concept maybe there’s a formatter; like, “I now have this fancy thing that formats my comments, but it’s got all this logic in it.” And it feels like – it’s related to comments, but it’s definitely its own concept at this point; it’s too big to belong in here. And what I’ve seen is I’ve seen people create a Comment_formatter package, or something like that.

This is where that nesting comes in, that we talked about earlier. It doesn’t have any purpose in Go, it doesn’t mean anything special, but under Comments package I would put a Formatter package. So now it’s kind of weird, because you’re gonna do formatter.whatever, so it’s gonna read like that in your code; it doesn’t have any comments-specific thing in that naming, but it lives underneath the Comments package. And what I find is that I like to drop that into that nesting structure, and then I always find I reach down for my packages, but I never reach back up from my packages. So Comments can reach into Formatter, but Formatter should never reach into Commenter or up the chain anywhere. In fact, Commenter typically won’t reach anywhere up anywhere, in my entire system. That’s one of the patterns I picked up at work, and I’m curious if you’ve picked up any patterns like that.

That’s where your internal package would benefit as well. For those who don’t know, Go has this mechanism whereby if you put Go files inside of an internal package, only the things that are in that package and below are accessible in that project. That’s a nice way of actually hiding and preventing things that are in that internal package from peeking out, so to speak, into other things.

[46:15] I’m curious just on that one right there - and I’ve gotten bit by the internal package only when it came to black-box testing… And I’m curious – maybe it’s a convention I’m not aware of, but I once put all of the .proto files in internal, and then when it came to testing, you had to have those structures, but you couldn’t get at them in a black-box test… So how do you solve that problem?

That is a good one. I haven’t come across that particular issue myself.

And again, I just tend to do black box… The way I solved it was I did internal testing. It was the only way to solve it, because I had to have access… But that is where internal has bit me before, and I find that I reserve internal for something truly like “This is private, private. I don’t want anybody to ever touch this.”

On the naming of those sub-packages, Cory, I actually will repeat the name. So if it’s Comments, and then you’ve got Comments Formatter, I will call it Comments Formatter. So you do get a bit of that repetition in the folder structure, but I think having the package name clear when you come to use it is worth it. The HTTP test package is an example in the standard library that does this. The package is net/http/httptest. In a way, you feel like that’s redundant when you look at it at that point, but in your code you get to say httptest.something.

Yeah, there’s some real validity to that, because I think at the end of the day when I read code, especially when I’m coming to a new project and I see something being referenced and I don’t know what it is, and then when I finally track it down like “Oh, it’s this sub-package over here”, it makes total sense where it was, but I frustrated that I had to chase this down to find out what I was looking at.

Yeah, it’s kind of optimized for writing, not really reading… And I like to optimize for reading.

Yeah. I completely agree.

So we talk about having a folder structure like this, and how you’re always reaching down… Go is probably the first language I’ve used that doesn’t allow cyclical imports, which I know coming from Ruby I feel like everybody just does it naturally there all the time… So it was a big change. I guess my question for you guys would be “Do you agree with that decision?” I think now to get used to it, it’s like “Okay, we’re designing all these things to go and work around that…” Is there a time where you kind of wish “I wish Go just had cyclical imports”?

There has been for me in the past, but usually that’s because I’ve tried to break things out too early. One of the advantages of just having everything in one package is that all your dependencies are just there. You’re not importing things, so you can’t have that cyclical dependency thing. But I do like it because it kind of forces things to be more simple, having that rule. But again, I might just be used to it, or it fits with the way I already write code anyway.

[50:29] In Ruby - yeah, you can do anything in Ruby, can’t you? You can literally do anything. I used to do Ruby, by the way; I loved Ruby. But you can’t do anything. It’s like in JavaScript - if you have a JavaScript function that takes a string, you can just pass the browser in instead, you know what I mean?

[laughs]

So yeah, I like type safety, I do. And I like these kinds of rules that constrain us; I find that helps me become more creative.

Yeah, I think that from a cyclical dependency standpoint, when I first started Go - and I came from Ruby as well - I got bit by that a lot, and it took me a while to figure it out. I can’t remember the last time I’ve ever had that error creep up when I compiled… Because it just becomes so ingrained, where everything is its own concept; everything is contained in its own package. But using simple interfaces and decoupling - it just becomes second nature.

But yeah, the only time it really bit me was when I was first doing Go, and I accidentally used the name of the package inside the package I was in, the foo package and I did foo.something, but I didn’t realize I did it… And I got this massive set of errors for cyclical imports, and I didn’t know what that meant. I spent three days refactoring it, tearing it apart, only to find out I just had to remove the foo. because that was just a typo, basically. [laughter] So that was my first learning with cyclical dependencies.

That’s true, I’ve come across that one. Thankfully, I haven’t spent three days on it, but I did come across that. Usually, with the cyclical import stuff - that usually a hint that I have a design problem, I have an issue. Maybe there’s another type that’s screaming to get out, maybe I need to leverage interfaces more… But that’s usually a strong, a loud yell that’s basically saying “Hey, you have a design issue in this code.” That’s when I usually take a step back, sit down, figure out exactly what I’m trying to do; maybe I need to introduce a popular abstraction, or something.

Then usually, once you sit down and think about it, the issue - at least for me - will typically rear its head out… And it’s like “Oh, okay, I’m trying to do this, when what I really wanna do is that.” So it’s really a sign of a design issue.

It’s interesting. There’s another thing that drives us towards these problems, and that is our obsession with writing dry code. If we have two packages and we’re – or actually when we have a package and we’re writing a new one, and then we want to use some similar concepts that we’ve used in another package, for some reason our very natural tendency is to immediately create a third package, and that can become a dependency if these are the two packages. That’s certainly how I think about – that’s my kind of initial reaction to when I encounter that. I want to dry the code up. And I’ve found that if I can resist that temptation, I end up with much better code.

So instead of taking out things that are common, leave them there. Repeat them. Even copy and paste from another package. It’s okay to do that, and I call it moist code. I think we should all write moist code. [laughter]

[53:50] I like that, I like that. I feel like there’s always this need - especially if I’m training developers who are kind of seeking feedback, and input, and mentorship kind of thing… They say “Well, I know you said don’t refactor too soon, but when is the right time to refactor code that I’m seeing that’s being repeated?” Eventually, I had to come up with a rule and I said “You don’t refactor this code until you’ve seen it at least three times. Then you can refactor.”

Obviously, it’s arbitrary; I came up with that. For me, that’s been a sweet spot, so that’s sort of what I recommend… But usually, I won’t bother refactoring repeated code at all until I’ve seen it at least two or three times.

I don’t think three is arbitrary. What three is, is quite interesting. I think it’s right. So the first one is just the first time you’ve done it - fine, that’s number one. The second one is where all the temptation is, because now “Oh, we’re doing something similar”, so that’s the one that needs the most resistance. And then the third time when you come to look at it, now you’ve got three different examples of where this is gonna be used; you’re in a much better place to design an appropriate abstraction at this point, and you can do it based on real data. You’ve got real code, you’ve got real stuff. You’re not trying to imagine it.

I think the third time is also important because you get a chance to see if either one of the first two evolved or changed. Because I think there’s a lot of times where we see two things and we’re like “These are the same”, but realistically, they’re slightly different in some subtle way, and you don’t know it until a little while later. So if you wait until the third time, you’re giving that code time to actually show you what the differences are going to be.

Just jumping back to cyclical imports though, if you don’t mind… Where I see it come up the most is –

[unintelligible 00:55:39.05]

Where I see it pop up is people will get this idea of, okay, I’m gonna try to split my code into like “Here’s my comments package, and here’s my users package”, and it usually stems from things like relations in like a SQL database, where a user has many comments, and a comment has a user. That’s almost always where it comes from. And it’s hard at times when you see people doing that, because you’re like “I get why you’re doing this, but we need to think about how we’re structuring our code”, and some different stuff like that to just kind of get around it.

So I guess one of the bigger reasons I bring it up is I don’t want people to feel bad if they run into it; that doesn’t mean you’re necessarily designing poorly, or anything. It is a challenging problem at times, and there are languages that make it easy to ignore that problem, or just to move on, it doesn’t matter. Go is just unfortunately not one of them in that way.

And that’s where you end up with the user/comments package.

Well, sometimes…

[laughs]

Yeah. Well, you end up with a lot of code that’s just switching between types as well.

Yeah. Well, I think the other thing is - like Mat said; or maybe it was Johnny - people don’t like to repeat stuff. People like to have one struct, and this maps to the database, and that’s it, and they don’t like to rewrite that anywhere. And I’ve seen code where having multiple different versions of the same struct that’s in the database - sometimes that’s useful.

I love that you said “As Johnny has already said, people don’t like to repeat things…” [laughs]

Well, I like to repeat things, apparently…

It’s worth repeating.

Apparently, I’m really good at cycling and repeating. Okay, so one thing I did find interesting - we’re talking about code structure, and one of the first things that almost always comes up is folders, file names, that sort of stuff. Mat, you had me look at your talk you’re giving at GopherCon, and one of the things that was interesting to me is that you’re talking about how you write your apps, and a lot of that is how you structure code, and where things go… But you never once really mentioned folders, that I’m aware of. But yet, it was still really insightful. Why do guys think that’s the case? Do you think people just don’t understand how to organize things, and they just get so caught up on the folders, or is there something else going on?

[58:03] Well, I tried to write about that in that blog post, and I tried to talk about that in my HTTP talks… But the problem is, as we’ve talked about, if you provide the end state of something, then that doesn’t necessarily help junior developers to see the rationale and the reason why we’ve ended up in that position. And sometimes, depending on where you are in the project, so really the context of the project, sometimes I do it differently myself.

I don’t find that there was enough of a common set of patterns… There’s a couple I talk about. I talk about the fact that I keep all my roots in one file, called roots.go. This is where I break the rule of having things grouped by responsibility… And I do that for a good reason, because I get a landscape; I can see visually the entire service in one place. I find that to be very useful. So I sort of break my own rules sometimes as well… I just never found enough commonality for folders and files and things.

Okay. So I guess we don’t have a ton of time left… Do you guys wanna talk about approaches you’ve tried that you’ve come to regret, or mistakes you’ve made? Because I feel like – you know, we always talk about “Here’s the end state. This is what we should be doing”, but we don’t really talk about what we’ve tried, what didn’t work well about it, things like that… Or even whether or not trying it was worthwhile.

From my standpoint, there’s some approaches I always want to try, because they sound fantastic, and then it comes down to I don’t try them, because I just have to get the code shipped. So there’s always kind of that regret where – there’s a couple talks, especially some of the talks by Kat… She’s got some really interesting approaches out there, and I really wanna give them a go one of these days, but I also know that there’s a lot more work involved there, because everything’s abstracted significantly more than what I normally do… So I don’t know where to go with that. One of these days I might try it, and I’m really curious how it works out in practice.

But fundamentally, I do end up with a lot of the same packages that I always have. I have my cmd package, I have my domain package, I have a lot of the same ones. I don’t have models, I don’t have utils… And it took me a long time to agree with not having a models package; it really did. I fought that for a long time, like “What is the problem with having the models package?” And it really just does come down to “It does say nothing. It says nothing.” And I think if your package says nothing about it just in its name, you’ve made a mistake. That’s a really hard one to understand, and I can’t convince you of that. I’m not gonna try to convince you of that, I’m not gonna tell you what’s wrong or right… I’m hoping that you just come to that conclusion on your own, after you’ve done it enough times. But that’s a hard pill to swallow for a lot of people. It was a hard pill for me to swallow.

So you’re saying they have to do that enough on their own to come to that conclusion. It’s almost like I would get the impression that if you hadn’t done it enough times, you might not have ever come to that conclusion. You almost had to make those mistakes to start to learn gradually over time “Oh, I can actually express this better in this other way.”

Right. And here’s the way I would like to explain the models. Let’s take a user, because we all have a user in most of our packages… And when you have a models.user, it’s just this data struct. There’s no behavior around it. Maybe there’s a validation, maybe there’s some simple things, but it tends to be this really shallow package of just a bunch of data structs and some really simple tasks… And it just kind of feels like it’s just kind of hanging out there. But when you have a user package, suddenly it becomes a very rich package, it says a lot about itself and it manages its own behavior. And I think that’s where when I flipped over to that type of paradigm, that my code became better, more understandable…

[01:01:58.00] I mean, I may have a domain package that in there has a user package, and has a comments package, or whatever, or maybe they’re flat within domains… It doesn’t really matter. But the idea there is it’s no longer just a data struct. It’s not something simplistic in its concept. It’s the entire thing, it’s the whole concept which makes the difference for me.

Early on this was one of the first mistakes I was making - having package bloat. I’d basically create a package inside of a package inside of a package… And I had to basically ask myself “What value am I getting from splitting up and creating this deep hierarchy of things? Why am I going about it that way?” That stems from being able to say “Well, I can see myself reusing this bit, this portion, this package out of this project in a different project.” And I’m like, well, then I’m creating these dependencies between things that I don’t need to have. If I really need to use some of the concepts inside of this project, just copy the darn file. [laughs] Just go put it back over there. Why am I creating these nested dependencies and hierarchies in my code…?

So I started basically – other than having this ritual really for certain projects of creating my cmd folder and putting things in there, I really started out flat, and I’d let the domains inform where do I need to create packages, and packages under packages. I think that notion right there is something basically I’ve carried over from other environments, other languages, and I kind of needed to leave it at the door. So many things in Go – you kind of have to leave a lot of things at the door, and learn to appreciate Go for what it is before bringing things and expecting the language to sort of bend to your will, so to speak.

You can write Go like you write Java, like you write Ruby… I don’t recommend it, it’s gonna end up biting you, but you can do that… But I’d caution that. Again, Go was meant to be a simple way of coding, of programming really, so approach your design the same way. Start a sample and let the domain you work in to inform where you set up boundaries.

I think that’s great advice. One of the mistakes I’ve made quite early - I really fell in love with Go interfaces, and I never really fell out of love with them, to be honest. So if I had a package that had, say, a greeter type, and a new greeter constructor, something like that, I would always have an interface there, too. And sometimes I’d hide the struct and I would only return this interface… Because I just felt like “Now there’s an interface, so the people can write greeters, and they can write mock versions if they need to for testing”, and sometimes I would even provide the mock version in the package, to help with testing, because it’s so important… And eventually I got some code review probably from somebody on Twitter which sort of pointed out “Actually, you don’t need to do that. You can just return the concrete type, the struct, and if somebody else needs an interface, they can write their own interface.”

[01:05:23.03] In Go we have this duck typing, they call it structural typing. The only requirement for a struct or a type to implement an interface is it just has to have the same methods. You don’t have to explicitly say, like in some languages, “Implement this interface explicitly.” So users can write their own interface, and then they can use that interface in their code. You can even use that as a storytelling opportunity and not include every field. If this greeter struct had ten methods, but I’m only using one of them, I could just have an interface with that one method. It makes it dead clear what I’m gonna be using this type for, and it’s less work when it comes to mocking or writing new implementations to replace it.

So I think one of the approaches – Johnny, you said Go is meant to be a simple way of programming… And I think it’s worth thinking about “What’s the easiest thing to do?”, and be a bit lazy, too. Do less. Do the minimum you can do. So I wouldn’t then bother with the interface if I was doing less… That tends to be quite a good way to think about it.

Now, of course, there are times when you have to do some work, hopefully. That means we’ve got a job, so that’s good… And then you have to do some work, so then you do it. But again, do the minimum. Do the absolute minimum, if you can, and you’ll find that you defer a lot of decisions to the time when you’re better-placed to make them.

Kind of like using defer.

Yes. [laughs] That’s my favorite keyword, by the way.

Alright, so I think that’s about it. I think we’ve pretty much hit a little bit over an hour mark, so thank you everybody for joining us. If you haven’t joined us on the Go Time Slack channel and the Gopher Slack, you should definitely check that out, too. Ben Johnson and some others have been chatting, and Ben has written a very awesome article about structuring your Go applications. I think most or all of us have probably read it. I know it’s definitely influenced the way I’ve designed my code, and it’s really helped… So you should check out that sort of stuff, too. Ask questions. We really appreciate you guys tuning in.