Since airing this show, Pieter passed away due to his battle with a metastasis of bile duct cancer in both lungs. But rather than listen to this show with sadness, listen with a happy heart and let’s celebrate Pieter’s life, and what he has accomplished. Thank you Pieter from the bottom of our hearts for your time on this show and for all that you are. You are loved by us my friend. This show will forever be a very special show for us.
Pieter Hintjens is the creator of ZeroMQ and The Collective Code Construction Contract (C4), a writer of many books and protocols, as well as a developer with decades of building software and communities – he’s someone who’s given so much, and continues to give - even up until the time he is planning for his death.
Featuring
Sponsors
Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com.
Linode – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018
. Start your server - head to linode.com/changelog
Rollbar – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog.
Notes & Links
- A Protocol for Dying
- Donate to make Pieter’s childrens’ lives easier, when he’s not there any more
- Planned Death
- Building Online Communities
- Books by Pieter Hintjens
- ZeroMQ Homepage
- The ZeroMQ project on GitHub
- Doron Somech, Pieter’s successor
- ZeroMQ: Modern & Fast Networking Stack by Ilya Grigorik
- ZeroMQ - Get The Software
- ZeroMQ Language Bindings
- ZeroMQ - Solve a Problem
- ZeroMQ - The Guide
- C4 - The Collective Code Construction Contract (C4)
- Changelog Weekly #12
- Changelog Weekly #72
- Changelog Weekly #73
- Changelog Weekly #83
- OpenWrt
- The Changelog #203: Jewelbots and Getting Kids Coding With Sara Chipps and George Stocker
Transcript
Play the audio to listen along while you enjoy the transcript. 🎧
Alright, we’re back for another show. Pieter Hintjens is joining us today. This show, Jerod - like many show we’ve said before - has been pretty much years in the making. I remember linking out to Pieter’s blog several times over the past couple of years in Changelog Weekly, and if you’re a reader of Changelog Weekly you’ve seen lots of great recent posts from Pieter from that blog there. What’s the best way to open this show up, Jerod? We’ve got a fellow who’s been in decades of software development leading communities, ZeroMQ, Living Systems, several books, an expansive blog… How do we open this show up, what’s the best way?
Yeah, it’s a tough one to know exactly where to start. Of course, we always like to talk about people’s origin stories, so Pieter, we definitely wanna hear that from you. But maybe we’ll first mention that a large majority of Pieter’s work is in building distributed systems and protocols, and he’s written 30 protocols, and even recently on his blog he’s written a protocol for dying, which I think is something that we’ll definitely have to talk about, as well. Pieter, thanks for joining us.
Glad to be here, thank you guys.
I guess the way we typically start a show like this, especially with someone with such deep history - I’m sure you’ve shared it sporadically or specifically in your blog; I haven’t seen the exact post that just shares your history, there’s been some recent posts that have touched on that, but how far back should we go to find out what got you into software development?
I was very lucky because I chose computer science in the days when this was just a very young thing at university - I think it was the first year they had this in York University in England where I studied - and it was the most boring, tedious, horrible stuff. It was just better than the math, which was even worse; stuff like database normalizations, and I couldn’t stand it. My mom bought me a little computer, a VIC-20, and I began writing games in BASIC, as many kids do. Then I began writing games in Assembler, because BASIC was limited. Then Commodore very kindly sent me a Commodore 64 and a printer and a disk drive for free. And I was selling these games. The Assembler, it’s a lovely CPU, it’s RISC, it’s got 8 instructions I think, it’s almost got nothing in hardware, you do everything in software. I got into this obsessively, and I was doing really badly at university, I was getting like 30%, 40%, 60% - really bad results. So I took a year off and I just programmed for a year, and made games and sold these games and paid off my university costs and so on.
After a year of that I went back to university and picked up… And it was all new, I had not seen this stuff before, it was all from the year that I’d missed, or the year after me, kind of. And it was so easy, my brain kind of switched into absorbing more, I understood how things worked in the computer, I figured it all out and I could just pick up a syllabus and read it through one time and understand it. That’s how I went through the exams and got really good results, although my end degree was pretty useless.
So there was this kind of love with making difficult things work, learning, and I was really good at that. I got a job building tools - this was in Belgium, where I ended up by mistake, thanks to military service. I got hired by a company - although I was much too young, but they hired me anyway, despite that. We were building tools in a small team for this huge company selling really crappy business software, but using our tools it was pretty good business software.
I was always building infrastructure, from the very beginning, tools for making other things.
What do you think the reason is for that? Why do you think you were attracted to the infrastructure side, versus a frontend side?
Well, selling games is fun, getting checks in the mail with money, or even cash in the mail is a great thing, but when you make stuff that you can give to other people to make stuff with, then it’s somehow more fun for me. You see the results of your work multiplied, and you can make things that can last longer, that can be reused. I really like that, when you make things that can be reused over and over again, the tooling and the layering in software has become for me the most important thing, the reusability of what I design.
You start with “This is going to last for two weeks, it is a very simple thing. People will use it and then they’ll throw it away.” Then you think “Maybe it will last for six months”, and at some point in my work I said “Okay, this thing I make I want to last 50 years” - which it didn’t, what we were making then was a protocol which was quite trashy in the end, the MQP. But the concept of making stuff that lasts 50 years came out, and I think that’s a valid goal.
Do you think anyone else entering software today has that kind of goal? It seems like we’re in a instant gratification, kind of keep-it-and-throw-it world today, where something new comes out every year, and everybody says that a year in software or in the internet world/internet age is like seven years, they compare it to dogs, or whatever. Do you think people have that kind of perspective and if not, why?
I don’t really think it’s a generational thing in the sense that there was a time when there were more hackers, and there are. I think that’s a certain slice of people that just really like to understand how things work and are interested by solving that kind of problem, and have the capacity for it, because it can be very difficult. We’re talking about concentrating on things for months and months, and concentrating on the right things - that’s also very difficult, to know what to actually work on. I think that mindset exists constantly, and I think before computers it was doing other things. Today it’s still there, and it’s still looking at problems which are very difficult; maybe not the same level of problems as we were on when I was 20 or 30, but I don’t think it goes away.
That’s an interesting perspective to say, to find the right things. Jerod, you could probably share this symptom with me - I think it’s the journey to find the right thing for you to work on; people say their dream job, or whatever term you want to use to describe that. I think it does take a while, especially as chaotic as technology is these days to get into it. There are so many layers, there are so many things… Just to get to ground zero, which is writing code, is sometimes a journey on its own, people have that problem every day. But finding the right thing to work on - it seems like, Pieter, you found the right thing fairly early, and then just sort of iterated on that right thing for you.
Yes, it’s pretty random, and there’s all this constant pressure to do the wrong thing, and people paying you and telling you what you should be working on. There are these huge waves of fashion in software, that “This is the new thing and everybody should be working on this.” I guess I was very lucky in my first job to work with a guy, Leif a Danish, kind of a crazy astronomer/programmer, and he had this very simple rule that everything is bullshit, and the more people are shouting something, the more it is bullshit. His thing was to always look at the mass market, and then look where they were wrong and look at the little areas where you could do things that everyone considered to be impossible or crazy or stupid – where there was actually a real profit. He had a real talent for that, and I think I picked that up and used that for a long time. Things that were impossible, things that were outside the mainstream experience were almost by definition going to be more interesting than what the mainstream was doing.
Pieter, the right thing for you - if I’m following along - is low-level communication protocols. I’m sure you do other things as well, but that’s kind of your “claim to fame”, especially around ZeroMQ and the legacy that you’ve built up with that project, and the communities that you’ve built over the years. In a recent post, when I reference A Protocol For Dying, you mentioned that you were lonely as a young man, and perhaps even somewhat autistic, and I always think about people who focus on protocols as their life’s interest or as the thing that excites them or that they’re good at, because they’re so necessary and they’re so low-level, and of course ultimately it’s a mechanism or a way in which two parties can communicate, and I wonder if this loneliness, this autism, or your desire to communicate to people and you didn’t have that - does that play into your fascination and your life’s work around protocols?
The protocol I’m most proud of actually is one called C4, which is a contract for collaboration; it’s the rulebook that we use in the ZeroMQ community. It’s a protocol, it’s to find us an RFC with all the formality, and it explains how the group of people who are trying to make some software should work together. I really like that protocol because it works really well; we’ve tried it now for years and it works almost magically well. It’s written kind of from an autistic viewpoint where you don’t really care what people feel as such, but you do care that they do feel. So there’s this kind of very brutal approach to human nature in there, where… You know, how do you solve bikeshedding? Well, you look at the emotions involved, you look at why people argue, you look at the fallacies that people have in their minds and then you reposition things so they go away. And to do that takes a certain distance. If you’re involved and you really care too much, then you can’t do that.
I love working with people, and I think I’m hyper social. I’ve made hundreds or thousands of friends in the last years as I’ve gone around the world with conferences and so on, and that’s one my great pleasures and happinesses in life, it’s people, other people.
I appreciated what you had to say in this article, especially around your desire to reach out when you go to conferences. I had never really thought about it like that, like “everyone there wants and expects to talk” - it’s a quote I’m pulling from that post of yours. You said you rarely talk about technical issues; read the code if you want. It’s kind of funny, you’re on a podcast like this to obviously talk about this and many things, and we’ll probably talk about some technical details, but I never really thought when I went to conferences what people’s desires are for going there. Maybe even in your case, Jerod’s question to you and how you were saying you’re a bit more hypersocial - that is kind of interesting, to pursue these strange… Chatting with strangers or being in unique situations to sort of force yourself to talk to people like that.
Yes, and this hit me a while back. I was in a panel debate in a conference in Greece, and someone was asking about the physics of software and why bridges don’t fall down and software does, and I just had this wisecrack answer where I said “Well, the physics of software is people and psychology, and you figure that out and you can make solid software.” Everyone there was just dumbfounded, they had no answer to that. This struck me as like “Okay, that’s it. That’s the key to what I wanna do - people.” Our limitations and how we work together lets us build big systems. I think this is becoming proven and obvious now, but then it was kind of a bizarre thing to think about. We don’t work well alone, we work well in groups, and how do you organize that?
It strikes me, this interest of yours… I was surprised when you agreed to join us on this show, and just to kind of point out the elephant in the room, if people are wondering why Pieter is writing a protocol for dying, is that you’re terminally ill. You said you have a metastasis of bile duct cancer?
Yes. I’ll tell you the story briefly. It must have been about 7-8 years ago, I ate some bad sushi. That’s what I think happened, and there’s this little parasite that lives in fresh farmed fish inside East Asia, and if you don’t cook the fish or really freeze it very solidly, then this parasite gets into your duodenum and attached itself to the bile duct, and it begins to produce carcinogens. This is one of the major killers of men, specifically around 50, in many South-East Asian countries. But these are very poor people, and this disease is basically just an ignored disease. It’s almost unknown in the West, and it’s almost unknown in wealthy countries, so there’s been very little study into it, and there’s very few people who survive the diagnosis.
My family has no history of cancer, we have no liver problems. I do like to drink, I’ve always enjoyed that, but I have no liver issues and I have no other reasons for cancer, so… I guess a few years fast forward and I began to turn yellow, so I checked myself into the hospital, and I had a diagnosis eventually of bile duct cancer. It’s operable, they said “Yeah, we’ll cut you open”, and it’s a terrible, terrible operation. They rewire my whole digestive system basically, and I shouldn’t have survived that. Normally that was… If you’re already diagnosed with bile duct cancer, you’re dead. It’s very fast-moving and it’s very difficult to see it until it’s quite late.
I’ve had five years to put things in place… I’ve always felt that this was like borrowed time, I didn’t take it. I was very happy to survive, of course; I was kind of euphoric for about a year and then I began to figure out “Okay, what happens if I actually die?” and I began to put things into place little by little and aim towards being replaceable, being redundant, in as far as that’s possible. Then I’ve had scans every year, every six months blood tests and so on and nothing, and then this year in February-March I have this cough which won’t go away and I go get scanned finally, and it turns out it’s come back and it’s metastasized in both lungs and it’s pretty invasive. So my oncologist looks at me and says “Well, yeah…” Huh! Now we’re doing chemotherapy, which is also pretty brutal, but it’s really just for show, and I think for data there’s no real evidence that this cancer responds to any chemo at all.
So yeah, you have limited days, so I was quite surprised that you’d be willing to spend an hour and a half on a Skype call with two people you’ve never met before in your life, but then I started reading more about you and your life’s work around building communities - not just around building software - and this interest that you have in meeting people and seeing new perspectives, and how that’s one of the things about your life that you’ve loved. So then I’ve started to clarify why, why you said yes.
Oh, this is awesome.
Yeah, it was even tough to ask him to come on this show; I don’t know if that made it actually into the show or not, but just… We’ve been fans of Pieter’s for years now, and it was just a touchy subject. I was like “I don’t know if that’s okay…” Is it okay to ask somebody to come on this show, considering that we’ve wanted to have you on this show for years and it just never became a thing, we just never reached out yet? Do you always feel - I mean, this is a side note, just in general - do you always feel like you have time? And whenever you’re hit with the moment when you don’t have time, it’s obviously not a good thing, but you sort of kicked into high gear to make some action and do some things.
I think you hear my kids in the background…
That’s okay, that’s perfectly fine.
My three lovely little kids. Well, I have time actually, because I’ve kind of stopped programming for the moment. I don’t think I’ll start again. I’m still writing… I got a delivery today from China, a thing called Freewrite which is this bizarre, steampunk typewriter which I’m trying out, and it’s quite fun.
So I’m writing, and I have time, and to be on the podcast with you guys is just absolutely awesome. I would do this every day if I could, seriously.
Okay, well we won’t feel bad about it then. Let’s crack open the idea of this protocol. I noticed near the end you said you’re not gonna open an RFC for this, which is a Request For Comments… I guess walk us through how you actually - since you’ve had such a history with protocols - wrote a protocol for this.
I had these biopsies done, and I got a bronchoscopy and it was the most horrendous thing. And from that I got these infections in my lungs, so I went into the emergency with my whole right side inflamed. They pumped me full of antibiotics and I went through this whole thing. I was lying there in bed, and I finally got my PC after about a week, my laptop. And one of my readers said “Look, why don’t you write an update on the article you wrote about surviving cancer five years?” So I just turned at this article, and I said yeah, a lot of people talking to me and I see there’s a lot of strange interactions when you talk with somebody who’s obviously very, very sick and he has a terminal diagnosis, so let me try to frame this as a simple, human-readable – I didn’t want to make it technical – story that people can look at and say “Yeah, okay. Okay.” And I just turned this out, I didn’t really edit it or go through it very much from where I was lying in bed, very sick.
The thing is when you’re feeling really bad in hospital, at least for me, having visitors and having people talking to you is a real pleasure. The worse you feel, the better it is to have company around you, that’s the irony. I don’t think everybody has that experience with me, that was how it was. I had days with ten, fifteen people coming in to see me. People coming in from everywhere - from America, from Finland, one guy, Consta, flew down to see me. It was amazing. And I wanted to capture the positives of that, what felt good, and the negatives. Some people were very strange, and I wanted to document that and say “Okay, these things that people do sometimes are very strange.” If you document it, maybe people can learn to avoid doing it.
Yeah. I certainly appreciate the way you framed it, with Bob and Alice. You certainly removed yourself from it; it’s not like “This is what Pieter wants.” It’s the thread that’s ran through that, with you writing, obviously, but Bob being the dying person and Alice being the person who - what she should not say to Bob. I’ve been sadly in a family where we’ve had several people die, and we’ve had some experience in my family with loss and people passing from us, whether it’s from cancers, instant death, heart attacks or things like that. I’ve been in a family who suffered loss, and on Facebook - whether it’s early news or it’s actually after someone’s passed away - they say all these weird things like “Hang in there” or “Keep hope”, all these things that… They’re just trying to be polite, but it’s just not constructive, it’s not helpful, and in the end it’s just not loving. So speaking of this protocol you’ve written in a human-readable way, and I think that the way you’ve written it from Bob and Alice’s perspective, even removing you from this, where it’s not like a personal offence to Pieter, nobody… Jerod and I talked earlier about how we had this uncomfortable feeling of having you on this show not so much because of you, but because we didn’t wanna offend you. I think that’s what’s interesting about this protocol you’ve written in a human-readable way, to share with people what to say, what not to say, and even how you’ve embraced lots of people in your hospital room when you were in the hospital, and what that’s like for you. I think it’s an interesting perspective when it comes from this protocol you’ve written.
I think there’s this real taboo about… Well, it’s a taboo caused by distance; we’ve lost this kind of naturality to dying. It used to be that we died mostly of old age and surrounded by our families, in our own homes - probably through lack of medicine and proper care, but death was something everybody participated in, and we’ve lost that. Most people die away from home, they die in hospital, they die in road accidents, but the dying isn’t a social process anymore, and that means the living are very, very uncomfortable with that. It’s become something taboo and hidden when you have to cross that boundary and talk over it - what do you do, what do you say? There’s just no culture, no background for it anymore, we’ve lost that. I think that’s avoidable.
I’m really lucky to be in Belgium, as well. I didn’t like this country in the beginning, I came here by force, I was conscripted, and I was like “Okay, screw you guys”, they stole a year of my life, but it turns out to be a pretty good country. They have laws for euthanasia and ‘planned death’ as I like to call it - this notion that you can construct your own death properly.
I saw this with my father who died at Easter, and this is how we did it, as a family. He had euthanasia; he was very feeble, very old, and it was a very loving and calm and untraumatic way to end someone’s life. He was going to die anyway, so it wasn’t about killing him. It was about changing from a total lack of control, as a family, to control. I think that’s something which will become more accepted worldwide as a kind of a human right, this notion that you can control your own destiny, you can control your own death. I like that aspect of involving people in that process, little by little, that dialogue with people. Even this podcast, we’re discussing my death, and that’s an interesting thing, which I find healthy.
It’s a first for the show, honestly. It’s something we definitely haven’t done before, and it’s something I never even expected that we would do. When the show first began, our tagline was “Open Source Moves Fast. Keep Up.” It’s still part of our mantra, but just that we cover the fresh and new of open source. This is the most unique open source subject that we’ve ever covered, which is a protocol for dying. It’s one of the many things we’ll talk about with you today, so it’s not the only thing. I think the reason why it makes sense to bring that up early is because it doesn’t make sense to talk about it later in this show. It’s something that the listeners are gonna know about right away, as a foundation for the conversation we’re about to have, and go deeper into it.
Actually, this is probably a good time to take a break - we’re about 20 or so minutes into the show - but let’s come back and touch on a couple of the things that you say that are positive ways to talk to someone who’s dying. I think you mentioned… Things you like, for example. So let’s take this break, we’ll come back and talk about some things that are constructive to talk to you, so that way when other folks feel invited to speak with you, whether it’s on Twitter or through some social way, or actually face to face in these upcoming meetups you have, they have a sign from your post, they have you actually saying some things that help them understand how to talk to you and how to treat you. So let’s take this break, we’ll be right back.
Alright, we’re back from the break and we’re talking to Pieter about a protocol for dying - it’s obviously a unique thing to speak about. Pieter, you’ve done very great with making this readable by the world and sharing different ways to talk to someone and not to talk to someone who’s passing away. Walk us through a couple of the things that are on the positive side, how to speak with somebody that is in your situation.
It’s actually quite simple, it’s company, it’s having people around you, this is how I see it - having the presence of people, having people besides you. It’s funny, the worse you feel, the nicer it is to have somebody there who is just present. If we’re gonna talk, it’s gonna be about the things which are, in my mind… I mean, being Bob then I’m very selfish, of course, I’m obsessed by the tubes in my arms and the effects of the chemotherapy and how much I’ve been vomiting that day, and how it’s better than yesterday, and how tomorrow will be. My focus is very limited, I’m not thinking about next week or next month, I’m thinking about the next few minutes, the next few hours - that’s when I’m sick, in the hospital.
If I’m at home, we can talk about other things. We can talk about stuff that we’ve done, we can talk about… If you’re in the hospital and you’re sick, then it’s, as Bob then your mind is on that, and if you’re at home and you’re feeling better, then your mind is elsewhere. But what it really comes down to is having company and having the excuse to be together and to chat about just random rubbish, without it being heavy, or difficult, or painful, and without there being this emotional burden. That’s the worst, the emotional burden - people who are very sad, or who feel very confused. That’s almost impossible to deal with.
I think the key thing about the protocol was… I mean, the part about what to say, what not to say, they’re quite trivial almost. The key thing was involve other people. Don’t hide your sickness, don’t hide your situation, don’t be afraid of what people will think of you. If you’re actually that sick that you’re going to die - we’re all ashamed of being vulnerable and being seen as vulnerable, but if you’re going to die, you’re going to die. If your doctors tell you that you’re terminal then tell people about it, be honest about it. Don’t assume the worst as the worst possible case, but be practical about what’s the likely outcome, and involve people in that. Once people understand that you’ve accepted it then they accept it, and it gives your community, your family, your friends time to grieve and time to prepare and time to go through that process with you.
I have a friend who has breast cancer, and seven years ago she was diagnosed, she had a small scan. And she just refused any treatment, and that was her solution - to deny it completely. She said, “I’ll fix it myself with diet, and whatever”, and seven years later she’s still alive, but the cancer’s eating her up. She’s not told her parents, she’s not told her friends, it’s a big secret. She’s really ashamed of it, and when she dies - and she will die - it will be absolutely catastrophic. Nobody will have the chance to have said goodbye or to even have understood it.
I think that’s the key thing about this protocol, is be honest - be honest with yourself, be honest with your community, your social circle, your family, your friends.
One aspect of that, and preparing for death, is sorting out your life - closure, wrapping things up, making sure that your desire is moving forward or held. That’s something that many people that aren’t ready to deal with their own death, or they feel like it’s far away from them, often won’t do, so you leave people in the lurch because they don’t know what you wanted. For instance, you’re supposed to create a will when you’re healthy, not when you’re sick, because you never really know when your life’s gonna be over. So in many ways, it’s an opportunity that you have - all of our days are numbered, but yours are particularly numbered, so far that you don’t think you have very many of them left, that you can actually put your things in order and you can pass the baton. You have kids, you have a family, you have all those aspects which are the most important things in terms of passing down your desires and your thoughts, and making sure that your kids are well-established in who their father is and all that. But from the technical software community side, these things that have really been your life’s work in terms of community-building in the software space - there’s a lot of stuff to think about there as well, and it sounds like you’ve done some thinking on how you’ll actually pass the baton for instance in the ZeroMQ community. Can you talk to us about that?
Right. This has actually been a long process. I’ve been working for years to make myself redundant. I’m the gatekeeper of the community, I’m the benevolent dictator for life - haha! - and for a long time I had to feed the community; I had to put a lot of effort and quite a lot of money into it. I got the money back, I got good business from being the ZeroMQ guru, but it was a big upfront investment, and it was a continual investment in keeping things straight, and keeping problems away from the community that were potentially very dangerous.
So it was a very deliberate thing over the last few years to make myself less important. I was always a contributor and I love writing code, but I didn’t want to be the person making the decisions, like “When do we make a release?” or “Do we merge this patch?” or “Is this person a valid contributor?” I wanted these decisions to be made by the group, in a decentralized, autonomous, reliable, robust fashion. That succeeded, and that’s kind of the core of our process, the C4 process.
We still have, of course, dependencies for things like domain names, trademarks, and ultimately the deciding voice when there is a real conflict or a real issue. I’m very lucky to have the time to do this, but I’m also very lucky to have a group of contributors who are absolutely amazingly good. The level of the people who’ve come to… We have these hackathons once a year in Brussels - before this huge open source meetup called FOSDEM we had these hackathons for a few days. The people who come there are just awesome, they are amazing. One of them is my friend Doron, who is an Israeli - he’s a bit younger than me and has more hair than me, and smiles more than me, but he’s been using our process, he’s been using ZeroMQ, he’s the author of the Windows NetMQ library which he’s rewritten three or four times. The man is a fantastic developer. I’ve seen him work with people, I’ve seen him basically as a younger version of myself, so I’m like “Okay Doron, you idolize me, I’m gonna give you the baton”, and he’s like “My god, what do I do with that?” I’m like “Okay, basically I’m going to pass you the domain names and I will tell the ZeroMQ community that I’ve chosen you as my successor. They can argue, they can bitch, but they won’t, they trust me. It’s a good decision.” And it comes down to actually this very interesting situation where we have no foundations and we have no committees, and we have no centralized infrastructure. We have one guy who owns a few domain names, he owns essentially ZeroMQ.org, and pays for that, and has the consensus agreement of the community that he is the guy in charge, and that’s it. It’s a very lightweight baton to pass on. Imagine that there was a foundation with a board - we would be in years of argument.
Wow, yeah. I mean, that’s actually an interesting thing you bring up, because just a year back we can rewind to io.js and Node, that separation, that forking process, fragmenting the community, then the obvious corporations involved, and now there’s Node.js Foundation. They’ve done a lot of great stuff, but in light of this scenario here, I guess it’s maybe a bit more specific when you have a BDFL model, right? Maybe that’s a little different, whereas in the Node.js Foundation - if that’s the example we’re given, since I’ve said that; it may not be the only example, but it may be a little easier to pass that baton because there’s no particular BDFL in place. There was, but there isn’t anymore, and I guess that’s sort of the walls they’ve been breaking down, and that’s an interesting on its own.
The foundation model is one that I’ve used and I’ve been part of several times, and they can work very well. The problem with it is that it has specific weaknesses, and it’s vulnerable to hijack, basically. It’s vulnerable to interference by bad actors. It’s actually quite simple, you can just get onto a board by pretending to be a good actor for a while, and then you start disrupting things and you can destroy an entire community just by sending a few emails and praising the wrong people, and attacking the right people, and it’s so easy to do. I’ve seen this done, and I’ve been at the receiving end of that in large communities, which were destroyed as a result of that. And the cost of setting these things up, the cost of having regular meetings - it’s just endless, endless friction.
One of my ideologies for ZeroMQ has always been utter decentralization. Every project is autonomous. Projects agree to use the rules they want to. Any project who doesn’t want me as a dictator and didn’t want me as a dictator, just didn’t choose me as dictator, it was very simple. I was only there by approval of each project, and there are hundreds of ZeroMQ projects, thousands even.
So it’s a model that I designed very deliberately designed based on having seen many worse models in my life, and it seems to work pretty well.
How about things – I mean, this is going a little too far on this subject, but how about things around your corporation and some things I notice on the ZeroMQ site with the support models and contracts like that. How does that change, how does that translate as part of this?
I never tried to make my company a very dominant… It’s visible, but it’s kind of visible on the margins of ZeroMQ. We’ve always kept ZeroMQ as a first-class citizen, and the business as a second or third-class citizen around that. Most people that use ZeroMQ never ask for support, they don’t need it. There’s a few customers now and then, people who are potential customers who want support, they need it, they want a contract of some kind; they hunt around for a contact address and they email me and ask for consultancy or support-level agreement, whatever it is, some sort of agreement. And it has to be that, it has to be some kind of commercial hotline for people that want to pay. Because otherwise you leave money on the table, which is just silly. So I assume that whoever has taken over - Doron - will replace iMatix with his own company. My company will be folded at some point in the future. It has no real structure, I’ve always kept it very small. Just myself, really.
Let me loop back around to your C4 protocol. As you’ve said, it’s the thing you’re most proud of. Perhaps I’m a little ashamed, I haven’t even heard of it until you’ve mentioned it, so maybe it deserves to have some light shed upon it. Collective Code Construction Contract, which is a hell of an acronym…
C4.
C4…
Yeah, yeah. What’s that word - alliteration? When you get everything starting with C’s. This seems to be a model that you said is working really well for ZeroMQ. I’m curious exactly what the model is, and if there’s other projects that are using it, if it’s something that you think more people should be using but they aren’t - give us the details on C4.
Where C4 came from was we’ve been trying to codify best practice in our community, and of course I like writing protocols, I like writing, well, contracts, honestly; I like writing contracts, weirdly enough. Protocols are contracts. And we had a number of really really big problems in our community for years, and these problems were difficult to understand, because they were hidden by all kinds of layers or stuff going on. But it boiled down to the conflict between a few very dominant contributors and the mass of, I would say, less aggressive and less ideological contributors and users. This conflict is very real and it happens in many projects, and you’ll see it when the core maintainers go off on an ideological tangent. They annoy their whole community and it can be very disruptive and very destructive. So we had this problem, and I was the person trying to keep things calm and trying to make stable releases while there was two or three major unreleased versions still in the pipeline, each one breaking APIs and breaking wire protocols, and completely out of control process. And it ended in a fork, which I thought was very satisfactory. As it ended in a fork - which I provoked by writing the first version of C4 - where I said “Okay, look, I own ZeroMQ, it’s my trademark, I’ve invested in this, and everyone here is welcome, but you are basically working on my property, and these are the rules as of now. The rules are that we will aim for maximum participation, that’s our first goal. I don’t care about the software in terms of its code, I care only about the people and the way they organize, and I trust they will make the right software.” That already is a very brutal statement to say to people who are focused on software first.
It seems so counter-culture.
Yes, it’s a very aggressive thing to say, and that annoyed a lot of people. Well, a few people. Then I said, “Right, this is how I see it working. First of all, we use pull requests. I’m tired of this cargo cult model we had, where you post your patches to the mailing list and somebody may or may not merge them - well, it works for Linux kernel, it will work for ZeroMQ. It’s a trashy model, it’s based on no understanding of actual dynamics, it’s just based on trying to imitate other projects. Well, you’re gonna use GitHub pull requests. That’s it, end of discussion. Everybody can use GitHub. Everybody will use GitHub, or whatever platform we’re using - it may not be GitHub in ten years time.
I’ve been using this approach for my book, the guide, where I’ve basically been merging people’s contributions. I had pull requests, people sent me code, and I merged it. And I noticed that there were very few mistakes. People got things wrong, people made bad translations of the sample code. I wrote the sample code in C, and people would translate it to every other language and send a pull request. And people got it wrong - that’s fine; others would come in and fix it. But there was almost no misbehavior, and I was merging pull requests. I just stopped reviewing it, I just began merging everything. Just blind merging, and it worked very well. I’m like “Okay, this model actually works.”
So I began discussing with other maintainers and writing blog posts about this notion of being much more accepting to contributors, and not reviewing their code aggressively, not making value judgments above all on their code, and moving towards a kind of a contract where you can solve any problem you must solve, you can bring in changes to fix problems, but you may not break existing applications. That’s the core of C4, this process for welcoming contributors, and defining rules that forced maintainers to be welcoming, even though they might have their personal opinions, and to give this kind of right to anyone.
If you’re using our product and you have a problem, and you have a solution, and you send a patch for that solution, it will be merged onto master, on our main development branch if you follow a few basic rules about style, about the size of your patch and so on, commit message. But we will not judge your patch. That’s completely radical in our business, this notion that the ideology of architecture is a problem, rather than an advantage. And this notion that your elite ninja programmers are your actually your handicap and not your asset.
It made a lot of people very worried when we started that, and then we noticed that our contributions began growing. We had more and more patches. The email lists became very happy, there were no arguments in the email list since then. No more strange discussions about who did what, and why. Now and then there are pull requests with threads which get a bit long; if they’re not merging it, I go in, I just press the merge button, end of discussion, and bikeshedding is finished.
That’s awesome! [laughing]
And it’s very simple. If you have a problem with someone’s pull request with their patch, you make your own pull request on top of it, and it will get merged as well. This notion that you can fix problems by moving forwards, not by focusing on the problem, it’s so powerful.
Right. I think so many people get fractured and paralyzed by that. You seem to have really struck a chord with this. One thing, I loved the way you opened up with focusing on the people. Sure, you’re creating software, but in the end of the day it’s the humans behind it that’s creating this thing, and only by the stability of that community around a project or a codebase will the actual source code become positive in its own mission. That to me is so profound, to think like that, and so counter-cultural. I kind of mentioned that earlier as I tried to interject a little bit, but I didn’t want to interrupt. I just think that’s so counter-cultural. It doesn’t seem like that’s how it is out there in the open source way, and now it seems like this is so different and unique.
I’m more extreme than that. I’m kind of way out there sometimes, so I had this… Although my job has been for many years software architect, and I build architectures and software, and I can plan out whole systems, and layers and layers and layers, and they will be built and they will be shipped and they will work - that’s been my job for many years - but I’ve come to believe that architecture is a fallacy. It’s a bit like this meme I saw of some kind of a weird dog that’s been bred, and it’s cross-eyed and it has broken ears and it can barely walk. Then there’s a wolf, and the meme is ‘Evolution’, and the second one is intelligent design, the bred dog.
Basically, I’ve been pushing kind of silently for people to understand that evolution is where we want to be going towards, rather than intelligent design. By that I mean that as individuals we’re basically incompetent to be solving the right problems. We think we’re competent, we believe we’re competent, we tell everybody how competent we are, but in reality we’re blind, we’re limited, we are only looking at a very narrow area, and we’re distorted by all these assumptions and fallacies and beliefs which are wrong. And it’s only by incremental trial and error with a large enough group and a diverse group that you can actually begin to solve the right problems. If you accept that, then you understand that the starting point is the group. You need the group, the collective, the diversity, the lack of friction in communications, you need the ability to challenge any assumption, to think quickly together, like an ant nest - I love the ant nest as a model. Ants are extremely successful and they have this tiny genome, I suppose - I mean, ants aren’t very complex and yet they do really complex things. I think that’s a satisfactory and accurate model for human behavior, and for technological development. I think that the other model where the ant is somehow a super intelligent thing that can create out of its own brain is a lie, it’s a fallacy. It can work by good luck, but it’s actually just bullshit. My friend Leif had a thing from Texas called a “Bullshit Repeller”, which is a spray he’d just spray around when someone talked rubbish. I think that’s the conflict between the ninjas and the community, this kind of lie against the truth.
Well, you definitely have the experience that bears it out. You’ve been probably there with many opportunities to spray some repellant. Let’s take our last break and we’ll continue talking about this, as well as community stuff, and we have some good closing questions for you as well. Lots to talk about, we’ll probably run out of time and think “Man, I wish we had more time with Pieter”, but let’s take that break and we’ll continue the conversation in a moment.
Alright, we are back with Pieter Hintjens. Pieter, we’ve talked about a lot of things on this call. We’ve talked about or around ZeroMQ, which is the project that I knew you for. Now I’m starting to think maybe C4 will be your legacy, but nonetheless ZeroMQ - a very interesting project, and we’ve been talking about the community around it and the way you’ve managed it and the way you’re passing the baton on.
We haven’t given too much to the project itself, its technical merits, its history, why people should be interested in it, and even where it’s headed. Can you give us the rundown on ZeroMQ?
Yes, so the story is kind of Prometheus and stealing fire from the gods and giving it to the mortals to cook their hamburgers on. Me and a couple of my friends got involved years ago in making messaging for the financial industry - these were my first major protocols. I think at a certain point it became clear that what we were building was actually really interesting for other people, too.
The first people we were aiming at as customers were smaller financial businesses, small trading houses, so we focused on raw power and speed, and making it open source and trying to get license fees from these people who of course never paid anything because they were so greedy, they wouldn’t pay a cent, but it was great fun.
Then we kept making it simpler and more accessible, and at a certain point for me it was like “Okay, we’ve actually cracked the problem of building distributed systems. We know this stuff works.” Our software ran the Dow Jones Industrial Average for years and it worked. It was really efficient, it was straightforward. ZeroMQ is better than that. It’s 25 or 100 times faster, it’s simpler. This stuff is really really valuable, but our target audience doesn’t really understand the problem space yet, and one of the biggest handicaps is the complexity of this whole thing; distributed systems are very complex. So we’ve been working towards making that simpler and simpler and simpler, as a metaphor, as a model, the learning of it. When that works, it’s a thing of beauty.
One of the stories in my article - people have been writing on my article all kinds of comments, and one of the stories from somebody is this guy who was asked to make a prototype for the NFL. They have put RFIDs on the players and they’ve put little scanners around the field, and they can collect all the data from where people are, where every player is, and they wanted a system to bring that data in and pump it out to their machines. So this guy wrote in a weekend a ZeroMQ program in C++ that pulled in and distributed. It kind of worked, and then he went off and forgot about it.
A while later his company folded, he was fired, and his friend at the NFL said “Come, come work, come back.” It turns out that his code was being used as the basis for the NFL’s Next Gen statistic system, which is one of their products, and it’s the same code; essentially it hasn’t been changed very much, there’s a team around it but the core of that was 500 lines of C++, and it worked basically the first time and it’s still working. The same model has been working now for a year and a half very successfully.
And deployed as proof of concept…?
Deployed, in production already for a year. So it’s being rolled out, it’s a product. This is thanks to a little library which is solving the right problems in the right way - for a certain class of applications, not always. So our journey with ZeroMQ has always been to make that more accessible, easier to use, less wrapped in mystical terminology and strange concepts that you can remove. It’s been removing concepts and removing concepts, making it simpler over time.
Do you consider ZeroMQ complete, or do you feel like there’s a lot - it can continue to remove concepts or continue to innovate beyond your direct involvement?
ZeroMQ is a community of projects, so it cannot be ever complete, there’s always going to be new things. We were speaking of Node.js and one of my last gigs was to build a Node.js binding for ZeroMQ, a new one, which I didn’t manage to finish - ha! So if there’s somebody who knows Node.js and C++, because it’s a C++… Man, it’s a horrible beast! And if someone is up to that, there’s a lovely piece of work there. Probably impossible to do, probably crazy. In fact, I don’t think anyone can do this.
ZeroMQ is also the core library, it’s the core messaging patterns, and in there we’re actually still simplifying those patterns. For example Doron Somech, who is the new dictator, actually implemented a bunch of new patterns which are very simple, very nice, their names are nice. It’s a client-server pattern; there’s a radio dish pattern which uses UDP. There’s a Scatter-Gather pattern, and these are simpler than the existing patterns; they only take single-message parts, there’s no multi-framing, there’s no strange structure in the messages. You send one thing, you receive one thing. These are still raw, it will take a year or two before they get into broad use, but they are very elegant and they are very nice concepts. In fact, we stole that idea from the fork of ZeroMQ, from Nanomsg, which had this simpler kind of approach. So I’m not averse to people forking stuff, and then us stealing those ideas back, I think that’s great as well.
Why not, right?
Yeah.
It makes sense. I’ve always wondered this myself, as someone who doesn’t go this deep into system software, and I kind of get the idea of what messaging is, but can you walk us through an overview for those out there who are listening and thinking “I’m following along, I get it, but what exactly is messaging in a system like this?” Why is ZeroMQ needed, what is the true problem it solves?
Typically you use TCP to connect systems. If you’re using TCP, then you have to solve a bunch of problems to make it actually work. You have to frame - because TCP is a stream of bytes, and messages typically aren’t streams of bytes, so you have to frame… You have to handle errors, you have to handle asynchronicity, because you’ll have multiple threads in your application. One will be sending, one will be producing data, and the two have to talk to each other, with buffering. You may send a thousand messages in one go, but the network may be quite slow, so you have this asynchronicity happening at all sides. You have to be able to deal with different languages; maybe you have C++ talking to Python, or maybe you have Java in there somewhere. Do you want to just impose Java on the whole ecosystem? Maybe, people do that. And so on. You have these 20 or 30 quite specific problems that you do have to solve if you want to make production quality communications between two or two million pieces of software.
People do solve these problems over and over again, and mostly they solve them in an ad-hoc way. They have a library in their application which they maintain, which is theirs, and which is never really looked at by other people, which is their best guess. “If I had to do this - it works, but it doesn’t really scale, it has performance issues.” They can push maybe 5,000 or 20,000 messages per second through this stack.
If you take that stack and you open source it and you put it into a community who are actually experts in distribution, then what comes out is much, much better. It’s much better in every dimension - not just in performance, but also in stability, also in exceptional conditions, also in error-handling, portability and so on. And you get this thing which you can plug into applications, or you can build applications around rather, because it’s a design tool. It solves the problems in a way which is really good, effective, in a way that you can rely on, and which lets you build now your distributed system without having to think about that TCP stuff or that UDP stuff happening between your pieces.
I’ve told developers this in conferences for years now, that if you’re still building monolithic systems, then you’re building for the past. And this notion that there is a computer in the building on which stuff will run, it’s the 1950s. The modern world has billions of computers. Even my house has hundreds of computers by now; just my kids and their mobile phones, and tablets. For software, too - use this profitably, it must be distributed; it must be decentralized. This problem of communication between these moving pieces is still really unsolved until you have something like ZeroMQ.
What you can build on top of ZeroMQ becomes then really interesting. Once you’ve taken away this cost and we build stuff on top, like Zyre. Zyre is a kind of ad-hoc, a clustering library where you simply join a network and you will discover peers automatically, and you can talk to them and broadcast to them. It’s reliable, so you can chat with peers and send them stuff, and they will receive it if the network is very flaky. It’s simple to use, it’s relatively hard work to make that, but it’s doable because of ZeroMQ.
I’ve used Zyre for example to build IoT demos with clusters of devices that you switch on and switch off, and they find each other, they can talk to each other, they can do interesting things. And it’s just straightforward to build that kind of stuff now. It used to be impossible. It used to be literally years and years of work, and now it’s a few weeks, a few days even, sometimes.
So take us through what systems are out there using them, and based on that, it sounds like this is like the - which makes total sense to have Ilya on the learning page (Ilya Grigorik, who’s been on this show before; ‘Internet plumber’, so to speak, that’s his self-professed title). It sounds like that whether we may know it or not, that ZeroMQ is the foundation for which a lot of the stuff we might do day to day on the internet is built upon.
Ilya was one of our first fans, and wrote about it, and helped us to get some footing. The thing about stuff like ZeroMQ is that we don’t know who uses it.
Really?
Literally, we have no clue. There’s no registration process, we don’t keep track of downloads. It’s completely invisible and when things work, nobody tells us, obviously. We only see a very small percentage of users; we see people who are absolutely incompetent, who are asking basic questions which we’ve answered often in the documentation, so they get told to read the guide and that’s it, most of the time.
And we get people who want to contribute improvements. So they have hit the limits in one or other dimensions, they’ve gone through the code and they’ve understood the code, and they’re like “Okay, I’ve got this problem. Here’s a patch.” And we have people using it in a way which creates a problem, and they are asking for help. Then we have this very small number of people asking for commercial support, and that’s it. So 99.9% of the users are just off the radar, and we have no way of - and we don’t even want to keep track of them, it’s not interesting.
Yeah. Let’s pull something from this post, since we’re talking about Ilya. He talks about different ways that ZeroMQ is being used, and I guess for the listening audience hearing what it is and hearing how it’s used is two different things. Let’s talk about it a while. What are some things that people have built with it that you’re aware of? His example is Mongrel. I remember when Mongrel was a big deal with Rails back in the day. But just some different examples of how they used ZeroMQ to build something else.
One of the largest deployments of ZeroMQ - and I mean this physically, is the Large Hadron Collider in Geneva, in France and Switzerland, run by CERN. It’s a fairly classic industrial control scanner problem. They have this 20-something kilometer ring - I don’t know how many miles that is; a lot of miles anyway - running around the countryside, and within this ring they have a whole series of machines which are producing energy beams and amplifying them and focusing them and steering them. And when they do an experiment, machines run in real-time, they’re steered and guided in real-time by a software network which has to be able to talk to them and tell them what to do. That’s the control system. The experiment runs for a few hours - I don’t know how long it runs actually - and they collect the data by smashing the beams together, and they have this huge, massive receiver thing around the smashing thing. [laughs] And that produces a lot of data which then gets sent off, and that’s the data part. We don’t deal with the data part, we deal only with the control part.
So they control structure is they have these machines; each machine has custom drivers, bizarre serial port control interfaces built by who knows what manufacturer somewhere. These feed into computers, just normal little Linux boxes. There’s I think about 500 or 1,000 of these computers on a network, which then talk to central systems which tell the operators the status of every machine, and allow the operators to guide the machines and the software which is doing the guiding for them, obviously. And that network is built using ZeroMQ, currently. Actually CERN went through a review of different things they could use, and they were using CORBA before, which is old, but it was a well-known quantity.
They rebuild their systems every five or ten years, and they went through and looked at ZeroMQ and said “Well, it’s the best software, but it has above all the best community. We like that and we appreciate that.” Community translates into lower cost also; you have more access to experts and so on. It turns out that there’s a whole bunch of these ring… There’s also an X-ray factory in Grenoble, called the ESRF, and they’re using ZeroMQ in a similar kind of control system called TANGO. TANGO is now built by a consortium of different X-ray synchrotrons around Europe and around the world now, and they’re all using ZeroMQ as their messaging system underneath that.
So it’s become the basis for systems like this. These are large systems, these are big applications, and the scale from that weekend project I talked about, with the NFL - sorry, this is confidential. It’s on my blog anyway, so I guess that’s not confidential.
[laughs] Too late, I guess…
Yeah, too late. So those weekend projects, which there are hundreds of people working on them… Whenever there is bunches of computers to get talking together and some smart person says “A-ha! We can use ZeroMQ for that!” then there’s a win. There’s a win for that person, he gets competence in his career and solves important problems, and his organization wins money.
Let’s talk about an unusual topic, but a topic everybody has to deal with, which is bugs. I was thinking about, as you described ways that ZeroMQ is built, or used in the wild, so to speak, how those - you and the rest of the community behind ZeroMQ - must feel whenever they have to reproduce or test case a bug, to deal with something. I’m on your “Solve a Problem” which is essentially your support page - not the contractual/commercial support, but bug fixes. The first one says “Make a reproducible test case if you possibly can.” So I’m thinking in a scenario where you have that ring and several machines, light beams smashing, how do you even deal with that? So what is bug fixing like for the community dealing with this?
First of all, when people are using ZeroMQ, if they’re smart enough to choose it and use it successfully, then they’re smart enough to build those large systems out in ways which shake out all the bugs early on. They’re obviously not spending a weekend wiring up the Large Hadron Collider and then trying to see what happens, right?
Right.
So they have simulations and test benches, and everything can be reproduced in isolation. Actually, ZeroMQ lends itself to that really well. If you look at the examples that I write in the guide, you’ll see that they’re always built out from simple to more complex, and you can take every aspect and test it out, and take the next aspect and test it out. So it turns out to be quite simple to isolate behavior, in most cases. It’s rare that you have really bizarre, complex bugs caused by a bunch of different circumstances. That’s the first thing, that’s always been the case.
But what’s more interesting is that in the last years we’ve really seen a dramatic dropoff in the number of issues and bugs in our code. I think that’s deliberate, but that was kind of a consequence of this magical process.
The C4 process.
Yeah, yeah. It’s actually quite obvious. Bugs come from code which hasn’t been properly tested, which has not been properly hammered by people, and which has got speculation in there, code which is inaccurate. And when you hit it with actual reality, it performs wrongly and things crash or go weird. The trick is to break down your development process into very small steps, where each step can be tested. I don’t mean formally tested by scripts, I mean tested against someone’s actual problem, and validated before you go further.
We used to have in ZeroMQ, like I said, these waves of raw code coming in; here are 5,000 lines of new code, here is this whole bunch of new changes. And the bugs came from that, and they came from the time it took to weed out all of the inaccuracies in that code. Now we don’t have 5,000 lines of new code, we have maybe 200 patches; each of them are 5 or 10 lines. In some cases they are longer, but they tend to be quite small, and they’re testable in each case. The patch is modifying some behavior which can be tested by a test case. So you’ve got a before and after for every change. It’s very hard to make that buggy, it’s very hard to introduce strange behavior when your changes are small and they are testable. It’s like the ants going out - this is why I get back to intelligence - and foraging for food. If they make a small step in the wrong direction, they can correct very cheaply. But if you take a bucketful of ants and throw them into a field, they’re all completely lost. That’s the buggy code, it’s a bucketful of ants, and what we build are these very efficient little lines of communication where the ants have identified some interesting problem and they go for that. And the more they work, the more accurate they are.
Quite literally bugs in this case too, using the ants as the model there.
Yeah. We have people who basically build on the master version of GitHub, different projects. That’s what they’re using, and they go into production with that when it’s properly tagged. Match that to our strategy of accepting pull requests onto master, and it gets quite freaky.
[laughs] Yeah, those two worlds collide.
Big time.
Well, they only appear to. It turns out they don’t really collide at all.
It’s not as big of a problem as you think it would be.
Well, the thing is that if you accept pull requests onto master, then when there’s a problem someone will fix it really quickly and that will get merged onto master. So in fact what you’re doing is you’re correcting master much faster than you would otherwise.
I assume people who are building large financial systems or monitors or what not, probably are not on master too often.
Probably not.
Let’s give some encouragement to those… I mean, what you just said there with the major dropoff of bugs is certainly encouraging for those who have heard of ZeroMQ, are into or want to get into this space more so, and we always like to give a good shout out to getting started. I know that you have a Get The Software page on ZeroMQ.org that certainly tells you how to get it, but what is your favorite getting started guide that you can recommend to the listeners? What’s the best way to get started?
I would say that it depends on your language. What you do is you have to look for the language binding that you like best of all, and then use that. Because the actual raw ZeroMQ is C++ wrapped in C, the API is kind of weird still, and it’s not the best place to start. Even the documentation I wrote is kind of… It starts there and then it very quickly switches to using a higher level C API. So the best place to start is, you know, if you’re a Python developer, you take PyZMQ; if you’re a Java developer, you take the JeroMQ; if you’re working on .NET you take NetMQ and so on; if you’re a C developer, you take CZMQ. Start with those examples and start using that, and then only go deeper as you need to. Only go deeper if you have a reason to.
You’ll find examples for whatever language you’re using in the guide; the guide examples are in every possible language. But each of these bindings has a community also, and they will be able to help you get started.
In terms of learning ZeroMQ there’s just one book, the one I wrote, and it’s called The Guide. That’s the guide to ZeroMQ, it explains it. That’s where you start if you’re actually going to go through the actual process of learning this as an experience.
So that’s it, zguide.zeromq.org. We’ll put that in the show notes of course, but I just wanted to mention that here. The bullet points on the front of that page say:
- Explains how to use ØMQ
-
Covers basic, intermediate and advanced use.
-
With 60+ diagrams and 750 examples in 28 languages.
So you’ve obviously got I guess most of your languages taken care of. It seems like you’ve got - let me see if my notes are correct here - 40+ languages that you’re covering in terms of bindings.
- Available online and in PDF format - is it like your other books where you can read it online but buy it if you want?
Yes, it’s available online. That’s where I started it, so it’s a Wiki… Actually it’s one very, very long page. People prefer that, it turned out. I had first of all written chapters, and people were like “No, no, I want one page. I can print it off, I can go back and bookmark it, whatever.”
It’s also available as an O’Reilly book, called ZeroMQ, and there’s also PDFs that you can download. O’Reilly were actually very happy with us having it available online at the same time as they made the printed version. The printed book is really nice, it’s quite a fat book, it’s a large book, to be honest, but it looks good in the bookshelf.
Is it as thick as the JavaScript book…?
…the JavaScript Bible.
I don’t have the JavaScript Bible.
That’s awesome. I guess one more topic here on ZeroMQ - we talked about the community, we talked about technology… Not just getting started, but what areas of the community can people fill in? We always ask the question of “How can the community help?”, that’s essentially the basic question. Where are the needs in ZeroMQ and how can people step up to fill those needs?
That’s a really good question. I think it’s very difficult to come in and contribute to a project which has already got so much mass, it’s hard to know where to start. Certainly the bindings - again, the bindings are the frontend deliverables to different language communities - always need help. Like I said, the Node.js binding I was working on. Every binding needs help, and being updated in documentation and guides for that, and so on. If you’re using a particular language in ZeroMQ and you want to help other people to use it, then that’s a good place to start.
I think also what’s - I don’t know if it’s missing or it’s just not necessary, but meetups. Finding other people who are using ZeroMQ or working on distributed programming in general, and meeting up regularly and discussing what you’re doing and how you’re doing it.
There’s the big problem of confidentiality, and most businesses don’t like talking about their infrastructure projects, but apart from that, I think that meetups are really an interesting thing to do. It’s great to meet people who have the same kind of problems as you.
Yeah. It’s interesting you should say that too, because you’re so community-oriented and you have such an influence around building community that that’s a need. It seems like it’s out of place to have that need.
Yeah, it seems out of place, so I’m not really sure if it’s absent because we’ve not worked on it, or because nobody really needs it. But I’m involved in other meetup groups and it can be a fun social experience simply to have people who have the same kind of problem domains as you, and who are working on the same kind of stuff. ZeroMQ may be a good excuse for that. If I was using it and not really confident enough to contribute code to it, then that’s probably where I would start.
If we’re clear here, I’ll switch subjects a little bit. I have a question for you, Pieter, that I’ve kind of been waiting to ask. One of the things that you mention in your protocol for speaking to someone who’s dying is to focus on the good things, to focus on life and camaraderie, and history, and talk about old adventures - discuss the good things of life. You and I just met, so we can’t really go back and talk about that, the good ole days when we were doing anything together, because we haven’t. But. Through you work with ZeroMQ and your writing and your teaching, like you said earlier, you’ve been going to conferences all these years, through your company you traveled the world, teaching, coaching and so on and so forth - surely you’ve built up some good adventures, some good stories. I’d love for you to share something with us, whatever comes to the top of your mind, a story of yours, whether it’s software related or not. I know you have lots of non-software interests - piano, guns… Even though you don’t seem to own a gun, which I think is kind of interesting, even though you’re a certified pistol instructor. Drums and other things… So if you have a good story for us, I’d just love to hear it.
Oh gosh, where do I start? I don’t own a gun because in Belgium it’s very complex to own guns. You have to have permission from the chief of police and you can only one or two. It’s a real hassle.
I actually was working in Texas for a year for a company, Samsung, and there’s not much to do in Dallas, Texas. You can drive and you can go shopping, you can go to nice places to meet people, there are lots of places to hang out and so on…
A lot of food.
A lot of food, a lot of beer…
And in Texas it’s almost illegal to not own a gun.
Yeah, it’s a requirement. [laughs] You can’t get a driver’s license unless you own a gun.
And I was doing this workshop with a bunch of people from Samsung, it was our first workshop there. We were trying this kind of prototype distributed system, and it was what eventually became Zyre, I guess. They’d been trying to do this for some time and they kept failing, trying different ways to do it. So they had five or six of these really clever guys - there was one from Kazakhstan, there was a bunch of Americans, there was one from… I don’t know, it was a very diverse mix of people brought in for this. And after three or four days we had built really amazing stuff and it was all working; little video games working across four mobile phones, and everybody was very impressed. Then just very randomly, I was like “There’s got to be a gun range somewhere”, and it turned out that the whole class were gun fanatics, shooting fanatics, in one way or another. There were maybe one or two who had never shot, but who are into it and willing to try it.
So we found a range, went there and shot. I was like “Okay, I love Dallas.” [laughs] It was cheap, it was easy, they were friendly, the guns were great, I was able to shoot. Over that year I must have gone shooting a hundred times. I would just shoot 200-300 rounds of ammunition. And I got really good at it; it’s just a matter of practice.
Then I decided at a certain point - this was a kind of an insane project; we all went kind of batshit crazy in this project for different reasons. It was incredibly political. We had been trying to do stuff that everybody wanted to do, and when we had got some success they were trying to kill us and take it over from us, both people from inside the company and outside the company. It was just endless, endless politics, so my job was to navigate that politics and just beat off all the bullies and keep the team together, alive and fed. So at a certain point I’m like “Okay, I’m gonna get my pistol instructor badge.” I become an NRA member…
Really?
Yeah, because the NRA are the people who do the pistol safety, the gun safety in American, which is a really good job. I mean, I don’t like their politics - their politics is bullshit, at least as far as I’m concerned - but their work on gun safety is really excellent. So I went down to Waxahachie, Texas, and there’s this amazing little gun school there, I think it’s called the “U.S. Weapon Something-Something”; although we don’t say weapon in the NRA. We never say weapon, we say gun or pistol.
Gun…
Gun.
See, I was in the military, and it was against the rules to call it a gun, or things like that. You had to call it weapon, it was a weapon.
It was a weapon, yeah.
“Where’s your weapon, soldier?”
But in the NRA, in the training, you don’t call this weapon, and everybody in the class with me, they were all ex-military, all infantry. You know, these big, strapping men with tattoos on their forearms; their forearms the size of… I don’t know. And they’re all aiming to get their instructor certificate so they can go off into security, or become an instructor. And there was this stupid, scrawny Belgium trying to figure out how all these things worked. I knew nothing about pistols in detail. Then I came to the practical exam, going out and shooting. Of course, if you’ve been in the army you’re an excellent shot with your right hand - your dominant eye and dominant hand, as you know, right? But if you have to switch to your left and one eye, the wrong eye and the wrong hand, then you’re kind of lost, I guess. That was the case here.
So as this test went further and further, they were just completely useless, and I had been training both eyes, both hands, so I was the best sharpshooter. They were all just so angry with me. I was like “Yeah! Belgium for the win!”
And then my family came to visit, my kids came out. I took my daughter shooting when she was 7, and my parents were so pissed at me. They were like “You can’t take your kids shooting? What is this, are you crazy?” I’m like “Well, she can shoot. It’s good. She’ll never forget how to shoot. She’s not afraid of guns anymore and she knows how to point the gun in the right direction. If ever a gun falls on the floor beside her, she’ll pick it up and she’ll know exactly what to do with it.”
I’m personally neither gun, nor anti-gun. I went shooting once, at a friend’s behest, like at a range. I had a good time, but it just didn’t really trip my particular trigger, some trigger… What is it about it that’s so fun and so interesting? Is it stress relief, is it the challenge, is it the noise? What is it about shooting a gun that you enjoy?
Well it’s the pure machismo, I mean, for a nerd to be able to go off and shoot things… No, I’m just joking, it’s not that at all. [laughter] Technically, it’s difficult. When you’re stressed, or if you have a lot of things to think about - it’s the same reason I enjoy drumming, actually; it’s the same thing with drumming - then to be able to shoot properly your mind has to completely switch off at a certain level, completely switch off. Even your reactions when you’re shooting, your body knows a noise is coming, so you’re gonna jerk that pistol up - you have to switch that off as well, so it’s meditation. If you get in the zone, and you’re not tired, and your arms aren’t tired and you’re shooting, then it’s just absolutely magical how well you can shoot. Then you lose it again, and so it’s kind of aiming for that zone. It’s a form of meditation. It’s fun, I like that. Also, for a European to be able to go shoot is kind of illicit - illicit pleasures for us. We don’t have that really. Maybe. I went shooting in Vilnius, in Lithuania, where it’s even more random than Texas, I have to say. But it was great fun, as well.
It takes a lot of breath-control, it’s like the number one step for shooting properly, and you can’t breathe well if you’re fussy or thinking about other things, or fidgety, or frantic, or whatever. That’s why motion-based shooting is really tough. The show 24 (Jack Bauer) - that’s not real. That never happens, you’re never that accurate.
Yeah, shooting teaches you… It makes it harder to suspend your disbelief when watching movies and TV shows because it’s so outlandish that they can do anything that they’re doing.
That’s cool. Pieter, I would have never expected you to come on this show and tell us a story about guns. You just blew my mind. Especially even here in Texas - that’s where I’m at, Houston, Texas - so Dallas is like four and a half hours away from here. Like I said earlier, it’s not true, this isn’t a fact, but it’s basically illegal to drive without a gun. It’s just so common here in Texas. We even have a saying that has a canon in front of it, and it says “Come and take it”, because we’re so relentless with our guns, we have to keep them.
I don’t so much have this belief myself, because I’m not a true Texan - this is a total tangent - I’m from Pennsylvania originally, so Pittsburgh area (Steel City), that’s my home stomping ground, so I’m a transplant to Texas, but I kind of feel some similarities between Pittsburgh and…
I was in Pittsburgh last year. I went shooting in Pittsburgh and it was fantastic.
See? Very similar.
Very similar.
I mean, it’s the US. Not all states share the same things, but…
You’re right.
…“come and take it” is a thing that Texans say.
That’s machismo.
Yeah, it is. Come and take it, come and take it.
That’s machismo. I wanted to make a conference called “Guns and Code.” I had the domain name GunsAndCode and I was gonna do this probably in Oregon, rent a cabin, have a bunch of friends up there, and just rent a bunch of guns and ammunition, and randomly code and randomly shoot.
Could be fun, I would attend.
Yeah… I got told it was a bit too politically-sensitive.
It could be dangerous.
Yeah.
I was recently invited to a bachelor party, and it was “Bring your guns, we’re gonna be shooting, and… There’ll be food there.” [laughter] That was the invite. I was like, “Well, I don’t have a gun, but I guess I’ll go and just kind of sit back from the main group a little bit”, and keep my safety. But we are quite a bit upstream on that. Let’s ask you this, Pieter - so you’ve wound your career down, by choice of course, because you’re ill, so you don’t have any career work ahead of you, aside from that you’re writing, which I assume you’re gonna continue to do until you can’t or until you don’t want to anymore. So you’ve accomplished a lot, you traveled a lot and all that, written books, given talks, so on and so forth, but what mountains are out there, or what goals or problems are unsolved? That if you had 25 years of a career ahead of you, and if you weren’t looking back and you were looking forward, what was something that you didn’t get to solve, or a mountain that you haven’t climbed yet?
I guess the thing I was working on, which I had hoped to be able to push forward was the internet of shit - I mean, internet of things, sorry.
[laughter]
I have this vision that – it’s not even a vision, I’m just predicting what will happen – the number of smart devices, the number of computers out there will keep doubling every 18 months, and it will keep doubling every 18 months, and every 18 months again, doubling and doubling. And that’s just the way it’s going to go until the world is so full of computers that nobody wants more, and that’s a long way off. And these computers have to talk to each other and they have to be organized into networks, and there’s this whole fight going on right now about who controls these computers, who controls their data, who has the right to update them and so on.
So we were working on this this year earlier and I had to stop, I was really getting too sick to work on that, but it was about building networks of little computers - we were using OpenWrt routers, the GL-AR150. It’s $20 and it’s absolutely wonderful; it comes in a small plastic box and it runs on a battery. Fantastic little device. Completely programmable - I don’t know how many megabytes it has, but enough to run our software on properly.
So my idea was look, you have a house full of these things, controlling - they all have GPIO, you can control stuff - controlling lights, controlling temperature, sensing rainfall, whatever you want. And they cluster together. You don’t have to configure them individually because there’s too many of them. You just plug them in the wall and possibly connect to a Wi-Fi and that’s it. You don’t program them, instead you send code to them. You don’t program them by updating the firmware - whatever is involved right now, with this horrendous process of sending your compiled code to the thing. That’s finished, that shouldn’t be happening. Instead, these little devices should be picking up code from the network like a browser does, and running it and then throwing it away. That’s how I wanted my kids to be programming the light bulbs - write a bit of code, send it to the light bulb. Switch on - become green. Switch off at 10:30… I don’t know. And if there are a hundred light bulbs, they will all respond the same way. If there’s two, they’ll respond the same way.
That notion of the internet of things becoming the web of little devices speaking some language - probably JavaScript; it will be JavaScript… Which is kind of sad, but that’s how it’s gonna be, I guess.
I wanted to make that happen. Then I wanted to sell - a-ha! - make money. How did you make money from open source? So I wanted to sell security on top of that, and certificates so that you could get properly signed code, that’s been authenticated. This is a lovely area, it goes into things like mesh networking and so on. Devices can be also Wi-Fi routers, they can talk to each other, they can build whole structures like that.
We’ve just talked about mesh networking actually in… 204, Jerod? What number was that we just released the episode? 203 - Jewelbots. It’s an interesting little project. I guess I shouldn’t say little, because it’s quite big, but it’s essentially programmable bracelets for kids - primarily girls right now - getting kids into coding and programming those things, and they build their own little mesh network through them. So if you walk up to another friend and they’ve got their bracelet on, and they’re your best friend, then they change color, or they have some sort of taptic pulse or something like that - some unique interaction that happens based on proximity and this mesh network, which is kind of interesting. That’s totally up your IoS or IoT, whichever you wanna say.
Yeah, it’s that proximity, something that we lost in the internet, the concept of proximity. I think it’s been possibily the biggest hurdle to people understanding what it really is. We used to have proximity, we used to talk, and then we had phone calls, but it was still conceptually a wire. Even this, we have a wire that’s connecting us somehow. But you know, you send an email to some address which represents some person who may pick it up at some point in time - it gets kind of weird, and I think that there’s something to be won back by recreating proximity as your connector, even if it’s virtual. And mesh networking does that, up to a point. If you have very large meshes, of course, you’re back to the internet.
It’s interesting. Have you documented any of this stuff? This isn’t a one-to-one, but similar to Elon Musk, which must be some sort of flattery mark to you because he’s got unique, big ideas that he’s only got so much time to do so much: Tesla, SpaceX and whatever else. I think these are some interesting ideas, and you’ve obviously got some wisdom and some bloody knuckles from being in the trenches so long and doing what you’ve done. Have you put this down anywhere that somebody can pass the baton of the idea, or the possible idea?
I write a lot, but it’s all over the place. If you look at my GitHub profile you’ll see lots of projects I’ve worked on. Zyre implements this concept of proximity; CZMQ implements this concept of simplicity. We have different projects that do different things. Then I have my blog posts…
It’s one of the problems, that if you’re writing lots of stuff in lots of places, you have fragmentation of anything which is a grand concept. Which brings me to my latest book actually, which we’re gonna talk about. Somebody asked - and I have this question happening a bit too often - “Okay, how do I build communities online?” And I have these different pieces of text I’ve written on that, here and there and left and right. I eventually put them together and published it as a book, Social Architecture, which is quite a slim… I actually got 20 copies from CreateSpace, I ordered them… And it’s a nice little slim book, but it’s like this is my best practice for this particular problem. It’s all in there. But that’s extra work. Somebody will have to go through my stuff and figure out what I was thinking about. [laughs]
This is a late article from you, May 17th, Building Online Communities. This is sort of - in your own words - an early draft from your book, Social Architecture.
Right.
How close is this to that copy?
That’s I guess about a third or half of the actual book. I wrote the article first and then I went back the next day and re-edited it, added in more stuff, and published it properly, with a cover and so on.
Gotcha. We have some we have pulled out here for talking through. We’ve talked already through fair authority I guess to a degree, with ZeroMQ and C4, but something that we’ve talked about probably maybe almost too much - I don’t know if you can actually say that, but we’ve talked about it quite a bit in the last 30 episodes, it’s come up at least several times, which is funding, or ‘sane funding’, as you put it in the toolbox. Maybe let’s talk about that one piece there from the table of contents. I know there’s quite a bit of stuff going on in there, but what are your thoughts on this sane funding?
Yes, I think it’s a problem which has solved itself in some respects. It used to be that to build a community you had to have infrastructure, which you had to pay for. That used to be a fact of life. You had to have servers, you had to have email servers - you had to have stuff, and it cost money, and the money had to come from somewhere. That was the first problem.
Second problem was that you had to do a lot of the work yourself. We hadn’t yet figured out how to really bring in people smoothly and cheaply. Until we had C4, we didn’t know how to do that properly. So the cost of infrastructure has gone away, almost completely. Right now, like I said, the baton for ZeroMQ - it’s a domain name, or two. The rest is hosted by GitHub, or we could be using GitLab; we have access to free email lists in different places and so on, so we don’t have to pay for infrastructure any more.
The second things is that when it would come to the actual cost of work, that if the process focuses on people who are using what you have, solving real problems and coming with their solutions, then it’s self-financing. That was a real breakthrough. At that point we realized that we didn’t need to be funding the work upfront at all. We could leave it, and if people were not improving it, that’s because they were not using it. And if they were using it, they would improve it, and they would improve it for their own benefit, on their own pay time, and almost all of ZeroMQ is built by people working at office hours, being paid to improve it because it’s in their own interests, and those improvements come back because of the license. Our job is to make that flow work as smoothly as possible; that’s a certain point, maybe three or four years ago, where I stopped having to invest money. So this thing of sane funding kind of has gone away, at least in software, where you can work with pull requests and incremental progress.
Now, when I wrote that toolkit originally, the toolbox on Social Architecture, I was working in political NGOs (non-governmental organizations) fighting software patterns in Europe. The problem we had there was that we had people who were working full-time on very complex technical legal issues, preparing drafts of amendments to laws that could be pushed through and voted on, analyzing what’s going on. There was a really large amount of working done by unpaid volunteers, who were being basically exploited by their own organization, and it was very bad for them, it was toxic. So my first job was to fix that problem and find money to pay these people, or stop them from working. It had to be “Either you stop working on this, someone else takes over, or we find you a salary.” So that sane funding concept really came from that. If you have people who are necessarily working on something, they can’t be volunteers, they must be paid. Otherwise they burn out, and it’s a cult that you’re building, and we don’t like cults. Cults are bad.
Pointing to the last sentence in that section there:
“Finally, watch out for individuals who take on too much risk without adequate reward – they can be vulnerable to burnout, something I’ll talk about in the next section.” So that’s kind of leading into The Market Curve, but I think that’s a risk anybody in any community has to watch out for. Anybody who takes on too much without reward. Because we’re humans, we want some sort of affirmation back, even if it’s a pat, even if it’s not money. It’s some sort of reward system that has to… In this case with NGOs though, obviously they’re getting not paid by volunteering, so that’s a whole different version of burnout.
Well, I’ve looked at burnout from different angles over many years, and what I actually think now is that it’s not so much to do with “I work too hard in this job” or “I gave too much to this thing.” What it really is is the dynamics of a relationship with a bad actor.
It’s true.
And the bad actor is using a situation to exploit people, and eventually they crack. If I look at my own periods of burnout in my career, and I’ve had several of these, I realize that in fact it was that. There was somebody there who was being very exploitative and manipulative who was using people, and I was one of many who burnt out around that person. Hence my interest in bad actors as a topic, because they seem to be very, very influential.
There’s some people I know too that are trapped in scenarios like that. It’s so troubling to hear you say this, because I think about those kinds of people - and I’ve been there too, of course; none of us are immune to it. But you just think of what a trap it is, and it’s almost like… Jerod, we’ve talked about these ones before, where you love the deceptor, in a sense. You’ve been offended by somebody - I’m thinking of things where it’s domestic abuse, or something like that - where you don’t go away… What was it called, Jerod?
You’re thinking of the Stockholm syndrome?
I think we talked about that at some point recently…
Stockholm syndrome, yeah.
…where you fall in love with the…
Someone is kidnapped or taken away…
You have this unusual love or compassion for your offender essentially, and you just don’t see the truth. You kind of get stuck in this trap.
Actually the last book I wrote before the Social Architecture book was about psychopaths - because that is what we’re talking about here, bad actors and the assholes in the world, the people who can use emotions as a weapon. They get really angry and then it’s gone, and it cost them nothing. They are never to blame, they are always the victim. They are innocent, they will never ever see themselves as a problem; they’re always the innocents in any situation, but they manage to hurt everybody around them. At a certain point I’m like, “Okay, this shit needs to be documented.” I can’t say I’m gonna stop it, because that’s impossible, but what I do really well is I take complex questions like this and I break them down and I try to find decent, workable answers.
I’m looking around the internet for answers in how to deal with psychopaths, and there aren’t really any good answers. This is kind of weird, it’s like the only answer that people will agree on is run away.
Yeah, evade.
Evade. Break contact. Leave. Okay, that’s fine, that’s easy to say, but don’t you get it that the basis of this relationship is this bond which you cannot break? The first thing that the psychopath does to you if you’re in a job or if you’re in a relationship is make sure you have no alternatives, and make sure you’re stuck.
So who’s this book for then? The Psychopath Code.
It’s for everybody. It’s not written technically, I don’t use long words and sentences.
What I mean by that is like is it for those who are trapped, since we kind of was on that subject? For people who are in these robbing relationships where they’re being exploited and they’re near burnout? Is it an early warning system, so to speak, for those people?
I suspect if you’re actually in that situation that you won’t be…
Aware.
…aware, until a certain point, and then you start looking for help, you start looking online. You see this in forums on relationships and on other situations, people are looking for help. At a certain point something tweaks in their mind, like “Okay, this is not normal. This has got to be something else, this can’t just be me.”
Hm. I am literally gonna buy this book when we’re done with this, and I wanna read this, because the last line gets me. This book delivers practical tools and techniques to survive the most difficult people. Jerod’s not one of them, but we’ve got difficult people around us often.
We all do…
It’s not Jerod, so don’t think that Jerod, but…
We do, that’s right.
We all have that problem. If it’s specific to our industry then even better.
No, it’s not. I think at least in open source we’ve actually filtered out many such people, but there are a lot of them around. What’s the number one talent of a psychopath? It’s stealth, it’s hiding. It’s not to be caught, not to be seen, and it’s this cryptic thing. You look around and you see normality, but it isn’t really normal. There is this population of people who are - I guess they’re innocent, I’m not assigning blame here. In fact, in my book you’ll see that there’s a lovely backstory to psychopathy, which is really quite fun. I won’t spoil the surprise for you, but it’s a fun thing.
This population of people, they cause a lot of pain. And I’m like “Okay, I’m not a fan of pain.” I think pain is something you can reduce, so there’s a painkiller. This is morphine. It’s not going to stop psychopathy, but it’s going to let you at least deal with it in a way which reduces the pain.
You can actually be in the same situation, with the same difficult person, but once you understood what’s going on, and once you’ve got tools for protecting your mind against this continuous attack, then it stops. I’ve done this; this is written from experience, it’s not just me fantasizing about stuff. It’s written from experience. I had to make this book for myself, and it works. It doesn’t solve the problem of difficult people, they’re still there, but their power goes away almost completely.
So you’ve written several books, obviously. A lot of them stem from either answering a question that you’ve investigated, or direct advice from your mind, and similar to the question that Jerod asked earlier about mountains left unclimbed, let’s kind of tee this question off to you, in a similar vein but different - what advice can you give to the former self, to the developers out there, to the community of open source, whomever you tend to speak to whenever you get into book mode or you get into advice mode? What advice comes to mind most for you, to those who are trying to be in this rich community of either creating software or pursuing creativity in software? I think it can’t be out there simply because our audience is software developers primarily, people who make things in and around software, or even companies. What’s some good advice you can give that isn’t in a book you haven’t written yet, because you haven’t written this? What’s some advice that comes from maybe a book you won’t write?
Most things are bullshit in our industry, that’s my best advice. [laughter] I learned this as a young developer. Don’t trust the mass opinion. Seriously, trust your own intuition and look for areas where you can be special, where you can find yourself, where you can learn or you have space to practice. Just because things are fashionable and everyone is doing it - in fact, that’s a reason not to do that stuff. Do stuff that’s hard and that’s different, that’s freaky. Of course, most of the time you’ll be completely wrong and you’ll be doing stuff that no one cares about, but you will learn, and you will learn the hard way and you will learn well, and you’ll internalize those lessons.
I consider myself a happy programmer. I program happily. I write code, it tends to work first the way I want it to work, and that took like 30 years to get there. So you need to be patient, as well. You can’t just learn something and then become a master in it. You have to internalize so many little lessons.
Making mistakes - that’s fun. Getting it wrong is fun. Just don’t make huge mistakes. Just don’t bet your company, or your house or your family on something that you can’t prove. It takes small steps, make small mistakes - you learn. You make huge steps, you make mistakes - you die.
Then my last advice is to trust other people. Not blindly, of course, as we’ve discussed. But you’re part of an ants nest, and your power is other people, it’s not you. It’s how you can bring other people into your world and be in their world, and how you can share and build stuff together. That’s the real power of a human being, whether you’re a programmer, or a writer, or a cook, or a taxi driver - it’s other people. And that’s my inspirational thought for the day.
I love that, thank you.
Very good. One last question for you here, Pieter, before we close off. First of all, this has been tons of fun. We appreciate you giving us so much of your time. If you were able to pick your own legacy, what you’ll leave behind and what everybody remembers about you, what would your legacy be and what would people say?
Honestly, I’m very content with what I’ve been able to do so far. This was a decision that I took maybe three or four years ago, to stop working for money as a first goal, and to go out into the world and become a writer. I wasn’t a writer before that. I’d written a few things, but to actually publish books and do that as a focused job… To go to conferences, and to meet people and to go out there and… I began writing my blog three, four years ago. Everything I’ve done since then is what I’m most proud of. I think before that it’s mostly - there’s good stuff, but it’s mostly lost to history. And it deserves to be lost to history. I think that’s one of the lessons - be yourself, be happy to be yourself and be proud of being yourself. We all have unique stories to tell, and I think that’s our power. I don’t think that you have to be an expert in someone else’s story. We’re experts in our own stories.
Well, Pieter, I think it’s been like close to two hours. It’s definitely been roughly 50 minutes since the last break. I didn’t think we’d go this long, but honestly I couldn’t hang up on you, because you were just so much fun to have on this show, and I was telling Jerod in the backchannel, “I’d date this guy. This guy is cool, man!”
[laughs]
I really enjoyed this conversation with you.
I dig you guys too, this has been a real pleasure.
Awesome. We’re glad to share - even though we don’t have a history with you, we’re glad to pull out some of your history…
Well, we do now.
Yeah, we definitely do now.
That could be our legacy: Pieter Hintjens digs us.
Yeah.
I dig you guys, yes.
Thank you, Pieter. It’s been a pleasure to have you on this show. Any final closing thoughts before we start tailing out? Anything left unsaid that you wanna make sure we bring up?
No, I just wanna thank you very much for this, I really do. I feel very privileged to be on your podcast. I have to go and listen to it because I haven’t listened to Changelog before, I have to admit. I’m a modest person, and this is very flattering and it makes me feel very happy, so thank you very much.
Good, it was an honor to have you on this show. And like I said, we almost didn’t ask you simply because we weren’t sure how sensitive the situation was, but we’ve been fans of yours for a very long time, and now is the best time as ever to get you on. In closing too I want to mention a couple things, and you’ve mentioned this in your blog, the last couple blog posts around your death and planning for it, that you’ve got a Paypal link that we’re gonna share in the show notes. Kind people have already sent you over 10,000 dollars in donations, and this money is going to go to your children’s lives and making their lives easier when you’re not there anymore. So we’re gonna put that link the show notes. Paypal.me/hintjens Maybe you can step in here as well Pieter, but ways that we can give back to you, and just this podcast alone, the advice you’ve given through your books, the ways you’ve given back to the community, C4, RabbitMQ… Or not RabbitMQ - I had that on the brain. ZeroMQ. There’s a lot of MQs out there, but that’s the one that came to mind when I was saying that, so I’m sorry about that. You know, all these things you’ve given back, I just think if somebody, if anybody could give back anything - whether it’s buying a book, whether it’s donating to this PayPal fund for your kids - what are ways that you would like to see some action, in light of your obvious scenario? What are some ways that you wanna ask the community to step in.
Buy the books. I prefer you buy the books. Actually, I get a good slice of the price in the paperback or an Ebook. Amazon is very good for that. And when you buy the book then I can give you something which you can share with other people. That’s the nicest thing you can do.
You heard it here first, buy the books, people.
Buy the books, people. I think that’s a good thing too, because I’m gonna go and buy several of these. I don’t know why I haven’t bought it sooner. I guess now I’m getting a bird’s eye view on what The Psychopath Code and your latest is about. It rings home to me.
There’s a thread on Reddit where I did an ‘IM me’ on The Psychopath Code, and I was absolutely trashed.
What?
I was trashed, I was murdered. I was told by people to kill myself. “You don’t have the qualifications for this, you’re not a psychologist! How dare you write about this stuff?” and I’m like “Wow, wow…” And I got my bullshit spray out and I was like “Psh-psh-pshh!” [laughter] I love Reddit, but sometimes… Nah, I like that book. I’m actually really proud of that book, I think it works really well.
Well, I think we’ve all been there, so talking to that with you opened my eyes to… I’m always a fan of finding great advice and books that speak to stuff like that. It’s an unhealthy community, and it’s part of the charge of this podcast - of the many charges - that we try to help, foster and lead, is just encouragement, to fend down the negativity, and these are all phrases that you’ve put in there, positivity being in that latest book, Social Architecture. When you talk about people or things, don’t talk about it in negative ways. People don’t have to be that way. I’m not on a soap box here so I’ll get down… Again, Pieter, it was such a pleasure to have you on this show. Thank you so much for all that you’ve done. We definitely appreciate to have you on this show today, and with that let’s call this show done and say goodbye.
Adam, Jerod, thank you so much. Bye-bye guys.
Thanks, Pieter.
Our transcripts are open source on GitHub. Improvements are welcome. 💚