KubeVelo 2022

Last month I posted the best bits from KubeCon EU 2022 - there was a link in the show notes - and halfway through that post I mentioned about Johann, my favorite person at KubeCon EU 2022. Welcome to Ship It!

Hi, Gerhard. Thanks for having me. I’m really honored. It’s the first time being on a podcast interview, and I’ve been looking forward to it pretty much… Please also know that I’m not an English native speaker, so bear in mind that sometimes I spell things or pronounce things wrongly… But let’s go with that.

I think that just adds more to the charm… Because I think we spoke through KubeCon, all the lunches, in the mornings, we would meet in the afternoons, I remember that… And I could understand everything perfectly well. I had a lot of fun talking to you. But the story which really just fascinated me, and which is why you were my favorite person - because you were the only one that I know to have cycled to Valencia from Bern. Like, that is crazy. I don’t know how many of our listeners tried to cycle for even like a hundred miles, or two hundred miles, but you cycled 764 miles. That’s over a thousand kilometers. And you took eight days. Why did you do that? That’s a crazy idea. Why did you do that, Johann?

Well, I remembered when we talked about it at Valencia, and I think at first you didn’t believe me that I cycled from Switzerland to Valencia… Now, the story is I was planning my trip to KubeCon, and flying wasn’t an option, so I checked train options, and other options, and - well, then I came up with the idea “Well, I could cycle to Valencia. It would be a great trip.” I checked it with my wife, with my family, if that was okay, being absent for ten more days, and they said “Yeah, it’s okay”, and here you go. So I went, I cycled to Valencia.

[04:22] I think that’s it, right? The family has to sign off on this idea, and others have to support you. In what way did others support you, which are like outside of your family? Like, from your job for example, or the people that depend on you outside of family - what did they say when you told them “You know what - I’m going to be away for like eight days, plus the conference.” Like two weeks.

I just said I’m being on vacation for two weeks. I didn’t do that much noise on social media. I considered it, but it was also some sort of self time. You know, I’m this type of engineer which is a bit introvert, and I don’t have a problem with being alone… It was still some time to being on a trip alone, so I recognize that as well. But there was some pretty good feedback. I got some gifts for the journey from some close colleagues… And some guys I also think they didn’t believe me that I’m really going to cycle to Valencia. And some colleagues I just informed after the trip, and they said “Wow, that’s stunning.”

Yeah. I mean, I used to cycle when I had way more time than I have today. I used to cycle every weekend; I would take the whole weekend and I would spend the Saturday, early morning, late night, and Sunday again, as much as I could do. And I couldn’t do more than 200 miles, and even then - this was like road and mountain combined. So when I hear someone cycling such a long distance… And you’re not a professional cyclist, right? Just double-checking…

No.

I don’t think you are… Okay. I mean, I don’t think many listeners could cycle for 700+ miles in one go. I think many of us would just give up after the first hundred or maybe 200… Because to be in a saddle for that long is just something else… And it’s not as comfortable as your chair, even if you don’t have an [unintelligible 00:06:27.18]

So I know that this is important to you, again, based on what we were talking at KubeCon; this is important for you from a sustainability perspective.

Yeah.

Can you tell us a bit more about that?

Yeah. Well, if you look around us, the recent heat waves, they’re happening - it’s really getting warmer, it’s a fact. Climate scientists are telling this for a long time; we know about it, but nobody is really doing something about it. I’ve read some books, I informed myself based on some scientific facts, and - yeah, what can you do about it? There are a couple of problems, and one is fossil fuels, because we just get CO2 emissions based on those. So if you want to cut CO2 or to reduce your Carbon footprint, mobility is one of the major problems. Flying is actually pretty bad, also because in the higher atmosphere CO2 is more effective at heating up our planet.

Okay. So reducing the carbon footprint, having a more sustainable life… Because I know that you don’t drive, which was like another surprising thing. You don’t even own a car, right? It’s bikes. One, two? I’ve seen the one, I’ve seen the Specialized, which is a very nice one…

I would say like three to four bikes at the same time.

Three to four bikes. Wow. Okay…

[07:58] I decided not to drive anymore, because if you don’t have a car, you can also have a huge impact on the Carbon footprint. I know pretty much everybody loves his car, but if you look on the streets - it’s been crazy. For a cyclist, you notice that there’s a lot of traffic going on, there are a lot of congestions and a lot of traffic jams… And we are paying a high price for our mobility. We have to solve that somehow, and if you don’t even own a car - and I know this is not possible for some places on Earth, but in Switzerland it’s quite possible, because we have a good infrastructure. So it only goes with a good infrastructure. We have good train infrastructure, public transport to near everywhere… So there it’s possible. And that’s why we decided to go without a car.

Okay, okay. The thing which I like most about this is that you show initiative in how you’re going to handle this problem, that by the way, affects us all. The initiative is you showing us how you are solving it. A responsible person would ask “Hey, how can I help solve this?” and an optimist would say “We can solve this.” They are very optimistic about what they’re going to do, but they may not have initiative.

Obviously, the pessimists and the cynics - I think they’re like the lower ones in this ranking - are like “We can’t solve this problem. That’s it, it is what it is.” And the cynics will say they don’t want to solve it. They don’t have this, and they didn’t do that, and they should be doing these things… While you - “You know what? I’m just going to cycle everywhere. Let’s see what’s going to happen.” I really like that. Okay, so we already established that you’re not a professional cyclist, but you love bikes…

Yeah, I do love bikes. It’s one of my favorite hobbies, too.

Okay. Steve Jobs was saying a very interesting thing about humans and bikes… And we talked about it before we started recording. There was the condor, the human - not that efficient, but put a human on the bike and then they are the most efficient mammal in motion when they are paired with a bike. But I think you had one more. There was actually something even more efficient.

Yeah. So I read this book “How Bad Are Bananas?” and it gives facts about Carbon emissions for various products or activities. I was pretty astonished to find out that electric bikes are even better, because an electric motor is even more efficient as a human cycling a bike… Because you have to consider the calories, and this means food, and this, again, is not as efficient as electric support. It sounds counter-intuitive, maybe it is… And you have to consider the energy to produce the battery. It’s problematic as well. But still, it’s more efficient.

Okay, I already know what I want for Christmas. But that’s another conversation, with someone else… [laughs] Alright, so thank you for indulging me, especially to the listeners… We talked for about ten minutes about this subject, so I would like us to switch gears and go into the cloud-native landscape, CNCF… Because I know that you’re a CNCF ambassador. So first of all, how did that happen? How long have you been a CNCF ambassador, and how are you finding that?

[11:45] So the short answer is I was somewhat lucky to become an ambassador… But I can start with the longer version, which goes back to 2015, and it started actually with containers. Let me tell you that story, when a colleague of mine demonstrated to me how Docker works. He just ran a container, some command, and it was so fast I was blown away. After that, I talked – at the time I was working for an insurance company, and there were some ops people, or infra people, and they immediately jumped on this train, like “Yeah, let’s do that. Let’s go with containers as the universal format for production as well.” After that, the discussion came up, “If you have containers, how do you orchestrate those?”

So there was a discussion whether it should be Docker Swarm, Kubernetes, or OpenShift. Finally, we decided to go with vanilla Kubernetes, which was pretty young… It was already there, but it was still a pretty young project… And to decide to go with Kubernetes at this time for an insurance company was a bold move. So that was between 2015 and 2018, and then in 2018 we thought “Well, there’s not only Kubernetes. There’s a whole ecosystem that’s going on.”

With two colleagues we went to New York, and we met some CNCF folks, like Cheryl Hung, and late Dan Kohn… And we thought “Well, we have to do something about that as well”, so we started our own meetup in Bern. It’s the Cloud-Native Bern Meetup. It’s a community now more than a thousand people. And if you know Switzerland, it’s not that large. It’s a small place. And that’s a huge community for such a small city like Bern. And we went on, and made like 13 meetups in between, and somehow I was lucky to become an ambassador as well. The selection process is not transparent, and I think it’s just because of the community work we were doing in Switzerland. I think that was the main reason for becoming a CNCF ambassador.

I really like how you start with “It was luck, most likely”, and then you go on to describe the last seven years of your life, why you pick Kubernetes, and you do this, and you try that, and it works… I’m sure there’s many, many problems that you have to figure out, and you have to start organizing things… And then 30 meetups later, and I’m sure like a couple of weeks later - there you go, “It’s luck. I became a CNCF ambassador. I don’t know how.” [laughs] Well, I think the clues are in all the work that you’ve been doing, and bringing people together.

As you mentioned, Bern is a small place. Beautiful, beautiful place, but a small one nevertheless. And there’s only so many people that are into tech, and into the cloud-native space. But we do know how tech-savvy and infrastructure-savvy specifically the Swiss people are. They want things to be proper, they want things to be nice, and just so. And they invest in things that have a future, are sustainable for the long-term. So I think there’s something there.

Yeah, I mean, if you compare me to some ambassadors - they are working for the big tech companies, they are influencers, I would say… I’m not an influencer, and I don’t intend to be an influencer…

Well, hang on, this podcast hasn’t gone live yet. Let’s see what happens. [laughter]

[15:50] No, I mean, my motivation to do that is – it’s actually fun to bring people together, get engineers to talk to each other… And we also started this thing, Swiss Cloud-Native Day, which emerged out of the meetups, when we thought “Well, we could invite some famous people to Switzerland, to Bern, so that engineers don’t have to go to conferences in the United States. They can just watch them on Mount Gurten, which is the location where Cloud-Native Day is taking place. They can just come there; it’s pretty local, so in a way it’s also sustainable, even though speakers are flying to Switzerland… But yeah, you can see it in life, in your own city, so…

I really like how your values and principles are aligned, and they’re all coming together - sustainability, the long-term vision, sustained efforts… A lot of things coming together there.

So you mentioned about Swiss Cloud-Native Day, which is a one-day conference. How long has it been going on for?

So we planned the first edition in 2020, and we did some peer pressure on Twitter for getting Kelsey Hightower to Switzerland… And he actually agreed publicly on Twitter. “Yeah, I’ll come.” And we were so excited to have Kelsey, because he’s such an amazing speaker. You had him on the show as well, and he did those KubeCons… It was like – we didn’t believe it.

Kelsey is something else, for sure. I know what you mean.

I’m still in contact with him, but the problem was in 2020 Covid happened, so we had to postpone. We said “Well, we don’t do it virtually, we postpone it for a year.” So we had to talk to the sponsors, and they were okay with it, and we shifted everything to 2021. So last year we had our first edition, and from the feedback we got - I mean, it’s not only myself, it’s like a team of around 6 to 10 people preparing for it… I mean, it’s not professional–it’s a community event, so we do it all in our spare time. We’re enthusiasts, and we believe in cloud-native, so… Yeah, it happened, and the atmosphere was vibrant. It was really one of the best experiences I had in my professional life.

Wow. Well, okay, I’m so sad that I missed the first one, but I’m not missing the second one. So that’s happening this year, in September, 14th of September?

Absolutely, yeah.

And two very important questions - are masks required, and will there be coffee?

The first question is pretty controversial, because there was a lot going on, even for – I mean, I talked to the organizers for KubeCon as well behind the scenes; I cannot imagine what they went through. It was a difficult decision. Currently, no masks are required, because regulations in Switzerland don’t ask for them, and we also informed all the speakers and asked them multiple times if that was okay for them.

There are variants coming up, and it might be possible that we have to do a mask mandate, but at the time there was no mask mandate.

Yeah. So people – it’s voluntary, that’s what I know. The reason why I ask is because, you’re right, it is controversial… And I think at KubeCon EU there was like a whole controversy around it, and follow-ups, and all sorts of things.

So I am curious to tell that story, and we will come back to that story in another episode, when we look at the facts, when people are maybe less in the heat of the moment… You know, things are being said, and things are being pooled all together… So we will go there, at the right time… But for now, I think it’s important to emphasize that if, for example, your region where you are doesn’t require this; like, there’s no regulation to require this. It’s very difficult to force people to wear them… Because that’s what effectively you’re doing. And some people may not be okay with that. And then I think you will fall on one side or the other, and then what do you do? It’s a very tough decision to make.

[20:26] So if you comply with your local laws, I think that’s a good first step, I think it’s a good default… And then making sure that everyone is comfortable with that decision, and giving them enough time to express their concerns, to go through those conversations so that people are not surprised… Which is why I’m asking you now, which is at least a month, maybe even more before the conference, depending on when this airs, just so that people know what is happening, why this is happening… And I know that you have on your website - I think here’s something where you mention this, like on the CloudNativeDay.ch. And again, obviously, this can change, so that’s like another thing. But right now, you’re just complying with local regulations, masks are not required, they are voluntary, everyone can wear them, or anyone can wear them… But that’s where we stand on this. Okay.

Yeah, absolutely. And we will only add a mask mandate, but we won’t remove it, like it happened at KubeCon, where they planned to remove it and then re-added it because of the feedback. We wanted to be an inclusive event, and it’s a pretty hard job. I think this is one of the toughest decisions at the moment as an organizer, whether you’re requiring masks or not.

And maybe an answer to your second question - of course there will be coffee. I mean, no conference without coffee.

The reason why I asked that is because these two are linked. It may not seem that they’re linked, having masks and having coffee, but if you are forcing masks, and the catering staff says “No, I’m not going to wear a mask”, what do you do? What do you do if people let you know “We’ll be there”, like the baristas, or whatever the case may be… The people that provide the coffee, the food, all that. If they don’t want to wear masks, but they have to, what do you do then? Anyways, we will come back to this. I don’t think there will be baristas at this conference, because this is a small one, as you mentioned… Community-run, enthusiasts, people doing it in their free time, which I’m a big fan of. I really understand the power of that. People doing it because they believe in it, not because they’re paid to do it… So I really like that story and I really like that angle.

Also, I like the engineers angle, because this is a conference from engineers, for engineers, because it’s run by engineers.

Yeah.

So can you tell us a bit more about that? Who are the organizers? I mean, it’s you, but you’re like – okay, we already established, you’re not a professional cyclist. CNCF ambassador, conference organizer, but also engineer. So tell us a bit more about that, and about your colleagues that you’re organizing this conference with.

Yeah, I’m really enthusiastic about engineering. So if I would have to describe my professional life in one word, it would be like engineering. It doesn’t even matter it’s computer science and not civil engineering; it’s just engineering. I love this stuff, and I’m doing this – I mean, I got a degree in computer science.

And the other colleagues are also from different companies around Bern. So when we started the meetup group, it was actually Phillip and I. Phillip is working at an insurance company in Switzerland, and he’s also the guy who co-founded this meetup. He’s like more from the infrastructure, I’m more like from the software engineering background. So I developed applications for quite some time, and I’m still developing, but it’s more like infrastructure right now… We also went into this DevOps movement, but we thought “Yeah, cloud-native is a bit more catchier for engineers”, because DevOps is also more about process, and how to work together, and methodology…

Engineers like to talk about technology, to be honest… So the group of organizers grew, and we thought “Well, we need representants from the major companies in Switzerland”, so we reached out to SBB, which is this federal transportation system, or Swisscom, which is one of the major telco providers in Switzerland… So we formed a group that major companies are being on the committee.

So you mentioned that you’re an engineer at heart… What does that mean to you? What does it mean to be an engineer?

I mean, I love problem-solving. It’s like solving puzzles all the time. Once you get into the zone – it’s pretty difficult to get into the zone, but the experience of getting into the zone, and then time just goes by… I love it.

What was the last problem when you felt this way?

Yesterday.

Okay. What did you do? Tell us about it.

So currently I’m working on supply chain security, and I dug into Sigstore, which is an amazing project, by the way. And I’m doing Go programming at the moment, and I had some problem to solve with my Go library… So I wrote some extension so that we can promote container images from one registry to the other. Pretty much the same way as Alfonso told us at KubeCon about promoting images for Kubernetes releases, but it’s like for a company I’m working for.

The difficulty is there are quite some libraries, like Go content registry, and then there’s [unintelligible 00:27:42.08] Sigstore, and they are kind of difficult to handle. So you have to dig your way through a solution. Everything was working, I started – I mean, I tried to do test-driven development, which is not that easy if you really try to practice it… And I wrote my unit tests, everything was okay, they were green, I was happy, I did some refactoring, it was still green, and then I tested it in production and boom, it failed because I didn’t consider the authentication of the registry, so I’m not done with it yet.

[28:21] Okay. I know what you mean. When things come together you keep digging and think “Oh, I have it. Oh dang it, no! I forgot about this one thing.” And then the deeper you go, the more you start appreciating how all those elements come together. And when you have that moment, when everything’s clear, you see the whole picture; it’s almost like seeing the matrix. You see how things combine, you have that mental image of how everything interacts, you have an image of all the potential failures that can happen, and you don’t have to obviously cater for every single one of them, but let it fail, let it crash, whatever the case may be… Be very deliberate about what you will handle, and what you won’t handle… And again, the happy path, keep focusing on that, while making the failures very clear when they happen again.

So the engineering - how do you build good systems? How do you build something that’s reliable, something that will keep doing what it’s supposed to do, in the simplest way possible, forever? I really like that moment, I really, really do.

Yeah, I think it’s an art. I once called myself also like a software artist; I don’t think I’m the best programmer, I also don’t think I’m the best engineer, but still, I love it.

Yeah. So there’s a small correction which I have to make… [laughter] When you say “Alfonso”, I know exactly who you, because you mean always Adolfo. [laughs]

Oh, I’m so sorry.

No, that’s good, because in your brain I know your connection that you’ve made… And I keep telling you, “Do you mean Adolfo?” “Ah, yes, yes, I do mean Adolfo.”

Sorry. Yeah, I mean Adolfo, of course, yeah. I consistently get that wrong, and I don’t do it on purpose, so sorry to Adolfo.

Sorry to Puerco, because he also goes by the name of Puerco… So Adolfo García Veytia, the full name, episode 53, where he’s talking about securing K8s releases. Adolfo, we really enjoyed, both of us, talking to you at the Chainguard booth. Priya was there, and I think a couple others were there, but I remember the four of us talking and that was a great conversation. And Johann says “Alfonso”, he means Adolfo, okay? Just so that you know. He has a very special name for you, and I really like that. No one else will call you Alfonso except Johann. That’s a great one.

So really, my apologies, and… Adolfo is actually – now I got it right, okay?

Yes, that’s it. [laughs]

He also loves cycling, so he does it there in Canada, and in the United States, and he’s enthusiastic about that one as well, and he knows what it means to do multi-journey trips.

Kindred spirits. Kindred spirits right there. Okay, so now he needs to find a name for you that only he calls you that… [laughter] And you’re even. I think that’s what needs to happen. Okay. So you mentioned Sigstore… How do you use Sigstore in your day-to-day?

So I would say – or maybe it was a tweet from Chris, when he tweeted 2022 is like the year of the SBOM. And it’s all about supply chain security; we had these attacks… And quite some companies emerged, and they are trying to tackle this problem. One thing I especially like is specifications, and there’s like the Salsa specification, which is an amazing point to start to harden your pipelines, essentially. So it gives you a framework, it gives you levels, it gives you instructions, what to do… And if you start doing that, you will soon find out that Sigstore is the underlying technology which enables all those recommendations.

[32:24] Yeah. And how do you use it day-to-day? How do you use Sigstore and SBOMs in your day-to-day? Where do they fit in?

So we started to harden the pipelines. And a colleague of mine and me tackled this problem, and we thought about “Well, we’re already building containers… Let’s create those attestations, those manifests, sign them, and put them into a content registry as well”, and that’s what we did.

So we didn’t do this only with provenance from Salsa, but also for the SBOMs. If you have a look at vulnerability management, what you’re currently doing is you’re scanning your images, for instance, and you check it with a vulnerability database in one step. And what’s going on right now is this step is becoming two steps. First you create the SBOM, with all the components of your image, and then in a second phase you are scanning or checking against vulnerability databases. And this is a way more powerful possibility, because it gives you more options.

You can check for instance just the SBOM and check what’s being deployed in production; you just have to know which artifact is being deployed in production, you don’t have to scan it at runtime and then get the SBOM and check it against a vulnerability database.

Okay. So when you say “integrating with pipelines”, hardening your pipelines, what is the pipeline for you, and what are you running through this pipeline?

Yeah, pipelines, it’s one of my favorite topics. I do CI/CD for a pretty long time. I remember when the book came out, in 2010, from Jez Humble and Dave Farley, Continuous Deployment, and they established this metaphor of the deployment pipeline. I did it before 2010, but they gave it a name and some instructions how to do it, and I was pretty amazed.

So what is a deployment pipeline? Jez Humble says it’s – I don’t know if I can quote him correctly, but you have some idea, or… Yeah, it’s the ability to get essentially changes no matter which type of change into production in a safe manner.

Okay.

And you do it with stages, or steps. So the pipeline gives you feedback; early on it gives you quick feedback, and afterwards it gives you slower feedback… Nowadays, I think deployment pipelines are pretty much commodity, even though people still confuse continuous deployment with continuous delivery, for instance. They don’t know the difference.

What is the difference? Let’s go for that.

Yeah, continuous deployment is if you go into production in a fully-automated way. So your pipeline is fully automated you don’t have [unintelligible 00:35:25.04] deployments in between. Continuous delivery has those safeguards where people can say when to go into production. It’s also automated almost all the way down, but deployment to production, for instance, can depend on some people deciding to go into production and push that button.

Yeah. Okay. Which one do you mostly use, and why, out of the two? Do you use them both, or…?

Yeah, 5 to 10 years ago it was more like continuous delivery, because – yeah, change, and bringing that change into enterprises… And now I’m striving for continuous deployment, because I think it’s one of the best ways to ship software.

[36:14] Yeah, pretty much. And do you go to staging environments or other environments before you go to production, or do you go straight into production?

No, it’s with – I mean, I’m talking now about enterprise customers I’m working for, and it always has staging environments. It’s a requirement to have those in between, and to do some quality work in between.

Yeah. Have you ever pushed into production a change in the continuous deployment pipeline that went through staging fine, but then there was a problem in production? Have you had that happen? And why was that?

Yeah, security was one thing… I don’t have an example at hand, but it happened to me. There weren’t enough tests, for instance; that happened, too. So that’s some examples which happened… And then you go into production, or – what I didn’t have, I have to admit, is production load or something like that. It’s more like wrong assumptions or a lack of a certain type of test.

It can fail in production, and you can also test in production. The question is what you’re doing after you fail in production. What the deployment pipeline gives you is the abilities to revert and go back to the previous day that worked, and that’s a powerful thing. So we have to be able to react upon that, but even that one can fail. I think a colleague of mine had it just yesterday. I was working on some infrastructure, and I removed some code from the pipelines, and he told me “Yeah, well, there’s a pipeline failing. I don’t know why. I thought it was because of my change”, and it wasn’t. I mean, he reworded it, and it was still failing. It was actually because of my change; I didn’t notice. So my bad, I did mistake and I immediately rolled it back. It wasn’t a critical component, but still, it shouldn’t happen; or you should monitor it and go back if there’s a problem.

Yeah. Whenever I think of continuous deployment, if there is another environment before production, I think of it as – again, my perspective… A lot of work that needs to happen to get it as close to production as possible, knowing that it can never be production because of the amount of traffic. So production will always have a special amount of traffic, there’s always something special about production. And then I’m trying to think “What can I do to push straight into production?” And then I try to configure everything and approach everything in a way that I know that this will go into production, so then what is the smallest amount of change, how can I ship slices of a feature or slices of whatever I’m doing, whether it’s a fix, so that it’s always out there, it always goes straight into production, and I can start figuring out “Am I going in the right direction?”

Because whenever there’s like another environment, now you have to keep it in sync, there’s data, and we know how difficult that is. So if you need to test something, test it locally, sure, if you need to.

I think having a production which spans more than one system, so that you can basically take one of ten - or one of two, whatever they case may be - down, update it… So this is like the canary deployment - I prefer those approaches, because you’re always checking; whatever you’re doing, you’re checking your assumptions (because that’s what they are) against the real thing. Not something which is pretending to be the real thing. There is a lot of hidden work, in staging or in other types of environments, where if you invest all the time into making sure that you’re changing the Go out into production, the blast radius is as small as possible… That’s my preference, let’s put it that way.

[40:19] It’s not what I think is better, it’s what I prefer; and I’ve always had good success with it. So I know that most enterprises will not go for that. They say “No, no, we can’t do that. We can’t sign off whatever regulation we need to sign off on. We need to have this other environment.”

But then the question is “So how much is that environment costing you?” And I don’t mean to run it, I mean to operate with these two things, or maybe even more things. So what do you think about that, Johann?

I think one has to distinguish between corporates, and startups and small companies, because startups and small companies - they have a system which is not… I mean, it may be large, but then they’re not a startup anymore. I would prefer to do it that way as you told as well, but it’s not viable for corporates, because they are not ready yet to do that, and they don’t want customer impact, or think that that’s a bad thing.

But at the same time, they are accepting maintenance windows over the weekends. So the whole system for a corporate is not working, like e-banking or something like that, for a whole weekend. And if I try to do e-banking exactly this weekend, then I think “Oh no, I would have liked to do my tasks. Now I can’t. I have to postpone it.” That’s some sort of an outage too, so why not having smaller outages, but keeping them minimal?

And I also like culminating or aggregating changes into one big change is a much higher risk than just doing it continuously. I fully agree with you.

But I think that’s the thing - if you have an outage, it’s an incident. So if you have to update something, and as a result of you updating that thing you’re causing an outage, you have an incident. You’re not updating anything, you’re not doing anything good. You’re actually doing something bad. So why is your system designed and running in a way that you pushing a change is creating an outage? That’s the problem.

Absolutely.

Not the fact that you’re pushing out updates. So how do you architect truly resilient systems, that change without them going offline? And I’m thinking about humans. Biology and humans are fascinating, where the cells change every seven years. You never die when you regenerate. So you’re continuing to function, all the body functions don’t get affected, but you’re regenerating continuously and constantly, and you’re always a better version (hopefully) of yourself. Smarter, more experienced, hopefully keeping those workouts, cycling… Whatever you do to keep healthy and energized.

So why shouldn’t a system that is like a well-engineered system - because we’re talking about engineering - what does it mean to engineer systems so well that it will not go offline? What do you need to do so that it doesn’t go offline?

Yeah. When we talk about high availability or stuff like that, we don’t have a second hard. I fully agree with you. We don’t have a failover hard, or two hards running concurrently…

We should, if you ask me… [laughter] I think we should.

So we can inspire ourselves from biology, it’s always a good thing. You mentioned it, application architecture - how do you design a redundant system? How do you architect it? It has some costs, but I think we should do it nevertheless. Architects should be able to design those systems. There are good guidances, like The Twelve-Factor App, which tells you how to design your system, keeping it stateless, and so on. Handling state is always the hard part.

[44:26] Yes, for sure.

Writing stateless systems is one thing, and you can scale them horizontally, that’s not a problem… But handling state is a huge challenge, I would say.

Yeah. But I think even for that… For example if you had read replicas, you can continue operating in a degraded way. For a bank, what would that mean? Well, for a bank - you know what, no. That’s too long. I’m not going to open that can of worms. I’m not going to open it because I know there’s so many things; there are regulations as well, penalties, all sorts of things. And then people say “Okay, we have to do this, we have to do that” and people are just scared. But I see a lot of fear there, and I see a lot of like “This is too hard. Let’s just do what we’ve always done. Maintenance windows - okay, job done.” And then you know someone has to be around during those maintenance windows. Anyways… For another podcast maybe. Not another podcast; for another episode, maybe.

So what projects from the cloud-native landscape do you like most? And it should be the ones that you use day in, day out. I think we’ve already established it is Kubernetes…

Yeah, absolutely. I like Kubernetes, even though there was recently a blog post, I think it was also on Hacker News, why Kubernetes is not recommended for startups, because it has some complexity. And I’ve also heard about Fly.io… So let’s come back to one thing, and it’s containers. Containers are a good thing, and they really – they were a game-changer. Kubernetes is one way to run containers, but there are many other ways, like serverless solutions, then public cloud providers have their platforms, there are PaaS solutions… So in essence, it’s containers.

Yes. We agree on that.

There’s some really nice things about Kubernetes, and they are not that complex. Like, Kubernetes supports deployment strategies, for instance, pretty well. You just have to know how to do it, and then it’s actually pretty easy. I mean, it supports rolling updates out of the box. You can just do it.

And apart from containers and Kubernetes, I’m working a lot with TerraForm. I think it’s also for infrastructure as code provisioning the best solution… Even though TerraForm has some flaws, in my opinion. What I don’t like about it is that you have to manage separate state. If you compare it to GitOps tools, they don’t maintain separate state. You just have the state in Kubernetes, and then you have the desired state in a Git repository. And this makes TerraForm handling a bit more difficult, but I think it’s still the best solution that’s available for infrastructure as code.

When I was using TerraForm I was committing the state file in the repository. And it wasn’t great, but at least I could see the state there. So even before GitOps became as mainstream as it is today, there was something about committing the state of whatever is managing your infrastructure. In that case it was TerraForm.

I think in 2019, something like that, maybe 2018, we used to do this. I remember that. And I was thinking “Hm… I wish this was not as difficult.” But also because you had like two sets of changes most often, you had a lot of unrelated changes when it comes to TerraForm reading the state from the resources it was managing…

[48:07] But the thing about TerraForm which always got me was the plugins. Oh, my goodness me. The amount of pain that I had upgrading plugins, and they stopped working, and I didn’t pin it correctly, and “Oh, now I have to change my config…” That was painful for me. Or the API that the plugin is using has changed, so you have to upgrade the plugin… That was the one thing which was a pain point for me. But tell us more about you, because it is about you.

Yeah, absolutely. I mean, the problem with changing APIs - you’re using some provider, typically for like a public cloud, and the public cloud [unintelligible 00:48:42.06] is changing APIs, and you run into some problems. This can happen pretty often… But it’s not the problem of TerraForm, it’s actually more a problem of the provider of the APIs, and this makes it hard to handle.

So TerraForm… What else is in your toolbox, in your favorite cloud-native toolbox?

Yeah, I used to worked with GitLab and GitHub about as well. One can discuss whether these are cloud-native technologies. I mean, what is cloud-native? There is a definition, but it makes it still hard to grasp…

What is cloud-native to you? Let’s start there, that’s a good one.

It’s nowhere mentioned, but I think Kubernetes or the way how Kubernetes works, and containers, is a major part of it, and it’s generalized. I mean, cloud-native is mostly about application architecture to leverage the possibilities that you have in cloud environments. There’s the cloud-native landscape, which mentions tons of projects… A lot of innovation is going on. First there was Kubernetes, then there was GitOps, and then you have the whole monitoring part, with Prometheus, and Jaeger… I mean, there are so many areas to cover. To me, it’s the modern way to tackle application architecture for modern cloud infrastructures. And open source is a very important thing as well, and also that there’s a neutral ground with the CNCF, and we don’t have to face proprietary technology anymore.

Yeah. Do you use something day-to-day which is not in the cloud-native landscape? Something that is dependable, something that works for you? And I don’t mean like your code editor, or anything like that. Something for your applications, when you run them, for your stateful services… Is there something which is outside of this space?

[54:27] Maybe my development environment. I’m doing Go development, and previously I was like a Java developer and got some proficiency there… But I’m still using the JetBrains IDE, and I think it’s still the best one available. I see so many people using Visual Studio Code, and it’s nice. It’s really nice. But if you know the possibilities and the power of a JetBrains IDE, it’s just a different level.

You still haven’t changed my mind, I’m not giving up my Vim. [laughter] Not now, not ever. I think where we start in our careers - like you mentioned, Java - and if you spend a significant amount of time in a specific community, I think to some extent that determines your development environment… Because you become familiar with something, and you like it, and you just find ways that work for you… And then after 5-10 years later, the chances of you switching are slim, because you already – like, you do things automatically, you don’t even think about it, and if someone asks you, “Hey, how did you do that?” “I don’t know, I just did it. It’s magic. My editor did it for me.” And it’s actually you doing things, maybe hitting some keys, some shortcuts, clicking some buttons, whatever the case may be.

The point is that I think a lot of people that do infrastructure work, they’ll be like Vim, or Vi, or Nano maybe… Emacs of course, you do have to do that… The point being, depending on where you start, you may start going in a certain direction, and then changing that becomes so expensive that you may never have the chance; time, opportunity, whatever the case may be.

So JetBrains IDE - I remember some people were using it. GoLand as well. I think that’s somewhat related.

Yeah, it’s one part of their tool suite, yeah. I actually started with Eclipse, and that was when I finished my studies. And a colleague early on told me “Yeah, well, as a Java developer, you have to switch from Eclipse, which is open source, to a proprietary toolchain.” It didn’t fit into my mental model, so I stuck with Eclipse, but eventually, I changed. And I have to say, I also use Vi or Vim, but it’s more like for if I’m on the terminal and I have to quickly edit some text file. But for development, it’s still GoLand, and so on.

Okay. If we switch gears a little bit now and we go away from our development environments, because I’m sure we can talk about it for at least another hour, and then I can start showing you my Vim plugins, and “Oh, look at this, how it integrates with that in Nvim” and “Ah, there’s like a huge difference…” There’s actually a pretty good episode which I was looking up recently, “Why we love Vim”, and I’m sure there’s a few others on the Changelog. That was #450. But I think IDEs - they do have their place as well, so I’m very accommodating when it comes to them…

I used to be more like in one camp versus the other, but not anymore… Because I did come to appreciate some of the things – like, seeing someone use GoLand, or seeing someone use JetBrains professionally, you realize there’s something there. There’s some craft that happens. We’re being artists with our editors, and it’s a very personal thing. So if you use a palette knife, or a brush, or a chisel, it is what you do. It’s still art at the end of it.

Okay, so switching gears - what are you most excited about in the cloud-native landscape this year? Is there something coming, or something new? I mean, we already mentioned supply chain, we already mentioned SBOMs… Is there something else besides those that you are excited about?

[58:14] Yeah, so besides supply chain tools, I already mentioned Sigstore, which is really amazing work. It still has some rough edges, I have to say, but it’s getting there. And I think Cilium is doing a pretty good job as well. They are starting to solve the service mesh problem from a different perspective, which might be another game changer… Because services meshes - they have been around; there are quite some solutions, like Linkerd, and Istio, and Consul from HashiCorp, but they’re not being widely adopted, in my opinion. I might be wrong, but I think they add another layer of complexity onto Kubernetes, for instance.

Cilium solves this at the kernel level. They’re replacing the IP table stuff that’s going on with eBPF, which is a lot more flexible, and they have a lot of – how do I put it…? New possibilities are opening up to solve problems.

Okay. Well, I’ll make sure to dig into that when I talk to Liz and Thomas in a future episode, because I think there’s something there. I think eBPF grew in very many directions, unexpected ones, and I think service mesh is, as you mentioned - it has always been, from my perspective as well, a controversial space, because it adds a lot of complexity, and very few are able to pull it off successfully… But it is a space worth watching, for sure.

And I really like what you mentioned, because from your perspective, based on the projects that you’re involved with, based on the people that you know, it’s like, you’re experienced, but still, within an area. And I like that regional feedback from what you’re seeing, where you’re at; I really like that perspective. I think those are really valuable. Because based on where we are in the world, we each see things slightly differently, based on our communities, based on the people that we work with or know… So okay, that’s a good one.

What about the CI/CD space? Is there something in the CI/CD space that you’ve been using for maybe a while and it works well? I mean, is it GitLab, because you mentioned it, or is there something else?

Yeah, I don’t work directly with it anymore, but I used to work with it for several customers. It was like ArgoCD, or let’s say GitOps tools, Flux as well, they share the GitOps engine, so… I mean, there’s this GitOps approach, and it’s a pretty amazing tool. And what’s not being solved yet is how to combine GitOps with deployment pipelines. That’s an open question, in my opinion. There’s no solution to that. But what GitOps excels at is they do not only concentrate on delivering or shipping software, it’s also running software. So this part of the software development process, so to speak, is not being covered. I mean, you just deploy, and then the pipeline is over. So what do you do if something fails in production? Do you just redeploy? And GitOps is actually the tool which excels at this problem.

What do you like most about GitOps?

It’s declarative. So I was skeptical about declarativeness… Is that even a word? I don’t know… A couple of years ago, when I tried to understand the design principles of Kubernetes, for instance - and it’s purely declarative; I loved it. And on top of that, it’s a natural step forward. You just tell what you’d like to have, but not how. And the system tries to find out what to do. It’s much more powerful, because you can essentially automate operational tasks with that.

[01:02:10.17] Yeah. Something that continuously converges, as you mentioned, is really powerful. You don’t have to tell it what to do and keep doing this. You just tell it “This is what I want you to do”, and it’ll figure it out for you. You’re right. That mind shift is really important, the imperative versus declarative.

I remember Ansible… Ansible felt very imperative to me. “Run this, then run that, then take…” No, no, no. None of that. Just tell it “This is the state of the world. Just make it so.”

Exactly. I mean, your infrastructure has to support it, and that’s not an easy task. But if the system itself can figure out how to self-heal itself, you get more automation and it’s much more powerful and flexible, I would say.

Yeah.

I also like about GitOps that the desired state is maintained in a Git repository, hence the name… And it gives you full traceability. You don’t do manual actions, everything is automated; I like that one as well about GitOps.

We set up a system - it was for a finance institution - which had to be PCI-compliant, and they couldn’t do deployment pipelines because it would push it into production, and this was prohibited… So GitOps actually solved that problem with this firewall in between, where you just deploy indirectly by committing into a config repository.

Yeah, the pull versus push model - I think that’s also a really important distinction… Where your CI/CD pipeline doesn’t deploy anything, it just makes a commit and there’s something that watches what is the desired state, and then it applies that desired state. But you don’t reach out into production and tell it “Run this thing” directly. It’s the indirect, “This is the state that I want”, it sees the state, it updates it, and then it applies it. I think that’s one of my favorite features, where you don’t push changes, you basically pull them, and it’s a very nice way of scaling things. So when you add another system, you don’t have “Oh, I have to update my pipeline to basically start pushing into this other target.” The target just knows, you just configure it “Hey, just keep watching this, and when there’s a change, pick it up and apply it.”

Absolutely, yeah. I think that’s a pretty powerful concept.

Are there any projects that you can talk about or want to talk about? Projects that you’re involved with, projects that are exciting, projects that are going well, or the ones which you wish would go better?

So how does my professional look like? I’m not being paid as an open source contributor by some big tech, so I have my own company and I’m basically consulting customers in, for instance, adopting a cloud-native journey. So do you mean that type of projects, like custmers I’m working with?

Yes. Without mentioning any names, obviously. Things that are going well… And you don’t even have to mention the industry.

Yeah, I’m mainly working in the finance industries, and also public sector. I can tell that. And what I’m doing right now - I already mentioned it, for one company, which is in the finance sector. They have a public cloud strategy; they want to move everything into the cloud, from on-premises solutions. They’re’ facing a couple of challenges, of course, and they also have huge security requirements. And one thing is, as I already mentioned, the whole Salsa thing, where we try to rethink the whole supply chain security part. So that’s one of the major projects I’m working on right now.

[01:06:13.08] There’s also quite some TerraForm stuff involved in the public sector. So let’s say - I specialized in consulting customers that are trying to adopt public clouds, or cloud-native concepts. It’s more in the public cloud sector right now - how do you get into the public cloud safely? So I’m working in the public sector there as well, with like “How do we automate everything, like provisioning, cloud accounts, and so on?” What controls do we add? And there’s a plethora of solutions that you can choose from. Policy-as-code, for instance, is such a thing, which is really great. You can use OPA, you have different [unintelligible 01:06:56.09] languages, like Rego, or CUE… We haven’t leveraged everything of that yet, but we are starting to dig into those solutions, and they are pretty powerful. OPA, for instance - it helped us to add guardrails to our infrastructure pipeline, so that we don’t delete resources, for instance.

Okay. Yeah, I haven’t used OPA, but it keeps coming up. How are you finding Rego? Because many people complain about just how difficult it is to use it. When I looked at it, it just looks a bit funky… But I haven’t used it myself; it was just like from afar.

So it wasn’t me personally who wrote the Rego stuff, but the guys who did it actually said “Yeah, it’s pretty hard until it’s up and running… Or if you can copy it from somewhere else.” But once it is in place, it’s really nice.

So you would describe it as – or what I’m hearing is it has a steep learning curve, but once you go past it, it’s fine. It’s like any other thing - once you learn it, it’s easy.

Yeah. What we haven’t solved yet is like [unintelligible 01:08:09.00] using CUE or Rego to check those manifests, and we haven’t decided yet which way we want to go down. I remember you mentioned CUE, so I wonder which solution we will take for this specific problem. But at least we can choose from several solutions.

There is a recent talk that I’ve seen from FOSDEM 2022… Marcel and Paul are giving it, about CUE. I forget the exact title. But if you look it up, “FOSDEM 2022 CUE”, you’ll see how they – I really like the Kubernetes example which they give, with the schema, and how they enforce certain constraints to what gets deployed in Kubernetes and how they can add certain defaults as well… I thought that was interesting. So that’s the one which I would recommend watching next.

Okay.

As we prepare to wrap this up, what is a key takeaway for our listeners?

Sustainability, I would say. Think about sustainability. I’m not a missionary, I don’t want to preach… As you said.

You’re just doing it. I really like that, yeah.

Just do it.

If I wasn’t as excited, I don’t think we would have told this story. But when I heard it, I was like, “Wow, there’s something there. I have to tell that story.” It was okay… You didn’t think it was a big deal, and I loved that. You were like, “It’s no big deal… It took me eight days, so it’s okay. Yeah, I cycled like a thousand plus kilometeres…” Actually, 1,230. I did the maths; 1,230 kilometers.

[laughs] And apart from sustainability - yeah, if you’re an engineer, you’re a puzzle solver… I mean, this is the greatest puzzle we have to solve, like “How do we tackle global warming?”

[01:10:00.21] Now, that is a problem worth solving, for sure. I hope that this point - we’re not still thinking that it’s not real, I hope. Some will, but that number is less and less by the day. So if we can all agree that it’s a problem worth solving, I’m very curious to hear others how they solve it… Because I know that we have such a big role to play on the infrastructure side, on the apps side, we can make them more efficient, we can autoscale them, or maybe only use as much as we need… We can do so many things there. We can use – maybe be more conscious about the languages that we choose. Some are more energy-efficient than the others.

There was a great talk, and a keynote - the KubeCon EU keynote from Intel, where they ranked the various languages, and they explained the Carbon footprint that they have. Now, I’m not suggesting everyone switch to Rust or C, because they are the most energy-efficient ones… Go is fine, Java too. Surprisingly, Java too; very, very efficient. But there’s something to be said about those, and there’s something to be said about the small choices that we make everyday… Including that burger, which I may not have today, after our talk. I know that’s like the worst food you can possibly have… To maybe cycling more, versus using the car.

Well, thank you for today. It was a great pleasure to tell this story, to re-live a bit of that fun. We did cycle when we were in Valencia, all the way from our hotels, our respective hotels, where we were staying, all the way to the beach. That was a long one. I think it was about like an hour.

It was fun.

It was fun, that’s what I wanted to say. Especially at night. That was the best. Those scooters - they were so fast, whizzing past us at night, and like ding-ding! The electric scooters - they were maniacs.

Yeah, absolutely.

We were trying to have a conversation, which by the way - maybe not the best idea at 12 o’clock at night, as you cycle, on a cycle lane… They weren’t that wide, but still, it was a lot of fun… And thank you for coming on today, thank you for the great conversations at KubeCon, and I’m very much looking forward to meeting you again at Swiss Cloud-Native Day, 14th of September. Everyone welcome, by the way. Anyone listening to this, if you’re around, if you’re local to Bern, France, Italy, Germany, close enough… You don’t have to cycle, by the way; it’s not a requirement. You can take the train, or the plane. That’s okay, too. I’ll be flying, by the way. I hope to see you there.

Thank you, Johann. Thank you for today.

Yeah, thank you very much for having me. And yeah, Cloud-Native Day in Switzerland - if you want to check out Tim Hockin, for instance, one of the fathers of Kubernetes… We have Thomas from Isovalent talking about Cilium service mesh, Priya from Chainguard, which I’m pretty much excited about… We have also a Rust guy, Tim McNamara, who’s doing a Rust programming workshop… So that will be, I hope, amazing.

Yes. Thank you very much for that. ArgoCD workshop - that’s the one which I’m thinking, like “Oh, that sounds very interesting.” Will they be in German, or will they be in English, the workshops?

They will be in English, I think. It’s left up to the workshop speakers, but all of them are capable of doing those in English. They will be in English, yeah.

Okay, excellent. Well, I’m looking forward to meeting y’all there. See you in a few weeks, a month, or two, based on when this comes out. See y’all, see you, Johann.

Yeah, pretty excited about it. Thank you. Bye!