Ship It! – Episode #74
Vorsprung durch Technik
with Sebastian Kister, Team Lead, Kubernetes Competence Center, Audi 🚙
I don’t think that you can imagine just how excited Gerhard was to find out that Audi, his favourite car company, has a Kubernetes competence centre. We have Sebastian Kister joining us today to tell us why people, followed by tech make the process.
The right thing to focus on is the genuine smiles that people give in response to something we do or say. That is an important SLI & SLO for reducing friction between silos.
How does this impact the flow of artefacts into production systems that design & build cars?
Raygun – Never miss another mission-critical issue again — Raygun Alerting is now available for Crash Reporting and Real User Monitoring, to make sure you are quickly notified of the errors, crashes, and front-end performance issues that matter most to you and your business. Set thresholds for your alert based on an increase in error count, a spike in load time, or new issues introduced in the latest deployment. Start your free 14-day trial at Raygun.com
Sourcegraph – Transform your code into a queryable database to create customizable visual dashboards in seconds. Sourcegraph recently launched Code Insights — now you can track what really matters to you and your team in your codebase. See how other teams are using this awesome feature at about.sourcegraph.com/code-insights
FireHydrant – The reliability platform for every developer. Incidents impact everyone, not just SREs. FireHydrant gives teams the tools to maintain service catalogs, respond to incidents, communicate through status pages, and learn with retrospectives. Small teams up to 10 people can get started for free with all FireHydrant features included. No credit card required to sign up. Learn more at firehydrant.com/
Notes & Links
- 🎬 Container Days - September 2022: A culture beyond SLA and T&C - Sebastian Kister, Audi
- 📄 How Audi provides developer self-service - April 2022
- 🎬 NGINX Sprint 2.0 - Nov. 2021: Audi Harnesses the Full Potential of Kubernetes with NGINX App Protect
- Celebrating 50 years of “Vorsprung durch Technik”
- Audi Digitalization
- 🚙 Audi Films: An Avant Story
|4||07:18||What is it like to work for Audi|
|5||10:53||How the transformation happens|
|6||14:50||When projects onboard|
|8||21:04||What about the documentation|
|9||25:09||Learning from the engagements|
|11||29:54||What runs here|
|12||34:40||Changing how people approach problems|
|14||41:33||How large is your team?|
|15||44:01||Importance of small teams|
|17||48:51||Working in this ecosystem|
|18||56:00||Getting to the end users|
|19||59:16||The stuff you wish you knew back then|
|21||1:04:40||Where will we see you next?|
Click here to listen along while you enjoy the transcript. 🎧
I want to start by thanking Taylor for kicking this one off. Today I’m able to bring two of my great passions together - cars and technology. And it’s not just any car, it’s my favorite car, Audi, and my favorite type of technology, cloud-native. And I cannot think of a better partner for this episode than Sebastian Kister from Audi. Specifically, the product team lead for the Kubernetes Competence Center. Sebastian, welcome to Ship It.
Thank you so much. Thanks for having me. Pleasure to be here.
So why Audi?
You mean why I chose to work for Audi, or why we joined?
You can answer it in whichever way you want, the first thing that comes to your mind, and then I can ask a more specific question. But why Audi for you?
Well, I’ve been a decade in startups, ranging from privacy by design data infrastructure startup, to IP television startup, which we brought from the first customer to 1.25 million customers in just two years. And after that, there were many possibilities to join different companies, but I wanted to go to where I came from. Actually, I’m born in Ingolstadt, and I wanted to get away from Munich… In Ingolstadt you have two companies, and the most famous one is probably Audi. There are a couple of more companies, small companies, but…
What’s the other one, by the way? Because that’s the only one I know.
Yeah, the other one is Media-Saturn which is a retail, like electronic retail in Germany. But retail business is not really an end product I could identify with at all. And the Audi design language, this progressive design, the E-Tron design, or in general the Tron design from the movies in the ‘80s and today, this culture of foreseeing society in 20 years, and imagining it, and living from a design language from the future is something I totally loved. I love the backlights, the design curves… I think we have met in a different meeting before, where I had this E-Tron GT backlights as a background. I just fully identified with the design language from Audi… So this was a great fit for us. And coming from startups, Audi obviously wanted the expertise in an enterprise, and create startups in an enterprise, and I said, “Yeah, I think we have a good match here. Let’s do it.”
[06:10] So I haven’t mentioned this to many people, but now I think it’s a reveal moment. I know you like telling secrets too, or making things sound as if they’re secret… But for me, it’s the way the doors close. It’s a really weird thing to say, but I think when you close an Audi door, it’s just so different from anything else. There’s like so much attention to detail in how the doors close… Can you imagine how all the other things are? And that’s what got me. I really liked the way the door shut, and then from there, it was discovery process.
In all honesty, my wife is a big Audi fan, and she said, “We’ll have a single car, it will be an Audi, and it will be an estate. And you can choose whichever one you want.” So that’s how we went through an A4, and A6, an RS6, always an estate, and I’m so happy. But there’s something in the design, there’s something in the detail that Audi does, which I haven’t seen any other manufacturer do. And that’s the why for me.
Back to you, because this is about you, let’s be honest… What is it like to work in a big company, in a big enterprise? It’s over 600,000 employees, it’s huge. What is it like?
Yeah, the entire VW Group, with direct and indirect employees, is over 600,000. Audi, specifically the Audi Group, is 96,000, something like that.
And only Ingolstadt, our main factory, and the business central, has 44,000 people. So yeah, it is big.
Still a big place.
And I’ll tell you the biggest advantage of it - it’s awesome food. You get the right food over there. It’s really good.
The campus! [laughs]
I totally enjoy it over there. And apart from the great food, obviously, because I’m not going very, very often to the office… But when I do, I stop by the canteen, or one of the many canteens; there are really many canteens. And they offer everything from very simple food to very sophisticated foods. Obviously, you have to choose your canteen, that fits your desire… But there is something for everyone.
And working in a big company also gives you so many possibilities… And I tend to focus always on the positive things, I create something out of every free space that I find. I create something out of every opportunity that I identify. That’s just my – that’s me. That’s what I do. And that’s why being in startups was probably very beneficial, to work on that and elaborate that. But in Audi I had a team surrounding me that were fostering this philosophy, and I had complete backing to do the sort of startup culture in my teams as well, which is not everywhere the case.
So I really must say that we started as an exception, because we wanted to change the culture. And obviously, you can’t change culture top-down by saying “Now we think different.” It won’t work. So creating germ cells in a big enterprise, and protecting them at all costs until they can deliver and supply product, create value-add, create business, but as well have like a gravitational effect, a cultural gravitational effect to draw other people into that sort of working, sort of thinking… And I say it is a critical mass now.
[10:12] We can absorb culturally projects when they onboard on our platform. So we are one end of a non-existing end-to-end responsibility in a silo-structured enterprise, but we can transfer our culture all the way to the other end… And it’s appreciated a lot. I’ve found a lot of appreciation along the way for this sort of cultural change, because it is a delivery process, and it is a business; it is not like idealism, and unbearable idealism with the current situation.
So when a project onboards onto your platform, that’s how the transformation happens, is my understanding.
You could say so actually, because you need touch points to do so, and you need to make people understand the value that you bring to the table by our way of working and thinking, and adjusting, and putting people in the center of everything that we do, actually. So the focus is really on the people. People first. When you think transformation in enterprises, what often happens is they processualize something, think process first, and impose something top-down. And - well, what happens is you change your address fields, you change your departments, you change maybe some managers, but on an operation level nothing changes, usually. Maybe sometimes when you say “We have now agile methods. We need to make an agile product creation process.” But it’s, again, a process. And even if you say “We’re working with agile methods now”, often they are not like philosophically applied, they’re just followed, processualized. And that is just not the transformation; that’s not a real transformation at all.
If you put the people first, and then look at the tech that they want to explore - you have passionate people all over the place. You ask them, “What do you want to do? What do you want to explore? Think of a product, from scratch, whatever you want to do”, and then you create out of what they want to do business value, because then some guy like me, who creates startups, or business cases for epics and whatnot, and brings back the passion of the people and makes a business out of i
When you have passionate people exploring tech, what happens naturally is a process that follows. So you have people, tech, and then process, because passionate people use great tech, and once you automate the great tech, the automation is the process. It’s a process prone to human error, actually, even. And that’s how we onboard projects, to come back to the topic. We onboard projects by enabling them in the bleeding edge technology, actually. Enterprises never use bleeding edge technology. They are afraid of it; early adopters - not with us. But you can’t be progressive, you can’t be… Audi. You can’t be Audi if you follow an enterprise paradigm; you need to be progressive in a very fast-paced environment, and if you want to say that being progressive is an attitude, or progress is an attitude – it’s an Audi hashtag, actually. If you want that and live up to that, you need to be bleeding edge, at least in the bottom layers. In the end, in the car, fair enough. I mean, there’s so much going on until a car is being built… But in the digital service portfolio… And we want to bring this customer excitement, from the closing door sound, to the digital service portfolio as well. The engineering is unrivaled. It’s amazing. And the love of detail our engineers have is very annoying for their managers, actually.
[14:39] I can imagine. “No, it doesn’t close properly enough. No. Let’s do test 1,125,000.”
Okay. So when a project onboards, and it comes into the Kubernetes Competence Center, how does that work? Someone has an idea, someone is passionate about it; you’re on the cutting edge of technology, and you say “We’ll make this work”, and you’ll have a great time doing it. It will put a smile on your face, at least one, I can guarantee you that; there will be more. How does that work? Can you run us through that?
There are many, many dimensions to it, actually. It’s not like that’s a one-dimensional thing. But let me go through it from from the beginning. First, when we onboard someone - and I say just standard; standard workload, easy thing, nothing bleeding edge. Let’s separate the onboarding process from being bleeding edge. And if you have an onboarding and you need to show someone how to create a container, for example, because it’s the first time they worked with Kubernetes; for them, that is bleeding edge already, probably. And then you have also something that we call shift left, that you enable projects in a sort of shared responsibility manner, to take care of things before they happen in the cluster. For example, container image scanning, container runtime scanning - you can use a couple of tools locally on your machine already in the build process, in the whole supply chain, actually, before you throw it at the Kubernetes API… And we say no, because in our cluster, we have enforcers deployed, with policies, and sometimes your container image wouldn’t check out. And that causes friction; that kills your time to market. And it’s not very good for continuous integration at all. You can’t deploy like three times a day if you’re not sure your image will check out.
So if you have all this shift left toolchain and the stuff that makes you – you have to be able to use it. That’s how we enable the project. So once they’re able to use it, they have the time to market in their own hands, and they don’t bother our developers and our operations engineers, our SRE teams. They don’t get tickets, because they have it in their own head, and know if it will check out, because they can enforce the policies or check if the enforce policies on their image will work or not, before they push it. That way, you reduce friction, communicational overheads, you have a high level of automation, you can deploy faster, more often and in smaller increments, and you can react faster on customer expectation, actually. That is the core of today’s development or digital service portfolio creation in general. If you have a digital product, and customer expectation is a fast-changing phenomenon, you need to react fast. That’s a core competence, and that’s how we do it, actually. And if there’s something bleeding edge, like nobody knows how to do it, it’s just perfect with us, because we love to explore it with him. We take his hand, we never let his hand go until he’s successful in the cluster, happy in a runtime, and we face the issues together and learn from it together.
So that focus on good, smooth onboarding is important, right? That’s how everything starts. Is there a documentation that developers – I’m trying to figure out how much of this is self-service, versus “Let’s work on this together”?
There is a lot of documentation, especially because you have this [unintelligible 00:21:34.03] necessities to document things… But let’s be honest, your project won’t read that. It’s like buying Lego and then really, you read the manual; or buying a new phone and you first read the manual. Nobody does that. And projects are people too, and they won’t read your manual, even if you have one. So make it a people business.
Obviously, the first argument you hear now, “Okay, Sebastian. Nice one. But can’t scale. This guy can’t scale”, right? I can prove that wrong, because it scales way, way better than self-service, and anything like automation and anonymous pages can do, because you create shared competencies with every onboarding. When we onboard a new team, that’s people. And they learn from us during that onboarding. If it’s a self-service, they dig through stuff and they don’t learn anything. Basically, they just make things happen. And maybe they find some errors, but they won’t report it, as long as it works.
But during our onboarding, it’s like a professional services layer; we get in touch with the people, we deploy the first workload together. That’s what I mean we take their hand, and we lead them to success, whether they want it or not. And you’d be surprised how much are not too interested in having success. But once they had it, and have the feeling again, they start to burn for something, they become passionate about something. And that’s also this human component, if you put that first; this real, real human component, and with bad days, good days… And you have a positive attitude in the team, and you ignite that spark to them… And they love working with us, they really enjoy it, and they learn from us. And the second, the third, and the fourth project works like a self-service afterwards, because they know what to do already; they’re completely enabled with everything. And it’s way faster than him reading through a manual, is us showing him how it’s done properly… And every every project is different. We have so many projects, and we kind of didn’t identify a single project that is like 100% the same as the other one.
[24:14] Sometimes they have a different database, they want some different cloud-native service connected with their cluster, or with their namespace, or with their container, whatever. Everybody has some slightly different stuff going on. And if you have a self-service, you have a different problem. You have projects putting requirements on your platform, and you have an ever-growing backlog that will rip you apart in two years.
So if you individually solve all these problems 20 times, with 20 projects, then you have a community, a community of happy people, successfully working with your platform. And that is scaling way, way better than self-services and ticket bonanza because they help each other, you know?
Yeah. How much do the developers that work within the Competence Center, the ones that maintain the platform, learn from those engagements when they work with someone who’s trying to onboard something?
That depends… Obviously, we have developers with a very deep insight into app development as well, to really understand the other side. And this great mix of our principal operations engineer and our principal platform engineer, cloud engineer, makes the perfect combination, actually, to have a guy really automating everything on the infrastructure, and the other one saying “Dude, I need to keep that alive. Let’s slow down a little.” And these guys, in combination, they obviously learn a lot about the requirements projects have… And once we identify an 80/20 case, or say, “Okay, 80% of the after projects need this and that”, then we create a managed service on the platform level. But it needs to be 80/20 first. When it’s below 80% of the projects demanding something, we just don’t do it, because you can do that yourself. We are very open. Our platform focuses on just two things: security and runtime. And when you focus on security, you can – I mean, I can’t reroute some cloud-native services; it just doesn’t make sense, because it will be more expensive than from the cloud provider directly, and it will lose a couple of nines behind the comma. So I cannot create a better service, and it won’t be cheaper. So I want to it. It’s not a business case.
I see. So why did you pick Kubernetes?
I didn’t pick it.
The Competence Center… Why is it called the Kubernetes Competence Center? Why is it in the name? I think that’s important.
Yes, I can explain the history. I joined a team of two passionate Kubernetes engineers… And that’s it. That was it; it was just us three. And they created something for the E-Tron charging cables and charging function in 2017, and they used Kubernetes for it. And when I joined Audi, they already created an Audi proprietary Kubernetes control plane, focusing on deploying into EC2 directly, or then when EKS came out, on the first abstraction layer on EKS.
So from there on, we went and we focused on Kubernetes… And that’s what I told you before - I was not saying what we have to do, I was asking them what they wanted to do, and they want to do to Kubernetes. And I was like, “Okay, then let’s be the best. I don’t care what we do, but when we do something, let’s be the best.”
[28:15] And from there on, we grew, we delivered, we had more projects, we had platforms, we created bigger platforms… At a certain point, we consulted other platforms, we delivered infrastructure as code to other platforms, we have a cross-platform delivery center for middleware, for example container image scanning, container runtime scanning… And we supply consulting to other platforms… And that was the point where our product was not the platform itself. It was being there for all the platforms. And there was the point where it was a Competence Center. And now we consult platforms that will be created, and we allocate platforms in the right layer… Because some business platforms focus too much on the infrastructure layer that they can actually abstract. They don’t need an infrastructure; they can consume like 12 clusters from us, with all the heritage that we give them. It’s very beneficial if they don’t do their own infrastructure in this case, and focus on their business line and say, “Okay, we have a platform focusing on security and on runtime. Yeah.” So if you have a business platform and you want some machine learning, or data analytics, or whatever, you have a couple of managed services that your projects want to focus on. Also, you have a different set of burst workloads. You probably want a consumption-based chargeback model, and not like 80% capacity and hands-off.
So in terms of workloads, what runs on these platforms? Because it sounds it’s like more than one. There must be a lot of workloads that run. Are they stateful? Are they mostly applications? What do you run?
Well, the company is quite big, you know? There’s literally everything. There’s literally everything.
Just how big? That’s what I’m trying to – like, is it thousands of applications? Is it hundreds of stateful instances of databases, or caches, or whatever you run?
In total probably yeah. In total, it’s, I think, 6,600 projects, or something like that. You don’t know if they’re all active. So there’s maybe many, many projects that are not even active. And in terms of our platform, it’s one of the smallest platforms in the VW Group, and we want to keep it that way, actually, because we want to have a big – how do you say that… POC case for new technologies as well. So we want to be early adopters in technologies, and if we have 6,000 projects running on our 200 clusters or something like that, that won’t work anymore. So we can’t be the Competence Center and have the biggest platform. We need the smallest platform, and that way we can really be bleeding edge and create POCs, with new technologies, with new vendors, try out things… And then we can consult new platforms and old platforms in adapting new technologies. And this adaption product is actually a new product. It’s a cross-platform delivery center. So something that we do for our own platform in terms of security, often in other platforms is just a license, but zero adaption, so we need to help them in the adaption.
[31:48] We’re spending on our security experts a tremendous amount of money, actually, and other platforms can leverage that. Just imagine if everybody needs to have two or three security [unintelligible 00:31:59.11]you shoud have two security experts on each platform. We have 33 platforms. 66 security experts. We would be the biggest IT security company in Germany, probably. And if you know the market, that’s not possible.
So this kind of thinking is really leveraging – I avoid the term “synergies” since a couple of years, but that’s the way to leverage the benefits of our infrastructure as code, our adaption and our maintenance, our efforts in general that we put into security; the others can leverage that very, very easily.
Okay. When it comes to things that Audi runs, there’s like a lot of stuff. There’s the cars becoming connected, there’s like more and more data coming from the car, data being sent to the car; there’s more cars that are coming online. How much of that do you get to run on your platform?
Like I said, the platform itself, we have many workloads that develop that kind of stuff… But the connected car stuff, the in-car software, that actually runs in the car, in a productive state and customers are using it - that needs to run somewhere else, and on different kinds of systems. You don’t want that – like a comparison with an airplane, you don’t want to ride an airplane that has an overdue upgrade. Maybe. I wouldn’t…
You got the idea, right?
I mean, [unintelligible 00:33:44.03] is a very nice topic, by the way. It’s controversial, it’s not that easy, and obviously, for some parts of the software it’s great, but other parts, you don’t want it. You want an engineer looking at it physically when you bring that to the workshop. And in terms of software, these kinds of upgrades are great, but only for a couple of things, not for all the things.
In terms of connected car or in-car software in general, there are many, many engagements and things going on in the VW Group, on a group level, but also at Audi itself… And I think there will be many workloads developing that stuff, but just a few very dedicated platforms running that stuff.
So when you take those happy, passionate people, and when you take technology that’s secure, that you really understand how it works, and there’s a process that ensues when you combine these two - how does it translate to day to day? We all have incidents, we all get to see them, to deal with them, there’s always zero-day exploits… How does what you do change how people approach problems like those?
Well, first of all, the shared responsibility philosophy in the platform or in the culture itself, with a higher enablement of projects and uses, and workload teams, project teams - it creates a net of people who are actually understanding what just happened. Most of the incidents that we face in infrastructure are led by hysteria, a sort of hysteria… They have no idea what just happened, but it must be bad. “Let’s escalate it!” And many, many things just are not that bad.
[35:53] So in terms of alerts, first of all, just create an alert when you want it to wake you up at four in the night, and you want a reaction time of one minute to solve the issue. So the alerts - they need to be really, really in place and trustworthy. Then if something really doesn’t work anymore, worst case, like production failure; we can’t build cars anymore in the night. Then obviously, that’s a different kind of story. But to talk more abstract about people working with it, the higher, I would say, competence, or the more broad competence of everyone working together, and everyone knowing each other, because the onboarding was not a self-service, creates a family-like, team-like cross-silo family of people that solve something, that don’t blame something to other silos. They have been there together, deploying that, and they will be there together solving it… And that is just a way different approach than throwing tickets around, operations blaming development, backend saying frontend, “Hey, what did you do?” and then the guy in the middle says “I have to manage three different companies, and my own company, who does operations, blames a fourth company that I haven’t even heard of. What’s wrong here?”
So this is really different. On our platform, even all the companies that contribute to the platform - I want them to know each other. I want the SRE from the team that manages the cluster to know the guy and the content from the container image scanning software, and from the layer seven five [unintelligible 00:37:43.20] provider. I want them to know each other, I want the engineers to know each other, and in the worst case, I can bring them together within 10 minutes. And it’s a different kind of story if you have contracts; obviously, you have SLAs… We have contracts, and if something really, really bad happens, obviously we can rely on them. But nobody pulls up a document that says “I’m not responsible for this in my team”, ever. We don’t do that.
How do you scale and maintain that? Because all I see is like lots of people meeting and talking and figuring stuff out… When people work together, like multiple people like this, are they – I can’t imagine them being in meetings constantly. Right? They must be working together. So how does that happen? How do you get people to work together, that maybe are not in the same physical location, on the same problem, that crosses multiple departments? How do you do that?
Actually, I even have to add their calendar is empty. In my team, I take care that they usually don’t have anything else in terms of meetings than 15 minutes in the morning with me. That’s it. Obviously, we have review meetings, we have refinement meetings, we have some “Look at this technology” or some breakout sessions here and there… Then we have onboarding meetings with projects, and stuff… But usually, it has nothing to do with an enterprise [Bleep] that usually takes place, or that consumes 50% of your time.
Then again, we create 15 minutes meetings. We have like a 15-minute on Wednesday with all the OpenShift platform owners, for example. Just one control plane; we have many control planes, but I just mentioned one now. And with all – they face a couple of issues that are similar. Obviously, they use the same control plane, which is the control plane focusing on this one. And in 15 minutes we don’t solve the issues; we organize the people that need to go to different breakout sessions.
[39:56] So I just need to know who knows what, and who wants to work on the problem with the guy that has the problem. And that’s it. I can do that in 15 minutes. And then off you go. And the next problem - okay, here. Who can help? Here and there. Okay, off you go. And then in 15 minutes, we sorted out everything. And just imagine, 15 minutes for the entire VW Group, expert group, on one control plane technology. That usually takes – some people do days. Breakout sessions, workshops… And we just do 15 minutes a week. That’s enough. Consistently, continuously coming together is more important than one workshop every three months, and nothing happens.
So that is how the calendars are empty. And then you have time to really work on stuff, actually; to be efficient. To get into a certain kind of flow that is very rare in enterprises, because all the time you get calls, and whatnot, people talking to you. And I get really, really angry with people that call my developers directly, unless they invite them for lunch, or something. That’s fair. But if it’s a work call, and they stripped them from their flow, that’s where I step in and say, “No, this entire team is being protected by me. You talk to me. Anything you want. And I decide if we need a developer to answer that or not. Or if an intern can do that.” Because most questions, to be honest, my interns can answer that.
How large is your team?
Both internal and external we have 14 people right now, and the team working on the different platform products. But that is not the real scale of it, because there are so many other people contributing to it. This is just a core team, you could say. We have a very, very strong core team of three people, and then we have an operations department that is being created in Audi Hungaria, where we enable and create a cloud capability and cloud competence center. In Audi Hungaria - we’re recruiting there right now, so if you’re from Hungary, you’re near Győr and you’re listening to this, you will join my team. [laughs] Indirectly. And maybe we should promote it a little more over there… I will talk to the Hungarian guys.
And then obviously, we have external people, because hiring at Audi or VW Group is not that easy, actually. The positions are always combined with so many other things that it’s sometimes easier to just go to suppliers and say, “Guys, I need some architects, I need some developers. Give me a price.” And the difference is that they are really working in the core team. It’s not like we treat them any differently. From a processual point of view and from a legal point of view, obviously, we do that, so we are compliant with all the rules about treating external people. They cannot sit – we’re completely remote anyway, so nobody comes to the office, so all these problems, we don’t have them. But like on events, where we meet with our suppliers or stuff, it feels like a family coming together, actually. And that’s the nice thing over there, in this team. And I’m really proud of having a team like this, where no matter from which company you are, you’re in that team, in the Kubernetes Competence Center at Audi, and it feels like family, no matter from which company you actually are, from which team you have contributed into the product, into the deliveries and deliverables. And that is really, really interesting to watch and see.
[44:00] I can see how a small, close-knit team is very important for this working as well as it does. And then they’re really passionate about what they do, they inspire others, and they create this community effect, where others join, they pick up on that, they learn this approach, they see how well it scales, and they take it further. And then it’s almost like you’re inspiring others in this approach. Less about growing your team and becoming really big, because once you have 50-60 people, managing 60 people is very different to managing 14, even up to 20 people, I think.
Yeah, I completely agree. Also, you must ask yourself if that is really useful. So when I look at all the other platforms, their teams are even smaller, but we have more deliverables. We have like a cross-platform delivery center. For security middleware, we have professional services. Obviously, we need to cover for more people. But in general terms, when I take the SRE teams, they are not in that team size, obviously. This is where I want to create value-add, to be honest. Any kind of control plane for me is not a value-add. We have some requirements to it, obviously; it shouldn’t be heavyweight, it should be easy to use, it should be flexible, it should be state of the art… In the best case, there is a Kubernetes upstream from that team, really going to the cloud-native community… So I also decide on the open source side, and if it contributes to cloud-native… And I strongly, strongly go into the direction of upstream technologies, actually. They are more future-proof than proprietary tech. And this is not a belief, this is just a fact that you can see from which technologies survive once they’re bought by a company. It’s more proprietary from a freemium model, or whatever - they survive another two years, and then they’re gone. We have seen that from 2017 to 2019, and from 2019 to today, with the control plane market in general - it’s just a couple of players left.
And the same with all the other things going on in cloud-native… It might be core technologies like Jaeger or other things (Prometheus) that we use constantly. And they’re incorporated by big vendors; they have to use the upstream technologies. And it’s the open source community against the hyperscalers, and the open source community proves to be stronger. That’s really great to see. And from a business perspective, it just gives you the edge if you go with the open source community, and you get an enterprise support by some vendor… Because you need to have that. Don’t forget the SLAs at all. They’re important to be in place, but you need to have a culture beyond the SLA to be very effective.
I’m wondering, what is it like to work with the cloud-native ecosystem, with the CNCF, Linux Foundation, and also the cloud-native community? What is it like? Because you’ve been doing this for a couple of years.
Yeah. I’ve been creating the partnership with the Cloud Native Computing Foundation in 2019… And also, I really learned and liked a lot how the CNCF community works, thinks, and builds, and creates. We also have in Audi something that we call inner source, where you can contribute to certain projects, eliminating bottlenecks, actually… Because the backlog doesn’t reflect your necessity in the project, but you can do it yourself, right? You can solve that issue yourself, contribute it, and we leverage it as a core maintainer for everyone. Something like this. And the same happens in the cloud-native community.
So everybody has a slightly different approach to some things, or some different priorities… But you can create that. You can solve that issue. You can solve your number one priority, which is maybe number ten for the entire community… But you can solve it and you can contribute it; then it’s there, and it’s leveraged. And you profit from leveraging project teams that are concentrating on the other ten priorities, or other nine priorities.
I must admit that working with the community and the technologies has proven to be the best business case out there. And it needs to be a business case. You can’t be sustainable without making it a business; you can’t be idealistic without making it a business. It needs to be a business in the end; it needs to be a business case. Otherwise, change doesn’t work. Nobody changes anything, unless it’s a f***in’ business. And that is something that I really saw happening, actually with cloud-native technology taking off in the enterprises, because it is a business case. And it’s an easy one; it’s a very easy one. You have two develop in your company contributing to that, and the other 4,998 developers - they do something, too. So you can maintain your fork for the next two years with your two guys, until the first breaking change happens, and then you’re done. Your project is done. You can never touch it again, and hope the entire ecosystem it touches will also never be touched again… But you get the idea; that doesn’t fit modern development at all. It doesn’t fit progress at all. So that’s not working.
[51:54] That’s from a technology point of view… I’ve also been in the CTO summit of the CNCF last year; no, this year, actually. It’s 2022 still. It’s so cold, it feels like it’s already next year. [laughter] And we’ve been working together with the CTOs or head of infrastructures from Allianz, Mercedes, Santander, Evori, TikTok, Spotify, Orange… And it was really very nice to see that we have similar issues, enterprise issues… Like, you can separate it into big company bull***t and into real technical requirements. And what is very fascinating is that our approach that I’ve introduced at Audi to put people first, let them use cloud-native, in this case cloud-native tech, and explore it, and automate it, and let the process follow from that… Like, the handover process between friction walls is completely eliminated through cloud-native shift left technology. And the automation is a very natural process. And when you do that in one enterprise, and you can inspire, let it be one CTO per keynote - and I do like 25 keynotes a year - that will actually make the life of all your operations teams, and the company, and the developers, a lot easier. And they will not be called at four in the night for some stupid incident, because they had a container image not checking out through a policy, and whatnot. Or less. Let’s say less. [laughs] And it will make their life easier.
The culture is just so positive, actually, in this way of working with projects, and not very anonymously self-service, clicking something together, and sharing competencies, sharing passions, working in communities… And obviously, it’s just more fun to be in teams like that. And people are happier. They’re more passionate about something, and it needs passionate people to create great products. It just won’t work with people not being passionate about something. Putting up a great shop maybe… You can do that for a while, but you will not do that for very long. And they’re happier. And I tend to think that if I create like 200 happy people with every keynote, in the end, that’s 200 parents that have happier children in the end, because they can be there for them, at home, or whatever, and have more time. They nail their goals faster, which is also paying into a different type of work ethic. Solve issues, then have time, and not trade time for money; like, you have to work eight hours a day, I tried eight hours a day. No, that’s not what I want. I want people to get something done. If the stuff is done, and you’re happy with it, and it’s a great product, then off you go. And if you nail that in two hours now, because you have the tech for it, and the process doesn’t hinder you - off you go. I don’t care. And then you can focus on the next issue, or take a half a day off, because nothing’s burning. And it’s more relaxed… And like I said, and the endquote here is really that people are happier. In this sort of culture, the people are happier, and you don’t have too many levers, you don’t have sick days… They’re just happy, and… Is it work-life integration in the end? I don’t know. It depends on each person. But they’re definitely happier.
[56:00] How does this bubble up all the way to the end user, this approach from the people that are building these technologies, that are putting all the things together, that are working with other people? Let’s not forget, it’s a very social activity, what we do. How does it translate to the end user?
It translates in hard business. And that is the charm of it. Because like I said, you can’t make change happen, cultural change, or people happier without actually creating a business out of that. And one of the strongest KPIs and the easiest to understand is time to market. Like I said before, you have friction in a silo-structured enterprise; you have hand-over protocols to operations. There is no DevSecOps. I mean, you can do separate DevSecOps teams for a dedicated use case, sure. But you can’t do DevSecOps in a Plan, Build, Run structured enterprise with silos. It’s just, they’re isolated, there are hand-over protocols where this SLA goes into that SLA, these terms and conditions end here… Maybe there’s even a gray zone in between two terms and conditions. But you’re still one company, right? You’re still one group. The VW Group is one group, and we don’t tend to think “This is Audi stuff, this is VW stuff, this is whatever.” We just solve things together.
If you go to the very end of this supply chain, to the developer, and you reduce all the friction between through enabled people, and communities that know each other, that talk to each other and that know the technologies that they’re using, and they’re using it for eliminating the friction walls in these hand-over protocols, then you have the advantages of a DecSecOps teams, and also the advantages of a silo. And a big enterprise can’t scale DevSecOps teams. They can only scale in silos, and that is organizationally not possible otherwise.
So it’s not “Fight the silo. Eliminate the silos.” No, you need this; a silo is not a bad thing. It has advantages; that’s why we do it for a hundred years already. This philosophy of creating one team doesn’t necessarily go against creating silos. So you can work your way through all these friction silos, culturally, and with people in tech. So once it’s automated, it’s even a cross-silo process, to deliver your build that you’ve created automatically into the cluster, and all the hand-over protocols - they’re done. All the scans, the container image scans, the ingress, the controls - everything’s done; you can automate it. And out of the automation, you can create a process out of that for the organization.
So you’ve been doing this for some number of years. Is there one or a few things that you wish you knew before?
No, the answer is no, but because of a very philosophical approach to life, in my case. I don’t want to regret or not to have learned something a different way. Because I could only be the subsumption of my experiences that I am today because I didn’t have them before, and everything that happened in life, the good, the bad, the ugly… It was just because I was that person, that moment, that very moment. And I’m very happy being the person that I am today. So no… Obviously, everything would have changed - life, work; everything, literally - if I knew the stuff that I know today at a different point of my life. And that would go in terms with regret, and I don’t regret anything.
[01:00:27.00] I think I would have changed your path, and you enjoy the path that you have been on, and I think anything would have changed that.
Yeah, it wasn’t very enjoyable, to be honest… But I learned very valuable lessons. Probably the very – the hard way, yeah. But it’s okay. I needed that.
Right. So for someone that stuck with us throughout the entire episode, is there a takeaway that you would like them to have from our conversation from today? One thing that they should remember, because it feels important to you, or it is important to you?
Yes. Personally, what I just said - I think it’s important to give everything in your life; one life, one shot, give it all you’ve got… And workwise, many people see walls where there are none. And only because something might be there, just go back and eliminate yourself from the equation. Be selfless. Professionally-wise, be selfless. You don’t exist; just think very abstract. That is the only way we can incorporate change in enterprises. Because it has been a system of headcount and budgets, there are so many political battles originating from egos, from vanities, from fighting for a year or more at the Christmas bonus, or whatever. But it really is something you spend so much time of your day… It needs to make you happy. If it doesn’t, go away.
Okay, so keep looking for your happy, and don’t regret anything, because everything is a lesson that propels you forward and gets you into a better place. Take it as such. Don’t be petty, about that Euro that you didn’t get. Life is too nice and too short to be worrying about silly things that ultimately don’t matter. Okay, that’s a good one.
If you work in an environment that you’re happy with, you can really create great things, and you can fight the status quo always. But choose your battles wisely. You can invest so much energy into something that you can’t change. That is in relationships - sometimes it hurts, but you need to go away, because it sucks out the energy of you. But it’s also in your work, don’t forget that. If you have the same feeling in your work, that you can’t change things that need to be changed, and you can’t accept them, this will drown you in second-hand problems that you don’t want to have. It will make you unhappy.
So you don’t necessarily need to accept things. Always fight the status quo, always abstract problems… Like, what if we do it this way? Or okay, we have a process here, but what if it were not here? And what can we do about it? And choose your battles wisely; invest your energy in the 80/20 use cases, where you can have 20% energy, but 80% impact. Because that is where the fun really is. If you optimize the last 20%, then you lose yourself in that. That is something that you can leave to Audi engineers, optimizing the sound of your car closing. But they’re passionate about it, for them it’s the 80%.
[01:04:12.04] So there will be someone that identifies your 20% as his 80%, and will be very passionate about it. If you’re not the one, tell people that you’re not the one. Find your happy place. Somebody will be there and finding something – usually, our companies receive back. There will be someone.
Yeah. That is a good one. That is a very, very good one. Thank you for sharing that. So for people that want to listen to you talk - I know you have a few conferences coming up, some keynotes… Any that you’d like to share with us, to know where we can listen to you next?
Oh, yeah. I’m very excited about it, actually. Just let me see in the calendar… I’m going for the first time to Eastern Europe. Is it Eastern Europe? It’s Bucharest and Belgrade in November.
So that’s for me for the first time in that tech scene. I didn’t have any touch points with the Eastern European tech, apart from Ukraine. And Poland, by the way, I see more like Central European scene. They’re very close to us. So it’s Belgrade in 15th, 16th and 17th of November. I’m really looking forward to that. And also, we’ll be joining Katie Gamanji from Apple. She will be joining there as well, I think we talked a little about it, I think out of the same interest. We’ve never been there, it must be fun, and it seems to be a beautiful city.
What’s the conference name, do you know?
DSC. Data Science Conference?
And I don’t know where the Bucharest one is, actually. Or when. Ah, the Go Tech Summit in Bucharest is on fourth of November. It could be an interesting one as well. You never know that unless you’ve been there. I’ve had conferences where it was like – you know, at Container days, it’s amazing. It’s my most important event after KubeCon, since we have KubeCon. In Valencia this year, for example, or next year in Amsterdam. And it’s in Hamburg, by the way. Container Days in Hamburg. It’s just the perfect city to have Container Days. And a very good event. But I have been to events as well that were like very, very strange, and I was like speaking hopefully in front of a wider audience online, because offline it was empty.
So you have these events where you speak in front of 10,000 people, 8,000 online, 2,000 offline, and then you have events where you’re just not sure why you’re even there, and you just want to go home…
I see that.
But that’s the same feeling as you have as a musician when you’re touring, right? You’re playing at good concerts, you’re playing at fun concerts, and you can make the best out of every situation. But as I said very initially, why I do this, what I do - if there’s a small conference and there’s just ten people, but you inspire ten of ten, or one of ten, that’s enough. If I inspire one of 2,000… I have maybe better chances of finding the one, yeah. But if I inspire one of ten, it’s the same thing. So it’s just a very – yeah, it’s a numbers play, right? So the chances are higher on the bigger events, but it doesn’t make a small event bad, actually.
A one-to-one talk could be just it. It could create a huge amount of change somewhere. So take every chance… And I’m really thankful for the people coming to these events and listening to me, sharing philosophy, knowledge, and also giving me their input and their thoughts about it. That is very valuable for me as well, and it’s the only way for me to get better and enhance with the more perspectives that I have… Because traveling - and everybody knows that… Traveling enhances your empathy, your perspectives, your way to reflect yourself, because you just have more comparisons out there, than having nothing because you just have yourself, and you just do stuff the way you always did stuff. And these travels - and there are a lot - they are not only a driver to inflict change somewhere else, but they’re also a driver to change me, to reflect on me, to reflect on what I’m doing… And that’s how I can enhance all these theories, all these ideas, and put them into action, and try them. Also to find new tech, right?
Yes, for sure. I’m super-excited for you. I was recently at Cloud Native Day in Switzerland. It’s not a big conference by any stretch; I think up to 300 people. Between 250 and 300, and it’s a two-track conference. But some of the conversations which I had were so amazing. I really enjoyed myself… Including the one with Katie. She was one of the speakers, too. I was a speaker at the conference. It was a great one, I had a great, great time, so I’ll make sure to add the link from your talk, the one from Container Days, and even the others, from the other conferences, if they will be recorded. I can add them after the show goes live. So if new links appear in the show notes, that’s what it is; that explains it.
I’m really looking forward to your future talks, the ones coming up. I hope they will be recorded… And it was great talking to you, Sebastian. An amazing conversation. I’ve learned so much from you. And the thing – my key takeaway is your calm manner. Whatever you’re doing, it works, because you’re so relaxed about it. And it’s not for lack of responsibility or lack of impact. It’s your philosophy, and I really like it. I think it’s a very healthy one.
Thank you so much. Actually, it’s peace on a meta level. You don’t need to know the solution. That’s operational level, right? You just need to know you can handle everything life throws at you; no matter what, life goes on, and it will work out. Even if it’s horrible at the beginning, if it’s a big problem, you just need to know you will be able to handle it. And I don’t need to know the solution. I just know I will find one.
That’s it. That’s exactly it. We will figure it out. And if you have that belief, everything else will fall into place. And everything is amazing. Even when it feels like it’s bad, you’re learning. People learn a lot more from mistakes, from failure, not when everything is pink. So that has its own purpose.
Once again, thank you very much, Sebastian. I look forward to next time. Thank you for joining us today.
See you soon. Bye-bye!
Our transcripts are open source on GitHub. Improvements are welcome. 💚