I donât think that you can imagine just how excited Gerhard was to find out that Audi, his favourite car company, has a Kubernetes competence centre. We have Sebastian Kister joining us today to tell us why people, followed by tech make the process.
The right thing to focus on is the genuine smiles that people give in response to something we do or say. That is an important SLI & SLO for reducing friction between silos.
How does this impact the flow of artefacts into production systems that design & build cars?
Featuring
Sponsors
Raygun â Never miss another mission-critical issue again â Raygun Alerting is now available for Crash Reporting and Real User Monitoring, to make sure you are quickly notified of the errors, crashes, and front-end performance issues that matter most to you and your business. Set thresholds for your alert based on an increase in error count, a spike in load time, or new issues introduced in the latest deployment. Start your free 14-day trial at Raygun.com
Sourcegraph â Transform your code into a queryable database to create customizable visual dashboards in seconds. Sourcegraph recently launched Code Insights â now you can track what really matters to you and your team in your codebase. See how other teams are using this awesome feature at about.sourcegraph.com/code-insights
FireHydrant â The reliability platform for every developer. Incidents impact everyone, not just SREs. FireHydrant gives teams the tools to maintain service catalogs, respond to incidents, communicate through status pages, and learn with retrospectives. Small teams up to 10 people can get started for free with all FireHydrant features included. No credit card required to sign up. Learn more at firehydrant.com/
Notes & Links
- đŹ Container Days - September 2022: A culture beyond SLA and T&C - Sebastian Kister, Audi
- đ How Audi provides developer self-service - April 2022
- đŹ NGINX Sprint 2.0 - Nov. 2021: Audi Harnesses the Full Potential of Kubernetes with NGINX App Protect
- Celebrating 50 years of âVorsprung durch Technikâ
- Audi Digitalization
- đ Audi Films: An Avant Story
Chapters
Chapter Number | Chapter Start Time | Chapter Title | Chapter Duration |
1 | 00:00 | Welcome | 01:00 |
2 | 01:00 | Sponsor: Raygun | 01:59 |
3 | 02:59 | Intro | 04:19 |
4 | 07:18 | What is it like to work for Audi | 03:35 |
5 | 10:53 | How the transformation happens | 03:57 |
6 | 14:50 | When projects onboard | 03:52 |
7 | 18:41 | Sponsor: Sourcegraph | 02:22 |
8 | 21:04 | What about the documentation | 04:05 |
9 | 25:09 | Learning from the engagements | 01:50 |
10 | 27:00 | Why Kubernetes | 02:54 |
11 | 29:54 | What runs here | 04:46 |
12 | 34:40 | Changing how people approach problems | 03:35 |
13 | 38:15 | Scaling up | 03:19 |
14 | 41:33 | How large is your team? | 02:28 |
15 | 44:01 | Importance of small teams | 03:32 |
16 | 47:33 | Sponsor: FireHydrant | 01:18 |
17 | 48:51 | Working in this ecosystem | 07:09 |
18 | 56:00 | Getting to the end users | 03:16 |
19 | 59:16 | The stuff you wish you knew back then | 01:32 |
20 | 1:00:47 | Big takeaway | 03:52 |
21 | 1:04:40 | Where will we see you next? | 04:51 |
22 | 1:09:30 | Wrap up | 02:04 |
23 | 1:11:34 | Outro | 00:53 |
Transcript
Play the audio to listen along while you enjoy the transcript. đ§
I want to start by thanking Taylor for kicking this one off. Today Iâm able to bring two of my great passions together - cars and technology. And itâs not just any car, itâs my favorite car, Audi, and my favorite type of technology, cloud-native. And I cannot think of a better partner for this episode than Sebastian Kister from Audi. Specifically, the product team lead for the Kubernetes Competence Center. Sebastian, welcome to Ship It.
Thank you so much. Thanks for having me. Pleasure to be here.
So why Audi?
You mean why I chose to work for Audi, or why we joined?
You can answer it in whichever way you want, the first thing that comes to your mind, and then I can ask a more specific question. But why Audi for you?
Well, Iâve been a decade in startups, ranging from privacy by design data infrastructure startup, to IP television startup, which we brought from the first customer to 1.25 million customers in just two years. And after that, there were many possibilities to join different companies, but I wanted to go to where I came from. Actually, Iâm born in Ingolstadt, and I wanted to get away from Munich⌠In Ingolstadt you have two companies, and the most famous one is probably Audi. There are a couple of more companies, small companies, butâŚ
Whatâs the other one, by the way? Because thatâs the only one I know.
Yeah, the other one is Media-Saturn which is a retail, like electronic retail in Germany. But retail business is not really an end product I could identify with at all. And the Audi design language, this progressive design, the E-Tron design, or in general the Tron design from the movies in the â80s and today, this culture of foreseeing society in 20 years, and imagining it, and living from a design language from the future is something I totally loved. I love the backlights, the design curves⌠I think we have met in a different meeting before, where I had this E-Tron GT backlights as a background. I just fully identified with the design language from Audi⌠So this was a great fit for us. And coming from startups, Audi obviously wanted the expertise in an enterprise, and create startups in an enterprise, and I said, âYeah, I think we have a good match here. Letâs do it.â
[06:10] So I havenât mentioned this to many people, but now I think itâs a reveal moment. I know you like telling secrets too, or making things sound as if theyâre secret⌠But for me, itâs the way the doors close. Itâs a really weird thing to say, but I think when you close an Audi door, itâs just so different from anything else. Thereâs like so much attention to detail in how the doors close⌠Can you imagine how all the other things are? And thatâs what got me. I really liked the way the door shut, and then from there, it was discovery process.
In all honesty, my wife is a big Audi fan, and she said, âWeâll have a single car, it will be an Audi, and it will be an estate. And you can choose whichever one you want.â So thatâs how we went through an A4, and A6, an RS6, always an estate, and Iâm so happy. But thereâs something in the design, thereâs something in the detail that Audi does, which I havenât seen any other manufacturer do. And thatâs the why for me.
Back to you, because this is about you, letâs be honest⌠What is it like to work in a big company, in a big enterprise? Itâs over 600,000 employees, itâs huge. What is it like?
Yeah, the entire VW Group, with direct and indirect employees, is over 600,000. Audi, specifically the Audi Group, is 96,000, something like that.
Okay.
And only Ingolstadt, our main factory, and the business central, has 44,000 people. So yeah, it is big.
Still a big place.
And Iâll tell you the biggest advantage of it - itâs awesome food. You get the right food over there. Itâs really good.
The campus! [laughs]
I totally enjoy it over there. And apart from the great food, obviously, because Iâm not going very, very often to the office⌠But when I do, I stop by the canteen, or one of the many canteens; there are really many canteens. And they offer everything from very simple food to very sophisticated foods. Obviously, you have to choose your canteen, that fits your desire⌠But there is something for everyone.
And working in a big company also gives you so many possibilities⌠And I tend to focus always on the positive things, I create something out of every free space that I find. I create something out of every opportunity that I identify. Thatâs just my â thatâs me. Thatâs what I do. And thatâs why being in startups was probably very beneficial, to work on that and elaborate that. But in Audi I had a team surrounding me that were fostering this philosophy, and I had complete backing to do the sort of startup culture in my teams as well, which is not everywhere the case.
So I really must say that we started as an exception, because we wanted to change the culture. And obviously, you canât change culture top-down by saying âNow we think different.â It wonât work. So creating germ cells in a big enterprise, and protecting them at all costs until they can deliver and supply product, create value-add, create business, but as well have like a gravitational effect, a cultural gravitational effect to draw other people into that sort of working, sort of thinking⌠And I say it is a critical mass now.
[10:12] We can absorb culturally projects when they onboard on our platform. So we are one end of a non-existing end-to-end responsibility in a silo-structured enterprise, but we can transfer our culture all the way to the other end⌠And itâs appreciated a lot. Iâve found a lot of appreciation along the way for this sort of cultural change, because it is a delivery process, and it is a business; it is not like idealism, and unbearable idealism with the current situation.
So when a project onboards onto your platform, thatâs how the transformation happens, is my understanding.
You could say so actually, because you need touch points to do so, and you need to make people understand the value that you bring to the table by our way of working and thinking, and adjusting, and putting people in the center of everything that we do, actually. So the focus is really on the people. People first. When you think transformation in enterprises, what often happens is they processualize something, think process first, and impose something top-down. And - well, what happens is you change your address fields, you change your departments, you change maybe some managers, but on an operation level nothing changes, usually. Maybe sometimes when you say âWe have now agile methods. We need to make an agile product creation process.â But itâs, again, a process. And even if you say âWeâre working with agile methods nowâ, often they are not like philosophically applied, theyâre just followed, processualized. And that is just not the transformation; thatâs not a real transformation at all.
If you put the people first, and then look at the tech that they want to explore - you have passionate people all over the place. You ask them, âWhat do you want to do? What do you want to explore? Think of a product, from scratch, whatever you want to doâ, and then you create out of what they want to do business value, because then some guy like me, who creates startups, or business cases for epics and whatnot, and brings back the passion of the people and makes a business out of i
When you have passionate people exploring tech, what happens naturally is a process that follows. So you have people, tech, and then process, because passionate people use great tech, and once you automate the great tech, the automation is the process. Itâs a process prone to human error, actually, even. And thatâs how we onboard projects, to come back to the topic. We onboard projects by enabling them in the bleeding edge technology, actually. Enterprises never use bleeding edge technology. They are afraid of it; early adopters - not with us. But you canât be progressive, you canât be⌠Audi. You canât be Audi if you follow an enterprise paradigm; you need to be progressive in a very fast-paced environment, and if you want to say that being progressive is an attitude, or progress is an attitude â itâs an Audi hashtag, actually. If you want that and live up to that, you need to be bleeding edge, at least in the bottom layers. In the end, in the car, fair enough. I mean, thereâs so much going on until a car is being built⌠But in the digital service portfolio⌠And we want to bring this customer excitement, from the closing door sound, to the digital service portfolio as well. The engineering is unrivaled. Itâs amazing. And the love of detail our engineers have is very annoying for their managers, actually.
[14:39] I can imagine. âNo, it doesnât close properly enough. No. Letâs do test 1,125,000.â
Exactly.
Okay. So when a project onboards, and it comes into the Kubernetes Competence Center, how does that work? Someone has an idea, someone is passionate about it; youâre on the cutting edge of technology, and you say âWeâll make this workâ, and youâll have a great time doing it. It will put a smile on your face, at least one, I can guarantee you that; there will be more. How does that work? Can you run us through that?
There are many, many dimensions to it, actually. Itâs not like thatâs a one-dimensional thing. But let me go through it from from the beginning. First, when we onboard someone - and I say just standard; standard workload, easy thing, nothing bleeding edge. Letâs separate the onboarding process from being bleeding edge. And if you have an onboarding and you need to show someone how to create a container, for example, because itâs the first time they worked with Kubernetes; for them, that is bleeding edge already, probably. And then you have also something that we call shift left, that you enable projects in a sort of shared responsibility manner, to take care of things before they happen in the cluster. For example, container image scanning, container runtime scanning - you can use a couple of tools locally on your machine already in the build process, in the whole supply chain, actually, before you throw it at the Kubernetes API⌠And we say no, because in our cluster, we have enforcers deployed, with policies, and sometimes your container image wouldnât check out. And that causes friction; that kills your time to market. And itâs not very good for continuous integration at all. You canât deploy like three times a day if youâre not sure your image will check out.
So if you have all this shift left toolchain and the stuff that makes you â you have to be able to use it. Thatâs how we enable the project. So once theyâre able to use it, they have the time to market in their own hands, and they donât bother our developers and our operations engineers, our SRE teams. They donât get tickets, because they have it in their own head, and know if it will check out, because they can enforce the policies or check if the enforce policies on their image will work or not, before they push it. That way, you reduce friction, communicational overheads, you have a high level of automation, you can deploy faster, more often and in smaller increments, and you can react faster on customer expectation, actually. That is the core of todayâs development or digital service portfolio creation in general. If you have a digital product, and customer expectation is a fast-changing phenomenon, you need to react fast. Thatâs a core competence, and thatâs how we do it, actually. And if thereâs something bleeding edge, like nobody knows how to do it, itâs just perfect with us, because we love to explore it with him. We take his hand, we never let his hand go until heâs successful in the cluster, happy in a runtime, and we face the issues together and learn from it together.
So that focus on good, smooth onboarding is important, right? Thatâs how everything starts. Is there a documentation that developers â Iâm trying to figure out how much of this is self-service, versus âLetâs work on this togetherâ?
There is a lot of documentation, especially because you have this [unintelligible 00:21:34.03] necessities to document things⌠But letâs be honest, your project wonât read that. Itâs like buying Lego and then really, you read the manual; or buying a new phone and you first read the manual. Nobody does that. And projects are people too, and they wonât read your manual, even if you have one. So make it a people business.
Obviously, the first argument you hear now, âOkay, Sebastian. Nice one. But canât scale. This guy canât scaleâ, right? I can prove that wrong, because it scales way, way better than self-service, and anything like automation and anonymous pages can do, because you create shared competencies with every onboarding. When we onboard a new team, thatâs people. And they learn from us during that onboarding. If itâs a self-service, they dig through stuff and they donât learn anything. Basically, they just make things happen. And maybe they find some errors, but they wonât report it, as long as it works.
But during our onboarding, itâs like a professional services layer; we get in touch with the people, we deploy the first workload together. Thatâs what I mean we take their hand, and we lead them to success, whether they want it or not. And youâd be surprised how much are not too interested in having success. But once they had it, and have the feeling again, they start to burn for something, they become passionate about something. And thatâs also this human component, if you put that first; this real, real human component, and with bad days, good days⌠And you have a positive attitude in the team, and you ignite that spark to them⌠And they love working with us, they really enjoy it, and they learn from us. And the second, the third, and the fourth project works like a self-service afterwards, because they know what to do already; theyâre completely enabled with everything. And itâs way faster than him reading through a manual, is us showing him how itâs done properly⌠And every every project is different. We have so many projects, and we kind of didnât identify a single project that is like 100% the same as the other one.
[24:14] Sometimes they have a different database, they want some different cloud-native service connected with their cluster, or with their namespace, or with their container, whatever. Everybody has some slightly different stuff going on. And if you have a self-service, you have a different problem. You have projects putting requirements on your platform, and you have an ever-growing backlog that will rip you apart in two years.
Yeah.
So if you individually solve all these problems 20 times, with 20 projects, then you have a community, a community of happy people, successfully working with your platform. And that is scaling way, way better than self-services and ticket bonanza because they help each other, you know?
Yeah. How much do the developers that work within the Competence Center, the ones that maintain the platform, learn from those engagements when they work with someone whoâs trying to onboard something?
That depends⌠Obviously, we have developers with a very deep insight into app development as well, to really understand the other side. And this great mix of our principal operations engineer and our principal platform engineer, cloud engineer, makes the perfect combination, actually, to have a guy really automating everything on the infrastructure, and the other one saying âDude, I need to keep that alive. Letâs slow down a little.â And these guys, in combination, they obviously learn a lot about the requirements projects have⌠And once we identify an 80/20 case, or say, âOkay, 80% of the after projects need this and thatâ, then we create a managed service on the platform level. But it needs to be 80/20 first. When itâs below 80% of the projects demanding something, we just donât do it, because you can do that yourself. We are very open. Our platform focuses on just two things: security and runtime. And when you focus on security, you can â I mean, I canât reroute some cloud-native services; it just doesnât make sense, because it will be more expensive than from the cloud provider directly, and it will lose a couple of nines behind the comma. So I cannot create a better service, and it wonât be cheaper. So I want to it. Itâs not a business case.
I see. So why did you pick Kubernetes?
I didnât pick it.
The Competence Center⌠Why is it called the Kubernetes Competence Center? Why is it in the name? I think thatâs important.
Yes, I can explain the history. I joined a team of two passionate Kubernetes engineers⌠And thatâs it. That was it; it was just us three. And they created something for the E-Tron charging cables and charging function in 2017, and they used Kubernetes for it. And when I joined Audi, they already created an Audi proprietary Kubernetes control plane, focusing on deploying into EC2 directly, or then when EKS came out, on the first abstraction layer on EKS.
So from there on, we went and we focused on Kubernetes⌠And thatâs what I told you before - I was not saying what we have to do, I was asking them what they wanted to do, and they want to do to Kubernetes. And I was like, âOkay, then letâs be the best. I donât care what we do, but when we do something, letâs be the best.â
[28:15] And from there on, we grew, we delivered, we had more projects, we had platforms, we created bigger platforms⌠At a certain point, we consulted other platforms, we delivered infrastructure as code to other platforms, we have a cross-platform delivery center for middleware, for example container image scanning, container runtime scanning⌠And we supply consulting to other platforms⌠And that was the point where our product was not the platform itself. It was being there for all the platforms. And there was the point where it was a Competence Center. And now we consult platforms that will be created, and we allocate platforms in the right layer⌠Because some business platforms focus too much on the infrastructure layer that they can actually abstract. They donât need an infrastructure; they can consume like 12 clusters from us, with all the heritage that we give them. Itâs very beneficial if they donât do their own infrastructure in this case, and focus on their business line and say, âOkay, we have a platform focusing on security and on runtime. Yeah.â So if you have a business platform and you want some machine learning, or data analytics, or whatever, you have a couple of managed services that your projects want to focus on. Also, you have a different set of burst workloads. You probably want a consumption-based chargeback model, and not like 80% capacity and hands-off.
So in terms of workloads, what runs on these platforms? Because it sounds itâs like more than one. There must be a lot of workloads that run. Are they stateful? Are they mostly applications? What do you run?
Well, the company is quite big, you know? Thereâs literally everything. Thereâs literally everything.
Just how big? Thatâs what Iâm trying to â like, is it thousands of applications? Is it hundreds of stateful instances of databases, or caches, or whatever you run?
In total probably yeah. In total, itâs, I think, 6,600 projects, or something like that. You donât know if theyâre all active. So thereâs maybe many, many projects that are not even active. And in terms of our platform, itâs one of the smallest platforms in the VW Group, and we want to keep it that way, actually, because we want to have a big â how do you say that⌠POC case for new technologies as well. So we want to be early adopters in technologies, and if we have 6,000 projects running on our 200 clusters or something like that, that wonât work anymore. So we canât be the Competence Center and have the biggest platform. We need the smallest platform, and that way we can really be bleeding edge and create POCs, with new technologies, with new vendors, try out things⌠And then we can consult new platforms and old platforms in adapting new technologies. And this adaption product is actually a new product. Itâs a cross-platform delivery center. So something that we do for our own platform in terms of security, often in other platforms is just a license, but zero adaption, so we need to help them in the adaption.
[31:48] Weâre spending on our security experts a tremendous amount of money, actually, and other platforms can leverage that. Just imagine if everybody needs to have two or three security [unintelligible 00:31:59.11]you shoud have two security experts on each platform. We have 33 platforms. 66 security experts. We would be the biggest IT security company in Germany, probably. And if you know the market, thatâs not possible.
So this kind of thinking is really leveraging â I avoid the term âsynergiesâ since a couple of years, but thatâs the way to leverage the benefits of our infrastructure as code, our adaption and our maintenance, our efforts in general that we put into security; the others can leverage that very, very easily.
Okay. When it comes to things that Audi runs, thereâs like a lot of stuff. Thereâs the cars becoming connected, thereâs like more and more data coming from the car, data being sent to the car; thereâs more cars that are coming online. How much of that do you get to run on your platform?
Like I said, the platform itself, we have many workloads that develop that kind of stuff⌠But the connected car stuff, the in-car software, that actually runs in the car, in a productive state and customers are using it - that needs to run somewhere else, and on different kinds of systems. You donât want that â like a comparison with an airplane, you donât want to ride an airplane that has an overdue upgrade. Maybe. I wouldnâtâŚ
Okay.
You got the idea, right?
Yeah.
I mean, [unintelligible 00:33:44.03] is a very nice topic, by the way. Itâs controversial, itâs not that easy, and obviously, for some parts of the software itâs great, but other parts, you donât want it. You want an engineer looking at it physically when you bring that to the workshop. And in terms of software, these kinds of upgrades are great, but only for a couple of things, not for all the things.
In terms of connected car or in-car software in general, there are many, many engagements and things going on in the VW Group, on a group level, but also at Audi itself⌠And I think there will be many workloads developing that stuff, but just a few very dedicated platforms running that stuff.
So when you take those happy, passionate people, and when you take technology thatâs secure, that you really understand how it works, and thereâs a process that ensues when you combine these two - how does it translate to day to day? We all have incidents, we all get to see them, to deal with them, thereâs always zero-day exploits⌠How does what you do change how people approach problems like those?
Well, first of all, the shared responsibility philosophy in the platform or in the culture itself, with a higher enablement of projects and uses, and workload teams, project teams - it creates a net of people who are actually understanding what just happened. Most of the incidents that we face in infrastructure are led by hysteria, a sort of hysteria⌠They have no idea what just happened, but it must be bad. âLetâs escalate it!â And many, many things just are not that bad.
[35:53] So in terms of alerts, first of all, just create an alert when you want it to wake you up at four in the night, and you want a reaction time of one minute to solve the issue. So the alerts - they need to be really, really in place and trustworthy. Then if something really doesnât work anymore, worst case, like production failure; we canât build cars anymore in the night. Then obviously, thatâs a different kind of story. But to talk more abstract about people working with it, the higher, I would say, competence, or the more broad competence of everyone working together, and everyone knowing each other, because the onboarding was not a self-service, creates a family-like, team-like cross-silo family of people that solve something, that donât blame something to other silos. They have been there together, deploying that, and they will be there together solving it⌠And that is just a way different approach than throwing tickets around, operations blaming development, backend saying frontend, âHey, what did you do?â and then the guy in the middle says âI have to manage three different companies, and my own company, who does operations, blames a fourth company that I havenât even heard of. Whatâs wrong here?â
So this is really different. On our platform, even all the companies that contribute to the platform - I want them to know each other. I want the SRE from the team that manages the cluster to know the guy and the content from the container image scanning software, and from the layer seven five [unintelligible 00:37:43.20] provider. I want them to know each other, I want the engineers to know each other, and in the worst case, I can bring them together within 10 minutes. And itâs a different kind of story if you have contracts; obviously, you have SLAs⌠We have contracts, and if something really, really bad happens, obviously we can rely on them. But nobody pulls up a document that says âIâm not responsible for this in my teamâ, ever. We donât do that.
How do you scale and maintain that? Because all I see is like lots of people meeting and talking and figuring stuff out⌠When people work together, like multiple people like this, are they â I canât imagine them being in meetings constantly. Right? They must be working together. So how does that happen? How do you get people to work together, that maybe are not in the same physical location, on the same problem, that crosses multiple departments? How do you do that?
Actually, I even have to add their calendar is empty. In my team, I take care that they usually donât have anything else in terms of meetings than 15 minutes in the morning with me. Thatâs it. Obviously, we have review meetings, we have refinement meetings, we have some âLook at this technologyâ or some breakout sessions here and there⌠Then we have onboarding meetings with projects, and stuff⌠But usually, it has nothing to do with an enterprise [Bleep] that usually takes place, or that consumes 50% of your time.
Then again, we create 15 minutes meetings. We have like a 15-minute on Wednesday with all the OpenShift platform owners, for example. Just one control plane; we have many control planes, but I just mentioned one now. And with all â they face a couple of issues that are similar. Obviously, they use the same control plane, which is the control plane focusing on this one. And in 15 minutes we donât solve the issues; we organize the people that need to go to different breakout sessions.
[39:56] So I just need to know who knows what, and who wants to work on the problem with the guy that has the problem. And thatâs it. I can do that in 15 minutes. And then off you go. And the next problem - okay, here. Who can help? Here and there. Okay, off you go. And then in 15 minutes, we sorted out everything. And just imagine, 15 minutes for the entire VW Group, expert group, on one control plane technology. That usually takes â some people do days. Breakout sessions, workshops⌠And we just do 15 minutes a week. Thatâs enough. Consistently, continuously coming together is more important than one workshop every three months, and nothing happens.
So that is how the calendars are empty. And then you have time to really work on stuff, actually; to be efficient. To get into a certain kind of flow that is very rare in enterprises, because all the time you get calls, and whatnot, people talking to you. And I get really, really angry with people that call my developers directly, unless they invite them for lunch, or something. Thatâs fair. But if itâs a work call, and they stripped them from their flow, thatâs where I step in and say, âNo, this entire team is being protected by me. You talk to me. Anything you want. And I decide if we need a developer to answer that or not. Or if an intern can do that.â Because most questions, to be honest, my interns can answer that.
How large is your team?
Both internal and external we have 14 people right now, and the team working on the different platform products. But that is not the real scale of it, because there are so many other people contributing to it. This is just a core team, you could say. We have a very, very strong core team of three people, and then we have an operations department that is being created in Audi Hungaria, where we enable and create a cloud capability and cloud competence center. In Audi Hungaria - weâre recruiting there right now, so if youâre from Hungary, youâre near GyĹr and youâre listening to this, you will join my team. [laughs] Indirectly. And maybe we should promote it a little more over there⌠I will talk to the Hungarian guys.
And then obviously, we have external people, because hiring at Audi or VW Group is not that easy, actually. The positions are always combined with so many other things that itâs sometimes easier to just go to suppliers and say, âGuys, I need some architects, I need some developers. Give me a price.â And the difference is that they are really working in the core team. Itâs not like we treat them any differently. From a processual point of view and from a legal point of view, obviously, we do that, so we are compliant with all the rules about treating external people. They cannot sit â weâre completely remote anyway, so nobody comes to the office, so all these problems, we donât have them. But like on events, where we meet with our suppliers or stuff, it feels like a family coming together, actually. And thatâs the nice thing over there, in this team. And Iâm really proud of having a team like this, where no matter from which company you are, youâre in that team, in the Kubernetes Competence Center at Audi, and it feels like family, no matter from which company you actually are, from which team you have contributed into the product, into the deliveries and deliverables. And that is really, really interesting to watch and see.
[44:00] I can see how a small, close-knit team is very important for this working as well as it does. And then theyâre really passionate about what they do, they inspire others, and they create this community effect, where others join, they pick up on that, they learn this approach, they see how well it scales, and they take it further. And then itâs almost like youâre inspiring others in this approach. Less about growing your team and becoming really big, because once you have 50-60 people, managing 60 people is very different to managing 14, even up to 20 people, I think.
Yeah, I completely agree. Also, you must ask yourself if that is really useful. So when I look at all the other platforms, their teams are even smaller, but we have more deliverables. We have like a cross-platform delivery center. For security middleware, we have professional services. Obviously, we need to cover for more people. But in general terms, when I take the SRE teams, they are not in that team size, obviously. This is where I want to create value-add, to be honest. Any kind of control plane for me is not a value-add. We have some requirements to it, obviously; it shouldnât be heavyweight, it should be easy to use, it should be flexible, it should be state of the art⌠In the best case, there is a Kubernetes upstream from that team, really going to the cloud-native community⌠So I also decide on the open source side, and if it contributes to cloud-native⌠And I strongly, strongly go into the direction of upstream technologies, actually. They are more future-proof than proprietary tech. And this is not a belief, this is just a fact that you can see from which technologies survive once theyâre bought by a company. Itâs more proprietary from a freemium model, or whatever - they survive another two years, and then theyâre gone. We have seen that from 2017 to 2019, and from 2019 to today, with the control plane market in general - itâs just a couple of players left.
And the same with all the other things going on in cloud-native⌠It might be core technologies like Jaeger or other things (Prometheus) that we use constantly. And theyâre incorporated by big vendors; they have to use the upstream technologies. And itâs the open source community against the hyperscalers, and the open source community proves to be stronger. Thatâs really great to see. And from a business perspective, it just gives you the edge if you go with the open source community, and you get an enterprise support by some vendor⌠Because you need to have that. Donât forget the SLAs at all. Theyâre important to be in place, but you need to have a culture beyond the SLA to be very effective.
Iâm wondering, what is it like to work with the cloud-native ecosystem, with the CNCF, Linux Foundation, and also the cloud-native community? What is it like? Because youâve been doing this for a couple of years.
Yeah. Iâve been creating the partnership with the Cloud Native Computing Foundation in 2019⌠And also, I really learned and liked a lot how the CNCF community works, thinks, and builds, and creates. We also have in Audi something that we call inner source, where you can contribute to certain projects, eliminating bottlenecks, actually⌠Because the backlog doesnât reflect your necessity in the project, but you can do it yourself, right? You can solve that issue yourself, contribute it, and we leverage it as a core maintainer for everyone. Something like this. And the same happens in the cloud-native community.
So everybody has a slightly different approach to some things, or some different priorities⌠But you can create that. You can solve that issue. You can solve your number one priority, which is maybe number ten for the entire community⌠But you can solve it and you can contribute it; then itâs there, and itâs leveraged. And you profit from leveraging project teams that are concentrating on the other ten priorities, or other nine priorities.
I must admit that working with the community and the technologies has proven to be the best business case out there. And it needs to be a business case. You canât be sustainable without making it a business; you canât be idealistic without making it a business. It needs to be a business in the end; it needs to be a business case. Otherwise, change doesnât work. Nobody changes anything, unless itâs a f***inâ business. And that is something that I really saw happening, actually with cloud-native technology taking off in the enterprises, because it is a business case. And itâs an easy one; itâs a very easy one. You have two develop in your company contributing to that, and the other 4,998 developers - they do something, too. So you can maintain your fork for the next two years with your two guys, until the first breaking change happens, and then youâre done. Your project is done. You can never touch it again, and hope the entire ecosystem it touches will also never be touched again⌠But you get the idea; that doesnât fit modern development at all. It doesnât fit progress at all. So thatâs not working.
[51:54] Thatâs from a technology point of view⌠Iâve also been in the CTO summit of the CNCF last year; no, this year, actually. Itâs 2022 still. Itâs so cold, it feels like itâs already next year. [laughter] And weâve been working together with the CTOs or head of infrastructures from Allianz, Mercedes, Santander, Evori, TikTok, Spotify, Orange⌠And it was really very nice to see that we have similar issues, enterprise issues⌠Like, you can separate it into big company bull***t and into real technical requirements. And what is very fascinating is that our approach that Iâve introduced at Audi to put people first, let them use cloud-native, in this case cloud-native tech, and explore it, and automate it, and let the process follow from that⌠Like, the handover process between friction walls is completely eliminated through cloud-native shift left technology. And the automation is a very natural process. And when you do that in one enterprise, and you can inspire, let it be one CTO per keynote - and I do like 25 keynotes a year - that will actually make the life of all your operations teams, and the company, and the developers, a lot easier. And they will not be called at four in the night for some stupid incident, because they had a container image not checking out through a policy, and whatnot. Or less. Letâs say less. [laughs] And it will make their life easier.
The culture is just so positive, actually, in this way of working with projects, and not very anonymously self-service, clicking something together, and sharing competencies, sharing passions, working in communities⌠And obviously, itâs just more fun to be in teams like that. And people are happier. Theyâre more passionate about something, and it needs passionate people to create great products. It just wonât work with people not being passionate about something. Putting up a great shop maybe⌠You can do that for a while, but you will not do that for very long. And theyâre happier. And I tend to think that if I create like 200 happy people with every keynote, in the end, thatâs 200 parents that have happier children in the end, because they can be there for them, at home, or whatever, and have more time. They nail their goals faster, which is also paying into a different type of work ethic. Solve issues, then have time, and not trade time for money; like, you have to work eight hours a day, I tried eight hours a day. No, thatâs not what I want. I want people to get something done. If the stuff is done, and youâre happy with it, and itâs a great product, then off you go. And if you nail that in two hours now, because you have the tech for it, and the process doesnât hinder you - off you go. I donât care. And then you can focus on the next issue, or take a half a day off, because nothingâs burning. And itâs more relaxed⌠And like I said, and the endquote here is really that people are happier. In this sort of culture, the people are happier, and you donât have too many levers, you donât have sick days⌠Theyâre just happy, and⌠Is it work-life integration in the end? I donât know. It depends on each person. But theyâre definitely happier.
[56:00] How does this bubble up all the way to the end user, this approach from the people that are building these technologies, that are putting all the things together, that are working with other people? Letâs not forget, itâs a very social activity, what we do. How does it translate to the end user?
It translates in hard business. And that is the charm of it. Because like I said, you canât make change happen, cultural change, or people happier without actually creating a business out of that. And one of the strongest KPIs and the easiest to understand is time to market. Like I said before, you have friction in a silo-structured enterprise; you have hand-over protocols to operations. There is no DevSecOps. I mean, you can do separate DevSecOps teams for a dedicated use case, sure. But you canât do DevSecOps in a Plan, Build, Run structured enterprise with silos. Itâs just, theyâre isolated, there are hand-over protocols where this SLA goes into that SLA, these terms and conditions end here⌠Maybe thereâs even a gray zone in between two terms and conditions. But youâre still one company, right? Youâre still one group. The VW Group is one group, and we donât tend to think âThis is Audi stuff, this is VW stuff, this is whatever.â We just solve things together.
If you go to the very end of this supply chain, to the developer, and you reduce all the friction between through enabled people, and communities that know each other, that talk to each other and that know the technologies that theyâre using, and theyâre using it for eliminating the friction walls in these hand-over protocols, then you have the advantages of a DecSecOps teams, and also the advantages of a silo. And a big enterprise canât scale DevSecOps teams. They can only scale in silos, and that is organizationally not possible otherwise.
So itâs not âFight the silo. Eliminate the silos.â No, you need this; a silo is not a bad thing. It has advantages; thatâs why we do it for a hundred years already. This philosophy of creating one team doesnât necessarily go against creating silos. So you can work your way through all these friction silos, culturally, and with people in tech. So once itâs automated, itâs even a cross-silo process, to deliver your build that youâve created automatically into the cluster, and all the hand-over protocols - theyâre done. All the scans, the container image scans, the ingress, the controls - everythingâs done; you can automate it. And out of the automation, you can create a process out of that for the organization.
So youâve been doing this for some number of years. Is there one or a few things that you wish you knew before?
No, the answer is no, but because of a very philosophical approach to life, in my case. I donât want to regret or not to have learned something a different way. Because I could only be the subsumption of my experiences that I am today because I didnât have them before, and everything that happened in life, the good, the bad, the ugly⌠It was just because I was that person, that moment, that very moment. And Iâm very happy being the person that I am today. So no⌠Obviously, everything would have changed - life, work; everything, literally - if I knew the stuff that I know today at a different point of my life. And that would go in terms with regret, and I donât regret anything.
[01:00:27.00] I think I would have changed your path, and you enjoy the path that you have been on, and I think anything would have changed that.
Yeah, it wasnât very enjoyable, to be honest⌠But I learned very valuable lessons. Probably the very â the hard way, yeah. But itâs okay. I needed that.
Right. So for someone that stuck with us throughout the entire episode, is there a takeaway that you would like them to have from our conversation from today? One thing that they should remember, because it feels important to you, or it is important to you?
Yes. Personally, what I just said - I think itâs important to give everything in your life; one life, one shot, give it all youâve got⌠And workwise, many people see walls where there are none. And only because something might be there, just go back and eliminate yourself from the equation. Be selfless. Professionally-wise, be selfless. You donât exist; just think very abstract. That is the only way we can incorporate change in enterprises. Because it has been a system of headcount and budgets, there are so many political battles originating from egos, from vanities, from fighting for a year or more at the Christmas bonus, or whatever. But it really is something you spend so much time of your day⌠It needs to make you happy. If it doesnât, go away.
Okay, so keep looking for your happy, and donât regret anything, because everything is a lesson that propels you forward and gets you into a better place. Take it as such. Donât be petty, about that Euro that you didnât get. Life is too nice and too short to be worrying about silly things that ultimately donât matter. Okay, thatâs a good one.
If you work in an environment that youâre happy with, you can really create great things, and you can fight the status quo always. But choose your battles wisely. You can invest so much energy into something that you canât change. That is in relationships - sometimes it hurts, but you need to go away, because it sucks out the energy of you. But itâs also in your work, donât forget that. If you have the same feeling in your work, that you canât change things that need to be changed, and you canât accept them, this will drown you in second-hand problems that you donât want to have. It will make you unhappy.
So you donât necessarily need to accept things. Always fight the status quo, always abstract problems⌠Like, what if we do it this way? Or okay, we have a process here, but what if it were not here? And what can we do about it? And choose your battles wisely; invest your energy in the 80/20 use cases, where you can have 20% energy, but 80% impact. Because that is where the fun really is. If you optimize the last 20%, then you lose yourself in that. That is something that you can leave to Audi engineers, optimizing the sound of your car closing. But theyâre passionate about it, for them itâs the 80%.
[01:04:12.04] So there will be someone that identifies your 20% as his 80%, and will be very passionate about it. If youâre not the one, tell people that youâre not the one. Find your happy place. Somebody will be there and finding something â usually, our companies receive back. There will be someone.
Yeah. That is a good one. That is a very, very good one. Thank you for sharing that. So for people that want to listen to you talk - I know you have a few conferences coming up, some keynotes⌠Any that youâd like to share with us, to know where we can listen to you next?
Oh, yeah. Iâm very excited about it, actually. Just let me see in the calendar⌠Iâm going for the first time to Eastern Europe. Is it Eastern Europe? Itâs Bucharest and Belgrade in November.
Okay, okay.
So thatâs for me for the first time in that tech scene. I didnât have any touch points with the Eastern European tech, apart from Ukraine. And Poland, by the way, I see more like Central European scene. Theyâre very close to us. So itâs Belgrade in 15th, 16th and 17th of November. Iâm really looking forward to that. And also, weâll be joining Katie Gamanji from Apple. She will be joining there as well, I think we talked a little about it, I think out of the same interest. Weâve never been there, it must be fun, and it seems to be a beautiful city.
Whatâs the conference name, do you know?
DSC. Data Science Conference?
Okay.
And I donât know where the Bucharest one is, actually. Or when. Ah, the Go Tech Summit in Bucharest is on fourth of November. It could be an interesting one as well. You never know that unless youâve been there. Iâve had conferences where it was like â you know, at Container days, itâs amazing. Itâs my most important event after KubeCon, since we have KubeCon. In Valencia this year, for example, or next year in Amsterdam. And itâs in Hamburg, by the way. Container Days in Hamburg. Itâs just the perfect city to have Container Days. And a very good event. But I have been to events as well that were like very, very strange, and I was like speaking hopefully in front of a wider audience online, because offline it was empty.
So you have these events where you speak in front of 10,000 people, 8,000 online, 2,000 offline, and then you have events where youâre just not sure why youâre even there, and you just want to go homeâŚ
I see that.
But thatâs the same feeling as you have as a musician when youâre touring, right? Youâre playing at good concerts, youâre playing at fun concerts, and you can make the best out of every situation. But as I said very initially, why I do this, what I do - if thereâs a small conference and thereâs just ten people, but you inspire ten of ten, or one of ten, thatâs enough. If I inspire one of 2,000⌠I have maybe better chances of finding the one, yeah. But if I inspire one of ten, itâs the same thing. So itâs just a very â yeah, itâs a numbers play, right? So the chances are higher on the bigger events, but it doesnât make a small event bad, actually.
[01:07:55.29] Yeah.
A one-to-one talk could be just it. It could create a huge amount of change somewhere. So take every chance⌠And Iâm really thankful for the people coming to these events and listening to me, sharing philosophy, knowledge, and also giving me their input and their thoughts about it. That is very valuable for me as well, and itâs the only way for me to get better and enhance with the more perspectives that I have⌠Because traveling - and everybody knows that⌠Traveling enhances your empathy, your perspectives, your way to reflect yourself, because you just have more comparisons out there, than having nothing because you just have yourself, and you just do stuff the way you always did stuff. And these travels - and there are a lot - they are not only a driver to inflict change somewhere else, but theyâre also a driver to change me, to reflect on me, to reflect on what Iâm doing⌠And thatâs how I can enhance all these theories, all these ideas, and put them into action, and try them. Also to find new tech, right?
Yes, for sure. Iâm super-excited for you. I was recently at Cloud Native Day in Switzerland. Itâs not a big conference by any stretch; I think up to 300 people. Between 250 and 300, and itâs a two-track conference. But some of the conversations which I had were so amazing. I really enjoyed myself⌠Including the one with Katie. She was one of the speakers, too. I was a speaker at the conference. It was a great one, I had a great, great time, so Iâll make sure to add the link from your talk, the one from Container Days, and even the others, from the other conferences, if they will be recorded. I can add them after the show goes live. So if new links appear in the show notes, thatâs what it is; that explains it.
Iâm really looking forward to your future talks, the ones coming up. I hope they will be recorded⌠And it was great talking to you, Sebastian. An amazing conversation. Iâve learned so much from you. And the thing â my key takeaway is your calm manner. Whatever youâre doing, it works, because youâre so relaxed about it. And itâs not for lack of responsibility or lack of impact. Itâs your philosophy, and I really like it. I think itâs a very healthy one.
Thank you so much. Actually, itâs peace on a meta level. You donât need to know the solution. Thatâs operational level, right? You just need to know you can handle everything life throws at you; no matter what, life goes on, and it will work out. Even if itâs horrible at the beginning, if itâs a big problem, you just need to know you will be able to handle it. And I donât need to know the solution. I just know I will find one.
Thatâs it. Thatâs exactly it. We will figure it out. And if you have that belief, everything else will fall into place. And everything is amazing. Even when it feels like itâs bad, youâre learning. People learn a lot more from mistakes, from failure, not when everything is pink. So that has its own purpose.
Once again, thank you very much, Sebastian. I look forward to next time. Thank you for joining us today.
See you soon. Bye-bye!
Our transcripts are open source on GitHub. Improvements are welcome. đ