Ship It! – Episode #28

What does good DevOps look like?

with Romano Roth, Head of DevOps at Zühlke

All Episodes

This week Gerhard is chatting with Romano Roth, Head of DevOps at Zühlke, a company founded by Gerhard Zühlke in 1968. Nowadays they help companies all over the world build, ship and run anything from factory robots, to AI assistants in complex regulatory environments, and even medical devices that perform autonomous robotic surgery.

When Romano is not leading a team of 30 software engineers that specialise in operations, infrastructure and cloud, he is one of the organisers of DevOps Days Zürich, and also the DevOps Meetup group which is how Gerhard and Romano met in 2019.

Having started his career as a .Net developer back in 2002, Romano had his fair share of dev and ops challenges, and he always enjoys seeing real business value delivered continuously in an automated way. In recent years, Romano’s perspective broadened, and now he sees DevOps realities across many companies. If you are curious about what good DevOps looks like, and what are the real challenges, then Romano has some good insights for you.

Featuring

Sponsors

HoneycombGuess less, know more. When production is running slow, it’s hard to know where problems originate: is it your application code, users, or the underlying systems? With Honeycomb you get a fast, unified, and clear understanding of the one thing driving your business: production. Join the swarm and try Honeycomb free today at honeycomb.io/changelog

FastlyCompute@Edge free for 3 months — plus up to $100k a month in credit for an additional 6 months. Fastly’s Edge cloud network and modern approach to serverless computing allows you to deploy and run complex logic at the edge with unparalleled security and blazing fast computational speed. Head to fastly.com/podcast to take advantage of this limited time promotion!

Equinix Metal – If you want the choice and control of hardware…with low overhead…and the developer experience of the cloud – you need to check out Equinix Metal. Deploy in minutes across 18 global locations, from Silicon Valley to Sydney. Visit metal.equinix.com/justaddmetal and receive $100 credit to play.

FireHydrantThe reliability platform for teams of all sizes. With FireHydrant, teams achieve reliability at scale by enabling speed and consistency from a service deployment to an unexpected outage. Try FireHydrant free for 14 days at firehydrant.io

Notes & Links

📝 Edit Notes

Transcript

📝 Edit Transcript

Changelog

Click here to listen along while you enjoy the transcript. 🎧

So two years ago, in 2019, I gave a talk about making your system observable. And that was a DevOps Meetup Zurich, and I’ll add a link in the show notes. Romano was the organizer, and he put up quite the event, so I would like to thank you for that. It was a great experience.

You’re welcome.

And this year, my intention was to join DevOps Days Zurich, but the timing wasn’t right, so I couldn’t make it work. But again, Romano was one of the organizers, and I’m wondering, how did the event go?

It was absolutely great. So it was in the beginning of September when we had that event; it was also one of the first events which we could do in person, and that was amazing. The only thing – it was quite frustrating in the beginning to organize all of that, because you need to look at the Covid numbers, you needed to create a concept for Covid, and that was quite stressful. But in the end, we could manage to do the event, we had a Covid security concept, everybody needed to line up, needed to have their certificates… And it also worked with all of the people which were coming from around the world. We had people from the U.S. coming, and also from Israel, and everything worked well. We had 250 people in there, and it was absolutely great.

And this was a two-day event, so it wasn’t just like one day, which makes things slightly more complicated, right?

Okay. How many talks did you have?

I don’t know the correct number, but what we have is always a keynote, then we had a set of talks, I think it was 3-4 talks… And then in the afternoon we had the Ignite talks, so that’s the five-minute talks which we have… And then we usually have workshops and open spaces. And that’s over the whole two days. So I would say roughly 20 talks altogether…

And was it single-track?

Always single-track, yeah.

Okay. That’s nice, because you have to sit there and enjoy; you don’t have to change rooms, meeting rooms… Yeah, okay. Which was your favorite talk, do you remember? I’m sure there were many, but any one talk that stood out?

Yeah. I liked very much the talk about – what was it…? “Better. Sooner. Happier”, from Jonathan Smart. I liked that quite a lot, because during the talk he asked always a question, and one of the questions was “Are you doing IT transformation? Please, hands up”, and everybody was putting their hands up, and he said… “Don’t.” [laughs] And that was absolutely amazing. And he continued with these questions, for example “Are you using a scaled Agile framework? Don’t.” And so on. That was quite good, because he was going back to what really matters when you are doing an Agile transformation. That was cool stuff.

I want to ask you what that is. If you don’t want to spoil that talk for you, you can skip maybe a few minutes… So what is it?

[laughs]

Can you tell us?

Yeah, sure. The thing is, you really need to focus on the people. You don’t need to focus only on doing the transformation because you want to do a transformation. It is more focusing on what do you really want to achieve, and focus on changing these things. That’s also why he said “Don’t use a scaled Agile framework”, because there you focus on the process, and on changing terminology. It is more shifting to “What do you really want to achieve?” Identify really what you want, and then changing these things. Not having that huge IT transformation that many people are doing. So really focusing on what really matters for you.

I seem to remember – well, no, I remember that in the Agile Manifesto, as it was initially captured, one of the core principles were people over processes.

Exactly.

So that’s what this is. Okay, that makes sense. That was an interesting one. Now, I know that all the talks are available online, as videos, to catch up on-demand; again, I’ll add the link in the show notes. I’ve also seen the pictures. So if you wanna see how this meetup was, you can go and look at those pictures.

I’m wondering - this is a yearly thing, right? So next year it’s gonna happen again, also in-person, I’m imagining.

[08:06] Yeah, exactly. I think the next one will be on the 31st of May, and we will do it again in-person, and it will be again in Zurich, or in Winterthur, as we call it. In the same building.

Okay. When do you open call for papers? When do people start submitting their talk proposals?

[laughs] Very good question. We don’t know yet. We are currently closing off all of the stuff which we need to do for the pat conference… But my opinion, I think it will be perhaps around December or it will be January when we will open up the call for papers.

Okay. Are there any specific topics that you would like to see more of in the next DevOps Days conference? What do you call it - conference, summit?

Conference.

Conference.

Yeah… Don’t come with Kubernetes… [laughs]

Okay. So no Kubernetes.

[laughs] We have so many proposals on Kubernetes, usually… No, what I really liked about the past conference, what we are focusing on, is diversity. And not only diversity like women or men, it’s all about diversity also in different mindsets. That’s why we also have different talks, and that’s what I also like to see - big diversity, on different topics. We had talks on culture. We had talks on, for example, the role of UX in DevOps, which is also quite a special topic, but it’s an important topic. And that’s my wish - have that diversity on topics, and not only focusing, for example, on technology or only on the process; it’s more also on the people side.

I love that. I mean, that really speaks to my heart, because we keep forgetting it’s human beings, fallible, that get easily bored, and they keep chasing shiny, new things… And granted, Kubernetes may not be the shiny, new thing anymore, but then it’s comfort, right? People are comfortable with that. So yeah, there’s a lot there.

Now, I know that you’re into DevOps; like, big-time into DevOps. But I don’t know why. Why are you into DevOps?

[laughs] Very good point. When I started my career, I was a .NET developer. And this was back in 2002, and we were doing there development of applications, or rich client applications. And one of the things also in the early times which struck me is “How can I ensure the quality of what I’m doing?” And yeah, of course, you could do testing also there, but it was not so automated. And I was always a little bit lazy, and I wanted to automate things… So I went into this area where we were starting automating the tests, and then also the deployment. So I went into continuous integration, continuous deployment, and the applications were getting bigger, and distributed… I was becoming an architect, and slowly I moved in the direction of these continuous delivery pipelines.

And when the whole DevOps movement started, I jumped on that, because this was really one of my hard topics, where I wanted to create these pipelines to continuously deal that value to the customer.

So my understanding is that you were passionate about how value gets delivered, which got you into DevOps, which seems to have made that almost like the center of its activity. How do you move this code from a repository into customer hands, wherever that may be? And there’s like a whole lot of automation, because you can do it manually, but there is a better way, and automating that. Okay, interesting. Which was your first CI/CD system that you used? Do you remember?

[12:16] The absolutely first CI/CD system was actually a command line that I used.

Interesting.

Yeah, definitely. But I think what you want to ask is more the first product that I used, and this was – I think it was the Team Foundation Server.

So when you mentioned the command line, did you mean like Rsync, or FTP, or SCP? What exactly did you do on the command line?

Different things… For example, I had some scripts, command line scripts which I used to just compile or execute the tests. So my first build system was a batch file on my local computer, which I just could double-click and then it executed the tests. It compiled my code and it says “Yeah, everything is okay”, and there is the deployable artifact. And when it went into the distributed system, I usually added also an FTP, where I just could move the code to the server, and then it was on the server.

Okay. What about today? What do you use today?

Today I use quite a variety of systems. One of the products which I love is still TeamCity. I love that quite a lot, because you can do a lot of configuration. I use usually TeamCity together with Octopus Deploy, which I also love as a tool. But I see quite a strong movement in the moment into the direction of platforms, like GitHub and GitLab. So at the moment, when I look at the clients which I am working for, they are moving into this direction, away from Jenkins, Octopus [14:05] or CircleCI and into the direction of GitLab and GitHub. So these are now the big players. Customers are going into this directly because there you have a platform. Everything is there, and you don’t need to deal with different tools which you need to stick together.

That’s interesting. So I think that now we are starting to discover another side of Romano, because we know that you can put up a conference really well, as well as a meetup, but you also do other things. So when you don’t organize various DevOps-related events, what do you do? Because you mentioned customers. There’s more to it than organizing events, right? What exactly do you do?

Yeah. [laughs] I’m the head of DevOps at [14:52] and there I have a whole unit of DevOps engineers and DevOps consultants, and I bring DevOps forward at Zühlke. So there is one side at Zühlke - I do a lot of trainings of people in the direction of DevOps, so that we can deliver better quality, better software to our clients… And on the other side, I’m also [15:16] projects, and there it’s usually in projects where we are doing an Agile transformation, or we are doing a DevOps transformation. And I consult there different clients into this direction, and I also educate their people.

Okay. So how large is your team?

The team is roughly at the moment I think 31 people, at Zühlke. But this is only in Switzerland, of course; there are other people around the world which are also doing DevOps.

Yeah. So your team is 31 DevOps engineers that work with various customers that you consult, help with DevOps-related projects. How many projects do you have?

[16:01] Quite a lot, and the different people which are in my team, they work for different customers, and also with different engineers. So it is not only that these projects are under my responsibility, they are under different people’s responsibility, but my team members are working in these projects.

Okay. So I’m thinking that you must have seen many projects this year that went well, as well as many projects that didn’t go so well. Is this something that you can talk about, without giving any names? We don’t have to give any names. But things that worked well, and things that didn’t go so well. What do you think about that?

Yeah, sure. When we think about things that went well, there is - especially at one customer, where we are creating a whole transformation… And what we did there is they are going through an Agile transformation. And one of the things you need to have for an Agile transformation is technical fundament, so that you can do this transformation. And we were thinking how we can do that, and we built up an Agile release train for that, with different teams in there, which were focusing on different aspects of this transformation. There is one team which is more focusing on the governance part, one team which is more focusing on, for example, the continuous integration and continuous delivery pipeline, and one team which is more focusing on the containerization. So it’s about that. And that worked very well; we are now in the fourth [17:45] which we are doing, and that’s quite cool. We had also a very, very good learning in there.

From the beginning, we identified who the customer is, and we said “We want to deliver to this customer.” And we said, “Okay. Everything what we are doing must be something that the customer can use.” And we started with that.

The thing was that in the first sprints which we did we saw that we were delivering to the customer, but only to the customer, but the customer was not using it. So we changed that, and we said, “No, no. From now on, the customer needs to use it”, so we put that in the definition of Done that the customer needs to take that over. But that was also not enough, because we also had our system demos or our review meetings where we showed that, and that was not enough. And now we said, “Okay, when we are demo-ing stuff, not our people are demo-ing it. The customer needs to do the demonstration how he uses it.” And that’s a very strong thing you can do. So always deliver directly to the customer. Let the customer show what you have delivered and how he is using it. That would also be one of my recommendations to do in the future.

Break

[19:16]

So when it comes to the biggest obstacles, the biggest challenges to driving DevOps transformation or successful DevOps projects, what are they? What did you come across in your experience, Romano?

So one of the biggest obstacles that is out there is actually the middle management. What you can see is you have a lot of companies which are organized in different units. And these units - they have goals. And of course, there is a head of this unit, and he has built that unit up, and he is chasing his goals. But one of the problems that we usually see is that there is a lot of misalignment between these units, or you can also call it silos.

So now, with Agile transformation or with the DevOps transformation, you are starting to align the people around the value stream, and you bring people together. And this means that some of these heads of these units - they are losing power. And they know that, they see that, and this is something they don’t want to have. So they want to still be in charge, they want to have their budget, they want to see how things are done. And that’s a big challenge, and I can also fully understand these people… But sometimes they are completely in the way, or they are also attacking an Agile transformation or a DevOps transformation, or they are doing stuff which makes it very difficult to bring that through. That’s a big obstacle that I see, and it is very important to bring them on board, to educate them, and to also show them how their new job looks like in the future.

So how do you succeed with that, bringing them on board? How does that even look like?

So one of the things you need to analyze is what are the goals these people have. And usually, it’s not their goals, it’s the goals of their bosses, and you need to change these goals. So when we look at an Agile transformation or a DevOps transformation, it is very crucial that it comes from the top management, and the top management has a clear vision, and also clear guidance what they want to do, and in which direction they want to go, and they need to change the goals of these people. Only by doing that you can change how these people are behaving.

Okay. And what about the type of person that doesn’t want to change? What do you do then?

This is a very difficult case, when you have that. The only thing you can do in that case is try to educate, try to convince, but if you can’t, he/she potentially needs to leave the company.

[24:02] I see. Okay. So coming back to the Agile transformation or the DevOps transformation that you mentioned about - the first example that you gave us, of what good looks like in practice, is when you connect the value that the team builds, the customer(s) they build it for, and have the customer not even verify, but make sure the value is what they expect it to be. So that’s what good looks like. So is there more to it, or is this basically the core of what you’re referring to when you say Agile and DevOps transformation.

It’s more. One of the most important things that I always say is what you usually have is you have bright ideas. The business has ideas, the customer has bright ideas… And usually, you have a lot of these ideas. And what you want to do is you want to transform these ideas into value - value for the customer, value for the company.

So behind an idea, there is always a hypothesis, and you need to identify this hypothesis which is behind this idea. For example, a hypothesis can be when we bring this feature or this shiny, new mobile app, then we can have 10% more turnover. So that could be the hypothesis behind that. And now, the important thing is to find out what is the minimal thing we need to do to prove this hypothesis. This is a very important thing to do, because with that you can reduce the batch sizes which you have. So by analyzing what is the minimal thing, you identify the minimum viable product. And you need to also identify what are the leading indicators which indicate us that we are on the right track, and that this hypothesis is true, and we should invest more money into that. And by doing that, you can reduce massively the batch size and also the amount of work which is going through your value stream. And that’s what you really need to do, you need to do less, but you need to do the things which you’re doing in the right way. And by having these hypotheses and identifying them, and also having an evaluation on “Is it the right thing which we are doing?” and early also stop doing things, you can massively change the things you are doing, and you can create more value for the customer.

The way I understand that is ship less, more often, and check if it works.

True. Absolutely.

That’s how I’d summarize it.

Yeah, perfect.

And in that case, you want to optimize the shipping cycle as much as you can. If it takes a day, try to go for an hour. If it takes an hour, try to go for a minute… No, that doesn’t work. I don’t think you can ship it in a minute. Maybe if you have a function, maybe… The point being, go as quickly as you can, but still it should feel like a comfortable pace. You shouldn’t feel like you’re rushing things out.

The scientific method is really important. Everything that you think is an assumption. And by the way, it’s most likely wrong. But that’s okay, because the quicker you can iterate on that, the quicker you’ll figure out what right looks like. And once you know one right, then you’ll have two, three, and before you know it, you have a set of things which work well together, and that’s the value that I see.

[27:49] Absolutely. One thing that I also find important - you said “Go quicker.” Yes, this is right, but you also need to recognize when it is enough. And I think this is also quite an important thing. You don’t need to chase a Google, Netflix, or so. It can be perfectly fine in your context to, for example, ship every day or every week, or so. You don’t need to deploy to production every second, like Amazon is doing it. So I think there is always a sweet spot, and you need to identify this sweet spot.

Yeah. In my mind, any code, any feature that is built and it’s not out there, it’s inventory. And we all know that zero inventory is the best type of inventory. So whatever you have, just make sure it’s out there; make sure people can start using it. Even if it’s not complete, it doesn’t really matter. Does it look right? And if it looks right, you have the confidence, “Okay, I’m walking in the right direction. Let’s just keep adding on top of it.” But if you can verify those assumptions as early as possible, the chances of you going terribly wrong are much less. You’re less likely to go terribly wrong if you have that approach. You will still go wrong, but that’s okay, as long as you can do those small-course adjustments. It’s like driving on a motorway, right? You do a little bit of left, a little bit of right, and if you have an autopilot - because we were talking about your Tesla - you can see those very small steering wheel changes without you doing anything. So that’s what you want, the small course adjustments, continuous; they happen every few seconds, and it’s okay.

Absolutely. And one of the things that I usually see - in many companies it is not allowed to make failures. And this is a huge pity, because when you always need to do things right, you cannot go that fast. That’s a huge problem, and this is also a culture shift, and culture shifts take a lot of time.

Can you think of an example when making a mistake was a great thing? When making a mistake or a failure, people have learned from that failure? And if they weren’t allowed to make it in the first place, they wouldn’t have had those learnings. Can you think of such an example?

Yeah, of course. I have such an example. So in one of the projects we needed to move fast. And in order to move fast, we said, “Okay, we have already an application in place, which we’re using server-side rendering.” But that was an old technology. And we said, “Okay, we can use that, but the user experience will not be that good. But we need to move fast.”

So we made an architectural decision together with the business and said “Okay, we will use this old technology, and move on, so that we can learn.” And we were building up the user interface with that. But soon, the business said, “Yeah, it looks okay, but we want to have a better user experience.” So a user experience specialist came to the project, he designed some very nice user interfaces, and we said “We don’t know if we can implement that with this old technology.” But we tried, and we failed. We failed very hard, we had a lot of bugs… And this was the time when I went to the management and I said, “Look, management, we have the iceberg in front of us.” Now we have three possibilities. We go left, and this would mean we need to change the UI technology of the whole application to a modern UI technology. I think it was Angular in that time. Or we go right, and right is we stay with this technology which we are having, but it will not look fancy, and remove everything that we just added, all that fanciness, and it’s just user interfaces, with loading time, and so on. Or we go straight through the iceberg and we say “No, we want that with this old technology”, but it can be that we will fail very hard.

[32:14] And we had a very good discussion about that, and we said we’d take the risk, we go to a new UI technology. Of course, we made some silly estimations, which were absolutely wrong; we completely underestimated it. But in the end, it was the right decision which we did… And the thing is the following - in the beginning, we started with this old technology, and that was of course, when you look back at that decision, but it enabled us to learn very fast what we’ve really wanted, or what the customer really wanted, and we were able to see, “Oh, okay, it looks like that.” We added the whole user experience, we saw with this technology we were not able to do it, and we were able to see that we now need to change.

Of course, now you can say “Yeah, you could see that already in the beginning, and you could change that already in the beginning”, but in my opinion it was not feasible.

Yeah. That’s a really interesting story. And I’m wondering what that story would have been had the developers maybe stumbled across something like LiveView, which is server-side rendering, but it’s a modern server-side rendering which exists in the Elixir ecosystem; it’s running on the Erlang VM. Very efficient. It keeps JavaScript at a minimum, so you don’t have to end up in the Npm hell, as some call it. You can keep things simple, you can keep things server-side rendered, but it’s still fast, it’s still modern. So I’m wondering what that would have looked like with this technology. But obviously, you need to know your technology, you need to know what suits you, and you need to own it. So whatever you decide to use, you need to be confident that “I will make this work. And if it doesn’t work, I will course-correct… Because hey, I was wrong.” And that’s perfectly fine. Saying “I was wrong” – I think a lot of people are so afraid of saying they were wrong that they never admit that in the first place, and as a result they can never course-correct, and then they hit the iceberg, and then we know what happens next… Right?

Yeah. And one of the important things is you should not be afraid of the sunk costs… Because that’s always a bad thing. And you always hear that term quite a lot, “Yeah, but then we have sunk costs.” Yeah, of course you have sunk costs, but throwing more money after a bad idea or a bad solution is also a very, very bad thing.

Yeah. It’s not gonna make it better, right? The focus is on learning. The focus is not on the time spent to learn. What did you learn? Is this a good thing, and can you build on top of that? So if you switch your mindset and you think “Well, that’s okay. We know not to do that again”, and we know that that’s an area that we’re not comfortable with… And the longer you delay it, the worse it gets. We all know that, right? Just stop thinking about things like that.

Okay… Now, talking about technology, I’m wondering what role does a specific technology play in these decisions. I know that many teams get excited about something like Kubernetes, or they get excited about (as you mentioned) Angular. I’m not sure who gets excited about Angular these days, but I’m sure there are people out there which love it… Or some other JavaScript framework, and they say “No, we have to use this.” How do you deal with those types of scenarios? First of all, have you been in those types of scenarios? And if you have, how did you deal with them successfully?

I have been in these types of scenarios quite a lot. The thing is the following - what you need to do is you need to understand what the real need is, what you need to do. So getting excited about the technologies is a great thing; trying out this technology is also a great thing, but you should not do that in a huge project, trying out things.

[36:11] What I usually do is I really want to understand what exactly our need is, and what problem we are trying to solve, so what is the underlying problem we are trying to solve with this technology. And there is technology out there which perfectly fits the problem, but just looking at the technology and not knowing what problems we are trying to solve is a very bad thing. So what I do is when we have such a case, I really try to identify what the problem is… And of course, you then have different technology or different decisions you can do.

What I then do is I do sort of an analysis of the different possibilities, where I say, “Okay, this is technology Y, and this has these advantages, it solves us these problems, but it also could potentially introduce these problems. And this is the other technology which we are having”, and then you have something which you can compare and which you also can say “Okay, should we go into this direction, or should we go more in this direction?” And after that, I usually also do some prototypes on these technologies, to get my hands dirty on that, so that I can see “Does it really work, or does it not work?”

Who decides which technology should be used? Do you let the developers decide, the ones doing the work? Or do you let the architect decide? Or the management? How does that look like?

In my opinion, it should always be the decision of the team. The team which needs to work with this technology, they need to take the decision. Because if someone else takes the decision, the team does not stand behind this decision. So that’s why I usually want that the team takes the decision, and also does the analysis, and everything. So they sort of need to come up with the idea, and also with the decision. Of course, there are companies out there where this is not really possible; then I also try to do that, but then I try to convince, for example, the central architecture or the management about this solution which we should do.

Break: [38:51]

I know, Romano, that you have a YouTube channel, which is growing in popularity. I’ve seen some really good videos. And I’ve checked today, and your most popular video to date is what are the DevOps trends that you have seen in 2021. I think “2021 DevOps Trends”, something like that. I forget the exact title, but it was DevOps trends for 2021. So why do you think that video is so popular?

Actually, I really don’t know why it is so popular, but I made some analysis and it looks like people are googling this title, so they want to know what the trends are.

So what are they? Can you tell us what they are?

Yeah, of course. So the trends which I brought up in 2021 was it’s all about automation; so we need to automate more, so that’s one of the trends that are pointed out. Security, so the whole dev sec ops - that was a huge one. And AI ops, that was also one of the trends that I pointed out. So we have a lot of data, and we need to deal with this data, so AI is a very good match for that, and these are the things that I see are coming up.

When I look back to the statements that I did, I think I was absolutely right with these trends. For example, when I look at AI ops, this is something that’s coming, quite huge. I also started using AI ops in some areas, and the results are really amazing.

First of all, what is AI ops, and second of all, how do you make use of that? What does that look like in practice for you?

So AI ops - as I said, usually you log out quite a lot of data. So you have a lot of log statements. And when you have a distributed system, you have distributed log files. So first of all, you need to put that all together into one logging system which you can have. But then you have a lot of logging statements in there, and it’s impossible to really see where problems are, or where trends are. And here come AI ops into play, because AI ops can do pattern-matching. There’s a ton of tools out there - I don’t want to do advertising here, but…

What do you use? That’s something–

For example, I use quite a lot Dynatrace. And I’m a huge fan of Dynatrace, because we have some very difficult projects out there, and we were chasing some performance problems, and also some problems where suddenly something didn’t work, and we weren’t finding it. And also with log file analysis, we were not finding it. But by using Dynatrace, Dynatrace was able in minutes to point us to the correct server where the problem was, and it was just a configuration problem on that server. We were like, “Whoa, how did that go?” And that’s quite amazing, how good these AI ops systems are already.

[44:13] Okay. Anything other than Dynatrace that you’ve used and you’ve liked?

I’ve also used Datadog, I like that also. Beside of that, no, I cannot –

Okay. So Datadog - were you using it in the same way, in that you were shipping logs to Datadog, and then Datadog figured out what was going on in the system based on the logs?

Exactly. Yeah.

Interesting. Okay. We talked about AI ops… Now, in automation, what tools do you find yourself reaching out for when you’re automating things? What is in your toolbox, or what do you find maybe that your team likes to use?

What we quite often use when it comes to deployment automation, we use quite a lot Octopus Deploy, of course… And when it comes to CI/CD pipelines, then of course we use Jenkins, but also TeamCity, ASH Devops is also a huge thing, and of course, GitHub and GitLab.

Okay. And another category was dev sec ops; I think I would call it like supply chain security… What tools do you use for securing the supply chain?

We need to understand that there are different aspects of security when we talk about dev sec ops. One thing is the application security. So when we do continuous integration and our continuous integration server is compiling our source code, we do static code analysis. There, for example, we use of course SonarQube, Checkmarx is also one of the things you can use, and of course, there are also other tools… I think there is [46:07] tool, but I don’t know the name anymore. A ton of tools are out there to do just static code analysis.

What you also need to do is you not only need to analyze your code, you also need to analyze the libraries, and the libraries of the libraries of the libraries…

Oh, yes… That’s a big one.

Exactly, that’s a big one. And you need to identify these vulnerabilities there. And there I use usually WhiteSource to do that, which is also quite good because you also get the information about the licensing, which is also a difficult thing.

Oh, yes. That’s a big one. You’re right - once you enter the enterprise world… You don’t even think about these things as a startup, but when you go in the enterprise, this is a big-ticket item; really, very important.

Exactly, exactly. And the second thing you need to think of is of course when you are in production. First of all, what you need to do is monitor your system, and therefore you need to have these enterprise security monitoring systems. There is also a ton of products out there, but usually what you use is Splunk. You configure quite a good alerting together with the security experts, so that you get alerted about any security vulnerabilities.

That’s interesting. So we have heard about the DevOps trends for 2021, and you gave us some great examples, some tools that you use in various spaces. I’m wondering, first of all, will you create a video for 2022?

Sure, of course. I’m currently preparing it. I’m gathering all of the trends that I see at the moment, and at the end of the year I will create that video and publish it.

Okay. Can you give us a couple of hints as to what you’re thinking about? Again, this is a draft, this is not the finished version, but a few things that you’re thinking for this video.

[47:59] Yeah. So first of all, what I will do is I will look back to what I said in my ‘21 video… I will have a look at that and I will say what kind of trends I see in the future. One of the huge trends that I see is hyper-automation. So it’s not only about automating stuff, it’s about automating nearly everything. So this is a huge trend that I’m seeing coming.

With the hyper-automation there is also another thing coming - you get a lot of data out of that, and you need to monitor that, and then you have again that big data problem, and again, AI ops comes into play, because with all of that automation you also need to maintain that, and you need to operate that. So your topic, observability, will be quite a huge thing.

Interesting. So hyper-automation - that is a great title. I’m sure that we could do an episode just on that - what it is, why is it important, what elements do you see in that… Interesting; okay, that’s a great idea, I think. Let’s run it by the product team, I think…? Because you were talking about ideas, everybody has one, so how do you figure out whether the ideas – how do you formulate a hypothesis? So maybe if anyone listening to this can tell us if there’s something they’re excited about; we can connect them to the end users, to the ones listening, and if they would want for us to do an episode on that. I’m excited.

What the next thing is, which we’ll have - this is the whole cyber-resilience topic. Of course, on one side we have that dev sec ops thing, so we bring security into the whole DevOps cycle, but when you look at all of the attacks that are out there on companies, I think cyber-resilience will be one of the big, big topics, and I think together with dev sec ops, we will be able to give the companies this cyber-resilience in their application, but also in their infrastructure.

Interesting. I don’t know enough about that topic, but it’s something I would like to research, just to understand a bit better. I know all the ransomware attacks and all the cyber attacks - they’re becoming more and more prevalent, and bigger, and they affect more and more users, but I don’t know enough, like more details, other than what you just get from like afar. So I think that’s something I would like to spend a bit more time in.

Switching subjects… Because I know that one topic that was top of your mind recently was how to allocate budget. And I forget the exact phrase that you used… It was a really good one. Let me check that, actually… Or you can tell me what it is.

Sure. It’s participatory budgeting.

Okay, what is that?

[laughs]

What is participatory budgeting?

Exactly. So participatory budgeting is a thing you can do to allocate budget. So what is one of the big problems that we have when allocating a budget? Usually, you have people who want to do stuff. And on the other side, you have people who have the budget, and who say where, which kind, and which amount of budget it gets.

The problem is that the people who have the budget don’t really know what exactly the impact is, [51:47] people who want to do something really has. And that’s a huge problem. What you usually get is the people who have the budget will just say “Yeah, we divide everything apart, and everybody gets the same amount”, and then everybody is sort of happy.

[52:05] That’s quite a bad thing you usually have. The better thing is to have that participatory budgeting. That’s an event, and in this event everybody who wants to have a budget and is part of a value stream comes together, and they get allocated the budget. They sit at a table, they get the budget pot, and then they on the table need to pitch for their budget. And then they have together - participatory - a discussion on “In which area are we going to invest the money?” And that’s a very, very good thing, because then the people are discussing about impact on value, and how much value this topic brings, and especially when you have, of course, OKR or a strategy, they are also coming up with the strategy and are saying “Hey, look, this initiative buys more into the strategy than the other one.”

So there is that entrepreneurial thinking which is coming up, and they start to think like it is their own enterprise, and they are more emotionally attached, and in the end you get a better budget.

That was a great summary; I know that you gave a whole talk on this… And based on that summary, I’m going to watch it. So thank you for that. [laughter] Great. So we are just about to wrap up. I have one last, very important question. What is the most important takeaway for our listeners from our conversation? What would you like them to remember?

A very good question. I would say don’t be afraid to take decisions; don’t be afraid to make a bad decision. Just constantly learn and react, and constantly adapt.

I love that. That’s amazing, Romano. Thank you very much. This was a pleasure.

Thank you also.

Changelog

Our transcripts are open source on GitHub. Improvements are welcome. 💚

0:00 / 0:00