Launching Dagger

So this is a story about launching Dagger, and it doesn’t often happen that I’m part of a group, I’m part of a company that excites me and thrills me as much as Dagger has. It’s been the biggest launch for me, it’s been one of the best experiences ever, and this has been weeks and weeks, and I think surely it’s going to die down; surely, something is going to happen and I won’t feel as great again. Then we find another level. And it’s been like this for months. The adrenaline - I don’t think I can take it anymore.

Anyways, today we are joined by all four Dagger founders. I’ll start in alphabetical order: Andrea, Eric, Sam and Solomon. It’s so great to have you on Ship it. Welcome!

Thank you.

Thank you.

Thank you.

So we are all here to tell about the story about shipping Dagger into space, and there will be a diagram in the show notes which shows why I say that. The vertical bar - that was an amazing screenshot. [laughter] So let’s go straight into it. What did the Dagger launch mean for you? And anyone can take it away. What does it represent? Because it’s so much more than just putting Dagger out there. Who would like to start?

I’m just glad we’ve finally launched. [laughs]

Okay… It’s done, finished…

Finally.

Why do you say you’re glad it’s finally launched?

[04:02] We just worked on it for a long time. We had our system where we had people testing it privately etc. but it’s different when everyone can see it and you’re out there in the world. So it was scary, and we were not finished, but now that it’s done, we get to keep doing it in the open, keep improving it in the open, and it feels great.

Sam, you were saying something earlier?

Yeah, I think for me it’s related to what Solomon said; it’s something that we’ve thought about and we’ve been working on for a long time. On one side it’s the result of a lot of work, and a lot of experiments, and asking a lot of questions to people out there, trying to find the best possible thing we could bring them, and on the other side it’s the beginning of everything. So it’s not exactly a reason, it’s basically the first milestone; so the rest needs to happen.

Yeah. So when you say a lot of work, Andrea, you understand it so, so well. What did that mean for you, a lot of work? can you quantify it for us?

It’s been a long time in the making. Back in the day we had an alpha one, alpha two, then to beta, then back to alpha, and then beta again, and finally, the public launch. It’s been a few years doing that, but we learned a lot, we did many iterations… We did at least 2-3 private launches until we got it right, so we’re finally happy to be publicly launched. And as Sam said, it’s only the beginning, but it feels great to get external feedback. It’s not just us anymore, it’s a lot of people, and inbound feedback on what we should be doing next, which is great.

Yeah. Eric, I know that you wanna say something.

Absolutely, yes. Technically, I’m not an engineer; realistically, I’m not a user. I’ve seen this launch like the beginning of Dagger, and with all this excitement that I’ve seen before in the past, like in 2013, when we pivoted at Docker.

I think that’s something that many don’t know, that this – I don’t wanna say like idea, but the whole excitement that the world has been on with containers has started with Docker, and you have been part of that. So when Docker launched, you were there, and you have that experience. And now Dagger launched. I’m curious, how do the two compare? How is this launch different from the Docker launch?

It feels like a refined version to me… We spent a lot of time talking about our experience at dotCloud and Docker; we were dotCloud, and then we pivoted to being Docker. And we went through both of those things. We went through the launch of Docker, but before that, we went through the long periods before launching Docker. We were already working in Docker, but we didn’t know it yet.

For me, it was the defining experience in my career. We talked about a lot what would we do different, and “Oh, that worked really well. Let’s do it again, even better.” Basically, one or the other. So it definitely has echoes, but it’s cool to get a second chance at getting it better the second time. And you wonder “Okay, how much of all that was luck?” I guess we’ll find out. [laughs]

There’s that element of “We don’t know what happens, even if we’ve done something similar before.” Similar people, with all this experience, doing it all over again, in a new setting; the world is very different today than it was in 2013, so we still don’t know, and you still don’t know. You’re like, “I’m going to try this. I’m going to put this out in the world and see what happens. I’m going to work on this idea for a couple of years”, and I think that’s again something that a lot of people miss… This has been refined, and there have been mini-launches, and mini, like, shipping this, for a long, long time, until the point came when Solomon said “I’m so glad it’s finally out there.” This has been in the making for a long, long time.

Okay, so now that Dagger is out, what happens next? Is it done?

[08:12] Well, as I was saying before, it’s just the beginning. There is a long list of things that we’re going to discuss actually within the community in terms of improvements. One quick note about your previous question, about how different it was at Docker… I think an interesting thing - and it’s related to your question right now; you’ll see. We started to do a lot of experiments and do a couple of releases in the private community. So we started to experiment a lot of things, and getting feedback, and helping users already before opening the repo publicly, and I think that was a difference with Docker, that helped us figuring out what matters the most for users. And now we need to do that more, but with more people. And so we need to prioritize, we need to work on the right thing, we need to make it inclusive enough so everybody can participate from the community… So there is some work that we’ve done all together here to make it easier for people to contribute. That’s very important.

Yeah, that’s something that was really fun in the early days of Docker… It was very welcoming, and open source a lot of times is not welcoming; it’s intimidating. You feel like “Oh, it’s a scene. I’m not part of that scene.” Even if you’re an experienced engineer sometimes, but it’s even worse for beginners. It’s scary. I like that we’re making an effort to be welcoming to everyone, even if it’s your very first open source contribution ever.

I think a lot of people made their first open source contribution in the early days of Docker, and that was deliberate; it took a lot of work to make it that way, and we have the benefit of that experience. Then it became different - it was larger, more serious, also more contentious, because standardization started taking place… And you had businesses arguing over who was gonna make money with it, instead of individuals just having fun together. That still happened, but it was kind of overshadowed. So this time we get to do the same thing, but try to keep the focus on that.

I think a lot of people are gonna make their first contributions on Dagger. That would be really nice. That would make me happy.

We have seen a lot coming already. It’s only been a couple of days and we’ve already seen so many great contributions… And it’s not just code. It’s people just showing up, coming to say hi, talking to us, asking questions like “How does this even work? It sounds interesting. I wanna know more. Can you help?” And then people start to talk amongst themselves on Discord, or on GitHub issues. That’s so great to see. How do you do that? That sounds almost like secret sauce. Is there a secret ingredient to that? How do you make that happen? That sounds so important.

I think we got this question a couple of times, actually. I remember the days at Docker, after the –

Don’t tell them the secret ingredient! [laughter]

Yeah, that was pretty much the conversation. What’s the secret? Like, “Oh, we have this product, we need to bootstrap an open source community. What are the three steps you can follow to bootstrap an open source community?” And there is no simple answer, there is no secret sauce. I think it starts with a couple of opinions, but honestly, it’s not worth much, because it’s very – I don’t think there is a recipe, but I think it should start with a product that people need. That’s the basics. And then building this product in a way that can welcome contributions.

Open source is not just making the code available out there; open source is really building the product, talking about design discussions, arguing about GitHub issues, or anything; all of that happening in the open. I would say most companies that I’ve seen in the past asking these questions while we were at Docker was really this mindset of a closed source product on one side and a small, tiny project on the side saying “Oh yeah, that’s our open source community.”

[12:18] The way we started to work from the beginning with the Dagger team and the community was blended together, and this is what I think helps bootstrapping an open source community and showing to people that their opinions and the work that they do matter a lot. Because we’re not faking it; this is how we work, we do everything in the open.

Yeah, it’s gotta be authentic.

I remember something that Solomon put back… You know, this big screen, you were writing, it was “Build something people want.”

That’s a good one.

[laughs] Yeah, that’s the Y Combinator motto. We went through Y Combinator early on with dotCloud, and again with Dagger. I won’t confuse people with the earlier name of Dagger, because that’s a lot of different names. [laughs] I mean, the whole thing about YC and its influence on us is that we’re engineers, so our starting point is technology, and we had a strong gut feeling for what is good technology, what is important technology, and what is technology that can do completely new things, but in kind of a vague sense. And then you’ve gotta connect it to a sense of product, connect it to one specific problem, for one specific group of people, and then solve that, and then maybe solve a slightly bigger problem for a slightly bigger group of people, and then keep going.

Personally, that’s what YC did for me - they gave me a new frame. And again, as a team, they gave us this frame, I think. We were better at it the second time, but still, it was useful. And the way you figure out what people want is you just talk to them, and you listen… Which is what we’ve spent almost a year doing.

Yeah. And I think shipping it is such an important ingredient. You have to keep showing them and asking “Is this what you meant? Is this what you want? Is it still what you want? Do you want something else?” And the more often you do that, while it’s an essential ingredient, it’s working with everything else that happens before and after that. Because it’s just like a very small point in time; same thing as launching it. Getting to launch is such an important moment, but it’s just like a small blip on the big time horizon. Essential, but tiny.

Yeah. And it started a long time before launching. This secret sauce - we started very early on to have all the design questions in the open. We try also to inform new team members every time, like “Hey, no, you shouldn’t have the discussion in a team private channel. This should be public”, and steering everyone into public discussions, and public GitHub discussions. I think that really helps.

With the launch, you were asking what’s the secret… There’s really people contributing, and stuff like that; it’s because we didn’t start last week. It actually started a long time ago, to involve people, have discussions ready, having issues ready for the community to pick up… So we’ve put a lot of effort before the launch.

Yeah. We also decoupled it to not be a product thing. We launched as soon as we thought we were launchable. All the signs were green, or green enough, and so we launched it, just to be kind of done with it. We decoupled it from validation in the market, we decoupled it from product readiness… Like you said, it’s a step you go through.

I mean, you know, you get questions, and people won’t get it and they’ll say “Well, I don’t understand who would ever need this”, because it’s a big world, and lots of little audiences… So if you’re not in our audience, you might not get it.

Personally, I have great confidence when you’re in that mode where you’ve been listening to people, and shipping something in response, and then listening again and shipping again… Dozens and dozens of times. We’ve shipped I think 60 prototypes to actual people; starting with very small groups of people, but then kind of snowballing, for almost two years.

[15:58] Everything on our website, the way we explain the problem, what we solved etc. we’re literally not quoting people, we’re paraphrasing them. They told us “This is how I use your prototype. This is how I use your prototype.” So when someone says “Oh, are you sure that’s the right positioning? Are you sure people need this?” by definition, yes. Absolute confidence, because literally, we’re just conveying the words of our users telling us – you know, they’re almost dictating what we write. So it gives us great confidence, because it’s not just from our imagination.

So to me, it sounds like there’s a lot of refinements going on, all the time, over a long period of time. And if you don’t enjoy it, you can’t fake it. This needs to be sustained, and you really need to believe in what you’re building, with whom you’re building with, because otherwise you will not be able to sustain it long-term, which is a requirement to be able to get there… Whatever “there” means, by the way.

Why do you think that the world needs Dagger? And I would love to hear what Eric has to say about it.

It’s very simple. I think when Docker was introduced people didn’t know much about containers, and it was the same for me. Now, when I try to explain Dagger to the people that are technical, same as me, I’ve found a way to explain it, but it does very simple, and people get it just right away. So just for that, to understand the problem and what we are trying to solve. That’s why I think it’s absolutely essential that you can in one sentence describe what Dagger does, and then people, whomever they are, they can get it.

I think we’re ready to hear it, Eric,

Thank you very much. [laughter] So I’ll describe it this way, with my own words, obviously… So software people are developing what they call a program, and when they want to send this program for everyone to be able to use it - for example send them to the cloud, which is called deploying, then the other different way to make that happen, which is very complicated, from one company to another… It’s like different steps they need to follow to make that happen. It’s like climbing a ladder, but there are several different ladders.

Well, Dagger solved this problem - from the developer to the cloud, it gives a way to ship your program directly to the cloud in just one single step, that you can replicate anywhere, on whichever cloud you want.

[20:11] That is a very essential explanation, which I think the majority will just get straight away. It’s really complicated, even though maybe you don’t feel the pain, because you don’t deal with application deployments or code deployments. We all know that it’s hard stuff, instinctively, because of how many clouds there are, how many ways there are of doing this… And everyone does it differently. Do they really have to do it differently? And some say “Of course we do, because our environment is different.” But what if we standardized those things, just as we standardized containers? You know, the code that you used to SSH into things, and do things, or build VM images… I mean, people couldn’t even imagine what a world with containers would look like, and here we are today. I think the same is happening with Dagger. Andrea, I know that you’re ready.

That was a great explanation, actually. I would totally watch a YouTube channel with Eric explaining [unintelligible 00:21:01.12] technology. [laughter]

To all the VCs and everyone that wants to invest. I can totally see that. [laughs]

Yeah, I think Eric’s explanation really captured the essence of what we’re doing. Everyone is doing more or less the same thing, but in their own little way. And we’ve seen that with a ton of companies, they started the same way; they started as just only a few people, they do a few scripts here and there to build and deploy and do that kind of stuff, and eventually, if the company continues to grow, that becomes an entire team. And now they have a ton of engineers working on the problem.

It was interesting for us, because talking to different companies, we have seen them evolve in the same way, doing the same thing, but with a different internal product. And I think there’s a lot that can be reused and standardized. There’s so many ways to build a React application. You don’t need 12 different ways. You could have one component and just reuse it across, first of all, your repositories, but also across different projects. They’re doing the same thing.

And I guess I’m saying exactly what Eric said, but with a few more technical details… But I think that’s the essence of it, just being able to reuse components, be able to – also what Docker did, before Docker. It works on my laptop, but then I deploy it and it doesn’t work anymore. Nowadays, that problem is solved, but we have the new problem, like “Hey, the test passed on my laptop”, but then you open a PR and CI and then it fails, and then you push a few commits to try and fix it… We’re trying also to give you the same experience locally and on the CI. Exactly like containers did, from production to laptop. So that’s the other part.

Who wants to add the next refinement to that? There’s so many perspectives here, and they’re all valid.

It’s like – what’s that game? Jenga? You remove one thing…

Jenga, that’s it.

Keep refining…

No, no, this is different… [laughs]

Until the whole things…

Okay, now we’re going in the wrong direction, I think… [laughter] That’s the opposite of what we’re building with Dagger.

[laughs]

That’s a good one.

I would just add to what Andrea said that people working on a CI/CD pipeline, either for just linting code, all the way to deploying very complex microservices to Kubernetes - the people in charge of building this kind of pipeline are usually not equipped with anything. You have a lot of different tools that are very powerful on their own, but including everything together, and making everything consistent - that consistency usually happens in Bash. [laughs] It’s kind of a caricature, but not too much… And so we want that to be expressed in Dagger; at least that’s what we provide with Dagger. And it can be very simple; again, we have very simple configurations… Just calling a linter, or just examples like just building a binary, all the way to generating a Kubernetes manifest.

[24:17] So yeah, it’s really about bringing real development tools to DevOps people who really right now just have the feeling to just be able to do scripts, and they should not just have that.

Yeah, that is a good one. Solomon, do you wanna pull the next piece and see what happens?

Ha-ha! Yeah, I mean - the comparison to containers is good, because it’s about consistency, and “Hey, it works on my machine; why doesn’t it work over there?” But it’s applied to a different area. Instead of applying it to the application or applying it to the pipelines that ship the application.

We’re doing it because we spoke to people, and they complained that it was a terrible experience, so we got pulled into that. But then it made sense to also… It just kind of makes sense when you look at the current state – the state of the art, I guess. Because there’s so many different ways to organize an application. I mean, there’s literally software for everything. So it’s very hard to standardize everything about your application, to force people to do it all the same way.

So there’s a lot of component reuse, there’s a lot of standardization, but it’s very fragmented, because there’s so many different kinds of software. The pipelines are pipelines. So instead of a degree – it’s one degree removed from the application itself, so if you look at a banking software and a video game and whatever is monitoring your robot vacuum cleaner… Incredibly different software, different architecture, different languages, different everything. Different people even, different specialties, but one level removed, they all need pipelines to deploy that stuff to the cloud. And the pipelines - they’re different, but they kind of look the same architecturally. So that’s kind of where we’re focusing. It’s an opportunity to standardize and just waste less time repeating yourself, and we just think you can standardize more of it… Because there’s less of an excuse to not do it the same way as everyone else.

There was always an excuse to not use Docker. Or use it differently. So Docker and containers actually – you know, they ended up being a very widely adopted technology, and they still are… But it was never 100%, and it never will be. There’s lots and lots of software that don’t run in containers. And within containers then there’s all the competing ways to run containers, and thank God at least we standardized the base image, but we didn’t go much further than that.

There’s images, and there’s pods, and then there’s of course all the orchestration stuff, and all the – then there’s Kubernetes, and then there’s all the different fragmented ways of doing Kubernetes… So it’s like a fractal, and it’s only gonna get worse. So there’s limited reach. You can help, but you can’t help through ubiquity. So in a way, the original vision of Docker in that way was not realized, because we cannot standardize as widely as we hoped… But with pipelines I think we really can. Because like Andrea said, how many different ways of building a React app do you need? How many different JavaScript frameworks do you need? Yeah, a lot. Yeah, sure. That’s normal. But once you’ve picked one, how many different ways do you build it and integrate it into a pipeline? Hopefully not hundreds, because that’s too many. Anyway…

[27:45] For me, just the way Docker changed the way we come together when it comes to bundling our software and getting it out there, I see the same principles repeating themselves with Dagger, in that we get together when it comes to shipping our software. So it’s such a fundamental act of getting it out there, literally. Putting it out there and see what happens. Putting our software out there and see “Does it work?” Everyone has to do it. And we do it in so many complicated ways, because code - okay, we had a couple of packages, and there were some repositories which we had before, like package repositories, there were a couple of operating systems… I mean, the variability was lower than what we have with pipelines today. You can literally deploy anything anywhere, and it’s so hard. Everyone keeps complaining about the same things, and everyone keeps repeating it… So where do we put that common knowledge? How do we share it? How do we even talk about it?

Dagger is combining this know-how of how to combine these common ways of doing things, like deploying a React app, or building a React app, and putting it in the universe that we all can share. That’s so amazing. I love that.

Yeah, that’s the most important part… Because you need to be able to reuse someone else’s work, and also reuse your own work from last month. And until you have that, then fragmentation will continue. I mean, that’s where containers couldn’t get us all the way, because they’re binary artifacts, so you don’t wanna reuse other people’s binary artifacts. It’s not enough composition. So you need a composition model; that’s where we talk about Lego. That’s the one tough engineering and designing problem that we’re trying to solve. We think we’re solving it, but you know, it takes a lot of work. It’s binary - either we really solve it, and the DevOps community looks at this and says “Wow. Yeah, I can actually encode my expertise into this software component and then someone else can use it”, and vice-versa. It’s actually true, for everything I’m doing, or it’s not true, because the design doesn’t work, and then it goes in the pile of all the other attempts, which there are many. But I think that’s the fundamental challenge for us.

I’m wondering, how has your opinion of Dagger changed since - let’s take October, when we recorded episode 23. And this is a question for all of you. In the last five months, six months, has something changed the way you think about Dagger based on all the feedback that we’ve had, based on all the refinements, based on everything that we’ve learned in the last six months?

There was this major – you know, everything we did before October I think was very iterative… Except [unintelligible 00:30:30.08] a lot of rewrites, a lot of tests, a lot of experiments… But then I think in October we reached a point where we thought we were close to launching, but we got a lot of feedback from the community about the – I would describe it like the ergonomy to write a Dagger plan. There was something – whether we had too many concepts, or it was not very intuitive to start or to understand what Dagger was doing… So we had this Europa project that Richard Jones (a member of the team) led, with the community also, and everyone was involved in discussing that… So I think at some point we realized that we needed – this design was so good and so big of an improvement for the developer experience that we thought it justifies launching at some point, yes, but launching not before this thing is ready. That I think drove the launch date in some way. In some way, because as you see, there is everything else, but…

Yeah, without that we were not launchable.

I remember trying Dagger very soon after October and thinking “Hm, could this be a makefile alternative?” and it wasn’t. But what it was - it was the CI/CD that can run locally, and we could migrate the whole Changelog CI from CircleCI to GitHub Actions. So it definitely validated that that works.

[32:04] But you’re right, when it comes to the interface, it was like this very difficult to understand thing of what it can do. It wasn’t very straightforward, “Okay, so how do I approach this thing?” It was like, “Well, you can approach it any way you want.” And while that’s still true today, when it comes to getting started I need something simpler. I want to think that some of our feedback made it through, and then we said “Okay, we have to do a proper makefile alternative”, which I know it is now. It really is… Which in October I think the direction was there; yeah, it’s getting there.

It’s getting there. Your feedback was definitely important. I remember specifically a bunch of your questions around the container APIs; we had a bunch of different ways to run containers and build containers, and there was a lot of confusion around “Wait, so Dagger does things in containers, so I can have Dagger build and run containers through BuildKit as a built-in feature, but then separately there’s Docker…” We can talk to Docker Engine and do docker run, and that’s a very common use case. That’s completely different, but also kind of the same.

Then there’s an additional semi-official API on top that was there because the basic one had rough edges… But that was like the third thing that you can to kind of grok. I remember you had questions about that, and that was definitely influx. Because you were not the only one; that’s the thing.

Yes, exactly.

You give feedback, and then two other people give feedback, and then we start thinking, “Okay, there’s a pattern. It’s confusing.”

In October it was definitely raw. Like, “How do I structure this?” “Well, any way you like.” Or “How do X?” “Well, pick your choice.” Which was extremely powerful, but too much of a choice. You had to make too many decisions. I think since then it’s more opinionated, yet you can still do exactly the same things, but at least it gives you some structure to begin with… Like, “Here’s a blueprint that you kind of need to follow”, but then you can break out of the blueprint and just put some code on the side. But at first at least it gives you some guidance… Like the landing lights, basically.

That is an artifact of our very specific way of the design process, and the discovery process. So the early days of creating the product. There’s different ways to do it, and there’s one way that’s 99% of the times the correct way… And that’s the one we’re not doing. [laughs]

Okay…

And then there’s the way that’s 99% of the time really stupid, and that’s what we’re doing. [laughs] But it’s also what we did with Docker, and it worked really well, so that’s why we’re doing it again. Dagger is two things - it’s a tool, but it’s also an engine. And the engine is kind of inside, and it has all these capabilities… And then around it is a tool. And if you notice, all the stuff we’ve just talked about, “Oh, this API was confusing. Oh, it was [unintelligible 00:34:58.20] all of that stuff is the tool. So it’s kind of the wrapper around the engine.

So there’s two approaches. Either you start from the tool and then you iterate and you basically almost fake it; there’s almost no engine, or it’s like a really basic, made-up engine that does the bare minimum. And then you validate the need, you validate the right UX, and then, only then, you ship the absolute bare minimum tool, with the shitty engine, because otherwise it’s over-engineering. You’re investing too much in the engine, in the backend, and probably you’re gonna throw it all away, because you don’t really know it’s needed. And then over time, you’re gaining more confidence, and you re-engineer, you scale etc.

That’s the normal, correct, proper, common sense way of filling a rut. And then there’s the other way, which is that you’re actually inventing a new engine, and you’re just going a leap of faith that that engine is so much better in some way than what existed before that you just have this unstoppable gut feeling that “Okay, this engine needs to exist, and there’s a thousand different tools we’ve built on it, and we’re not sure which one, but we’ll figure it out.”

[36:17] So you start from the engine, and then basically you – I’ll just take the analogy of a car. You invent the new car engine that just blows away everything else, and then you just kind of slap together a shitty car around it, and then you ask a few people “Hey, do you wanna drive my shitty car?” Like, “Yeah, I guess… Wow, you’ve got really a lot of work to do.” And then they go inside and it goes from 0 to 1,000 miles an hour, or it flies, or something, but there is no doors. [laughs] So “I’m very confused by your dashboard design, but wow, I’m flying.” [laughter] So you say, “Okay, good, you’re flying. Okay. Yeah, don’t worry about the dashboard for now.”

So that’s kind of our approach, but it’s very dangerous, because it’s a leap of faith that your engine actually does something better. So it relies on your understanding of what people want. Like, do people want to fly? I hope so, because that’s what our engine does.

Teleport, please… Teleport me.

Teleport, yeah. That’s what Docker was. It was about containers. “Wow, you can really do a lot of cool stuff with that.” And then that started in 2008, and then we had so many versions… And eventually, it became a PaaS, like a Heroku competitor, and then we pivoted away again, from “Oh, let’s just open source the engine and figure out a new tool.” Like, invent a new car. So we’re just doing that, but we’re used to it, and also, we just did it faster. Instead of five years, I guess it only took three. [laughter]

That’s a huge improvement, by the way… And I do have to say, this is the thing that the majority misses. And I think Eric has something to do with it. So the moment Eric showed me how the company finances run on Dagger - there’s a pipeline for company finances - I was blown away. By the way, this doesn’t exist.

What?!

Exactly. [laughter] So no, seriously…

Not yet.

Exactly, not yet. So that is the leap of faith that you have to take, that these pipelines are more than just about shipping code, are more than just about shipping software. And I had a glimpse when I’ve seen you, Solomon, build the company objectives in CUE, and generate them in a bunch of formats… Because - are you telling me that now I’m able to join my documentation with my automation? And I think there’s something there. There’s something that I so want to explore, that shows you your entire automation in whichever format you want… And by the way, the documentation is built in… And that just blows my mind. The fact that that is possible - it just blows my mind.

Yeah. I think people are confused – it just takes some calibration, because usually, at this phase, after three years of prototyping, you should not be making last-minute rewrites of the whole UI, and some things should not be this rough in the interface.

And it should have some doors, right? After three years… Where are the doors, Solomon? [laughs]

Yeah, right. “Oh, we literally just changed! We’re doing doors that go like this now. But we’re launching tomorrow!” [laughter] But then the engine is – I mean, I’m not gonna say it’s perfect, but it’s way more mature, because we’ve kept it through all these different iterations… And actually, as usual, Andrea’s prototype one… [laughs] I mean, the engine is basically an evolution of Andrea’s – like, one of the early prototypes… Do you remember that, Sam? Sam and I were like, “Oh, what about this? What about that?” And Andrea said, “Oh, look - vroooom!” [laughter] We’re like, “Okay, let’s use that one.” [laughter]

That’s exactly what I imagine you did… Okay, that must have been an amazing night.

“Oh, you want some nuclear power?” “Oh, that’s pretty nice.” [laughter] “Oh, what did you work on, Solomon?” “Oh, nothing…”

[40:02] I just went on a podcast and I had great fun. [laughs]

I’m just more used to hear the noise that makes Dagger, since we know that Docker won, but not Dagger…

Then the next 30 prototypes were built on top that – I think it was called Envelope, wasn’t it?

Yeah. Yeah, yeah, yeah.

There was so many names…

Every prototype had a codename. It had to not mean anything, because otherwise – you can’t pick all the good names for the prototype, because then… You don’t wanna throw it away, because it’s such a good name, and it’s taken now. So we had the stupidest names.

Yeah, except one prototype was named Dagger. [laughter]

Exactly. What happened with that prototype? That’s what I’m wondering. “Okay, let’s just not change the name. It’s okay, it works.” [laughter]

I broke the Jenga. Sorry. [laughter] One too many.

I’m sure it fell. I can hear it. [laughs] That was a good one.

Yeah, I guess we did have one called Dagger. Well, there were so many… I’m telling you, there were more than sixty.

I like Envelope. Can we change it? Envelope sounds great.

Envelope… For a while – I mean, we shipped a real… Our first real alpha build of a product was from one of those prototypes, then we shipped a demo, and then we had actual users… You know, we had a whole year of that, I think. And that was not built on CUE, and it was not built on BuildKit. It was an in-house execution engine that was sort of like a BuildKit equivalent - again, prototyped by Andrea - and we used ACL instead of CUE. It turns out ACL - you can’t really drop in ACL the way you drop in CUE. It’s kind of like a –

HashiCorp configuration language, ACL.

Right.

Okay.

Yeah, and it’s different, because it’s very tied to one tool; it’s very tied to Terraform. There is an open source ACL repository, but it doesn’t work out of the box, at all. Every flavor of ACL is different. So we had kind of our own – that was DCL, I guess.

Yeah, DCL.

And wasn’t ACL of UCL, or – I mean, it’s a reimplementation of a reimplementation of a reimplementation…?

It may as well have been…

Tracing back to Marcel.

Yeah. One of the creators – or the creator of CUE?

The creator of CUE.

Okay.

I think it was just him for a long time, yeah.

Okay.

So one thing which really strikes me is how far in the future some of us are when it comes to Dagger. And from that future we came back into the present, and we’re building the future. Literally, that’s what’s happening. And the best way of inventing the future is by building it; someone said it, it doesn’t matter who, but the saying is important. And I think this is what the majority is missing, just how advanced this tool is, and the trajectory which it’s going on, and the possibilities it will unlock… Which was one of the things which made me really excited, as well as the people. I think that has something to do with it.

It is a good team.

These crazy, crazy people, that we have so much fun with, and it doesn’t feel like work, and it’s just… Crazy. It feels like I’m dreaming.

Well, that’s how it started in the first place, not feeling like a job; that’s how it started before the prototypes, the first thing.

I think that’s the secret sauce, I really do. You can’t fake this. All the interactions, all the individuals and how we come together - there’s no way you can fake that. And that attracts everything else. The crazy ideas, having the courage to try things out and see if they work; and if they don’t, throw them away. It’s okay. That experimentation, that courage to build an engine that no one has built before. They don’t even know that it’s an engine, and all they see is it doesn’t have doors. That’s what they get hung up on. “Where’s the docs?” [laughs] Oh, there’s like a huge, huge – yeah, okay…

“They’re already there. Oh, you mean docs for this version? Oh…” [laughter]

There’s so much there. We could do an episode just on that. The docs that we didn’t write, and why we didn’t write them.

“These are docs for helicopters…” [laughter]

“But look, it’s a convertible!”

So we talked quite a bit about this engine, and I’m wondering, Andrea, if you were to describe this engine from an engineering/software/technical perspective, how would you describe it?

Good question. It’s hard to describe in a few sentences, but the core idea is how can we work with DAGs. Typical pipelines are actually step by step, like “Do X, wait a little bit, then do Y, and then continue like that.” Here we’re trying to first of all leverage BuildKit, which is this amazing, understated project that powers Docker Build. Docker Build uses maybe like 5% at most of what BuildKit can do… So we wanted to leverage BuildKit as much as possible.

Also, on the other hand, how can we express pipelines, but as a graph. And that doesn’t make your brain explode, because brains are not meant to–

That’s an optional, paid feature. [laughter]

[48:04] Yeah, exactly. Spoiler - it’s not YAML. And if you don’t see YAML, that’s okay. It’s by design. It’s not missing.

That’s the airbags, you know…? [laughter] And yeah, the engine makes it possible to – on one end, the user-facing side, like “How do I express a DAG with dependencies without actually having to explicitly put my dependencies?” Analyze what’s going on and figure out the order of execution too, like on the backend, how can we ship this to basically BuildKit, and make sure that if stuff was already executed, don’t do it again. And if it was already executed, but with different arguments, then do it again, and handle the caching, and all that stuff.

So that’s mostly what the engine does; there’s the frontend, user-facing, and the backend, container-facing, and it’s just trying to mesh the two together in a way that makes sense.

Is BuildKit the engine?

Everyone is an engine, I guess… Dagger is an engine, and BuildKit is our engine, but containerd is BuildKit’s engine. [laughter] But runc is containerd’s engine, and then this libcontainer somewhere. And then you have cgroups… But then there’s the kernel. So I guess everyone’s got an engine at some point.

I see, I see, I see.

I think of BuildKit as a VM. A VM in the machine sense. It’s a new kind of programmable machine, and we’re building an operating system for it… Or that includes it. I don’t know exactly. But yeah, if we talk only about long-term - so ignore for a second our focus today on specific problems for specific people; to solve that problem, deployment is painful, DevOps engineers reinvent the wheel, all of that… That’s kind of an opportunity to build an engine that solves the problem better. And then we have an engine, and we have a community, and a product, and we’re launched; we’re on orbit. So that’s gonna be, I’m gonna say, a good year of work to finish that. But then you’re in orbit, and you have this community, using this engine, and that engine gets better and better.

I think you can do a lot more with Dagger if you think of it as a new kind of operating system… Because a lot of engineers I think have this feeling that the state of the art of software needs to evolve; we’re all stuck with these old paradigms and interfaces, but there’s this gravity pool of compatibility. It’s gotta be POSIX, it’s gotta be libc… You’re kind of stuck in that world. But then there’s all these distributed systems, and all these incredible capabilities now, but your software is not native to them.

So the hard part is not inventing a new operating system from scratch that’s native to distributed systems, and the cloud, and all these incredibly powerful things computers can do now. The hard part is getting developers to actually write software for it.

That’s right.

Because if you start from scratch and it’s not compatible with anything, then no one wants to be the first to write for it. And also why you; you know, I can do my own platform over there. So what’s interesting to me with Dagger is it’s a bridge. You can actually solve a problem now using Dagger by writing software for it, and the software you’re writing - it’s pipelines that deploy applications etc. But really, it’s software. And then over time you discover new uses for it, because there’s a lot of different kinds of software, I think, that can be structured as a DAG, and you have to make the effort to structure it that way. But once you have, all of a sudden it’s faster, it’s more scalable, it’s more secure… You’re basically freed from the shackles of Unix, basically. And you still get Unix compatibility, because it can run Linux containers. One, or two, or ten, or a hundred.

[52:01] But now basically we’re taking Linux and we’re making it one component into a broader platform. And a lot of people are doing that. But they’re doing it from scratch, and they’re making a frontal assault on mainstream platforms. “Oh, stop using – for your next web application, develop it on this wonderful thing!” But no one does it.

But I think once you’ve developed your pipeline with Dagger, and you start realizing, “Oh, well, in my case I could ship my objectives – I can write a program that generates a PDF exactly the way I like it… Which I would never do in any other platform, but I did it, because it took me like 30 minutes.”

And it’s a pipeline.

Yeah, it’s a DAG. So I think there’s just a lot more of those. You know, who did that really well was Amazon with Lambda. If you look at Lambda in the early days, they didn’t position it as – I mean, today you have Axel applications running on Lambda. And it’s just like explosive in growth. They’re powering a whole new chapter of the platform wars. They focused on a very specific problem, which was the little bits of glue around your regular application. “Oh, I need a handler for logs.” Or “I need to fire a quick data cleanup task every hour.” You know, just little bits of automation around the application. But then over time you get used to it, like “Wow, this is actually really powerful.” It’s powerful because it doesn’t need to be compatible with everything. But over time, you realize “Oh, I have this one use case”, then two use cases, and you grow, you grow, you grow. And I’ve mentioned automation around the application - that’s exactly what Dagger does, in a different approach.

So that’s the part that excites me, it’s actually operating system work. We’re just sort of taking the long way there, because if we don’t take the long way there, we’ll get there faster, but we’ll be alone.

Shortcuts are the long way, actually. Don’t take shortcuts. They always take longer.

What’s hard with the long way is you have to accept not being taken seriously. There’s a lot of people that are really interested in this ambitious, big platform work, but they see the website for Dagger and they say “Oh, it’s just a CI/CD tool. I don’t care about that, it’s not important. It’s a toy.” That’s actually good. That’s focus. But if you’re actually working on something much bigger from the engineering point of view, and you’re very ambitious in that way, and you know what you’re building would actually impress that person, you’re sad that they don’t see it. So you’re tempted to say “No, wait! Look! It’s actually so much more.” But you shouldn’t.

That is the definition of the innovator’s dilemma. People don’t see it until they see it.

Yeah, they’ll see it eventually.

That’s literally it.

When it’s just you, it’s all in your head. But then your team… And then you also wanna attract very talented and ambitious engineers, and talented and ambitious engineers don’t wanna work on toys. They wanna work on something big and important. So you have to also explain “Yes, we’re building this thing. We’ve just changed the doors’ color last week, and it does look like a toy, but remember, it can fly!” [laughs] It’s a mindset. You talked about staying on the road for a long time earlier, sticking to it. That’s back to that.

I’m really glad that we dug so deep into the engine and into the vision, but I’m wondering what is the fuel equivalent for this. People, money or something else? The fuel for the engine? What keeps driving the engine? Ideas? I’m looking at Eric and/or Sam.

Phew! [laughter]

This is the hard part. The easy part we covered.

Well, definitely you mentioned the people… People and money - it’s related, right? Because they want to join if the project is fun, but they also need to be paid. I would leave the money aspect to Eric, but to talk about the team - yeah, for me it’s very important. Because you also mentioned ideas. Ideas come from people; money doesn’t bring ideas.

[56:14] So yeah, definitely I think it’s the people. We talked a lot about what kind of people we want to bring on board at some point, because - I tell you, the difficulty of that was really about… We created a culture, we bootstrapped a team culture together initially, just the three of us, when we started working on some prototypes, and flying objects, and all of that, without doors… Or just doors on one side… [laughter]

I made a door!

Did you have a flying carpet prototype?

Yeah, and we had fun. To be honest, we had fun. We were like, “Okay, it doesn’t feel like a job. It feels that we can see how we’re gonna help people.” We could see where this was going to be at some point. Obviously, you know that we cannot get there alone. So at some point we need to build a team… And it was stressful, for many reasons.

We built teams at Docker before, so we had a certain experience doing it. But we decided to do it exactly like prototypes, start again from scratch every time - it’s kind of what we did with the team as well, because what we wanted was to expand these environments that we created, this culture that we created initially. We wanted to expand it to a larger group of people, without showing to everyone that founders feel special because they are founders, they were there in the beginning… How do you expand that so people can join and don’t feel that “Oh, I’m not a founder, so I cannot participate in X, decide the color of the doors…”, whatever.

So that was really what was stressful for us in the beginning. And I feel that we – I think we’re doing a good work. You could ask other people from the team and see what they think, but they are purely the fuel of the engine, that’s for sure. A hundred percent. It’s never done, it’s never achieved. It’s something that I think about every day, and that we discuss every day as well. It’s very, very, very important. The most important thing. The engine can change, we can replace it at some point; the team will stay for the long run. That’s very important.

Yes, if you want my take on that - when you mentioned the money or the people… I will say it’s the money, the people and the culture that will make this thing work. Because the culture is not only within the people, but you can feel it within the product also, because it will resonate with people. So it’s embedded in the entire Dagger history.

Yeah.

And actually, I feel like the interactions with the community – you know, there’s the team and then there’s the community, and they’re very connected. One really feels like an extension of the other. And it just kind of happened that way because of how we work. But these days what’s on my mind a lot is just strengthening that, and – you made me think of that, Eric, when you said it’s embedded in the product. It feels like it’s already been done to have a product that’s open source, and has a great community, and you use the product, and you talk about it with the community, and you kind of go back and forth… And that’s great. We were part of that with Docker, others did that, too; everyone’s pushing the envelope. We wanna push the envelope even further.

[59:38] We mentioned we’re doing even more in the open. When you join our Discord server, it doesn’t feel like you’re joining a meetup or a community. It feels like you’re literally joining our workplace. Like, that’s where we work. Sure, yeah, there’s a private channel where we exchange memes and whatever, but when we talk about developing the product as a team, we do it in the open. People are there, they listen, they participate… I feel like the next step is somehow building that into the product experience itself. There shouldn’t be a dividing line between “I’m interacting with the community” and “Oh, I’m using the product.” I think the community itself should be part of the product, in some way. I know this is kind of vague, what I’m saying, but…

No, it does make sense. And especially when it comes to paying users, and for users, there should be as little difference as possible; hopefully none, other than you just pay for Dagger because you believe in it, and you want it to succeed, and you want to be part of it. A more inclusive way. You just literally want to be part of Dagger’s success. And that’s one way that you can do that. If you derive genuine value and it saves you money…

Oh, you mean like donations?

No, I mean like paying for it. [laughs] It’s called a subscription…

Yeah, yeah. No, but you don’t wanna–

Here’s a novel idea - let me use my card to pay for it.

Just because you’re paying shouldn’t mean you’re leaving the community and now you’re in another group of people, more serious, because there’s money involved.

Exactly, exactly.

It’s like team and community… For example Helder - Helder is a member of the team who we met through the community. He joined, and he started using it, giving us feedback; really good feedback… And then feature suggestions, and the pull requests. And then, you know, he joined the team, and that was great. He didn’t leave the community when he joined the team. He just went even closer to the core of the community. It has to be the same with customers, and it did not feel that way at Docker, for sure. There was a big, big division, at least in the way we looked at it at Docker. You were either talking to someone in the community, and then that went through the community people, or you talked to the customers, and that went through the customers people. It was almost like two different companies… And that was a big mistake, because it turns out a lot of the customers - they’re actually on their free time, geeking out on the community site. There’s a lot of cross-pollination. And if you ignore that, you’re missing out the big picture.

So this to me sounds like a great takeaway, and I’m wondering if there’s a few others that we can share as we prepare to wrap up. What are our takeaways for Dagger users, present or future? And even maybe to our future selves. I really like that idea.

We are looking to expand the team even more. [laughter] If people are interested –

“We are hiring…” [laughs] “You will have fun.”

You will have fun, for sure.

Building some doors. [laughs] Because you don’t have them. That’s a good one.

We’re big believes in incremental progress. If you tried Dagger and it was missing doors, so you’ve figured you would stop using it, try again in two weeks; maybe you’ll have a door this time. I don’t know. [laughs]

That’s a good one.

Maybe not. [laughs]

No, there’s so many takeaways; I’m not sure… But yeah, it’s a team-first and it’s a product-second. That’s super-important. We’re still building the team. Also true.

Please, Solomon, correct me if I am wrong… I remember when I asked you what Dagger is about. You said, “Well, Eric, it’s very simple…” It was 4-5 months ago; because I needed a shortcut to understand what it is about. You said “You know, with Docker there were VM, and then it led to container. That was Docker. And for Dagger, you have the scripts, and then you will have the DAG, and it will be Dagger.” I’m too sure whether it’s still relevant or not, but…

That’s true. I remember that. Two by two. We should put that on the website. We have so many good ideas, and we just forget about some of them. It’s inevitable. [laughs]

Yeah. We just have to record them somewhere, put them down somewhere.

Yeah, absolutely. We fear no one.

We’re the absolute best at humility.

I think we’ve seen this play out so many times, and we always wanted to be part of a team that is genuine, first and foremost, does what they believe in, through and through… Do what they say, and vice-versa, say what they do; literally, just be honest. There’s like so many things that we managed to capture in the culture, and capture in every single interaction that we have… And this is us. And this is it. Whether you join the community call, whether you listen to this podcast, whether you open a pull request, or an issue - it’s the same approach. It doesn’t matter who it comes from in the company, present or future. And that is amazing. Andrea, anything to add?

No, I think we covered it all, and it was super-fun.

Well, I’m so happy to have this pleasure as well… I mean, we get to talk quite a lot through the week, week in, week out… But this just feels special, everyone at the same time… It rarely happens, because there’s so many things that we all attend to… This has been so much fun. I’m looking forward to another one, maybe in a year’s time, maybe six months, whatever we decide… But this has been amazing. And if any listeners want to get in touch with us, we’re on GitHub. Just go to GitHub Dagger. There will be links to Discord, to the website… However you want. Whatever works for you. And if not, that’s okay. There will be a Dagger engine flying by you before you know it. It’s just a matter of time. Just give it time. [laughs] And who knows, this may even take you into space, because that’s what we’re aiming for.

Thank you, everyone. This has been very, very nice. I really enjoyed it. Thank you all for joining us.

Thanks, Gerhard.

Thank you.

Thank you.

Thank you.