Justin Garrison: It’s been a lot of fun in the last week. We’re recording this on October 24th, so you’re all going to read this a little later, but… There’s been an influx of just people and information and stuff going on on Blue Sky recently, that has been fun, as someone - I’ve been there for a little over a year now. And it’s been quiet, but I’ve still been enjoying the architecture of it and how people are integrating the app protocol, that sort of stuff.

I wanted to reshare the white paper from that protocol on Blue Sky, because I did read this a while ago. It got updated in October. I haven’t read it yet, but they tripled the number of authors on it. One I have a copy of, and I was like, “Oh, it had three authors before. Now there’s nine. What’s going on?” So I haven’t reread it, but I think most of the content’s the same. I want a diff for white papers. That would be great.

Adam Stacoviak: I don’t know, though. I think with my idea, if it truly is a good idea, I think you could do both. It doesn’t have to be just because you’re rejected, plan B is X. I think it could be both based on what I hear. Now, this is 20 minutes of podcasting, which I haven’t dug into the white paper, or the details, and stuff like that. But I can’t see, based on what I’ve heard so far, why it couldn’t be both. Because it’s already doing that. It already can be speculated against. If I have a project and Jerod wants to stake against it, he can. So that’s all you’re doing. It’s about perception and mechanics and marketing really a story than it is simply what it can or can’t do.

Justin Garrison: That collaboration piece I think is huge, because the way that I’ve learned, throughout my career, is just like working next to someone. And back at the data center I learned so much about resiliency. It was like “Yeah, you connect that cable to the switch over there, because if this one fails, and that power supply goes away–” And like being able to “Oh, that makes sense”, because we can figure out why and how these things are going to fail.

And infrastructure as code shifted me to an individual player game, where I just did asynchronous reviews and no one actually reviewed or ran my infrastructure as code. They’re just like “Yeah, it looks good to me. You’re past the lint, so we’re good.” But the feedback loop for learning how to do it was so much harder, because I had to copy and paste that other Jenkins file, to then go do it over here, because I don’t know how to start, I don’t want to write groovy. I don’t want to write Groovy. This is just a necessary evil.

One of the things that – the last time I was excited about an infrastructure diagramming tool was probably eight years ago. I don’t even remember what the product was called, but I remember reading their white paper… And they had this notion of being able to see the history of how the infrastructure has evolved, and being able to go backwards in time and say “What has changed here, and why did it change, or who made the change?” Do you see that as a view capability inside of System Initiative? Because you have these change sets, and you have this way to diff this stuff already… Could I zoom back to six months ago and see what it was like?

Justin Garrison: But just remembering how long it took you to learn that, and like putting in the work, and reading the man pages. Same thing for learning systems. I go read the white papers. Because almost every big system came out of some research that they wrote a white paper about. And I’m like “Oh, I’m gonna go find the white paper. I’m going to read it.”

And it’s like, yeah, that’s how I’m going to spend my time. I can figure out why they wrote the thing. What answer were they trying to solve? The man pages are going to give you that stuff. Like, “Oh, these are the built-ins, because we have to have this set. We don’t want to execute out to something else, because it may not be there, and we need to build that into the shell.” And that’s where it’s something like “Oh, that’s the problem you’re solving. Now, why would I want to execute out to a different command for that same functionality?” “Okay, let’s figure it out.”

Abi Noda: Yeah. That’s the white paper.

Justin Garrison: I don’t remember if it was a white paper, or whatever… It was “How to hear into a room of people using a laser.”

Daniel Whitenack: And on that front, I’m really excited, because Chris, on this show actually a couple of times I referenced the OWASP top 10, Gen AI top 10 sort of risk white paper. It breaks down security and privacy-related risks, into sort of different categories, and helps people think about it. This is a collaborative thing, with multiple organizations involved… But today we’ve got with us Dinis Cruz, who is the founder at the Cyber Boardroom, but also has been involved in OWASP and in various capacities over the years, and is aware of all this that’s going on, and contributing to it. So we’re just super-excited to have you with us, Dinis. This is one I’ve been really looking forward to.

Justin Garrison: I mean, that’s 300 gigs on my computer to like go play with this thing and see how it works, and kind of prompt it myself. And a lot of their testing in the paper is like - the 405 billion model obviously is the best one, and it’s competitive to a lot of the closed source models in various tests. But it was fascinating just seeing how they walk through this; again, with references it’s almost a hundred pages of white paper. I don’t recommend that everyone read it, but absolutely go put this in ChatGPT and ask some questions.

Autumn Nash: That’s so cool, to read a white paper and then get to talk to you about it.

Justin Garrison: Yeah, for sure. And the last place I was gonna point out that I get papers from is acm.org. It has a bunch of journals. They run conferences that I enjoy. SIGGRAPH is a gaming animation sort of conference, and so there’s a lot of stuff that comes out of that from talks… But also, I watch a lot of talks at conferences online, on YouTube. If they’re recorded on YouTube and I think it’s interesting, I usually will find one that’s interesting at a conference, I will watch it, and then I will look at that person’s sources. Because a lot of them will be educational, or research-based, and they’ll say “Oh, we’ve built some of our foundational stuff on top of this paper.” So I’ll go find the paper, and then from that.

Same with books - if I’m reading a book, and they have a paper listed of like “Oh, we’ve got this research from here”, if I’m interested enough, and I feel like I have the time, I will go ahead and find that paper. And my flow for like finding papers - I just find the PDF, and I put it in Apple Books, and I read it on my iPad with Apple Pen. And I just add notes to it, and that’s where I typically consume them.

I used to print them out, and at lunch I would go with a sharpie marker, and I would go sit outside, leave my computer or my phone at my desk, and I would just go out there with a white paper… I don’t do it to remember them, I don’t do it to like search through them again, I’m not looking at my notes, but just sometimes I want to like jog my memory, of like “Oh, what did I like out of here?” And I’ll find some highlights, and I’m like “Oh yeah, that was the key point that I thought was interesting.”

Thank you so much for listening to this episode. If you have people you would like to have on the show, or topics you’d like to have us talk about, please email us at shipit@Changelog.com. We do have a bit of a Plus Plus thing. Autumn and I go on a tangent here, talking about not just research papers, but just in general engineering blogs, and kind of how that relates to Dev Rel, and engineering, and some thoughts we have there. So stick around for the Plus Plus content, and we will talk to you all again next week.

Benn Stancil: Sure. So like you mentioned, I started a data company; it was a BI tool, basically, that was called Mode, about 10 years ago. When we first started it, there were three of us. One of the persons was the CEO, who was a good face to the company, and out talking to investors and customers and things like that, and was the sort of person that we could probably stroll out as the external face of what we were building.

There was a person who was the technical co-founder, really, who was chained to his desk, building the product, and then there was me, who was neither of those things, who neither was an engineer, nor was sort of fit for external consumption, I would say… And so I didn’t have anything to do. Why I was a founder there - who knows? That’s a question you have to ask them. But my job – basically, my background was in data, and as an analyst and things like that, and really what I was doing was kind of representing the customer in a lot of ways, where the product we were building was for people who were like me… And so I was to some degree PM-ing things, or helping the person who was the engineer do the “Here’s what we think we should build”, and testing stuff out. But that leaves you with a lot of time, in the early days when you’re basically moving as fast as one or two engineers can build it… And so I had a lot of time to spend on stuff, and so I started basically writing a blog, that - we didn’t really have a grand plan behind it, but what it ended up being was kind of like FiveThirtyEight-style analyses of pop culture.

[08:15] We wanted to do something that would get notes and visibility… I kind of just started doing this because I needed something to do, and it was kind of entertaining to me. And so the very first blog posts we ever wrote were things about like Miley Cyrus, and the VMAs, the Video Music Awards, and various things that were just like interesting things to me going on in the world, from a data-driven perspective. So it was sort of like “Here, let’s take a data-driven look at x thing”, or whatever.

People seemed to like it. I think it worked because this was a time when content marketing and sort of having company blogs was becoming pretty normal… But most of those blogs would be sort of transparently thought leadership with the intent of somebody clicking on the button saying “Download our white paper” or “Check out our product”, or whatever. It’d be like “Here’s five tips to build your engineering team”, and then the fifth tip would be like “Use our product”, or whatever. And so this was not that. There was no real call to action for anything related to Mode, it was just “Here’s a bunch of charts about the baseball playoffs”, or whatever.

And so people seemed to like it, I enjoyed it, I thought it was kind of fun to do, and the writing part of it was something I’d never really done… But it was like “This is kind of interesting.” Eventually, within, I don’t know, six to nine months of starting doing that, Mode grew, my job expanded, I started doing a lot of [unintelligible 00:09:25.09] started having customers… I basically didn’t do it for that long, because there became a point where writing a blog about Miley Cyrus is not the most important thing that you can be doing to grow startup.

Justin Garrison: Yeah, I’m gonna link to the white paper. I’ll put it in the show notes. Actually, I’m just gonna put the Wikipedia page, because the white paper is linked in the references there, and that’s how I’ve found it. And it was fascinating as an evening of how long can you go without eating.

Autumn, I don’t need to see the book. Oh, she took [unintelligible 00:11:04.05] I believe it would exist. I don’t need to see it.

Justin Garrison: And you’ve just reminded me that I read a white paper, which was probably one of my favorite white papers, which is one of the early Software Defined Networking white papers. And I for the life of me cannot remember what the title of it was. If any of our listeners know the foundational, software-defined networking… I remember where I was when I read it. I used to print them out and go mark them up. I would read them at lunch, I’d bring a Sharpie with me, I’d write notes… And then I’d basically recycle them.

Justin Garrison: Looking back on when I first saw blockchain – like, I read the white paper from Satoshi. A long, long time ago – I was doing Bitcoin from 2009, or something like that. I had Bitcoin. I was like “This is fascinating.” Then I saw the shift of like everyone jumped in for the money side of it… I’m like “No, no, I wanted the tech side. How did that [unintelligible 01:02:18.19]

Yoav Shoham: So first, I’ll say that I don’t think that everybody needs to be building foundation models. But as I said to somebody, organizations are technical, and want to remain relevant; even if they’re not building foundation models, they should understand how they’re built. And if they really put their mind to it and the resources, they could build one… Because it really gives you a visceral, deep sense of what’s going on.

Now, regarding the Jamba, we actually try to be very transparent. So this is our first open source model, and the reason we did it was that it is very novel, and there’s lots of more experimentation to be done here, optimization, serving the – you know, training use models can’t be done on every type of infrastructure. Serving them similarly. And when you do serve them right now, we’ve had several years to optimize the serving of transformers. We want to enable the community to innovate here. And so we were quite explicit in our white paper, perhaps unusually so relative to the industry. So to the listeners who want to kind of get the nitty-gritty, I really encourage them to look at the technical whitepaper.

But I can tell you there’s been a ton of experimentation [unintelligible 00:30:41.15] that our guys did, trading off lots of – people use the term hyperparameters; a lot of things are very different from one another. But how many layers do you want? And how many Mamba layers, how many attention layers, batch sizes? …all kinds of stuff that – you know, what actually makes the difference… It’s hard to sometimes understand what makes the difference. And again, we tried to share the – for example, I said that Mamba’s performance doesn’t compete with the performance of comparably-sized transformer models. But when you look at the details, it’s actually quite competitive on many of the benchmarks. But then there are a few that it’s really bad at. And that gives you a clue of why that’s the case. It can latch on to surface formulations and syntax that the transformers managed to just abstract away from. And so we describe how you make this observation, and you correct for it. So there’s a lot of details that go into making these decisions.

And then there’s also pragmatic decisions. For example, we wanted a model that will fit on a single 80-gigabyte GPU. That was a design decision. And from that emanated a few things. We did put a bigger model, and certain context windows will fit there, others won’t… Still, 256k is humongous, compared to the alternative… But we can also do a million and larger, but not on a single GPU. And so those are some of the design decisions and the rationale. Honestly, it is a process – although condensed, a process that involved hundreds of decisions that it led to what we put out.

Justin Garrison: I miss the conference side of Twitter, which was a big piece of it for me… But thinking about it, too – because I closed my Instagram a decade ago, and I never looked back. And I’m like “I’m missing a lot of conversations there, and I just don’t have the bandwidth, I don’t know the time.” And it’s fine. And I know that for me Twitter will be the same thing, where it’s just like “I didn’t want to support Facebook either, and I don’t want to support Musk, so I’m off. And it’s cool. I have other places that I can be.” And I will miss those conversations, but again, I am very appreciative of having DMs from people, and just like having real conversations. I’m like, I’ll jump on a phone call with someone.

And Twitter Spaces were fantastic. I loved doing that. I was doing like white paper reads, and all that stuff… And I ran a community; I actually ran like three communities on Twitter when communities were a thing. I was having a lot of fun doing that stuff, and then a lot of that –

Justin Garrison: Yeah, I mean, like I said, I loved what Gerhard was doing with the show. I loved the topics that he was already covering, and some of the guests he had on. And I want to continue that as well. I want to focus on that topic space. Everything after git push. What do we do? CI/CD pipeline, security scanning, system scaling, whatever it is, all the way from observability, SRE - all of that stuff’s involved in the not-writing-code side of things. Like, how do I debug a Linux server? Those are things that not a lot of places focus on, and I want to keep that focus of the topic of just shipping the code.

[01:11:55.20] Getting some great guests on, focus on areas that are running code; if you run production code in any sort of environment, I want to hear from people, because it’s not just a web service. And in my time at Disney and Disney Animation, we had almost no web services. Even the Disney Animation website wasn’t run by us, it was run by another – we just did rendering. We would render stuff, and we had some internal services. How does that look? Why is that different? People still wrote code, and we still did stuff. And I want to know what those different environments look like for people, because some running software is not the same thing. It’s not always just an NGINX with a backend app. A lot of places look really different, and there’s so many variables in that, that I want to talk to a lot more people, and give people more exposure to what it actually looks like. If you’re in a hospital, how is that different than a streaming service? Those things are very different environments, and have different concerns, and different needs for what they’re doing with their software and infrastructure.

So that’s the first thing - I want to keep some of those people coming in. I also wanted to have some things that – I love listening to podcasts in general, and the things that I want to hear… I don’t want just a news show, but I want a couple news topics. I want to know some things that are relative, or something that the hosts, whenever I’m listening to the show, I want to learn what they learned this week. Some of my favorite shows that I’ve listened to in the past always have something that is personal, that’s like “Hey, I did this thing, I solved this problem, and now I –”, whatever. It’s like a small thing. It’s not like “Oh, everything’s groundbreaking every week.” No. I learned how to make a dashboard on my Raspberry Pi. Here’s the thing I used, here’s an open source tool, whatever it is. And so I have some recurring segments that I have ideas for to make that fun.

I’m bringing on an awesome host [unintelligible 01:13:33.12] with me, because she has such a great, different perspective, and different experience than what I have, from running services in a different sort of environment, and with different constraints. So I’m really excited about that. And then just having those guests come on and learn from them about what products exist, whether they’re open source, or SaaS products, or just different ways of thinking about scaling things.

I used to also run a Twitter Space for reading white papers on infrastructure. I called it Paper Club, and it was a monthly “Let’s read a white paper, and then just talk about it.” It was like a book club for technical white papers. And that sort of deep dive into technology, and where technology comes from, has always been fascinating to me, because I can learn a lot about how or when I should use something based on what problem it solves when someone created it. Do you want to use Raft Consensus? Maybe, maybe not. What problem did they solve when they created Raft, that something else didn’t solve? And then you can maybe make a better decision about which tool is the right one for you.

So those sorts of deep technical topics are something I also would love to bring to the show, and have people come on and talk about them. I had on one of my Spaces Eric Brewer, writer of the CAP theorem, and we were literally reviewing one of his papers. Not about CAP theorem, but about scaling like AOL services. And he joined the Space, and I was blown away that I can just have access to someone like Eric Brewer on a Twitter Space. Like, are you kidding me that? That amount of shrinking of what the internet is is fascinating to me, where it’s just like “Oh, that was what was great about Twitter in the heyday of everyone was just there a lot of times.” And he showed up, and people were discussing it. And I learned a lot. I read the paper, we were talking about it… I said “Hey, why did you do it this way?” He’s like “Oh, because of this other constraint.” We didn’t even talk about the paper. “Here, let me tell you.” “Oh, that’s great to know.” I love those conversations.

I’m looking forward to having more conversation on Ship It around those things, about “Hey, this is what we said in the blog post about the outage… But here’s the thing that we didn’t say, or the constraint that we didn’t know about at the time”, whatever it might be. Those are all areas that I would love to talk about.

Ben Cohen: And more recently, we adopted Windows as an officially-supported platform. So that was actually a community effort. So it was driven by a member of the core team who actually now works for the Browser Company, who are themselves using Swift to bring their Mac and iOS browser to Windows. And they actually recently published an article where they’re using Swift to wrap the Windows APIs. They’ve actually got a really interesting implementation of COM, which is the way that Windows interoperates with its API, that integrates really natively with Swift.

I think one of the links I sent you, that maybe people can find online, is about how they believe - and we agree - that interoperability is one of Swift’s superpowers. So one of the things that Swift can do is interoperate directly with C-based languages. So this was actually how we bootstrapped the original ecosystem for Apple devices. So on day one, you launch a language - it was there at WWDC; you could download it that day. But when you launch a language from scratch, it doesn’t have an ecosystem, except Swift did from day one, because we have this ability to interoperate directly with C, and in Apple’s SDK’s case Objective-C. So when you import an Objective-C header file into Swift, it comes in and looks and feels like a Swift library. You can create the objects, you can call methods on them as if they were Swift-native methods.

[16:20] One of the things that’s nice about the Objective-C ecosystem is that they had these really well-adopted naming conventions for their methods and their types. And that was really nice, because we were able to – Swift has also some great guidelines around how to name methods, but they weren’t the same as Objective-C. But because the Objective-C ecosystem was so consistent, we were able to do some tricks where we basically renamed the methods, so that they actually come into Swift looking what people will refer to as Swifty. They feel natural. So that was actually the way that we bootstrapped the original ecosystem.

Now, sitting on top of Objective-C meant we also had to have C interop as well. And that’s actually a really interesting opportunity on the server side, because obviously, we have folks who have written code that they felt had to be written in C. Like I was talking about right at the beginning, people who actually really, really need that low latency, high performance, they would usually pick C or C++. And I think in this day and age, unfortunately, that’s something that’s a real problem, because of the lack of safety in those languages. And the NSA, I think, about a year ago, put out a white paper urging people to start moving off of unsafe languages. And Swift was one of the safe languages that they suggested, as well as C#, Java, Rust… But we feel that Swift has two advantages in this area. One is if you’re going from C or C++, maybe you can afford to go to a managed language like Java, but maybe you can’t. And so Swift, alongside Rust, has the ability to compile natively. But Swift has the advantage, one, that we think that the high-level feel of the language pays dividends in terms of productivity when you make the shift from C++ to Swift.

We actually have been slowly rewriting the compiler ourselves in Swift; when it first came out, it was all written in C++, obviously, because you can’t self-host if you haven’t got a language yet… But we’ve been doing that migration and I was doing some work on our parser recently, and we have to do it twice at the moment, because we have the old parser and the new parser, before we swap the new one in… And it’s so much nicer to be writing in a higher-level language, that feels a lot more productive. So there’s that advantage.

The other advantage is that Swift, actually as of last release, now has C++ interoperability, as well as C. And so similar to Objective-C, C++ types come in and look like native Swift types. You can call methods on them with Dart, and things like that… And we don’t have – you often hear this term FFI, Foreign Function Interface, that a lot of other languages use to interoperate with C. So if you use Go or something like that, you have to create bindings, and then go through this FFI layer. Swift doesn’t have that. We basically use the C compiler, Clang, that’s also part of the LLVM project. It essentially is a library to bring in C API directly. And that means that we skip the FFI layer. That has an efficiency benefit, but it also means that we can really integrate those things nicely, and you don’t have to generate bindings.

Now, why is that important? The key thing there is that means that just like with apps transitioning from Objective-C to Swift, if you’ve got a big C++ server installation or library, you can do the migration essentially function by function, file by file. So it’s not a big bang rewrite… Which is normally where these kinds of initiatives go to die, right? You’re like “Oh, God, okay, we’ve got this existing installation that’s all written in C++…” What are your choices? You can either break it up into microservices, which has consequences in terms of performance, and all sorts of things like that… You’re gonna have to monitor multiple things… Or you can try and like smoosh your new language in this existing service together, and that ends up being pretty painful. With Swift, we think that it’s a much easier migration, because you can just directly interoperate with your C++ code as if it was native Swift code.

Solana Larsen: Yeah, I think front of mind, a lot of people are curious now in a way that they weren’t before. I mean, you must experience this on your podcast as well, that people now have this hunger to know about AI, where a couple of years ago they were like “Oh, what’s that? How does that concern me?” Now, everybody’s like “This really concerns me, what should happen.” And I think there are a bunch of areas where nobody is entirely sure what to do.

The first topic that we took on in episode one is around open source in large language models, this whole question where you have on the one side folks who are saying “It’s got to be open. We can’t audit the models. We don’t know what’s happening with the data.” And then on the other, you’ve got people saying that it’ll be the doom of all of us, and everything’s got to be shut down and closed for security purposes. And so you have these – I think a lot of discussion these days, it’s really polarized sometimes… And so it’s trying to figure out how do you make a nuanced argument that kind of explains not just different sides of the story, but just explaining how there’s a spectrum. And there’s a lot of AI topics that get sandwiched together just under this umbrella that’s called AI, and it’s just so many different contexts, and so many different business purposes… It almost less and less makes sense to talk about it all as one thing. But we’re right on the cusp, where we’re still talking about it as one thing and we’re still trying to grapple with how we should regulate, how we should build, how we should design, what we should think about personally… And so it’s a really exciting moment to try and figure out those things.

[10:07] And the challenge as a podcast creator is that each of our episodes is like 20 minutes long. So we pack in three, four different voices, there’s some really deep analysis… We work with our host, Bridgette Todd, who’s great… A whole bunch of people work together on this thing, and it’s like this very highly-polished/produced, lovely kind of white paper in audio almost of a big issue, a big topic. So I’m really proud of it. And last season, which was a little bit ahead of the curve in terms of talking about some of these AI issues, we actually won the Webby for Best Tech Podcast.

Mark Kurtz: The part that I’m most excited about is that generative AI space, specifically in being able to augment humans. Obviously, there are a lot of privacy concerns, and data concerns, and bias issues, and things like that in this, which I don’t want to see LLMs deployed everywhere becoming a default response for like Google search, or something like that. But it is really exciting to see, even in my day to day, starting to use these actively to augment what I’m doing around content generation, and framing, and things like that.

So that’s one piece that I’m really excited for. And with the work that we’re doing at Neural Magic, we’re especially looking at these, because one, we want to see that continue to grow in open source, and I think that’s been the other push that’s been really big, and really exciting to see, is that whenever GPT4 came out, it was completely privatized; they’ve put out a little white paper on it that had no details about it at all. A lot of data concerns and things like that within that. But the open source community has already released - and I can name probably ten models so far that have been released since then, that are ChatGPT-like, or GPT4-like. So it’s really exciting to see that.

I think the next stage from those open source models is going to be making them runnable anywhere, right? So you don’t need this big GPU cluster farm to get something that is usable. And that’s where we’re really looking at going. We’re actively working on the LLM deployment issue right now, and hope to have something out in the next few weeks, next few months that people can start actively using, download it, they can run it anywhere they want, on any CPU, and it’ll be just as fast as GPUs.

Peer Richelsen: Yeah. I mean, we had a community before we had a product. When I started, back in the day, it was called Calenzo.com. I made pretty much like a visual white paper website where you could sign up for a waitlist, and that waitlist would also take you to Slack. So pretty much like, “Here’s what I want to build - an open source, developer-first Calendly alternative. If you think this is great, join our community.” And from that day on, we’ve been nurturing our community with updates, we sometimes do Twitch streams where we live code new features, which obviously also only works for open source, and celebrate Product Hunt launches. We’ve been Product of the Day twice now with two different launches, Product of the Week, of the month, just because we have this super-powerful community. We have investors from our community who put in as little as $2,000 in our seed round.

I think it’s really hard to build a developer-first company without a community… Borderline impossible, in my opinion. And that’s also why I don’t think Calendly would be able to open source and then compete in the open source space, because open source is not just like leaked code. The Twitch codebase was leaked. That doesn’t make it an open source company. It’s the entire values and visions and theses and community engagement. So it’s a real mode it’s a real strong feedback loop for us. It’s something we will always foster and prioritize. And also, obviously, it also helps us to do the right thing, because I think once you lose touch with your customer base, you may end up doing business decisions, like charging the wrong feature, or removing the wrong feature. And we’re always kept accountable by literally 2,000 people. It’s like having a board of 2,000 customers that constantly tell you, “Hey, we like this more than this.” Sometimes you need to go against it, for - well, good reasons, or some other reasons… But in general, it’s a really good guiding system to make the best decision.

Matan Peled: But on the other hand, code is very structured, it’s very hierarchical, it has properties… In order to compile, it has to be very strict in various ways. So giving up all that information, all that context is silly. You do wanna use it, and the (let’s call it) non-machine learning approach to static analysis, to dealing with code is called formal methods, which is basically taking ideas from logic and those sort of areas of math, and applying them to code. And that’s where all the things like type checking and that come from, all the theory behind it.

I don’t understand 100% how Copilot works. I’ve read their white paper, it’s very interesting… I don’t think that the – on the one hand, one of the points of machine learning is that they don’t do anything specific, they don’t say “Oh look, there’s a type.” They want the machine learning to somehow learn that themselves…

Mitchell Cohen: I don’t think any of it is fully replicated across anywhere else I’ve seen. We do document it, it’s in our white paper, so people are free to take a look at our whole architecture… But we’re really pioneering in so many areas here, especially over the past years, with our sharing features. And honestly, where we’ll be a year from now - and we’re gonna get to that later - is even more exciting than what we’re able to talk about now.

Adam Stacoviak: Cool. Evan, thanks for the deep-dive into all things Fauna. We really appreciate these technical deep-dives. Going back to the white paper, Dr. Abadi that you’ve mentioned as a board member for you… We’ll link up the blog post that we’ve kind of referenced to some degree in this call here, in our show notes. The Trust page, of course, and any other links we can think of that make sense… But Evan, thank you for your time. It’s been awesome.

Luis Villa: [laughs] Yeah, you can still be a jerk. And I certainly think – GitHub talked in that white paper that I mentioned earlier that they are implementing… I don’t know where this is out; I don’t know if it’s rolled out or anything, but they mention that they’re gonna try to implement some kind of “By the way, it looks like this probably is not original. It probably came from this.” Putting aside whether or not that’s legally necessary, you know, in terms of not being a jerk - like, hurray. GitHub should not be jerks. They’re an 800-pound gorilla, and I think maybe in their roll-out of this maybe one of the things here is they didn’t reckon with the emotional – you know, the haft that they carry.

They’ve been really good… I’m not a Microsoft apologist. I literally got into open source in part because I was convinced that Microsoft was evil. I’m still personally irritated at the Bill Gates image rehabilitation campaign… The guy has all this money to give to charity because he operated an abusive monopoly. That’s why he has so much money. So it’s nice that he gives it away, but let’s not forget that first part.

[48:05] So I’m not a Microsoft apologist, but I think GitHub and Microsoft in the past few years have mostly done really well by open source… So I think maybe they got a little laurel resting, a little too comfortable here, and didn’t fully think through how much this would really emotionally piss people off, even if the lawyers gave a full thumbs up.

Luis Villa: [laughs] Yeah, my little guy’s in camp, so… Yeah, you can still be a jerk, right? And I certainly think – GitHub talked, in that white paper that I mentioned earlier, that they are implementing… I don’t know where this is at; I don’t know if it’s rolled out, or anything… But they mentioned that they’re gonna try to implement some kind of “By the way, it looks like this probably is not original. It probably came from this.” Putting aside whether or not that’s legally necessary - you know, in terms of like not being a jerk… Like, “Hurray! GitHub should not be jerks, right?” They’re an 800-pound gorilla, and I think maybe in their rollout of this, I think maybe one of the things here is they didn’t reckon with the emotional –

Daniel Whitenack: Yeah, awesome. Well, Dave, we really appreciate you joining us. This is super-fascinating. I’m really excited by what Graphcore is doing. We’ll make sure and link a bunch of links in our show notes for listeners. Definitely check out what Graphcore is doing, read their white paper and all the information about the Poplar software framework; it’s really cool. And of course, the hardware.

I was just really enthused by the conversation, and I have a lot going on in my mind that I wanna think about more, so I appreciate that… Thank you for joining us, Dave.

Chris Benson: Yeah, they created this approach which they called automatic domain randomization, where they systematically created that randomization as part of their training process… And it was done in simulation, as we’ve been discussing. And it was interesting in that it was a technique that could increase the ability for the control policy to be able to generalize to the environment that it’s in. If they had not done that, for instance, and the articulated robotic hand had been maneuvering the Rubik’s cube around and any kind of interference was introduced - going back to your stuffed giraffe comment a little while ago, that could completely throw it off.

But if as part of the training process you are constantly introducing different types of interference in all sorts of different ways, and as part of its reinforcement learning process for its control policy it has to learn to cope with each of those forms of interference, then it is better able to generalize once you’ve completed learning down the road. I was fascinated reading through their white paper on how they approach that. I think it’s a great next step.