Jorge Castro: Yeah. Because that’s a layer up. You can’t remove things. However, we have this fella Amtheus, who’s been figuring out a re-chunker that will go back and reorganize all the layers and remove all the stuff that you’re not using, to get the image size down. So the image size and the updates are still not optimized, but Red Hat is productizing this for their enterprise stuff, so as they figure that out, we get the advantages. Because their [unintelligible 00:22:18.08] containers are like 300 gigs, or something. And we think Bazzite is bloated because it’s 20. [laughs]

Again, by using things OCI containers we could just leverage existing tools, optimization tools. On your OS image, you could run Gripe, the tool that you use to find the CVE vulnerabilities in your app containers… You can run that on your OS container and it’s smart enough to give you all of the versions of the packages, and stuff. They did a really good job normalizing the OS and what it would look in a container, in my opinion, which leads to our success.

Charles Martin: Well, look, I think what I’d like to have is a community of people who are training models, and getting them to interact with each other. A lot of the people - like I said, it’s hard to get feedback; people are doing things in industry… And because they are constrained by NDAs, they can’t really talk about what they’re doing. And I think it gives people an opportunity to really get into this space and learn how training of neural networks works, without being constrained by your employer, or your contract, so you can really do stuff. That’s really a lot of what this is.

I think there are other communities doing things like people building hyperparameter optimization tools, or people building reinforcement learning tools… We’d be happy to integrate the tool in. The challenge is always you want to make a tool that is self-contained. if people fork the tool and begin changing it, it ends up – I don’t know if you guys know the story of Emacs. I was at Champaign-Urbana when this happened; they wanted to port Emacs to basically X Windows, and Stallman didn’t want to do it, and they forked it. You have ex-Emacs and you have Emacs. It killed it. Forking Emacs killed it, because you have the ex-Emacs crowd… And these guys went off and started Netscape, and they’re probably all retired now, or they’re sitting at the top of – you know, hanging out at the top of Google or Netscape… But this is the problem - you want to make sure you have an open source community. You don’t want – I mean, I want people to contribute and feel they can do things. If we fork it and it goes into other communities, it kills it, because now those contributions don’t come back. You end up in these sort of weird battles, and there’s no value in that.

What we want to do is help people, and if it’s necessary at this point, commercialize the tool… I would then turn it into something which we can support, like Hugging Face. Hugging Face is a lot of open source, but any sophisticated technology needs maintenance; you buy a copier machine - it’s not open source, because it needs maintenance. So even a tool like WeightWatcher needs maintenance. So I would love to be able to work with people who would like to put it into production, and develop it, and then if at some point we realize “Look, we really need to put a service contract around this, so that we can maintain it and solve some of the harder problems for you, we’d be happy to do that.” And I think that that’s really what we’re trying to do.

There’s also a lot of opportunity for scientific research. WeightWatcher, a lot of it has come from doing research, statistical mechanics and learning theory. We have papers in JMLR, Nature, ICML, KDD… There’s a lot of opportunity for students. We have one student who is at a bank, who just did his master’s thesis on WeightWatcher… And so there’s a lot of that kind of opportunity as well, and I think there’s a lot of room for improvement.

Johnny Boursiquot: When it comes to benchmarking and performance optimization, I try very hard not to jump to that right away. I’ll try to solve a problem first, and then try to optimize… Basically, prevent premature optimization. And these tools, because they’re part of the standard toolchain, they make it very easy to just start using them right then there, just basically start leveraging them right away.

There was a time - maybe we’re still in that time - where it seems like there was a new HTTP [unintelligible 00:58:13.12] or router coming out every couple of weeks, and they were all like “Oh, benchmark. Compared to these other things, this one has zero allocation, and is 0.05% faster than the other one.” I kind of found it silly a little bit, because of all that going on, and I was like “Okay, we’re kind of missing the point here a little bit.” But yeah, having that tool I think is great, and like you, I don’t think I’ve seen that capability built in, part of the language, from the start.

I’m very careful with that, because it’s too easy to create a culture within an engineering team of “Okay, if I can even ship this thing, I have to make sure it’s super-optimized.” We’re kind of putting the cart before the horse a little bit there; it’s too easy to do that, so I tend to shy away from that stuff. I bring it in when I need to.

Johnny Boursiquot: [56:08] When it comes to benchmarking and performance optimization, I try very hard not to jump to that right away. I’ll try to solve a problem first, and then try to optimize… Basically, prevent premature optimization. And these tools, because they’re part of the standard toolchain, they make it very easy to just start using them right then there, just basically start leveraging them right away.

There was a time - maybe we’re still in that time - where it seems like there was a new HTTP Muxer or router coming out every couple of weeks, and they were all like “Oh, benchmark. Compared to these other things, this one has zero allocation, and is 0.05% faster than the other one.” I kind of found it silly a little bit, because of all that going on, and I was like “Okay, we’re kind of missing the point here a little bit.” But yeah, having that tool I think is great, and like you, I don’t think I’ve seen that capability built in, part of the language, from the start.

I’m very careful with that, because it’s too easy to create a culture within an engineering team of “Okay, if I can even ship this thing, I have to make sure it’s super-optimized.” We’re kind of putting the cart before the horse a little bit there; it’s too easy to do that, so I tend to shy away from that stuff. I bring it in when I need to.

Paul Dix: Yeah, sure. I mean, I think it’s a mixed bag for hiring Go developers. The thing is we have a number of different projects, and some of them, even though they’re all written in Go on some level, they require vastly different skillsets. For example, our user interface, Chronograph - there’s a bunch of Go code, and there’s this massive single-page React/JavaScript application. But the Go code in that is very different than the Go code you’d find, say, in the storage engine or the query engine of the database. Both are Go, but I don’t feel like Go developers are kind of interchangeable, so that I can just say, “Oh, go write the database”, or you take a database person and say “Go write these web APIs”, or whatever.

So generally, for the database what we try to do is filter based on a skillset that matches up with that, whether they have experience in Go or not. Although many of our database developers had experience in Go already coming into it, not all of them did. So for that it was basically like – Go is an easy language to learn; that’s one of the things I love about it. You can point somebody at the spec and the basic documentation and they can be productive within a few days… Particularly if you have code review and stuff like that as part of your process, because then they can just start doing pull requests and pick up idioms along the way, and have the developers who are already very familiar with Go and the idioms push them in the right direction.

So I would say, actually for most of our staff, it really depends. We don’t necessarily require Go experience, just because of the fact that you can pick it up so quickly. I think some of the other stuff around doing performance optimization and using the profiling tools and that kind of stuff is more advanced Go stuff. If we were trying to hire for one of those roles, we would filter for that kind of experience. So it would definitely be somebody who knows Go, who has experience doing those things. But those roles are kind of few and far between. I think for the most part it’s just smart developers who at the very least are excited to work in the language.

I wouldn’t want to hire somebody who didn’t know anything about Go, just because it’s entirely possible that you could get somebody coming from another language, who uses Go, who just absolutely hates it. And then at that point, if you’ve already hired them, it’s not a good situation for either of you.

I think for hiring we benefitted for sure from having open source bits all over the place. We’ve hired a number of people from our contributor community. We’ve seen commits and actually reached out to them and said, “Hey, your code looks great. We’re really interested to talk to you and see if you’d be interested in coming to work with us.” That’s nice, but that doesn’t really scale, and like I said, we kind of have to double our engineering team in 12 months, so at this point we use that, we use job boards, we use recruiters, and I feel like for some of these roles we’re definitely gonna be pulling in people who aren’t familiar with Go and training them up on the job.

Mike Basios: So the way we present the code optimization tool currently is part of your CI/CD process. So you make a pull request, then you will run some unit tests, you will have your integration testing, potentially you’ll have a security scanning tool like Snyk, or Checkmarx etc. and then the next step is, depending on where your application is deployed, we analyze your code and we tell you “Make those changes, because you can have this 20% improvement in CPU and execution time etc.” That is the way we present it currently as a CI/CD tool in the tool chain of the developer tools. However, if you think about the technology underneath, and all dev tools, the way I see it, are going to be using AI and they will take advantage of LLM-based solutions; from the moment you’re using LLM, if you generate code or you translate code, it’s a kind of the same kind of approach. It depends on the data that you will apply the LLM.

So code generation tools - practically, say for the PR example, the way those LLMs have been trained is that they see that there are some comments below the code, so you say “If I give you those comments, can you predict the code?” So in the code translation, where you don’t know about the speed of the code, you say “I have seen this C++ code, and the equivalent of Python code.” And LLMs do it. For example, Copilot or ChatGPT - they can translate code. If it’s perfect - probably not yet, but that is the fundamental technology.

[10:01] So code optimization is on the same set of tools, but it says “This is a slow code that I have seen. Now I have seen a variation of faster code, so I can recommend you this faster code.” But eventually, you can expose the LLM, like any other tool that is built on VS Code or LLM-based - you can expose it in an editor and then the developers from our LLM will get suggestion for faster code. That doesn’t mean that will be beautiful code necessarily; that it will be faster for the hardware that you need to run it.

Johnny Boursiquot: Which means like “Don’t really look under the hood that much, don’t run pprof, and don’t analyze what your application is doing, because that might be premature optimization”, right? “Only go looking for these tools if you suspect you have a problem with performance.” But at the same time, a lot of times you end up building things perhaps in a sub-optimal way. And that ends up pretty much receiving more traffic than you anticipated. Or maybe you go through a growth period, kind of like Pinterest did, where once upon a time you were a small organization that was not public, and now you are public, and you’re getting a lot more traffic… Those things - you don’t go back and rewrite some of those things you were earlier on just because you now have more traffic, right?

[27:50] So you end up with this sort of a legacy built-in performance, suboptimal processes and tooling and services that are doing things that is way more than they were designed to do. Now you kind of have to have an engineering effort to refactor things.

I’m curious if you’ve experienced this need to go back and change things and refactor things and make things faster, and obviously, how Go made that easier or harder?

Daniel Vassallo: Yeah, it’s something that is very highly underrated in productivity advice nowadays in the modern world. It’s mostly focused on trying to squeeze every second out of the day, and pack as much as possible, creating habits and systems and techniques, calendars, tools to try to be even more optimized. Optimization however, comes at a cost. Like a Formula 1 car. A Formula 1 car is very optimized to go as quick as possible on the circuit, but it’s very fragile. As we all know, they break apart easily, and highly unreliable as compared to regular cars. That’s the cost of optimization, and I think it’s a very similar analogy to us. If we want to be agile, and we want to have the capacity to pounce on opportunities as we bump into them, we need to have some intention and slack in the system, right? Some safety margin, which could be three days in the calendar, or a mental sort of capacity, or a combination of those things… So that’s when we bump into a good opportunity, when inspiration strikes, when we’re lucky - we don’t just put it on the shelf, because opportunities and inspiration is perishable, as is commonly said. But we’ll have the opportunity to pounce, work on them, and take action.

[06:11] I think sometimes we’re lucky, we get into these serendipitous situations, we bump into great opportunities, but we’re not able to benefit from them, because we don’t recognize them, or we don’t have the capacity to recognize them. I think when we have a very busy calendar, I think subconsciously, we’ll put the blinders on, so that we don’t get distracted, because our focus is on getting to do the planned things. So we become blind, essentially, to opportunities that might fall right in front of our eyes, and we literally just don’t see them.

So I’m a big believer in trying to arrange our life/work arrangements as much as possible to intentionally create slack in the system, which will look like wasted time when you look at them individually. You might look like “I didn’t do anything meaningful, day after day after day”, but then sort of the big things tend to happen, and eventually it might be five days out of 365. we might look at data, the 60 days of the year, and say, “Nothing came out of this slack”, but then there’s those five days when the important things tend to happen.

Luis Ceze: That’s another great question. So you’re right that a lot of the way that folks are learning about machine learning today and getting started with it, they’re not thinking about model optimization deployment, because luckily we actually create pretty good tools to arrive at models and test them and make sure that they have the right accuracy, it has the right properties. But chances are that most of those models created wouldn’t actually see deployment. They wouldn’t make it to deployment, right? So it’s good for people to learn, but chances are they wouldn’t be deployable. And honestly, I want it to stay that way, even for the sophisticated users.

So let me talk about the other users. These are folks who have already done this for a while, and now they are improving models in a significant way, because they’re doing things that haven’t been done the way they wanted before, right? Today, people already start worrying about performance too early. They have to look, “Okay, if I make this change in my model, am I going to be able to deploy it? If I could deploy it, should I deploy it on CPUs or GPUs?”, and so on. People should not worry about this. I want human creativity/ingenuity to go into making a better model, and not have to worry about systems aspects too early, because that’s going to constrain the way they see the model. It’s as if you’re designing a new feature for a car, you’re all thinking about all the different ways it’s going to be used too early, and you’re not innovative enough because you get constrained by practicalities before you’re able to unlock your creativity.

If you don’t mind, I’ll just go on a quick side comment… I like where Hugging Face is going, where it makes it very easy for folks to start from an existing model, make modifications from a foundation model, and then specialize on specific use cases. And then you don’t have to worry about some of the systems details yet. When you actually put it into deployment, chances are you going to have to worry about it, because performance is going to come and bite you if you don’t, but at that conceptualization stage and model creation stage, I don’t think people should worry about details of how these models are going to be implemented, right? So I want it to stay that way. But I want it to stay that way and still allow these models to actually be deployed.

Daniel Whitenack: [12:06] I’m curious to dive into a few pieces of that, but you mention in one of the blog posts on Determined AI about people still kind of living in the dark age of AI infrastructure, where certain larger companies have built sophisticated AI-native infrastructure for their own use, but everybody else is struggling. I’m curious if that dark age that you’re seeing is due to the fact that, like you say, there’s all these other pieces of the AI workflow - that might be data pre-processing, model deployment, model optimization, data labeling… Is it that there aren’t good tools for those other pieces of the workflow, or is it that they don’t play well together in a sort of all-in-one workflow? …or just people haven’t had enough time to develop standardized methodologies around these things. What do you see as the main contributor there?

Robert Ross: So yeah, I still have conversations with folks. I went to a meetup for a company called Vantage, a cloud cost optimization tool here in New York City. Both of the founders are awesome. And yeah, I like to stay in touch with those folks. It’s really the only group that kind of understands some of the other hidden stresses of the job, I would say, that can actually kind of commiserate and be empathetic.

David Sweet: For A/B testing there are lots of tools, both open source and commercial… And when I say open source, you’ll probably find tools that you can make use of in R, or in the NumPy / Python ecosystem… But there are also commercial products that can help you do this online. That way you can sort of submit results to a web service, or there are other ways of accessing it.

So A/B testing is well-known, it’s the most commonly used of all these techniques, and there’s lots of support for it out there. On the other end of the spectrum, Bayesian optimization - there’s an open source tool called Ax that’s written and supported by Facebook. That’s very good. That would be the go-to, I would think.

I suspect Google’s got something similar, although I’m not as familiar with it. I’m familiar with their paper where they use their optimizer to make better chocolate chip cookies. Vizier it’s called. I don’t know for sure whether there’s an open source version.

Michael Knyszek: [52:13] I don’t think it’s totally mad. I don’t know if the wider Go community knows (it probably does), but the Plan 9 C compiler somewhat famously just allocated memory and never freed it. It was written in C, and it just called malloc, but it never called free, because the assumption was by the time you’re done compiling - “Whatever. The OS will clean it up. It’s fine.”

So for short programs there is some wisdom here… And I’ve known other systems that have done something similar, because there are performance gains to be had. If you know you’re not gonna run for a very long time, then of course it works.

Of course, I will say that in most cases it probably doesn’t make sense. It could certainly be a premature optimization, especially if you have a command line tool that’s growing to do more things; one day it’s just gonna crash, and you won’t know why. But it can be valid in some circumstances.

Amal Hussein: [laugh] Yes. No JavaScript for that. Yeah, and actually, speaking of optimization, Party Town. That’s another show, that’s another really cool tool. I don’t know, it’s like black magic. They’ve been on my list for a while, that I’m like “I really want to have them on.” But anyways, so thank you again, Annie and Rick. We’re gonna have lots of links in the show notes. Thank you all so much for listening. Where can folks reach you if they want to kind of get in touch, or if they have questions? Annie, do you want to start?

Mat Ryer: It’s funny, because I wrote myself a little tool which took my Go code and did the first optimization that you talked about… And it just would comment out the defer line, and then copy the – it was very rudimentary; it was just kind of an experiment.

Peter Wang: …which is cool. Now, it’s [unintelligible 00:36:37.22] so like “What percentage of your code is pure Python code?”, and that’s the percent that we would be squeezing the air out of. If the rest of it is numerical code, then Pyston won’t help very much, because that’s already quite optimized… And if you wanna optimize that further, to fuse loops or things like that, then you would use something like Numba. And in fact, it was really our Numba Compiler project that led us down this path. We had many different kinds of users coming to us saying “Hey, Numba is great. I wanna do Numba for my whole program.” And we’re like, “No, that’s not what it’s for.” It’s there to hit the hot numerical loops, it’s to allow you to write Fortran(ish)-like element wise stuff without having to go and break out C extensions for NumPy, right? That’s what Numba is good at.

But then, as we looked to see “Can we extend some of the ideas in the Numba optimization toolkit - can we extend those into broader program analysis?” and then we realized that it’s almost this different project… And Pyston is essentially a project in that vein… “Can we just make the interpreter itself much faster at a lot of these common things that people do?” And then there’s a 1% or 2% improvement here, 1% or 2% there, you start shaving off 1% or 2% all over the place, and you can start making something that’s quite fast… Again, without making people have to rewrite any of their code.

So we’re really excited about that… And of course, it’s an open source project, we’re gonna keep it open source; that’s kind of how we do… And yeah, I’m really excited about the team; they’re really sharp guys, and we’re really excited about what’s to come.

Daniel Whitenack: One is for this sort of transformer language models, oftentimes if you go much beyond 7 billion parameters, maybe pushing it up to kind of 13 to 15 billion parameters, you’re not going to be able to run it very well, just by default, by downloading it and running it with the kind of standard tooling on anything but a single accelerated processor, like a GPU. And even then, most of the time not on a consumer GPU. However, the follow-up to that is that a lot of people have created open source tooling around model optimization, that may allow you to run these models on consumer hardware, or even on CPUs. And I’d like to talk about that here in a bit, that a lot of times you may want to consider this sort of model optimization piece of your pipeline when you’re considering how to run the model… Because sometimes the sort of default size and default precision of the model might not be best for you, both in terms of your needs, in terms of performance, or in terms of the hardware that’s available to you.

[00:22:10.25] But I would say, in this phase of like “What model is going to be good for me?”, go ahead and put that sort of hardware concern - although it’s important, put it a little bit to the side and focus on which model is giving me the output behavior that I want. Because you have a certain task in mind, and if you could figure out “Hey, this model kind of does what I want, and it seems like it’s giving pretty reasonable output”, and then you find out “Oh, well, I can’t run it on the GPU that I have, or I need to figure out how to run this on a CPU”, then that kind of narrows down the type of tooling that you’re going to have to use for optimization. Or you might not need to optimize at all.

So kind of start with the smaller models, and build up to something that fulfills the behavior requirements that you have by just using some of these demos, using some of these spaces… And then think about “Okay, I’ve now figured out I need Falcon 180 Billion… So what does that look like for me to run that in my own infrastructure?” Then there’s kind of a follow up series of things that we can talk about related to that.

Daniel Whitenack: Yeah, yeah, of course. So let’s maybe just start out with a kind of state of model optimization right now. So first off, could you kind of describe - when you’re talking about model optimization, or that set of tooling, what do you mean by that, and how does that fit within maybe the things that a data scientist or an AI person would want to do?

Tom Occhino: [00:27:56.04] Yeah, it’s probably a function of two things. One is that sort of window between innovation and generalization. But the other is actually the way that you deploy Next.js yourself and all of the things that you have to consider, there actually just is a lot associated with hosting a sophisticated application. And so the steps that you have to go through - we can document them all, and we have, and it’s gotten a lot better in even just the past month or so. But it still is a process. “Here’s a bunch of different things that you need to set up on your own.” And actually, I think that’s great for anybody who likes that type of work. We document it very well, and we’ll continue to refine and simplify over time. I think it’s going to get better and better as we go. Like, for example, you used to have to install special dependencies in order to do image optimization, and now it’s kind of taken care of for you automatically… We’ll continue to refine and make it better. But I think the value proposition of something like Vercel is “Okay, there’s a lot that goes into this, and that’s not necessarily how I want to spend my time.” If people love to spend their time that way, that’s great. We should compare notes, because we have some folks that like to spend their time that way as well. But for me, I’m more focused on the end user experience. I really just want to build my thing and I don’t want to think about how it gets supplied. I just want to git push and then it’s done.

So yeah, we’re trying to strike the right balance between making it fully self-serviceable and fully self-hostable, and for the people who don’t want to think about that, here’s a great way to host it. We think that Vercel is probably the best way to host Next.js, but if you want to tinker on your own, if you want to host it yourself, we’ll provide you with all the tools and we’re committed to continuing to make that better and better over time.

Austin Clements: Yeah, absolutely. So I do want to first say that stability is a very important part of Go. So I have no plans to come in and completely shake up the house. [laughter] So yeah, in terms of what’s sort of top of mind for me and what I’m thinking about… We like to say that Go is about engineering at scale, and engineering for scale. One of the things that was really top of mind for me coming into this role is how we can scale the engineering of Go itself without losing its minimalism and approachability. There are both technical and people aspects to this. For example on the more technical side, we’re starting to see dividends from the new diagnostics strategy that [unintelligible 00:21:58.09] has been driving. There we basically said “There’s no way that we on the Go team at Google have the resources to build all of the runtime diagnostics tools that people want and need. So how can we create a platform on which people can build their own tools?” That’s a very technical approach to sort of scaling the engineering of Go itself.

And then there are a lot of people processes. I’m thinking a lot about how we better communicate with and leverage the amazing developer community that we have. How can we improve our transparency? How can we better accept contributions? How can we keep people around better? How can we create long-term stability in the Go project, while scaling up the engineering of Go itself? Another thing that’s really top of mind for me right now is on sort of the engineering at scale perspective.

One of the sort of founding principles of Go was that programming should be fun. And while I do strongly believe that programming in Go is still fun, I also think that one of the costs of Go stability has been that there have been – there’s been a lot of experimentation in a lot of other languages. And there are places where I think other languages have been finding places that they can do better, that they become more productive, they become more fun. And I think that Go has never been about cargo-culting from other languages, but I think we can learn a lot from sort of the movement that other languages have had in recent years.

Go sort of started the development of more systems programming languages, of sort of more exploration in what can a programming language be… And I think it’s fantastic that we have both sort of Go on the more stable end of the spectrum, and also other languages that are willing to just like go wild and explore things. And I think we can learn from some of the best ideas in there, and bring them back into Go, and sort of keep things fresh… Because the bar for what makes programming fun moves. It’s not in one place. The bar for what makes programming productive moves. And I think we need to keep up with that, and we need to be learning and exploring.

[24:19] And then on engineering for scale - I’ve been thinking about how to engineer systems that scale for a long time. My PhD thesis was on the formal relationship between software API design interfaces and multi-core scalability. And I’ve been thinking a lot about Go’s approach to performance recently. We spent many years basically trying to float all boats, and we’re kind of running out of opportunities on that front. Obviously, we’re going to keep trying to do that, but it gets harder and harder over time.

So I think we’re sort of moving into a new phase with performance and scalability of Go, where we’re going to have to give people more mechanisms to do explicit performance engineering. But that said, I also deeply believe that performance engineering in Go needs to be incremental and composable. So engineers can trade higher engineering cost for lower resource cost just where it matters, without having to pay the high engineering cost everywhere… And so that they don’t have to be constantly thinking about the effects of performance optimizations as a system evolves.

Some of our experiments with memory allocation are actually a great example of this. A few years ago we put out this experimental support for explicit arena-based allocation. We never moved forward with that, because it really wasn’t composable. You had to be pretty careful to keep track of what was allocated how, and the details of that tended to leak out of APIs and libraries that used this. So that was kind of incremental, but it really wasn’t composable. For example, we’re now experimenting with a new variation of arena allocation that is composable, that works on these principles, so that an engineer doing performance optimization can drop in a few annotations in the code, and then not really worry about it again for a while.

And it’s possible to do it wrong, otherwise we could just do it automatically… But if you do it wrong or your system changes over time, it degrades gracefully, which also means that you can revisit those annotations as your software changes, but you only have to do it from time to time, and hopefully, the tooling can tell you “You probably need to look at this again and figure out what’s going on here.”

So those are sort of – there are a lot of things at the top of my mind, but those are a few of the things that I’ve been thinking a lot about.

Break: [26:39]

Matthew Boyle: Totally. And I think that’s a key thing to call out. There’s a lot of tools we talked about here, like debugging, logs, metrics… Like, they can be used for debugging, but they can also just be used for like system optimization, system health, learning… In some cases you may even find people use some of the metrics to do business metrics. So I mentioned about like transaction successful and AmEx. Like, if you’re a FinTech company and you’re trying to figure out very loosely or quickly what’s the volume of Visa versus Amex versus MasterCard for today, you could use metrics from a service to do that, too. So there’s a wide range of uses here, which is really cool. It’s not just debugging.

Jerod Santo: What I learned from looking at 900 most popular open source AI tools

Four years ago, Chip Huyen did an analysis of the open source ML ecosystem. Since then, the landscape has changed, so she revisited the topic. This time, she focused exclusively on the stack around foundation models.

This is a #longread, so I’ll just list her personal favorite ideas for you here and let you take the deep-dive on your own time:

• Batch inference optimization: FlexGen, llama.cpp
• Faster decoder with techniques such as Medusa, LookaheadDecoding
• Model merging: mergekit
• Constrained sampling: outlines, guidance, SGLang
• Seemingly niche tools that solve one problem really well, such as einops and safetensors.

You can find most of these on her cool-llm-repos list on GitHub.

Miriah Peterson: You’re being really, really pointed towards the benchmarking tool, I think… [laughter] But before we go to the benchmarking tool, I do want to say… I run the local – well, I don’t run, I used to run a local meetup here, but I prepared a talk on pprof for it, and I made some videos… And me personally, I could never remember what the pprof tools do. I cannot, for the life of me, remember what all of the things mean, what all the graphs mean… Go makes their own flame graph in pprof. So they have the traditional flame graph, and then they have what’s called the new flame graph. One of the things that tells you is “Is Go’s compiler optimizing for you? Did it inline functions for you?” So you can tell right off the bat, “Did the compiler optimize for me?” by looking at this flame graph. So step one, put pprof in your program. Step two, look at the flame graph and see “Are anything already inlined for me? Is the compiler already optimizing things for me?” And then I would say “Great, now I notice that I have this function…” You’re looking at the web graph, which is their other thing - again, if you want to know what it means, I have a video on YouTube; go look for it. I go through it in detail. It’s visual, that’s why.

[26:10] And you can see “Oh, this is an expensive function. It’s taking a lot of CPU, it’s taking a lot of memory… Great. I want to see if I can try and optimize this. Let me write a set of benchmarking tests, turn on that memory flag, and see if you can start dialing knobs to go fix that high problem function.” So I don’t think it’s one or the other. I do think the tools have to go in tandem.

Again, I always find benchmarking to be that last step, that last “Great, I’ve got a program, I’ve got it working… I’m trying to get it optimized… Maybe it’s not the most optimized it can be, but I’m trying to get it 80% of the way there. I’ve identified some heavy CPU functions, or some heavy memory functions”, which you can identify with pprof very easily. “Now let me pick these functions, turn on the right flags, and start writing benchmark tests and see if I can do that.” And the way benchmarking works is instead of just running it once, it by default runs it as many times as possible within that window, and it’s giving you that basic average performance over that so many times of runs. And so you can see “Great. On average, I’m allocating 3000 bytes to this one function.” That would be exceedingly a huge problem, and you should fix that. As opposed to “Can I get that to be lower? Can I get it to not be maybe this–” And then “Oh, it’s taking 300 milliseconds per run, but the function next to it is executing in 20 nanoseconds per run?” Maybe I can trade off things so that this function is not as much of a bottleneck for that whole system.

So when I teach on benchmarking, those are the two things I always say: look at how long it takes to run your function, and then look at how many bytes you’re allocating per system, and those are the first knobs to start tweaking, in my opinion. I don’t do it that much professionally. I very rarely get to the point where I am needing to prove optimization at the benchmark. I think I’ve never worked at a company big enough where they start losing money based on the speed of my things… I always am at the company that’s like “We need to move our infrastructure from here to here”, and so I’ve always got the free budget to build new things. Anyway. But that’s still where I point people.

Daniel Whitenack: I appreciate that, and I appreciate the support that you all have given. I think it’s interesting maybe for people out there that are less familiar with the various options for model deployment to understand that there is really good tooling. Like you say, whether it’s optimizing a model and deploying it on a CPU for an edge environment, or just a cheaper inference solution… Or it’s like all the way to these Gaudi 2 processors that we’ve been experimenting with. I think there’s a lot of interesting and approachable tooling for that.

So I first came across some of the tooling around Gaudi 2 by actually seeing blogs on the Hugging Face blog about Gaudi 2, and I think at the time the blue model, which is a very large model, and running it on either a single accelerator, or spread across eight accelerators, with really high throughput on the inference side, and doing that with tools like OptimumHabana. So for those of you that are out there and wanting to explore things, actually, if you look up the Hugging Face Optimum library, there’s a lot of great tooling that you can play around with there; even not for Gaudi, but for other processors, too. So whether that be CPUs, GPUs, the Gaudi 2 HPUs, the Data Center Max GPUs… Optimum kind of provides you a way - if some of you can visualize, maybe you’re writing in your code and you’re importing a model from Hugging Face, that’s just like auto tokenizer, or auto causal LLM, or whatever it is, with Optimum a lot of times either you can just do like a couple-line replacement and just replace that with the Optimum version of those classes, or do some wrapping of the various models with optimizers. And this allows you to run your model very fast on a wide range of architectures.

So I think, to your point, Rahul, I think one of the things that we’ve found really useful is the actual ease of use, and coming in and saying “Okay, well, we have this stuff running on a GPU. Let’s try it on these various other architectures.” I remember even like maybe two or three years ago trying to do some of this model optimization things for edge deployments. It’s very, very challenging. A lot of times I would try to optimize a model - at the time I was working on like speech models and other things - and it just wouldn’t work, because operations wouldn’t be supported, or something like that. But this tooling, which – it’s cool, because Intel’s working directly with Hugging Face on this tooling… And of course, the ease of use has just been ramped up drastically. And we’ve been applying that with really good results, particularly for inference for LLMs. So that’s been a key feature to see that change happen.

Michael Quigley: I don’t think I’ve seen much about zero trust architectures in AI at this point. I think most zero trust implementations are mostly trying to get people to understand a) the need, and b) the architecture and how you kind of adopt these things, and getting a good foothold for what they do. I don’t know that, by nature, a lot of these things are going to be tools to help you optimize a lot of what’s going on. I mean, I do have some interesting stories about that in terms of optimization stuff that we’ve done along the way, but…

Jonathan Norris: Yeah, I think it’s just that the V8 runtime for WebAssembly is the most mature runtime, so they have those development tools built up. And I’m sure that the Wasm time by Code Alliance team are definitely working on similar types of outputs for all the different runtimes that they support… But yeah, the common recommendation is if you really want to get that low-level optimization, plugging it into some type of a browser engine that has Wasm support is the best way to get that low-level sort of profiling information. And obviously, Node, being that it just runs on V8, is probably the easiest way from a server side use case to get that data.

Chris Ferdinandi: [13:56] Yeah, so I’m not sure, to be honest… I think the thing that’s most clear to me is the optimization piece. People are finally kind of latching onto this idea that “more HTML good”, “more JavaScript bad.” Or it’s better to – if you can reduce the amount of JS you’re shipping, that’s better for you, and for the kind of business goals of the thing that you’re creating.

The developer experience piece is tough, because on one hand, I know very seasoned developers often really like the ergonomics of state-based UI libraries. And there are times where they absolutely make things easier. I think where that falls apart a little bit is like the barrier to entry for beginners, from people who are not specialists in JavaScript, but are in related disciplines within frontend development… So accessibility experts, for example - like, the barrier to entry for them to be able to participate in processes that use these tools goes way up. And so yeah, I’m not sure if the mental model itself will go away or not.

I think where state-based UI potentially gets a lot easier - some beautiful future state - is where diffing the DOM is as easy as passing a string into like the inner HTML property is today. That obviously doesn’t diff the DOM; it clobbers the UI. But it would be really nice if there was some sort of browser-native way to say, “Here’s the element I want to render stuff into. Here’s the stuff I want to render as a string, or elements, or whatever. Please change just the things that need changing in the least destructive way possible.” Because these tools do have… Like, just the process of getting set up with React, and understanding kind of the – the mental models themselves are not super-complicated, but working with the tool sometimes is, especially if you’re someone who’s not particularly experienced, or the command line is a terrifying thing for you… Sorry, it’s a long-winded way of saying “I don’t know if the mental model will change.” I think state-based UI can be a very great thing, and I think thinking about things as components can be really useful in certain use cases… So that probably – I guess I’m talking myself into it here; that probably won’t go away, but I do hope that the way we ship these things and the way that we’re actually delivering the things we build - I do see that changing. And when I say JavaScript is the new PHP, I don’t mean that in a mean way; I mean that like maybe moving JavaScript more and more to the backend, and ironically, in the process, reinventing a lot of stuff PHP did for us a decade or two ago.