The first real-time voice assistant

Welcome to another Fully Connected episode of the Practical AI podcast. In these Fully Connected episodes we try to connect you with various things happening in the AI space, and connect you with maybe some learning resources or talk about some subjects that will level up your machine learning game. My name is Daniel Whitenack, I am founder and CEO at Prediction Guard, where we’re enabling AI accuracy at scale, and I’m joined as always by Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris?

Doing great today. It’s dog days of summer here in the US, and it is –

It is really hot and humid. Yeah. Super-humid and nasty.

I’m looking forward to weather control from AI, and it will keep all of us at just the right temperature.

Right. I can’t see anything possibly going wrong with that…

Of course not.

Only positives there.

And regardless, that is in the distant, distant future of 2025, I’m sure.

Yeah, exactly. Exactly. Let’s focus on the next two weeks for now, which is important.

That’s right.

I think one of the things that caught me off guard this last few weeks, which - you and I try to stay plugged in to various things, and maybe people think and listen to this podcast that we’re keeping plugged in with every single thing happening in the AI space… But I was a little bit surprised when I saw the release – I guess I just hadn’t really been following along with what the company or research lab Kyutai was doing… So this is an open research lab that researches AI, and they have funding, and some support in terms of infrastructure and all of that… But they’re a nonprofit research lab in my understanding, and they actually – so we talked on a previous show; we kind of got fooled a little bit, or maybe it was a little bit of a fumbling in terms of marketing, but it seemed like when Open AI GPT 4.0 came out, people were hyped because a lot of the demos were voice-based. But at least you know at the time of that recording - I’m not sure all of what everyone has access to in the paid and unpaid and enterprise version, but the actual voice assistant for Open AI was not out. And at least as far as the release date of Kyutai’s voice assistant, which is called Moshi, they were the first to actually release a version of their voice assistant, which - it’s similar to, in my understanding, what GPT 4.0 is on the multimodal side, and that it is a multimodal model. So it’s a real-time multimodal model that supports a voice assistant.

And this research lab, I think it’s like eight people or something like that… Of course, they have resources that are supporting them. I think it was 1000 GPUs, or something. They have resources, obviously. But they were able to beat what is now the Goliath of the AI space, beat them to market with this real-time voice assistant, which I think took a lot of people maybe by surprise, or maybe some people were following it closer and expecting it… But I think in this sort of six-month or whatever time period it was when they were working to get this out and beat the kind of Goliath of what is Open AI, which I think in and of itself is pretty interesting.

It is. I mean, so many try, and some of the other Goliaths, the second-tier Goliaths, if you will, are continually trying to compete… And they may touch it, they may fall short… I always love hearing when a smaller group, especially if they’re focusing on open solutions, comes out and is able to do well. And they’ve got a cool name, by the way.

Yeah, yeah. And it is interesting, because this does run… So when you see the demo - and we can pull it up here in a second and maybe ask a few questions, but when you see the demo or the prototype, it obviously still has some rough edges. So I think you have some rough edges that aren’t fully kind of productized version, like maybe what you get with the Open AI voice assistant in the forms that it’s in… But it is very impressive also because this is a model that I believe it’s models that are of a size that you or I could run them on even a single GPU… And they’re going to open-source these models. I don’t know what the timeframe is on that, what exactly that will look like, what licensing, all of those things… They do have a few talks online, so if any of the listeners know that information and I just haven’t run across it, then they can maybe update us.

[00:05:46.27] But yeah, they will be open-sourcing this, which I think will drive a lot more experimentation. And of course, as we saw with the first open LLMs that were released, with LLaMA and other things, there is of course a huge explosion of innovation and experimentation going along with the release of the open versions of those things. And so I expect that there’ll be a similar thing with these models, and what I assume will be other versions or other families of these types of models moving forward.

Yeah. I noticed, going back to your point about being able to run it locally, and potentially on a single GPU, they talked about it in their press release, they just say “compact”. Moshi can also be installed locally and therefore run safely on an unconnected device. To extend that a little bit, I think that there are a lot of larger organizations that are worried about IP concerns… These are topics that we’ve covered quite a bit on the show in days past. So Moshi may very well find a home in corporate environments, first of all where they don’t want to send information out and they want to get the advantage of that, because it can probably be run on a single GPU, a lot of edge devices make it possible… So great thinking there in terms of what’s possible.

And then finally, thinking of my own industry in the defense space, since it can be run in an unconnected or disconnected environment, there’s all sorts of things from a government standpoint that they may be willing to do. So it’s a great strategy. I love hearing these small companies that might be able to have a big impact in industry by accommodating those concerns.

Well, Chris, I find one piece of this whole Kyutai Moshi thing very interesting, which is almost like it feels a little bit like deja vu, because we back in whenever it was - I forget what year Open AI came about… It’s like there’s these big players in the AI space, and they were doing certain pretrained models and all this stuff, and robotic things and all of that… And then Open AI came along and said “Oh, we need an open, transparent, nonprofit-driven research lab to really promote innovation going forward.” And of course, as we have moved forward through that, we’ve seen Open AI kind of get away from that sort of pure nonprofit status, with a little bit more of a complicated corporate structure, which we’ve talked about on different shows… But then also, just their release of their work and their research and their models and their data, and those sorts of things of course has become very not open. And they of course have their own reasoning behind that, which at least publicly they would say is related to…

Microsoft…

Yeah… Well, at least publicly, they would say it’s related to safety of the use of these models… Of course, there’s various people that might guess certain other motivations…

Microsoft…

[laughs] But yeah, I do find this whole thing sort of like déjà vu. I don’t know if you’re having the same feeling here.

You and I both have a long history in the more than six years now that we’ve been doing the podcast of supporting open engagement from different organizations, whether they’re corporate entities or nonprofits or whatever… And we’ve seen that from others. I mean, famously Yann LeCun talks about – he works for Meta, which is Facebook’s parent, and talks about nonetheless having open models and all that. And so we tend to shine a spotlight on those organizations that do that wherever possible. We certainly went through that, because we’ve been doing about – just after we started the podcast, which was back in 2019, or 2018 actually.

2018.

[00:10:05.18] Yeah. And about a year later Open AI closed up. So we actually covered that in the early shows. You know, it is what it is. They’ve done that. They remain an amazing corporate leader in the space, but yeah, they did close all up. And we tend to turn more spotlights toward others like this. So I’m pretty excited to see what Kyutai is doing and is able to do going forward here… And I hope they’re able to viably play against that top-tier competition. I think that would be wonderful to have multiple.

Yeah, do you think that there’s any chance for this sort of open research in the AI space or in the technology space to survive as a sort of bulwark of open, transparent research and open source within the pressures that come, of course, when you release this sort of technology, and you’re a leader in the space, and there are actual dollar signs and corporate concerns, and certainly like partnerships that are necessary? So partnering with companies to do this work is almost a reality, I think, in the space, because – we talked about this a little bit with the Stanford AI Index, where they found that the bulk of AI research is still happening from the industry side. So I don’t know, what are your thoughts? Do they stand a chance at staying the course with this, or…?

I think there’s certainly a chance at it, and I would argue – it’s the same argument I’ve made in previous shows where we talked on similar topics, is that we’re seeing… As the AI industry is has been maturing these years at an incredibly rapid pace, but we’re still seeing many of the things occurring that we saw when the software world was really maturing over several decades. And the place where open source has really, really worked are in common touchpoints where all organizations or many organizations need a common thing. And they might build something differentiated on top of that, for their revenue, to drive profitability… But there’s so much that is underneath that point of differentiation that they and many other organizations can get the benefit out of a lot of effort, a lot of work; a lot of times they’ll have paid employees do it…

So there’s a point where working together and doing open stuff makes sense for business, and it drives profitability. It may not be your single point of differentiation, but if it’s anything under that, why not? Why not share the costs, and pool expertise for the best possible foundations? And so what I’m hoping is that we continue to see that play out in the AI space. We’re seeing – if you look at Hugging Face, we’ve already talked about the fact that a couple of months ago they announced that they were hosting a million models; those are all open source. Really, really impressive. And so I think that there is a good chance that a vibrant, open community around AI can and will continue, and it will have a lot of corporate players involved in it. So I’m very optimistic in that way.

Break: [00:13:19.26]

Alrighty. So as we change gears just a little bit, I had noticed a couple of interesting things. So I spend a lot of time talking to different folks kind of in the Fortune 500, Fortune 100 world. I work at a big company, but I have a lot of friends and former colleagues at other companies, and we chit-chat about these things… So something has really come up in a whole bunch of conversations lately for me. And I thought “Wow, if I’m talking about it this much with different friends of mine, it probably is a good topic to talk about on the show.” It’s an interesting observation, and that is, for those of you who are familiar with the organization Gartner - and that organization does a lot of prediction, and kind of identifying different technologies and things where businesses can use them effectively… And famously, they put out the Gartner Hype Cycle. And what that is, is it is a lifecycle for technologies. And they basically, across all technologies that they track - which is many - they put them on this hype cycle and track where they are in their lifecycle. And the short version of what that is - it has a steep upward curve that looks like an ocean wave sort of, that plunges down into a trough behind it, and then it kind of comes up without so much steepness, midways, to kind of a sustainable plateau.

And so what they would argue is that, for any given technology, there is an innovation trigger, which is this rocketing up on amount of hype associated with the technology. And then it gets to a peak, which they refer to as the peak of inflated expectations, where it’s really high, everyone’s talking about it, but maybe not a lot of productive work has happened yet. Super-cool. You can probably already recognize how AI might fit into this, with all the things we’ve talked about over time. But then those expectations have not been met, and people getting kind of frustrated with the technology, and it plunges down into what they call the trough of disillusionment. And that’s where they kind of go “Wow, I thought that thing was so great, but boy, it really didn’t pan out, and we wasted a lot of money on it. It’s just not really worked out well for us.”

But then calmer minds come along, and they say “Well, wait a minute, this technology has some really good uses. We just need to be a little bit more practical, pragmatic about it, and not lose our heads over the hype.” And that’s called the slope of enlightenment. And that reaches a point that’s called the plateau of productivity, where basically for the long-term a technology lives out the rest of its lifecycle being a productive technology, but without all the craziness in the early hype days.

So now that I’ve introduced everyone to that lifecycle, going back to the conversation that I’ve been having repeatedly with multiple people, that I had noticed that so many organizations, especially large organizations, are just plowing money into generative AI, with mixed results. Some are getting some decent results within the context of it being early days in the corporate sense… But I noticed that after peaking and holding a peak on the hype cycle for quite a long time, generative AI is now beginning to plunge down into the trough of disillusionment. And what that would imply according to Gartner is that people are beginning to get a bit frustrated. And I would say that’s panning out, because I’ve noticed many articles and social media posts over the last few weeks that people have been kind of going “This isn’t gonna lead to generative AI. This isn’t quite as good as we thought. It’s not magic.” It’s all the things that you see with people being a bit frustrated with it. And those are increasing in the number that I’ve seen.

So it got us talking about “What does that mean in a corporate sense, especially when you have a technology plunging down into the trough of disillusionment?” And not only that, but it’s a technology that has received a lion’s share of funding relative to other technologies that go through the hypecycle. It’s the coolest of the AI, over the last couple of years the coolest of the AI tools in the toolbox, and with corporations always lagging… They’re now plowing money into it, and yet expectations are falling.

And so not getting to the point of – we’ll obviously find that slope of enlightenment and that plateau of productivity eventually… But what does it mean over the next few months, as we’re looking at organizations that are still plowing money into generative AI, but maybe not in the most productive sense, or not as productive as they could given the dollar value that they’re putting in. So I’ve asked a lot of people what they think of this… Daniel, what are your impressions of that? It’s an interesting place to be if you’re in corporate America or corporate anywhere these days.

[00:20:25.23] I do think it’s interesting, and I think that in some ways, some of these feelings are healthy. In particular, what I mean is I noticed earlier on - so maybe in 2023, or last fall - still talking to a lot of people with a misconception that “Oh, somehow what’s going to happen is we’re going to get access to a large language model” or “We’re gonna get access to a foundation model in our company.” And somehow that kind of equates to a solution to them; like, this will now be a thing that solves problems. And I think that, of course, is a bunch of baloney, because basically a model does nothing; it’s how you implement it, how you integrate it, how you use it that actually makes it a solution.

So I don’t know how else to describe that other than people thinking that AI would provide a different type of solution than other technologies, which are softwares that people deploy within their companies. And so some of this I think is really healthy, in that people are realizing “Oh, wait a minute… There’s still a need to think about how we integrate a call to a large language model in the context of a larger engineering project.” And actually, there is engineering around the edges of the integration of AI… In some ways different than traditional software engineering, and in a lot of ways the same, whether that be hosting services, or testing and evaluating outputs, or versioning the way that we call these models or other things… There’s a lot of those best practices that are still really valid from the software world.

So to me, it’s not so much – and maybe this is just because I of course have a vested interest in the technology, because I’m building with it every day… But I think it’s not so much a disillusionment about AI functionality in the context of what people are building over the next year, but disillusionment around how that integration happens. Whereas before, it was sort of this fuzzy thing that we’re going to bring AI in, and somehow that’s going to like solve a bunch of problems, without really an understanding of how you would actually see return around that. Now people are saying, “Well, yes, we’re going to bring in AI, we’re going to bring in LLMs, but that’s going to live still in a software stack that we have engineers developing, and we’re going to develop that on some lifecycle…” And yeah, there’s still going to be – if anything, maybe increased engineering spend, because there needs to be extra engineering around these models. And so it is enabling efficiencies, it is enabling net new kinds of features or net new products, but these are still products driven by software that requires engineering. And so that realization, I think, is a really healthy one, and so maybe that thing that has the disillusionment wasn’t really ever a real thing that could have been gamed, I guess.

[00:23:50.05] I think that’s a fantastic insight. I think in a perfect world if we can help people along kind of get through their own trough of disillusionment very quickly, to climb back up onto the slope of enlightenment by following that guidance is essentially what I’m getting at… Before diving into it, I know that over time, as I’ve talked to people, it reminded me of – amplified beyond what I’ve heard before, but of previous technologies that were supposed to solve everything. You know, blockchain was going to solve the world, if you recall. Blockchain was amazing, we were gonna have it everywhere, it was gonna be everything… And having since reached that plateau of productivity at the end of the lifecycle, blockchain has a fantastic place in the technology world, and a vibrant community. But of course, it doesn’t solve all things, and I think people need to realize that the same with these kinds of models is that they can do that.

So I know that one of the things that I’m trying to get people to do is to get through their own trough of illusion quickly, and start recognizing in a really productive sense how to fit it in with larger systems. We’ve always talked about it’s really the software system around these models that makes it all work, that makes the value for the user… And even extending that, if you’re not in the cloud, it’s the hardware. If you’re out on the edge, it’s all about what you have on the hardware, and how does it integrate, and how does it integrate with the systems you already have in place, and what special value are you expecting generative AI to bring to bear that you haven’t already been trying to design and solve for?

And so I think as people really stopped and they kind of got out of their New Year’s Eve party moment, and they said “Okay, I’m an engineer, I need to start being an engineer again”, and thinking about it, and they thought “Well, maybe it doesn’t solve everything, like I thought, but I can identify some pretty cool things that it would help [unintelligible 00:25:42.02] And I’m hoping that people will start focusing on that, and bring engineering, to your point, back to bear on this, and solving it, but solving it in that larger ecosystem, that includes the overall stack that you’re in, the software… And, since we’re moving evermore out onto the edge, into all the devices that we use out there, beyond just our cell phones that are always everpresent, that we can find some good uses.

So maybe this is a chance for a bit of a resurgence - yes, of engineering, but also, this triggers all sorts of like data sciency things in my mind… Because as a data scientist operating in that industry for however long it was the thing, it was about sort of choosing the right sets of data tools and models to come up with a solution, or at least that’s how I think a lot of people viewed it. And that may have been a gradient boosting machine plus a SQL database plus some sort of data pipeline, and connecting that into infrastructure, and then eventually into products that get out into the world… Now, some people might view data science differently, and have different views, because it’s sort of an ambiguous term in and of itself… But I see one interesting thing on that hype cycle that you were mentioning - there’s a shorter time period that they talk about this composite AI reaching the plateau, than “generative AI”, which is interesting to me in that I actually had to look up this term, because I have no idea what that term means… There’s actually a number of terms on the Gartner hype cycle that I have no idea what they mean, which I wonder where they come from…

And I’m right there with you. So neither one of us know what composite AI is. So I’m sure that there are a few people out there that are very familiar with it, and are snickering at us… And we welcome your education and feedback on such. Keep going, though…

Yeah. But I looked up the term, and this appears to just be almost a term describing data science, which is just like using different types of AI or machine learning together to solve issues or create solutions… Which is sort of just descriptive of data science and kind of what it was for many years. So I don’t know that it’ll be called data science. Maybe it’s called AI engineering. I don’t know. But I do think that we’ll see kind of a return to this idea of composite solutions and a multifaceted way of looking at doing these things, not just with Gen AI, but that plugged in as an option into the solution mix.

I couldn’t agree more… Despite being an AI podcast, I know you and I are always a little bit eyeroll-y when it comes to all the hype around it. We try for our listeners to cut through the hype and talk about it… So yeah, a return to engineering and taking advantage of some of these capabilities in a holistic system, that is highly productive and gives your end users what they need is the way to the future.

Break: [00:28:46.09]

Well, Chris, I have maybe a related question for you, which - it’s not exactly related to the hype cycle, but I was at my local co-working space for a fundraiser on Friday night, and shout-out to Matchbox co-working if anyone’s listening… But yeah, so that was fun. But I got into a number of AI-related conversations, as I usually do, and what one of the guys I was talking to mentioned was, you know, there’s a lot of people talking about how this sort of wave of generative AI, this wave of AI and what people are referring to AI now is being compared to kind of like the surge of – it’s like the new internet. When the internet was brought about, and the type of change that created. And his point was sort of “Well, it definitely created a kind of new market, this space that was an is the web…” And it wasn’t just about creating efficiencies. And his point was it seems like most people are using AI to create efficiencies in the enterprise, whether that’s helping write reports, or automate certain functionalities that interns were doing before, or analyze a bunch of documents, summarize those, answer questions, get quick access to information. And from his standpoint, these are all kind of efficiency gains, and not necessarily creating any sort of new market that would be comparable to the huge shift that happened when the web came about. So I was curious on your take of that. Maybe it’s slightly related to the hype cycle stuff, but…

Yeah, I think so. There are two different qualitative things. There are a lot of common traits between them, but because I’m slightly on the older side, I was an adult when the internet became – not when it was invented. It actually was invented the same year that I was born, or a year before… But at the point where it hit the general public in a slight way - I was in college, and by the time it became the thing, I was well into the workplace.

So qualitatively, the advent of the internet brought about a brand new ecosystem upon which people could do all sorts of new things. I would say it was like putting up – like, if you’re in a classroom, it was like putting up a chalkboard on the wall that people can then go draw. They can draw mathematical equations, they can doodle, they can do whatever they want, but it gave them a new medium upon which to communicate and do stuff and interact together. And so it was that baseline.

AI is a bit different. AI is – it has a similar revolutionary quality, obviously, but it’s expanding on that connectivity and saying “How can we get you what you need faster, and more intelligently, with aid along the way?” So it’s apples and oranges, but they’re in the same fruit bowl. A little bit of a strange analogy there…

Yeah, it’s interesting that you bring up the element of sort of creativity and communication… So there’s probably some parallels, in the sense that I would say there are many people treating these sort of AI models and what they’re building with it as a very creative, new – I don’t know if you’d call it a new canvas on which they’re painting, but definitely they’re trying things that are new and interesting maybe, that haven’t been done before, and are very generative, and… But some of those things, even if you think about something very much on the creative side like the Udio type of thing, the music generator that we talked about a while back… I think you could make an argument “Well, that is an efficiency builder, because you could make a bunch of music really quick for your YouTube videos, or a bunch of music really quick for your ads”, or whatever you’re running online. But I think also some people are using it as a creative element in and of itself, and doing maybe new and different things, or mixing things in ways that people hadn’t done in the past…

And maybe there’s other examples that are better in my mind where it kind of – it’s almost a both/and type of situation. So I’m always maybe a sucker for the third option, where it’s not like clear-cut on one side or the other side… But this third option of “Yes, it is about efficiency gains”, but I think there is an element of net new things that will come out of the AI space with these models, that maybe are hard to predict right now, like they would have been hard to predict in the rise of the web… It was probably hard to predict what an Amazon would become…

[00:36:19.11] Agreed.

…when people were kind of goofing around and making websites to do this or that. Maybe it would have, maybe it wouldn’t have, but the level at which that sort of company has shaped culture at large… Not even just like commerce, but culture at large - maybe we just don’t know yet, is one way to put it.

I have a couple of thoughts on that. First of all, there is creativity in these AI models, and some people argue against that even today. They’ll say they don’t invent wholly new notions, and stuff. They take things that are already out there and they combine them, and stuff like that. And there may or may not be merit to that. But what I can do is I can compare it to myself and other humans that I know, and I’m an extremely creative human, but I’m creative – I have strengths in certain areas of creativity, and big weaknesses in others. And I’ve spent a lot of time trying to compare myself to these tools that I’m using in that way.

So I am very good at creating out of nothing a software system in my head, and understanding all the right things to put in place to do it, even if it’s a fairly new way of doing things. And that’s a strength that I have. I’m terrible at drawing a beautiful picture, or painting, and getting that out; even if I can envision it in my head, I can’t do that. And what it’s made me realize seeing these tools that I’m using that are producing these capabilities that we’re all using, all of us listening to this are using every day these days, is it’s made me really question the sanctity of human creativity.

At the end of the day, I’m a big believer that everything is mathematical, whether you agree or disagree with it, that we’re biology, we’re based on chemistry, which is based on physics, which is based on math. And that kind of science stack that I tend to think of us as, whether something is silicon and producing stuff from its capability, or is biological in nature, I spend a lot of time going “How special is what we as humans create?” So maybe we just kind of acknowledge that we’re bringing things to bear, and these new tools that we’re all using every day bring things to bear, and we can be more productive and capable by combining our talents and doing stuff.

So I don’t tend to be in either camp. I don’t tend to be in the “This is amazing new imagination from computers. My God, what’s the world coming to?” And I don’t tend to be in the “This is just more of the same. I’ve seen this before and there’s nothing magical about it.” I’m a little bit in the middle, and maybe a little bit more nuanced than that. It’s a long way around to an answer, I apologize.

Yeah, I definitely understand that. And I think from my own worldview, and even my faith perspective, I would think of a sort of different special way in which humans exist… But at the same time, we have created a lot of creativity with the tools that we create, and the technologies that we create. And I think there is something beautiful about the fact that we are acting out as creators creating things that are creative in and of themselves. And so we’re kind of acting out the – I don’t know how philosophical we’ve got on this show up to this point, but…

We can afford a moment here, at the end of the show.

Yeah, exactly. This is the end of the show. Yeah, I would say it’s kind of a beautiful thing that we as human beings are creative, and we create things that in and of themselves could be conceived to be creative also, and we co-create with those things. I think that’s really cool, and I think that’s an element of what we’ve done with technology over time.

And so yeah, I think my perspective is maybe we just haven’t seen what we are to co-create with this technology moving into the future, and how that will shape culture. I think that’s going to be a longer time period than maybe the one to two-year Gartner hype cycle time period, that we actually see “Yeah, this is shaping culture, because people know about it now”, but I think there’s a deeper way… People knew about the internet at the sort of hype of the internet coming out, but really how the internet would shape culture and shape things like what social media and other things have done took a long time to realize. So yeah, I think that we have to wait a little bit for that, from my perspective.

Great perspective you have there. And I would encourage our listeners – you know, we had a little bit a moment of finishing the show up with kind of sharing our views on this… But I think this is important, because we’re all going to see an increasing amount of AI capabilities coming into our lives forever going forward at this point. Our children, our grandchildren… The world is changing faster now than it ever has.

So these are thoughts that I hope you’re having as well, evaluating how you see yourself in this world, sharing a world with these technologies that are increasing… And if you haven’t already, I hope you will join our Slack community, where you can engage Daniel and myself directly, and share some of your thoughts on how all this might work going forward with creativity, and with these other topics we’re having. Because we’d love to hear your thoughts, and for what it’s worth, we build these shows off of a lot of those conversations that happen in the Slack community, where people are showing interest. So please, engage us there, share your thoughts, including the philosophical ones; don’t be shy. And I’m looking forward to hearing what some of you out there are thinking yourselves.

Cool. Well, thanks for having the discussion, Chris, and I hope you can have a good week as you enter into more fun AI work.

Sounds good, Daniel. Stay cool in the hot summer weather, since we don’t have that AI climate control quite yet.

Yeah.

I’ll see you next week.

Alright.