Today, Chris explores Shopify Magic and other AI offerings with Mike Tamir, Distinguished ML Engineer and Head of Machine Learning, and Matt Colyer, Director of Product Management for Sidekick. They talk about how Shopify uses generative AI and LLMs to enhance their products, and they take a deeper dive into Sidekick, a first-of-its-kind, AI-enabled commerce assistant that understands a merchant’s business (products, orders, customers) and has been trained to know all about Shopify.
Matched from the episode's transcript 👇
Matt Colyer: [00:23:46.07] So we’ve got several products that are AI-enabled or magic-enabled at Shopify. So Sidekick is kind of the main one, which we’ll talk more about… But to give you a general idea, it’s a tool that helps merchants find their way around Shopify, but also answer questions about their business. So you can think of it as like the co-founder they wish they had, that’s available 24/7 and isn’t judgmental. So that’s kind of like the Sidekick idea. And then we’ve got a variety of other ones as well. So we have a lot of imagery… So it turns out shopping - people want to see what the thing is before they purchase it; not terribly surprising. And one of the things that merchants often want to be able to do before they’ve scaled up to a whole team that has a studio, and a photographer, and the rest of it, is they want to enhance the pictures that they do have. So at the scale that they’re at - this is where technology, again, bringing back the best from the frontier, and making it accessible to all of our merchants is exciting.
So there is technology that’s out there now, that you can essentially describe what you want to do to an image… You’re like “Hey, my background’s a little bit messy. Can you replace it with a studio background instead?” Because we all know that the nice, white studio background that looks like the object’s floating in space. You can do that in real life, it’s just really hard and expensive, and you have to know what you’re doing. And there are very few people who know how to do that well. And so it turns out – we’ve created models that can do it fairly well as well. And so bringing that technology back - that’s one of the products we do offer integrated into Shopify today, is background generation.
So merchants can import an image that they already have, replace the background with something that’s more on brand… Like, say they want to set their coffee to the background of a jungle, right? They can place it on a table in front of a jungle, if that’s what they would prefer. Or they could do it into the void of white space. So lots of exciting opportunities with that.
Another area that we’ve been investing in is that we have a product called Inbox, which allows our merchants to talk with the buyers that they have on their site, and if a buyer has a question, like “Hey, what’s your return policy?” or “Where’s my order at?”, they can interact with the merchants through Inbox. And so one of the things that we’re offering today is that we look at all the merchants’ policies, and all the other things that they’ve given to us, and then can help formulate answers for those common questions. It’s like “Well, what’s your return policy?” It’s like “Well, we’re pretty sure that this is the answer.” And then we can suggest that to a merchant, who then says “Yup, that’s right.” Or if that’s not right, they can adjust it to be correct, and then send it. Merchants love that, because it saves them time for answering a lot of those repetitive questions. And then those ones that are a little bit harder, they can write them themselves just as they would before.
And then the last one that I’m thinking about is – or again, going back to that product and merchandising kind of task, is that oftentimes merchants are uploading a lot of these at the same time. So they don’t always capture all the metadata in the first go… And so this is, again, where we’ve created models that actually can help with that.
So if you upload an image of a white flower dress, we have a model that can actually understand what that picture is, and suggest that “Hey, maybe this is a white flower dress”, and this should be categorized under dresses, and under Cotton. And maybe it can suggest “Oh, it turns out –” If you upload multiple different colors, it’s like “Well, maybe you want to create product variants.” So that’s some of the other technology that we’re kind of working on today, is using the data that we have from merchants, enabling them to more expressively describe their products through our sites.