Generating "hunches" using smart home data 🏠

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International, and I’m joined as always by Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How are you doing, Chris?

I am doing fine, I’ve just been crazy busy. We have this beautiful spring weather, and I don’t know if I’ve mentioned this before, but my wife and I are planning to move, so we’re thinking about a new house that we’re gonna build on a five-acre plot of land… And you know how I am with all the animals I love [unintelligible 00:03:11.23]

Right, right… Following the trend of people getting a little bit further out after Covid… I know there’s a lot of people moving out of various places, like San Francisco and other places, after working remote for some time, and all that.

Yeah, the price of wood and lumber for building is at an all-time high, so we’re trying to figure out – we could do it soon, but we’re trying to figure our timeline… But I’m having a blast trying to think about what I want in my new home.

Is it gonna be a smart home?

You know, I have been bringing that up with my wife quite a bit in terms of what could we do – and of course, I bring it up and she rolls her eyes at me, because it’s the kind of thing she expects from me… But I’m pretty excited about this idea of a smart home. We’re building from scratch, we can do all sorts of cool things… And I need some new ideas, my friend.

Yeah, I mean, we’ve gone locks, thermostats, some other things… I know at my wife’s business they have a bunch of Alexas around the work, because they play various music around, and they have them in the offices actually… So there’s a lot of choices. I think there’s a meme from Silicon Valley where they have the smart fridge, and they all hack into it and display all sorts of profane things… But yeah, there’s so many choices out there now.

I’m excited, because I’m definitely interested in that topic, and today we’ve got a chance to talk a lot more about it with Evan Welbourne, who leads the smart home machine learning team at Amazon… So who better to inform you about your new house, Chris?

I’m looking forward to some ideas here. Yeah, absolutely.

Welcome, Evan.

Yeah, welcome.

Thank you so much. I’m so glad to be here.

Yeah. Well, before we jump into all that good smart home stuff, could you just give us a little bit of information about yourself and your background?

Yeah, sure. My background is actually focused primarily on the internet of things. I’ve worked in that space since the early 2000’s when I was a grad student at the University of Washington, studying computer science… And at that time I was working on things like sensor networks, RFID, GPS, some of the really early smartphone stuff. A lot of that work [unintelligible 00:05:30.10] for the context of the customer or of the user from all that diverse sensor data, inferring things like running versus walking, or are they at home or at work, are they commuting… And then trying to build and evaluate applications using that type of intelligence, so things like activity trackers, reminders, various types of assisted living, location-based social networks, things like that. So machine learning was always a tool…

I’m wondering, is that sort of activity tracking technology - is that really a product of some of the neural network/deep learning boom of recent times, or have people been trying to do this sort of activity tracking sort of thing for some time, with more or less success?

Oh yeah, it’s been going on for quite a while, in research at least. More than 20 years people have been just experimenting with simple things like accelerometers, or motion sensors, and then can we use – I was using decision trees quite often, just the most basic of models. You can get pretty far with basic models, but of course, when you’re really trying to scale something and make it work for everyone, you’ve gotta use the more sophisticated approaches.

Yeah, it makes sense. So how did that activity tracking sort of work lead into later things in your career and where you’re at now, thinking about smart homes?

Yeah, so by the end of my Ph.D. it was clear that machine learning wasn’t just a tool, it was THE tool… So a lot more focus on machine learning, and then at that time the iPhone had just come out and it was becoming clear that smartphones were probably gonna be the technology that would kind of carry us to the next wave of internet of things, and we started realizing some of this vision.

So from graduate school I went to work for Nokia Research, which was still at that time the big mobile phone company in the world, and I worked as a scientist there for a few years. Then I went to Samsung Research and I led the Device Intelligence group there. That was very similar kinds of work - a lot of on-device machine learning, running algorithms on the phone or on a wearable to infer fitness activities, or learn preferences of the customer to recommend content, that sort of thing.

Then fast-forwarding from there, I’ve more recently come to Amazon, attracted mostly by what felt like the next big wave in consumer IoT, which is smart homes… So yes, as you said, at Amazon I’ve been leading the smart home machine learning team.

Just as a start - because I suspect most listeners kind of think they know what it is, but I’m curious if you can tell us how you and Amazon think of a smart home. What is it? …just to get us all on the same idea of what that phrase means, because it’s been marketed about over the years. What is a smart home when you work at Amazon?

That’s a great question. I think there’s various ways to answer the question. I think to do justice to the area, I have to start with saying that smart home - really, it’s been around since at least the ‘80s. Smart home technologies - there’s like X10 networking, there’s devices that are connected and you can kind of program your home… That stuff’s been around for years, and there’s been a lot of DIY folks that are just really heavy into that, and they’ve been developing the technology for years as well. So that’s sort of one slice of smart home, is that really sort of techy DIY wire-up-you-devices to automate your home. A lot of that home automation.

And I think when we think of smart home at Amazon and within Alexa, of course, we’re accepting that history and also sort of looking forward about “Well, if we’re really gonna add intelligence to the home, what can it really do for customers? And how do we make it really easy?” One part, of course, is we’re gonna get Alexa into the picture, so now we have voice control of the home. It’s kind of like the great enabler, the great simplifier. You can say “Alexa, turn off my light” and she’ll turn off the lights; a very simple type of interaction.

But the other piece of that that we think about a lot is what we call an actually smart home, which is not just sort of a fancy remote control for your home, it’s not just sort of like “I push this button and then that automation happens”, it’s having a home with this intelligent assistant, Alexa, that can really do things on your behalf, things that are really valuable to you, to help you achieve high-level goals like living more sustainably, or just being more comfortable, or keeping your family safe.

Could you give some examples of how you might implement some of those ideas? It can be anything you want, but I’m just trying to wrap my head around it.

Real or envisioned, maybe.

Yeah. It doesn’t have to be something that you’ve done yet, I’m just curious what’s in your head; as we record this, we’re still in our houses, in the late (hopefully) pandemic period, and stuff… What might I be doing going forward with that? What could I do now, and what might be something that you’re thinking in the near term? We’ll talk about the distant term later on.

Yeah, so I think three things come to mind, three forms of control or interaction with the home I think is a useful frame to think about. One is what we call directed control, and that’s where we’re explicitly controlling a device with [unintelligible 00:10:23.27] with an app, or with voice commands. That’s where I would say “Alexa, turn on the lamp” and she turns on the lamp. Or you can control groups of devices, all of that. So that’s one mode of interaction that can simplify the management of them home in kind of an everyday experience.

Then we have, as I was describing, this more classic mode of smart home, which we think of as kind of programmed control. This is where the customer is pre-specifying procedures that they want to happen using a program. The customer is thinking about signals, logical conditions, the actions that they want to happen… Maybe like shutting down your house at night by saying “Alexa, good night” and then all my lights shut down, the temperature goes down… All the things that I want to happen will happen. So we’ve sort of exposed that type of interface as well with our Routines product. A lot of people really are excited about that product as well.

More recently, we’re also thinking about - well, okay, so what’s the next actually smart experience? …and we’re thinking of this kind of mode where it’s more intelligent control; this is where we’re trying to further simplify the experience for customers by having Alexa more autonomously manage their home. Here, Alexa is gonna have what we think of as algorithmically derived intuitions or hunches. We’ll probably talk about that; that’s a name of a product, Alexa Hunches. And then the customer - they really just have to focus on their own life activities and think about their high-level goals of living more sustainably.

Alexa may have a hunch about “Well, you might want to turn off the light in the basement. It looks like it’s on anomalously.” Or she’ll automatically turn down your thermostat to save energy while you leave home. Or you may have the goal to help keep your family safer, so Alexa might have a hunch that there’s a door downstairs that looks like it’s usually locked at this time; maybe you forgot to lock it. Do you wanna lock the door? And she’ll lock the door on your behalf.

So kind of starting out with simple things… But if you look forward, there’s all kinds of things that we can do on behalf of customers in a smart home. If you think of Alexa as really your personal assistant who has this superhuman power of knowing all about things like energy consumption, and how to reduce it, or observing at any one time how the status of your locks or your security system and all the other devices and sensors in your home that Alexa can manage.

One of the things that I’ve been thinking about more and more, partly based on some of the conversations we’ve had on the podcast, but partly based on some of the work we’re doing is human perception around AI technologies, especially when they’re introduced in a new sphere… So one of the things that I’ve realized over time is, for example, if you think about the smart home, I think there’s a lot of people out there that are used to having a dumb home, right? So you’re introducing this whole new way of thinking about your home and devices in your home, and essentially trying to introduce a new framework for people in terms of how they interact with things that they aren’t used to interacting with in that way.

I’m wondering if you have any sort of perspective or learnings on that front in terms of where are we at on that sort of human interaction spectrum with smart home technology in terms of maybe younger generations, older generations, or maybe geographically where people are catching on to this technology more quickly, and where the challenges are in terms of helping people adopt this sort of technology in a place where they’re not really used to having a smart device.

That’s a great question. There’s a lot there. I think there are many challenges, given the diversity of customers that are now interested in smart home technologies. The long history includes lots of really tech-savvy people who were gonna make it work for them one way or another…

Yeah, there’s a group that’s like “I really want a voice assistant, and even if it’s hard to use at the beginning, I’m gonna push through and I’m gonna use it.”

Exactly. Those are always the super-valuable, early adopters that are pioneering these new technologies. But of course, we’ve got all kinds of other customers that are less savvy. Their interests may not be programming their home and wiring up all of these devices. It may be more about just keeping their family safe, who are like “I’ve heard about smart home, it’s kind of interesting. I’m not too confident about setting it up and using it, but it just seems pretty cool. I wanna give it a try.”

So that’s one of our key challenges, is trying to help customers through that journey, from getting their first light bulb, to being the kind of fully enrolled customer that has maybe multiple devices, and is taking advantage of some of these higher intelligence features. So that is one of the things that my team focuses on, is facilitating that journey with customers, helping them set up devices intelligently, and helping them recover from really common kinds of errors that they run into when they’re setting up and starting to learn how a smart home works.

So I think that was a great segue there in terms of what you were just describing to talk about – I’d love to talk about how you’re really putting machine learning at the central enabler of being able to make all this happen to people, to where they’re able to do that. I know you mentioned earlier that there is a services called Hunches that you guys are working on, and I would love to understand more about that, because I think I’ve been waiting for something like that for years. I’m selfishly thinking about the new house that we’re gonna build, and how that fits in, so I can’t wait to hear this.

Yeah, Hunches is a great example of a new paradigm, within smart home at least… The idea of a hunch is that it’s an algorithmically derived intuition that Alexa has. So for those of us who listen to this podcast, we’re using a model to make a prediction, and there’s a confidence score associated with it, and then there’s this sort of designed experience around it that kind of supports that type of prediction. It’s a prediction with the confidence; we may not be 100% right, we accept that sometimes Alexa is gonna be wrong, but it’s a hunch. Alexa has an idea of something that might be helpful to you, and then there’s this kind of collaboration with the customer around not just “Do we take an action in the case of a certain hunch?” but how do we learn more about what the customer’s intentions are, and what their goals are through this series of hunches; each time we request feedback from the customer and we learn more.

There’s a few distinguishing characteristics to hunches. One is that, as you can imagine, they’re personalized to the customer; they’re always about some particular device in their house, and maybe corresponding to some behavioral pattern that Alexa has observed. They’re also dynamically adaptive to the home and to the customer’s current context; the example of the basement light is still on, and it’s at nighttime, it looks like a time when you might go to bed, and Alexa may reach out and let you know that your light is on, and ask if you wanted to turn it off.

Or in some cases - we can talk later - we also have automatic actions for hunches, where Alexa will just automatically turn it off. But that’s after we’ve sort of built a little more confidence with the customer.

They’re also – as noted, they’re non-deterministic, so we’re not going to deliver a hunch, we’re not going to take an action unless Alexa has high confidence. So that model has to be pretty confident across all the signals that it observes that this is something the customer would actually value before we actually surface it.

And then finally, as described, it’s refined in a loop with feedback from the customer. So every time we’re delivering a hunch, we’re inviting the customer to give us feedback, often explicitly; Alexa may ask “Did you want me to turn off that light?” and they can say yes or no. Or they can give feedback through the app. But sometimes there’s implicit feedback, so if we go and we lower the thermostat, we’ll kind of watch to see if they turn it back up later, and that’s another kind of feedback that we can constantly learn from.

It sounds like that feedback - is it the central mechanism for establishing trust with a customer for the new service? Because we have so many conversations for people who are doing these amazing things with machine learning, and so much of it now is advanced, and it’s requiring people to make that mental shift; we’ve talked a little bit about that already. But is that establishing trust so that you feel you can incorporate hunches into your life? What else do you guys think about in terms of how to get there and what the next steps are? Because obviously, the trust has to be a huge part of the strategy on moving this all forward.

Absolutely. Feedback is absolutely core to this idea of hunches. We think of it as a collaboration with the customer; we’re earning trust continuously as we get feedback from the customer, and learn, personalize, adapt to their patterns.

I think a few other things just about the UX - or the CX, as we say at Amazon, for Hunches - is the kinds of hunches we started with a couple years ago were really sort of these extemporaneous [unintelligible 00:19:45.01] the case of reminding you to turn off your light or lock your door. They don’t happen that often; they’re sort of targeting anomalies, but it’s kind of a delightful experience when they happen. Fortunately, we’ve been able to tune the models so that we’re usually right about those anomalies; the customer does wanna turn off the light or lock the door.

So that sort of spark, even though it’s kind of a simple thing, that spark really earns a lot of trust, and then the customer also feels in control, because they have that feedback and they can say no if it was not the right thing, and we’re not gonna ask them again in that scenario. That definitely kind of helps us build that trust as we go forward.

And then absolutely - the way we’ve continued to work on the Hunches product is to incorporate gradually more and more use cases, and then most recently (just this last year) we launched the Hunches Automatic Actions, where we’re working with the customer in advance to sort of help them understand that Alexa’s now going to be able to take action and adjust your thermostat, or turn off your lights based on her inferences about what’s going on in the home. You can consent to that in advance, and then Alexa will turn off your lights if she thinks you’re asleep.

So that kind of gets that sort of other element of that earning trust, which is explanations for the inferences. So if we have a hunch that we should turn off that light, that’ll show up in what we call the Hunches dashboard within the app, that shows “These are the actions that Alexa took last night, or the last 30 days.” And you can see a very simple explanation for each one; Alexa turned off your light because she thought you were asleep, or she turned down the thermostat because she thought everyone in your house was away… And then you can give feedback right in there. But you’re understanding more about how Alexa is working.

I’m curious just on the practical side of things… I’m assuming as more and more smart devices are integrated in people’s homes, I’m thinking about the data side of developing something like Hunches. It seems to me like there’s all these different customers that could have all sorts of different, unique combinations of devices in their home… So the data is not the same for customer A versus customer B, and maybe the history of that data is not the same… And there’s also geographic factors, or lifestyle factors… So how do you even, from your team’s perspective – do you have any good maybe workflow hints or tips for people that are dealing with this sort of complex data situation, and really exploring that data and getting down to the… Because you have to start somewhere; like you were saying, you’re building in incrementally more and more of these. Any tips for people out there that are maybe dealing with this sort of complicated data situation and trying to get down to “Where should I start in terms of creating value? Because there’s so many different varied ways that I could go about this.”

Absolutely. Well, it’s very insightful. That touches on a really key challenge for smart homes. One part of that is the challenge of providing that sort of consistently high-quality inference across customers, that have so many different types of homes, so many different types of devices, they use them differently, they live in different places… It’s a super-hard problem. And there’s a few – maybe three things. One, personalization is always useful in the early phase. We may not know instantly about your house, or we may not know about everyone’s house, but we can learn how you use your bedroom reading lamp. That’s one thing that we can learn over time some basic things about that, just on your data alone.

Secondly is if you are able to, if you’re empowered to design a feature including the kind of user-facing interaction, then it really helps to build that feedback loop right into it, just like hunches. We propose a hunch and then we get feedback on it. That’s great for gathering training data from the only people who can really label it, about the context of their home; customers are really the main person who can provide you that most accurate label about what they wanna do.

The third area of work - and this is really where I see one of the key scientific challenges for a smart home - is trying to infer information about the home, trying to infer activities in the home across such varying datasets from individual customers. It’s actually pretty sparse data; from an individual home, if we’re just looking at the lighting data, people just turn on and off lights four times a day, five times a day… You’ve gotta wait a while to get data on any given house. So you’ve gotta find ways to learn across customers, not just – you probably start with personalized models, but there’s a lot of investment we’ve made in deep models that are trained across millions of different customers… And there’s even – I could talk more about it, but there’s even some fundamentally new scientific discoveries about what’s possible there, like how much similarity there is between customer behavior in one home and a hundred or a thousand other customers.

I know a lot of the devices these days - people might have a mix of smart home devices in their home, that are from different brands even, and report different sets of data. Is that correct?

That’s right.

So things are coming in with (I guess) different feature sets, different things that are represented by different brands, and maybe even different formats… And maybe there’s some standards around that now. Is some of that even just like synthesizing some of that data together and figuring out how data from different brands of different devices reporting different data is matching up to “Hey, this is lighting information over here, and this is lighting information over here, and it’s reported slightly differently, but we can–” How much work is there in synthesizing that across all the varied devices these days? Or is it more standardized than I think?

Yeah, that’s a good question. So it is a little bit standardized. We have an API that partners use to report any kind of data like that. That’s certainly helpful. Of course, the reality is, across many different kinds of companies, partners, devices, the quality of the data varies, and certainly the content of the data varies. There’s more than 140k different types of devices that connect with Alexa today… So that’s just a lot to keep track of.

Yeah, no big problem… [laughs]

No big problem, yeah. There’s a lot of different data there… So it is true, there’s a lot of variation across partners, there’s a lot of variation across device types, and there’s not really any one universal solution to just immediately cleaning up all data of every type, unifying it all into one big model, but there’s sort of categories of data. A lot of it is time-series data, of course, and so you see these kind of point events of a light turned on, or changed brightness… That’s the same for all different kinds of lighting, so you can develop sort of a model or understanding of lighting.

Thermostats are a different picture, security systems are different, smart plugs - they’re kind of similar to lighting, but they also have different types of usage patterns; you might wanna model them a little bit differently.

So you tend to start looking at devices in categories, and standard practice - you might wanna build a layer kind of above that raw data to just kind of… If you have lighting devices, for example, you may wanna build a layer that tries to smooth out any kind of noise in the data, and offer you, for example, the state of the device with a confidence score, rather than just the raw information.

Okay, yeah. So you’re building that sort of middleware layer that does some sort of synthesis or correlation of that data together, smoothing and that sort of thing, with confidence scores… And I would guess that something like hunches would then rely on the fact that “Hey, I know about lighting data, and the trends of lighting data generally”, so I’m guessing that would help with something like that… Is that right?

Yeah, definitely. For the kinds of models we use in Hunches and in most of our products, we do try to incorporate as much information as we can, not just from this sort of time-series type of data, but even just the metadata about devices. Again, if we’re talking about lighting, customers are able to name their light. Sometimes they just leave it with the default name, like First Light, Second Light, but often they name them “Living room lamp” or “Reading lamp” or “Basement lights”, what have you. And even just in the name of the device there’s a lot of information. If we know that something is a bedroom lamp, it’s probably gonna be on in the evening for an hour or two, and then it’s gonna be off overnight, versus a front porch light, that often is on overnight and it’s off during the day. There’s a lot that we can kind of pull in to add to the model… And then by extension, of course, if you think about training the models across millions of customers and millions of different devices, you can start building device embeddings that kind of distill all of that information about not just “Is it a bedroom lamp?” but what kind of bedroom lamp is it. There’s a lot that you can pull together between the behavioral patterns, and then the names of the devices, the way customers interact with them.

So you were saying something a few minutes ago that my brain’s been spinning on a little bit, and I wanna go back into kind of a late follow-up on it… When we were talking about models that could generalize across thousands and millions of people in terms of these activities - turning the lamp on, and stuff like that - as we go forward in time and you’re moving ever more (presumably) into personalization, and really not just supporting all of our lifestyles, but starting to hone in on “What does Chris need? What does Daniel need?”, and our own differences, and stuff, how do you approach that in terms of – you have these tools that you’ve built that can handle these activities or tasks in a large sense… But over time, I might be going through the house and turning on lights in a different way Daniel does, because of some quirk of my own personality, and you’re having to tie my activities into those otherwise kind of mainstay things - light on, light off, you’re recording that, you know that that’s happening… How do you think about personalization as you’re moving into that world where it’s not just about turning the light on, it’s about why Chris would do it, versus when Daniel does it, versus other people? How do you think about a future of where that’s going, and approach it?

I think that’s really interesting… Part of that, if I’m understanding you correctly, is kind of understanding and perhaps modeling the customer’s intention; what’s actually going on here, not just “I think that specific light is gonna turn on”?

You said it much better than I did, actually.

Yeah, I know in a chat/dialogue sense there’s the idea of user intents when you’re looking at smart home data… Like, “Hey, they’re doing this on a Saturday” versus “They’re doing this on a workday, or something.”

Yeah. So the short answer is yes. At many levels we have that concept, and then we approach modeling of customer intents in different ways. One thing to note - of course, even just in the most simple kind of directed control scenario we have the concept of “What’s the customer actually trying to do? What’s their intention?” We can sometimes see that they’re trying to turn on a light and perhaps they didn’t get the name of the light correct, and it failed, but we can still tell that their intent is to turn on that particular light. So there’s just very simple kind of intentions. But if we get into activities on a weekday, or a Saturday, and what would they want to happen at this particular time - that’s where it starts getting really interesting. I feel like that problem – I think of it as modeling the context of the home and the customer. That’s where ideas of the activity of the home, the activities that the customer is engaged in - those are a key part of that picture, understanding that they’re having dinner now, or that they’re all asleep, or that they’re away from home, whatever the relevant activities are, is a big piece of that.

The other way I think of that, and the other thing that’s important and we’re just sort of getting to in the scientific agenda for our team is really understanding how to help customers achieve their long-term goals. So not just turning on a light, or not even just controlling their thermostat in a way over the day that makes them comfortable, but helping them actually save money on their energy bill next month, or helping them stay safe throughout the year with their various security devices and security system… And kind of balancing those goals with each other. If you want to save money, you’re gonna be biasing towards turning lights off quite often, turning the temperature down… But if you want something to be really comfortable, you might bias towards having the temperature up a little more often. Or if you want them to be really safe at home, you also in some cases might bias towards having the lights on, just in case someone’s there and you want them to be able to see where they’re going. So it’s a really interesting problem of balancing these long-term driving goals with short-term actions.

As we’re talking about this, I know there’s gonna be at least a few folks out there thinking about “What about the security as you’re doing this?” Because I’m all excited about getting my home able to basically predict to me ahead of time and do that; that’s very exciting. But there’s gonna be someone out there worrying about what happens when everything is voice-controlled and you have someone who shouldn’t be there. How do you think about it? And I really mean security not in just basic security, but like my voice versus a stranger’s voice, and whether there’s any kind of recognition built into services, or will be in the future, as that becomes more of a real-life consideration… How are you thinking about that level of personalization going forward where some people should and some people shouldn’t, and what’s the thinking around that at this point?

Great question. It’s always a lively area of work, the security and authentication question…

I’m sure.

Hard problems there. One caveat is that my team doesn’t own that space within Alexa…

Fair enough.

…but I can at least comment that – so regarding the security use cases, as you may imagine, there’s certain use cases we just don’t support. You wouldn’t wanna allow anyone to yell from outside “Alexa, unlock the door!” and then we unlock the door. There’s layers of security, whether it’s voice codes, or there’s speaker recognition - that’s another feature that folks have implemented for use in all kinds of cases within Alexa…

Another aspect of this that’s really interesting is personalization in general, and how do we accommodate or deny people who are not part of my smart home. If someone is a guest and comes to my home, are we gonna allow them to control the lights, and the music and all of this?

For the most part, I can say that’s a very interesting problem, and there’s really a lot of product thinking in addition to thinking about authentication and privacy… Like, what kind of experience do we want to create.

You just threw an out that I had not considered at all - it would be very different with different customers of yours, I would imagine - you have guests over and you said control the lights, control the music. “It’s a party?” “Yes.” And other people… It’s an interesting problem, because there’s a lot of nuance there to tackle…

Also, no one wants me choosing their music at their parties… [laughter]

Note to self, okay.

Note to self. [laughter] Chris, do you remember we had Nhung Ho on from Intuit, who is director of data science at Intuit, and she was talking about pandemic times in time-series modeling… And of course, time-series modeling depends on how good is the history of your data. This last year hast just totally blown all that apart for many people, so Evan, I’m curious, for the smart home machine learning team at Amazon - what has that disruption in the history of data meant for your team in terms of maybe new opportunities arose, things you didn’t realize before? But also in terms of thinking about “Hey, maybe we need to do things slightly different to pandemic-proof some of our processes.” I’m curious about that aspect of your team’s conversations over the past year.

Yeah, it had a huge impact. It kind of changed everything in the data that we see, because all of our data is from the home, and one of the implications of a global pandemic is that people are staying home a lot more often than they were, especially – we’ve got at least this big chunk of the early adopter folks who are often tech workers who are probably gonna work remote anyway… So one of the things we’ve seen, for example, as of maybe March of April last year is that suddenly weekdays started to look a lot more like weekends. People are waking up a little bit later, there’s more activity in the home throughout the day, people stay up a little bit later now on weekdays as well… And to that end, we’ve had all these models trained on customer behavior patterns across days, and weeks, and so on, and we had to switch all those up, because what looked typical back in February 2020 is very different from what looks typical in February 2021.

A couple other notes - one of the things that we’ve had to do is rely a little bit more on, again, personalization. We’ve gotta lean into what do we think this particular customer is going to do, rather than any customer… Because people, at least within the U.S, states are opening up and locking down at different times, and we can’t rely on people in Texas to predict what’s gonna happen in Montana. It’s just different places.

Yeah.

So we kind of leaned in a lot more on personalization to an individual customer. And then we’ve just sort of reset some of the assumptions the model has made about things like trips away from home, for example. There used to be, of course, this very established pattern of 9 to 5 work…

Yeah, commuting, and things…

[unintelligible 00:37:35.08] away from home and come back… And that doesn’t really exist anymore. A lot of the times now these trips away from home are kind of short trips, like running out to get groceries and coming back, something like that.

Yeah, it’s true. It’s kind of funny - just to your point right there, and Daniel knows this, and you may have heard this on a previous thing… I’m taking flying lessons, and I told my wife the other day - because I was commuting, I was going off on business trips, and I was off and away from home through the whole day, and maybe for multiple days… And my life has flipped so much that with my flying lessons and the short trips out I realized that I’m flying as an amateur private pilot student more than I’m driving my car at this point… Which was kind of a bizarre realization to do that.

That raises another point that I wanted to ask about, and that is - as you’re addressing smart home and you’re pulling the data from the home, we’re also seeing smart technology being implemented out of the home, and we’re getting automotive smart capabilities and various other things in our lives that are outside our houses… Any thought into how those integrate over time? And I realize that you’re working your way there, and there’s a lot of stuff that has to go forward, but at some point we’re gonna be moving around in smart vehicles, maybe fully autonomous vehicles, not terribly far into the future at this point; we already have some out there. We’re coming home to our smart homes, and so much of our lives are being automated, and yet they’re all somewhat disparate right now. How do you envision that coming together for more of an integrated feel, recognizing that it may not always be just Amazon. Amazon does a lot, and you guys may be in a lot of those fields, but if you’re looking even beyond that, you have a lot of different players doing different parts of life, if you will… How does the world come together in an integrated experience that is what that consumer wants it to be?

Yeah, that’s a great question, and great observation as well. You already see some of them; there’s Alexa for auto, for example.

Yeah, of course.

The way I see it - and also I think it’s aligned with Amazon’s vision - it’s all kind of part of the same problem if you’re focused on the customer, really. We’re just trying to help the customer live more simply, achieve their goals… And a lot of that we can do with just the smart home data, but if you pull in data and inferences we can make about what they do in the car, what they do out in the world, maybe how they’re using their smartphone, there’s just a lot more you can understand about their intentions and what they want to happen.

It’s sort of obvious for us… Alexa is kind of the unifying element here. We have this ambient assistant who is able to stay with you, whether you’re at home, whether you’re on the go, with your smartphone; she’s kind of present there across these elements of your life and can kind of help tie things together. I think that’s sort of the metaphor we’re using to unify the data, as well as the experience for the customer.

I think the opportunities are huge. As I noted, I’ve spent years working on smartphones and mobile inference; I know there’s a lot there to add to the big picture.

That’s super-exciting. I think that’s a really good way to tie things up here at the end, is thinking about that ambient smart technology permeating your life. I know it’s really exciting, there’s a lot of challenges, there’s a lot of questions ahead of that, but I’m really excited about it. I’m excited to see what Chris puts in his home, and we can hear about that.

I am, too.

But yeah, I appreciate you joining us so much, Evan. This was a really great conversation, and I’m excited to try out some of these hunches and other things, and see how they develop over time. I really appreciate the work that your team is doing, so thank you so much for taking time to join us. We really appreciate it.

Excellent. Thank you so much. I look forward to listening to the podcast.