Machine learning at small organizations

Welcome to another episode of Practical AI. This is Daniel Whitenack. I’m a data scientist at SIL International, and I’m joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

I’m doing just fine. How are you today, Daniel?

I can’t complain. It was definitely the first meeting-heavy day of the new year for me.

Me too.

I was really enjoying those, “not everyone knows that I’m back at work, and so I can get stuff done” days… So yeah, now everybody knows. But all good things. I’m working on fun stuff, so…

Hey, we’re getting to talk AI now for the next few minutes, so we’re good.

Exactly, yeah. And really excited to get to have Kirsten Lum with us. She’s co-founder and CPO of Storytellers AI. Welcome, Kirsten.

Thank you for having me.

Yeah, it was great to get to connect with you on Twitter and get you scheduled for the show. One of the things that we were chatting about when I was first talking to you about potential topics for the show was machine learning at small organizations, which - I definitely like the idea of like discussing this, one, because I don’t think we’ve like coherently discussed this on the show in the past, and kind of alluded to it at certain points… But also, I got my start as a data scientist working at startups, at smaller organizations, so I definitely know both some joys and some pains from trying to do machine learning or do data science at a smaller organization. What got you started thinking about this topic in particular, and - what I understand from Storytellers - also kind of engaging with a lot of these small organizations in this type of work?

So kind of like you, I started in this field in this range of companies, from small to large… In particular, one of the things - I’ll go a little bit into my background, how I got into data science and why small organizations ended up being my passion… I don’t actually have a degree in data science, I have a degree in English; that’s my background. So I came in sort of a transplant into this field… But I came up through startups; that’s how I got into tech, was working in startups. And one of the things that you learn at startups is to do kind of the task that’s at hand; you just figure out what that task is, you will do it, you make things work… And so I’m so grateful that that’s where I started my journey in tech, was in startups, because that really is the underpinning of how ML at small organizations works.

I ended up going into data science through analytics. I was doing marketing for a long time, got my feet wet, and “Oh, wow, if I have data about my marketing campaigns, I can do these things that, if I didn’t have the data, I wouldn’t be able to be nearly as successful.” So that’s where I really found my passion for data. And my first real data science project - I took this marketing process that was a bidding algorithm, that was being run out of Excel, and I converted it into a Python script. And that Excel process was taking like 30 hours a week, and with Python, it took eight seconds. It was magical.

Talk about return on investment, yeah…

Exactly.

Exactly. I actually learned Python to do it. It took me two weeks to learn enough Python just convert this process into a Python process. And that was like “This is just too fun and too powerful of a tool to not spend all my time doing this stuff.” So when I think about – that project was actually at a large company, but that large company didn’t have a lot of access to data scientists; it was a pretty nascent field at the time. So my knowing Python, being a marketing analyst, and just being like “I’m gonna roll up my sleeves, I’m gonna stand up this process in Python and just do it myself” had a huge impact on the business. They actually changed this entire part of the business to be based within those tools, because of how much more powerful it was to not have people clicking buttons in Excel for hours a week. So that’s where I got really passionate about it. I saw how one person who had these tools could come into an organization and make meaningful change, not just organizationally, but actually for the business itself, for growth, by using these techniques.

So out of those experiences - because you being a single data scientist in that context you were able to make a big impact… And so a small organization - I can imagine a lot of small organizations saying things “Well, we’re not a big tech company. We can’t support this type of work.” Or “We’re not in a position to do predictive things”, or “We don’t have enough data”, or whatever it is. What are some of those stories that you’ve heard or cases that you’ve heard where maybe a company is selling themselves short in terms of the opportunity that’s there around data science and machine learning?

[06:26] Yeah, I love that question, because I think it is really selling short, especially now. Maybe 10 years ago, I think that those were very valid reasons to not take a step into data science or predictive modeling. Now, the tools are so much better in terms of being able to have a single person who’s making a big impact with these techniques. It’s much, much easier, even than when I started, to be able to do that.

I would say the top reasons that I tend to hear are 1) “We don’t know how to even start, in terms of how to hire someone.” That’s a big, big barrier. Knowing how to evaluate if someone’s going to be able to come into your organization and make an impact is pretty tough. I also hear a lot about not knowing if their data infrastructure is ready, or if their data quality is ready… Those are two big questions that are fairly hard to answer. How do you know you’re ready, your data is ready for data science? Do I have enough data? Is it clean enough? Is it stored the right way? Those are all big questions. And then the final one is, “I don’t know how I would integrate this person into my existing business such that their output gets fed in as an input to all the things that are already running? All of my marketing campaigns, all of my website analytics, all of that. How do I get this new discipline integrated with all the other technology?”, especially with small companies having – you know, these people are always stretched very thin; your database administrators are stretched very thin, your engineers are stretched very thin… So do I have the margin to incorporate this new technique?

Do you think, to that last point, that just kind of FUD, fear, uncertainty and doubt really kind of play in from a management standpoint? I had a similar experience and I really walked away from that; it was the last small company I worked at, and I just don’t think they thought they could do it. And they didn’t. And I ended up leaving as a result of that. But do you think that’s a common situation that small companies run into?

I totally do. And to be fair, I think that – interestingly, I don’t think the data science community is doing a fantastic job at creating literature that is accessible to someone who’s in a small business to be able to say, “Here’s how you get started.” I think a lot of the literature in data science – we’re still really in this experimental phase of like “What new models can we build? How can we push the state of the art in terms of accuracy?” and that kind of thing. But the literature really doesn’t have an angle for, say, a CEO of a small company that has great analytics, but is actually ready to take the step. There’s not that bridge that says, “Here’s how you do it. Here’s how you go from an analytics, data-driven company to a data science-driven company or a predictive company.” And I think that’s right. It’s hard to dispel that fear when it feels so mysterious, you know? So that was one of the things I wish that the data science community had more of, is non data science-facing literature about data science.

That’s a great point.

So I just had this interaction… My wife is an entrepreneur and owns a small business, and we just had the conversation – because she just saw Jasper, which is a copywriting assistant, like generative language solution that helps you write, and that sort of thing. That was the first time – she’s, of course, been married to me for quite some time, but it was the first time where she kind of made the connection, like “Oh, I could have my own people be augmented by this sort of technology in a way that’s non-threatening, or not a lot of work.” And she was able to talk to her team about that.

[10:14] I’m curious - because in that scenario, it’s like people that are already inside the company, that are kind of… Now that tooling, sort of like machine learning or NLP tooling is getting more user friendly and marketed in that way, they’re seeing how they can use those tools to advance their business. But then there is a need where – like, at her company I help do some of the forecasting, or other things that really there isn’t a great kind of off-the-shelf tool for, but it also isn’t that hard. If you know Python, and you can import Facebook’s Profit, then you’re good, and boom, there it is. Okay, I’ve got it. But it’s not the same sort of approachable thing as like a Jasper, or something like that.

So where do you see, moving into the future, the limits of this low/no-code kind of people leveling themselves up, versus the things that are gonna be really valuable for a data scientist to do in a small organization moving into the future?

I love that question, because it reminds me… Back when I was learning to be an analyst, there was always this idea that eventually BI tools will get good enough and you wouldn’t need a BI, right? Like, you’d have these – like, Tableau is just a step towards whatever the no-code interface is, and some of these companies would be like “Your marketer can just go in an interface and click some buttons, and they’ll get a report that really just answers all their questions, no BI needed.” And that didn’t prove out to be true, right? Like BIs are still like this very important role. But there are these other roles and other tools that can fill some gaps around there.

I love the idea of Jasper as this friendly interface to be able to do this particular task of content generation. I think it’s a fantastic use case. And also, your example of Profit, needing someone to come in, use a prebuilt library for doing forecasting in order to get this very simple output that really changes a business’s ability to guide itself, as that example of “That’s a hard task to get rid of.” And the real core of that that I see is that data is different from organization to organization. When I was at Amazon, we’d always talk about “Solve for the constants.” That’s a constant. Like, you go from one business to another; someone’s using Stripe, someone’s using Square. This person is using Salesforce, this person is using HubSpot. And you have that problem that just multiplies when you look at all these different technologies that come together to underpin their business… That’s where it’s like the BI task; you’re always going to need someone who’s able to reconcile, make sense of the data, and then maybe push it through something that to them is like “Oh, this is easy”, right? Like training a forecasting model in Profit is easy for a data scientist, but it’s unreachable for a CEO of a company.

So that role, I think, just like we’ve seen with BIs - they were always needed, even when we had all these fantastic low or no-code tools for analytics. We still needed BIs that were able to do that heavylifting of reconciling, of telling the story, of creating those interfaces. I think data scientists have that same kind of role. They’re always going to need to pull the data together, build that model, explain it to the person, explain how to integrate it… And that role I think is the most fun. If I’m honest, that’s the most fun data science role to me, is that particular role.

Yeah, I’ve always really enjoyed that. I’ve used this analogy with Chris a lot of times, that oftentimes I really enjoy this sort of idea that data science is more like cooking than you’re at the chalkboard, doing math problems. So you have a recipe, but your recipe doesn’t quite work the way that the tutorial on Medium is telling you, because you don’t have Stripe, you have Square. But you kind of modify the recipe just enough to make it work for your scenario. And then you don’t have the same ingredients, but you adjust the recipe and you go from there. I think that’s a really good framework to think about, and I also really enjoy that.

[14:19] It does make me wonder though, for like a data scientist or a machine learning person at a small company - like, I could be a machine learning engineer at Google, on the Translate team, and my every day is like Translate, right? What I’m thinking about is machine translation. Whereas at a small company, like you said, you’re in an environment where you are the data science resource, or there’s a small number of those. How can a data scientist or a machine learning person at a small company deal with that sort of variability of like one day you’re dealing with dashboarding and another day you’re dealing with sales forecasting, and that sort of thing?

Yeah. It’s a fantastic question. I go back to – you know when we used to talk about T-shaped data scientists? Like, across the top you need to know a little bit about pretty much everything when everything was a little bit narrower… You’d know a little bit across the top, and then you’d have one area - your, again, translation, a large language, or you’re computer vision… I tend to think of these roles - there’s not even a letter that describes this, because there’s so many… You need to have a relatively shallow, but working knowledge of the entire cycle. And so instead of thinking of your role as a data scientist as training models, or even producing models, the role of the data scientist is to convert the data into some business value using data science techniques. And that means having a working understanding of all the elements of the machine learning workflow, like data infrastructure, having a working knowledge of how to stand up a simple database, how to pull data from various sources into that simple database, and then your feature engineering layer, which is really ETL. That’s one of the areas I hear a lot of data scientists say “No, that’s not my job”, right? ETL is actually at a small company, your job. It’ll be in a database, but you have to do your own really heavy duty ETL to build your features.

Then you’ve got your training, the thing that everyone kind of gets into data science for, it feels like; the training your models. But then afterwards, you also need a simple way of deploying your models and a simple way of monitoring and testing the impact of your models. And so I think about a role at a small company is needed to encompass all of those components, but it doesn’t need to be to the level that you would see if you were in a large company for each of those components.

So sometimes people hear that and they’re like “Wow, that just sounds awful.” But the reality is it’s because they’ve seen someone who’s specialized in MLOps and they’re thinking, “I have to do that, too.” It’s like “No, no, no. You don’t have to do MLOps like your MLOps peers.” You just need to know MLOps well enough to where you could deploy your own models. Have a very simple batch pipeline that you know how to stand up in a technology that’s available to you at your company, and a very simple - real-time, if you can do that - real time inference pipeline. And those are your recipes. Just plug your data into those two patterns, and you’re good to go. Very rarely do you actually need to come up with a whole new way of doing MLOps at a small company; you can usually stick to a few simple patterns.

I’d like to extend a little bit what we were just talking about and kind of ask… We were kind of talking about patterns or recipes if you’re in a small business, and you addressed a little bit about MLOps and the fact that you don’t necessarily need to go to what the large company person does, who specializes entirely… But that does raise the question of there’s a lot of those tasks to dive into. And so for a person to be successful in the small company environment, where they’re by necessity forced to be a little bit more generalist and cover a lot more things, but maybe not at that depth, what are some of the patterns or other recipes, other than MLOps, that you’ve identified, that like if you were bringing somebody in new, you would say “Focus on that, and that, and that, and your life off the bat will be a lot better than just diving into the deep end without any help.” What would you tell that person?

Fantastic question. So one unintuitive thing I would tell folks is build your project or product management skills. Having a very strong framework for how you manage a project from end to end is pretty much - you’re going to need that muscle to be quite strong… In particular because you’re going to be shepherding, sometimes from all the way at the beginning; like, data isn’t even in a database, and you’re just talking to the salesperson who has a problem you’re trying to solve, and like “Alright, we’re starting from the very beginning, from scratch.”

So having a very strong framework that really goes all the way, end to end, from data in to data out to your product is critical. And a lot of people use CRISP DM as sort of their framework for that… I find it’s not even quite specific enough for as a practitioner to move things from end to end. But that’s gonna be a little variable by company. But I do tend to break it up into these five stages of, you know, having an interview format - how do you know that you have gotten the requirements from the person who’s going to use this such that you have the inputs you need for your model. Having a framework for interviews; having a simple recipe for standing up a database, or if you need to set up your own database, or have a very simple architecture for that, that you can plug all of your projects into so they’re sort of centralized.

The other thing that I really recommend is – well, here’s just a side note. Almost all problems in this space are tabular. It’s tabular data, that’s what you’re going to do. And if you’ve been on Twitter about tabular data, you know it’s gradient boosted trees. Just use gradient boosted trees; that’s your baseline. Don’t worry about baselining with linear regression, or the simpler models, or random forests - don’t worry about that. Just stick to a very simple baseline with your gradient boosted trees. You’ll probably get a pretty good model out of that.

[22:02] And then the last part that I didn’t mention in that earlier list that I think is super-critical is having a very clear baselining process. How do I know when I’m done? And then when you’re done, put down your pen. Don’t worry about trying to get to state of the art on every problem. Just know what your baseline is. How do I know when I’ve built a model that’s actually going to improve this business, so that I can stop working on this one and work on the next one? Because two models will have a much better impact on the business than one perfect model.

I’m having all sorts of flashbacks to my work in various organizations, all the time while you were talking. Some good and some painful… And I’m thinking about a couple – so one of my experiences in like a small startup environment is like getting in this sort of… I don’t know what to call it; like, you become the replacement for Excel function, where like “Oh, email Daniel. I think he can like merge two columns together”, right?

Right… [laughs]

And then they send you to like Excel sheets, and you’re like “Okay, I can do that in one minute.” So you do it, but then you get this increasing number of tasks, and then you just do that all the time. And the second scenario is “Okay, I’m going to try to build out this roadmap, and like this project plan”, but things get shaken up all the time in small companies, and it’s like “Okay, I had this plan to optimize my pricing model over the next six months using these A/B tests”, or whatever I was doing, right? And then the CEO is lik “Oh, we’re not meeting revenue this month. We need to just change our pricing structure entirely.” Does that spark anything in your mind in terms of – like, the level at which you can do product or project planning within a small company while still managing that sort of flexibility and random tasks that come up - any recommendations there?

Yeah. It’s funny, because you’ve sparked a whole bunch of memories too of doing that. I feel like that’s a universal experience. Once you know how to use data, people will not let you do anything with data. Yeah.

Right.

I think number one is, going back to the idea of solving for a constant - that is a constant in small business, is that strategy is changing very rapidly, and so building that into the way that you approach data science I think is really important. I find that one thing that helps with both of those scenarios is being able to deliver a result that baselines everyone about what good data science does for the company.

So if a good data scientist is able to build a model that increases open rates on emails 50%, it’s much less likely that they’ll email you to ask you to merge the columns, because you’re the person who optimizes email open rates 50%. So that tends to work super-well, which kind of goes back to the idea of like you need to have an end-to-end process, you need to be able to deliver relatively quickly on a quick timeline, and know how to measure your results, so that when you do start getting those questions, you’re like “You know what - I’m super busy on this pricing model. If you can wait until such and such a time, I’d be happy to. Otherwise, here’s the Google search.” Not quite as abrupt as that, but “Here’s a tutorial on how to merge columns in Excel.”

And so that I tend to point you in a lot of scenarios as something that solves the instability problem in a small business, is make sure that you’re really focused on results, and that people know what the results you’re driving are.

But road-mapping, I think, is one of those things… I used to, as a project manager, hold on to my roadmaps very tightly… But as you learn – even when working in large companies, we would have those where you’re kind of feeling a little whiplash, and it’s like “How do I solve for a changing roadmap? How do I solve for that constant?” and really clear prioritization frameworks tend to really help with that, too… Especially if they’re known way up the chain, like “This is the metric I’m trying to optimize for this business, and therefore, this is how it reads into my roadmap”, and being able to give those cost trade-offs all the way up to the management chain tends to be a really effective solution as well.

[26:26] I have a follow-up question for you, as we talk about process… I love the way that you’ve kind of worked out the end-to-end system, and you have kind of go-to things that you can utilize, and you’re kind of simplifying and making sure everyone understands what is is, what the result is, how to measure it, and stuff like that. In so many small businesses they’ll have the data scientist, but they’ll also have the software person; and then they’ll have the infrastructure person, or whatever you want to call it; systems, or whatever.

DevOps Doug.

Yeah, exactly. Or DevOps Diana, whatever. That person is there. And so I like what you’re saying about kind of going through that system… Where do you bump in, either – two different ways of looking at it. You bump into that person, or find a way to integrate, and everything is Nirvana together… How do you navigate that in a small business where the person’s like “Okay, you’re on my toes now”?

Yeah, I love it. I love it, because it gets to this skill that I feel like is not talked about a ton in data science education, which is how much people are actually the mechanism by which things get done. More than any code or any framework, it’s people that get things done… And so knowing how to work in an organization with folks that have sort of their territory - how do I earn trust with them? “How do I think about handoffs between the components of this overall system managed by people?” is so important.

I think one thing that I tend to recommend for data scientists is that earn trust part. How do you break down the process of earning trust inside of an organization, and make that a repeatable thing? Overall, just knowing the architecture, people-wise, of your organization, is a task that some care should be taken with. Who are the people that are over these various systems, and meeting with them and knowing who they are, is like baseline. Even then, knowing what their goals are, what their blockers are, and how your work can actually make their life better is huge. There is a big advantage to knowing, “Hey, software team got this problem where they’re meant to optimize this part of the app flow. And they’re struggling with that.” Well, actually, that’s a place where machine learning could come in and actually solve part of that problem for them. Can I actually do something that helps them with their own KPIs, such that I build this trust with this organization?

And so as much as we like to talk about code, and “If it was just faster, if it was just easier, if it was just simpler, then we could get data science done.” Focusing on that data part, and being part of the team, is actually the skill that stands on its own, beyond any technical skill that you develop over your career.

This is Practical AI, and I have a very practical question, which I think is sort of a boring question, but I think it actually could be really helpful to people. So like you were talking about in part of data science education, machine learning, we don’t talk a lot about this project management side of things… And I’m guessing there’s probably even listeners out there, data scientists who maybe they have - let’s say it’s a data scientist who has like a background in science, or something. Or Academia. And their idea of project management is “I have a notebook with some things written down in it.” And then on the other end, you have maybe a data scientist coming from like a software engineering background, and their idea of project management is “Okay, I have a JIRA board, or Asana”, these sorts of tools. And maybe there’s other backgrounds as well. I could see how the notebook isn’t going to get you totally to a good place. I could also see how some of these other systems, like a JIRA or Asana - it could be overkill for managing what you need, especially if you’re like a solo data scientists working on projects.

[30:29] Do you have any recommendations in terms of some things that are not overwhelming? It doesn’t have to be a system or a product, but things to look into that can just make your project management workflow work for a data science scenario?

Yeah, it’s a good question. I’ll start with – I really like Trello. So if we’re talking about products, just products, generally, Trello is a fantastic place to start. If you are that person that’s just like written things down in a physical notebook, Trello tends to be like a good step forward in terms of something that’s shareable, and everyone can see it…

Not overwhelming…

Exactly. It’s really great. You can build templates in it, and so you know “These are the things I need to put together for my data science project.” But beside that, the thing that I find is very hard to beat is Google Sheets. Google Sheets is a fantastic tool for almost any workflow. Google Sheets is great. Usually, when I come into a new organization that doesn’t have sort of a project management muscle, I start with a spreadsheet, because it’s so easy to change, so easy to update… You can add a new column, remove a column you’re not using… It’s super, super-easy, and you do that for like a quarter. Like, manage 10 projects through a Google Sheet and see what actually is helpful in terms of making sure everyone knows when things are due, what the deliverable actually is, how do we know we’re done, what stages does our project go through… Iterate on those in a Google Sheet for a while, and then you’ll have this system that really makes sense to everyone, because everyone’s been using it, and then you can level it up with an interface like a Trello, if you wanted to. So that is my secret to almost every workflow, is putting Google Sheets somewhere, that’s connecting some pieces for a little while, and that’s what’s going to teach you what you actually need in order to manage that in the long-term.

Yeah, that’s awesome. I wonder too, with that - so project management is one thing, but then there’s like… We were just talking about the people side as well, communication-wise, within a small company… I’ve had experiences in the past where I am maybe managing my own thing, and I think I’m managing it well, but I’m doing that in a silo, and I sort of crank on something for like a month, and then try to lob something over the fence, or something like that. So do you have any recommendations with regard to that, and really kind of developing that empathy and good communication of data science within a smaller organization between key stakeholders?

It’s a fantastic question as well. You’re hitting on all of the things, all the pitfalls of working in a very small organization…

It’s only because I’ve hit the mines. [laughter]

That’s exactly right. Yeah. The thing that comes to mind with that is really the understanding that success in a project - you can do as well as you want in a project; your product itself can be super-good. But at the end of the day, that product won’t be able to make it to your end customer without passing through a few more hands. And when you really tie your project success not to the trained model, but to the deployed model that is in front of your customer, your end customer, it starts to really raise the priority of understanding what happens downstream of the output of your workflow. So the output of our workflow is like this trained model; even like a pipeline of inference, an inference pipeline for this trained model - that’s the output. But that pipeline has to connect somewhere, and there’s got to be someone who’s doing that connection. And that is one of the tricks that I used to really raise in my own mind the priority of having good relationships with people up and downstream of me.

[34:16] And so good relationships downstream, as you kind of pointed out, regular communication really helps with that. Having a project management framework where you’ve got deadlines, and you’re meeting your deadlines, and you’re giving people regular updates along the way really goes a long way in earning trust in that up and downstream relationship with anything like that. When you’re, as we’ve mentioned a couple times, that means that’s another task. I can imagine someone listening being like “Oh my gosh, not only do I need to figure out how to do my pipeline, but I also need to figure out project management, and I need to figure out a communication strategy.” Making all of those things as light-touch as possible is really important.

So if you have your project management framework in your Google Sheets, something as simple as updating that Google Sheet, copying those rows out, putting it in an email and saying “This is is my update for the week. Here’s where we are.” Really leaning on those agile frameworks, a simple stand-up “Here’s what I did last week. Here’s what I’ll do next week, and here’s when I’ll be done”, those very simple mechanisms and training yourself to make them simple and keep them regular is really, really critical.

I have a follow-up, which I’ll get to in a second… But just something that you said that was really resonating with me is I’m currently - though I’ve spent a lot of years in small companies - I’m currently in a large company, but just before we started this conversation, I was in a work meeting… It was with a development team, and I was thinking exactly that; I was literally saying, “We need to lighten this up. We’re too heavy-handed.” I think it’s one of those small-company things that could be used very well in many large companies, is don’t overdo it; light as well. So I just wanted to say, I really resonated with that when you were saying that.

You’ve done a really good job of talking about the need for trust, and finding those opportunities, and communicating… But as we move forward with data science in small organizations, and there are all these opportunities for growth of the small organization by really absorbing data science beyond just your role as the data scientist, but kind of affecting all of the other functions… As you build that baseline of trust, and you’re actively communicating that, how do you approach getting people who aren’t thinking about the benefits of data in a business context? So they’re not doing your job, they’re doing their job, but there’s so many ways that if you kind of get data-centric, in a non technical way, that you can get benefit from that… How do you bring people along on that journey? Because it’s a hard thing to do. It takes a very savvy touch to bring people to see something that’s not normally their forte (that’s your forte), but they can benefit tremendously? What are some of the ways of bringing all of those other people along in their own right?

Yeah. The reason why I like this question so much is in a small organization, when you’re kind of one of a few data scientists, you have this responsibility that you’re not just representing your work, you’re actually sort of representing the discipline within that company… And I think there’s a lot of companies, going back to like the beginning of our conversation, that are not doing data science, because they maybe dipped their toe in and really got burned. And that one project where there was a lot of promise, but it never panned out, turned them off of wanting to do data science, kind of, for a long time.

So these cycles tend to be really long. Trust cycles like that tend to be super, super-long. Maybe like two or three years later they’re like “Maybe something’s changed in the landscape that would make me want to try it one more time.” So knowing that is sort of serious. Like, when you’re the one data scientist, it’s kind of serious to be like “I not only have to do my work well, I have to convince folks in this organization that this discipline can do something for their organization, beyond what they’re doing today.” So with that in mind, one of the things that I think is really critical is thinking of yourself as not just delivering a product, but educating about what that product is, and its benefit, which is why having a very strong A/B testing framework, maybe unintuitively, maybe intuitively, is so important.

[38:19] If you’re not familiar with how to deploy something and A/B test it at the same time such that you can describe its impact, that’s one of the biggest differentiators I’ve seen in small organizations, that really just love their data science team, versus like “I don’t know why we are doing this.” That’s a really big differentiator.

That’s a great point.

Yeah. And so that education point - it’s very hard; I used to say this quite a bit when I would review resumes… There is a primacy to delivering results. Anything else when you’re reviewing someone’s resume, a lot of times that’s the output of kind of all of the inputs in the resume, what results have been driven. That can tell you a lot about how well do they manage projects, how well do they work in a team. All of those things kind of ladder up to delivering results. And so when you think about earning trust within a small organization, thinking about delivering results as being the output of earning trust, doing your work well, training your models well, having good relationships with the people along the pipeline - that’s what will really point you there. And once an organization sees those results, it’s actually very tough – you’ll have more work on your plate than you’ll know what to do it. That’s what I’ve found.

We’ve talked a lot about challenges related to being a data scientist in a small organization, or like things that you need to be thinking about… I’m wondering, what from your perspective – because I’ve definitely seen some of these things… What advantages does a small machine learning organization, or machine learning practitioners in a small organization - what advantage do they have compared with machine learning engineers, let’s say, at a really big tech company, in terms of what they’re able to do? Because I think oftentimes that’s not highlighted; some of maybe what you can do in that scenario - it’s maybe harder to do in a large tech company. Have you run across those things?

Interestingly, that’s part of the reason why I like doing it in small companies… Because when you do data science at a large company, you have to think about things like “How do I parallelize the compute for this so that I can actually get this pipeline to run?”

Through 3 billion users, or whatever…

[laughs] Right? So there’s this part of the machine learning tech stack that I find the most complex, the hardest to understand, the hardest to become expert in, the hardest to deploy; that edge where you have a very, very high number of users, an incredibly large amount of data, latency requirements that are very stringent… Like, this inference needs to happen in 300 milliseconds, or everything is off - those constraints don’t tend to happen in small businesses. You tend to be looking at tabular data, you tend to be looking at batch inference… And those are things that are actually fairly straightforward to learn when you sit down and you look at “What technologies do I have to do batch tabular inference?” That’s fairly straightforward in most platforms.

And so I think the advantage is that you actually get this vista of the machine learning discipline at a small company, that you don’t tend to get a larger organization, in a larger team. You tend to have a fairly narrow aperture, of like “I’m looking at - my features have been engineered by my data engineering team. I’m then doing some last-mile stuff on that data to put it in my model and train, and then I’m handing that model artifact off to my MLOps team and they’re doing all of the CI/CD stuff.” So your aperture is very narrow, and so you tend to not be able to see the innovation that’s happening in MLOps, or the innovation that’s happening in data engineering leading up to your model training, that I think is fascinating. It’s so interesting to do and be a part of.

And then you get the chance – like, if it turns out, for some people who did modeling for a while, that for instance worked for me, they will get a chance to do some of the MLOps stuff and be like “You know what - I like MLOps. That’s what I want to do!”

[42:12] Yeah.

And so you get the chance to actually see these different roles and try them out, and then you could go deep. You could say, “I’m going to do MLOps, and I’m going to be an expert in MLOps. That’s what I want to spend my time on”, and then go do that at any size organization.

So you got me thinking about this, because as you’re talking about that, you’re really making me think back to my small organization time, but I’m also in a large one right now, and as a comment, before I ask the question… In large companies you’re often at the mercy of arbitrary decisions of others, that may not be as informed as you are, as the data scientist. And that happens all the time. But you’ve kind of differentiated these kind of different opportunities at the different-sized companies… If someone came and was asking you for guidance on “What kind of companies should I target?”, there’s a clear set of kind of generalist, but a lot of opportunity at small companies to try different things, and then there’s this opportunity at large companies that you may have to accept what they give you, but within that scope you can go deep. How would you recommend someone try this or that? Someone’s looking to you for that mentorship - how do you steer them the right way?

I think it takes a combination of things to be a data scientist in a small organization. I actually tend to recommend that less often than at a more established organization. In particular, if folks who are coming just out of school, I tend to not recommend looking at smaller companies as their first job, particularly if that company is just starting their data science muscle… Which usually you can find out when you’re doing an interview with them, like “Am I employee 1, or sub 10 doing data science at this organization?” Usually, I’ll steer people away from doing those roles if they’re early in their career, because there’s so much about that role that’s not taught at, say, a university.

I tend to find people just out of college don’t know how to set up their own data science pipeline from end to end; how to interview someone really rigorously to understand “How does this requirement tell me how accurate this model needs to be?” and making that translation. I tend to recommend sort of mid or larger-sized companies for a first role, so that you can learn from other people that have done this for a little longer, and have built these sort of intuitive ways of doing data science; you can pick up some of those frameworks from them.

But if, one, it is a startup that is led by a CTO, or a CPO, or a CEO that has deep expertise in data science, has seen it elsewhere - that can be a fantastic opportunity to be mentored by someone really directly. So I tend to sort of put them on that spectrum of like if you’re brand new in data science, I tend to recommend a more seasoned organization, so you can pick up some of the stuff that simply is learned on the job. Sadly, as much as I wish I could point at like a blog, or a course, or something like that, that would teach you end-to-end data science workflows, I actually haven’t found one. Please send one to me if you have found one. But because that doesn’t exist, I tend to say you need to see it, and really take that opportunity to learn… Not just your narrow aperture, but really try and observe what the people upstream of you and downstream of you are doing, so that you know that end-to-end workflow. And from there, a smaller company can benefit from what you’ve learned at that larger organization, as long as you have this desire to really hands-on own it end-to-end.

[45:35] If your desire is that the person upstream of you is doing a lot of the data preparation, please do not go into a small company… [laughs] If you hope that someone’s going to tell a story of how good your work is, don’t go to a small company; that’s going to be your job. But if you really love this, like, “I want to be the one that takes this whole thing end-to-end”, a small company is where I would tend to send people.

That’s a great perspective. As we kind of wrap up here and get to the end, I’m just curious, as you look forward to the future, that could be something with Storytellers, and the work that you’re doing there, or just generally in the industry, what’s exciting to you and encouraging to you as you look to the future?

I’m most excited - as we’ve built this company and seen how much need there is in smaller companies for data science techniques, and how hard it still is for these companies to find, hire, keep data science talent, I’m super-excited for the opportunity to show how much data science can help a relatively small organization, and prove out that case. Like, “Data science really works here.” And from that, I think that will spark some ideas around - you know, MLOps tools tend to really focus on enterprise use cases; large teams that are solving very complex problems. I’m excited to see tools that are really aimed at solving these constants that small businesses run into; lots of disparate data that needs to be brought together in order to build these models and deploy them simply, like a very simple sort of layer for that. I’m super excited about that.

Also, as you mentioned, Jasper - I’m really excited about how large language models will play into this idea of deploying data science within organizations. Will it make people more familiar with the concept of data science, to where they’re more ready? They’re like “I see this value other people are getting. Can I try this in my organization?”

And then lastly, I would say I’m really excited for how I see the data science community generally maybe starting to pivot away from excellence as being measured as state of the art performance, towards excellence as being measured as impact within some sort of vertical.

There’s so many areas that data science could really – I mean, not to sound like the Silicon Valley stereotype… But really could make the world a better place… [laughs]

Of course.

You think about like how data science can help universities identify when a student needs some sort of service in order to help them get their degree. That’s a very practical thing, and something that I haven’t seen many universities really embrace. Yet, it’s one of those small organizations that still tends to have a little reticence around adopting data science techniques. But if we can point towards data science outputs as driving impact, not just the accuracy of our models, then I think we’ll see that adoption really start to grow within these smaller organizations.

That’s awesome. Well, thank you so much for joining us, Kirsten. It’s been a great conversation. And yeah, I know I’ve got a lot of tips that I can articulate better now, even with my own team, after the conversation, so thank you so much for joining us.

Thank you. I really appreciate it.