Broccoli AI at its best 🥦

Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am founder and CEO at PredictionGuard, and this is a pretty special and fun episode for me, because I get to kick back with an old friend of mine.

We went to the same university, for those that haven’t heard of it, Colorado School of Mines, in Golden, Colorado. Of course, a shout-out to all the ore diggers out there that are listening… But yeah, we have with us today, Bengsoon Chuah, who is a data scientist now, and working in the energy sector. I was really fascinated to talk over the years with Bengsoon about all the things he’s doing, and in particular his approach and learnings around active learning and NLP models… And I wanted to invite him on the show to talk through some of that and learn a little bit from him. Welcome to the show. How are you doing?

Hi, Daniel. Thanks for having me.

Yeah, it’s been a while since the days in Colorado School of Mines in Golden.

Good old days.

And now you’re working as a data scientist in the energy sector, and also working in Asia, which is super-cool… I’m wondering if you could give us a little bit of a sense of some of the unique things about doing data science and machine learning type of things in the context of the energy sector, in the context of an actual enterprise, real-world kind of situation… We talk a lot about – recently, we’ve been talking a lot about all of these Gen AI models, and APIs and such, and that is super-cool… But also, there’s a lot of on-the-ground work going on in data science, that maybe looks quite a bit different than that.

Yeah, thanks. So I work in the energy sector, and it’s pretty much a traditional type of a sector. A lot of the companies, as you go around – at least in Asia, or at least over here where I’m at, we do not actually even have things like cloud services, or subscription and stuff like that, due to different reasons, and stuff. But at the same time, there’s an appetite for machine learning, AI, data, and all of those things. You see people talk about Gen AI as well… But I think the ones that I’ve noticed, at least for me personally, that has really brought a lot of values are what you guys were talking about in a previous episode, of Broccoli AI.

Broccoli AI. I love it.

Yeah. Not so sexy, but still, really important; it really brings value… Particularly, I guess what we’re going to talk about is active learning in the context of NLP, natural language processing. I think that’s a pretty exciting place to be in, to do. It kind of translates into gen AI, to a certain degree… But at the same time, I think to at least the context that I’m in as well, in the sector that I’m in, we do not have things like cloud services ready for us. So you have to figure out ways to kind of bring that about, in an on-prem server VM. So how do you work around that, and how do you actually bring in cloud-native, modern technologies within a traditional kind of structure.

Yeah, and from at least my impression, even though there’s not this – whether it be for security reasons, or legacy reasons, or just connectivity, there’s not the type of connection to cloud services like you’re talking about, that others might be working with… But at the same time, at least my impression is that this sector, and maybe others - there’s other related verticals where they have been data-driven, to some degree, for some time… And I don’t know if you could speak to that, like the types of data that people have been processing, or storing, or are available in those contexts, but…

Yeah, that’s a good point, because a lot of these traditional industries, at least in the energy sector that we see, we have sensors that are constantly flowing in data all the time. The data is there. We are collecting data. And then going a little bit deeper, then you find that a lot of us have been collecting a lot of unstructured data, too. So in my experience, at least what was happening was when I came in, I was pretty much the only data scientist, and so I had to make my existence justified, in a sense… So I knew that I had to do something about bringing value within this organization, and I’ll be able to have proof that “Hey, look, data science actually does work, and does bring value.” But the quickest, I guess low-hanging fruit that I’ve found, at least in the context of where I’m at, is the whole unstructured data.

[00:08:07.24] So we have been collecting thousands and thousands, if not hundreds of thousands of unstructured data, but there has never been a way to really analyze that at scale. So people have been analyzing it, they’ve been able to do some sort of a human analysis on it, but there’s never been someone who’s able to say “Hey, what’s been happening in the past 10 years that we’ve been collecting all this data? What’s it telling you?” Nobody has really been able to do that. So I thought, okay, maybe that could be one thing that we could actually bring in, is that you have all this data that’s ready for you, that has all the insight, that has all the information that is locked up, waiting to be unearthed, pretty much, waiting there for us to just extract it or mine it.

So I did a quick POC for one of the departments, a company, and said “Hey, guys, you’ve been doing Power BI, Tableau, Power BI kind of thing… You’ve been able to do stuff with your structured data, and you’ve been able to plot it on beautiful graphs, and stuff like that. You’ve been able to analyze it in that sense. What about all the unstructured data that you’ve been collecting over the years?” And they said “There’s no way you could do it, right?” I mean, you’ve got [unintelligible 00:09:20.26] There’s no possible way for you to put it in Power BI.” And so I said, “Well, maybe we could explore ways that we could actually get machine learning to actually help you to scale that analysis from an NLP standpoint.” Thankfully, they bought it, and we took off from there. It was pretty cool. I mean, it was a journey, for sure; a journey to learn.

Yeah. So when you say unstructured data, give people a sense of the kinds of files, or… Not maybe the specifics, but I’m imagining a file store with some type of files in it that contain – yeah, something… Give people a sense of that.

Yeah, for sure. I mean, I guess unstructured data, typically we’d think about it as text. It could be text, or it could be something that’s just out of structure.

And then like Microsoft Docs, or…? [laughs]

So we have been storing all of those data within SharePoint, Microsoft SharePoint… And so what I’ve seen is these unstructured data - it usually comes in tabular form. You have a table that is collecting all of the structured data, but alongside with it there’s always that comment, or additional things, and you have to actually tell the story of what you’re actually collecting… And those are the ones that I think [unintelligible 00:10:46.29] It’s very typical in any kind of industry where you have tabular data, that collects sensor data and everything, or like reports of what’s been happening, and those are structured. But then there’s also a column that would bring in some sort of remarks, observation, or actions taken, whatever it is.

Comments…

Comments, yeah. But more often than not, we just kind of wade through it and kind of not really put too much attention to it. At least within the dataset that we were looking at, we found that there were a lot more insights in those data than the structured data that they’d been collecting. I’m talking about like safety data. So we’ve been collecting like safety reports every single day, a couple hundred sometimes, or tens of them, every day, over the years… And so this data has some sort of insight, and it brings in an insight of how is the safety condition in our operations. And so with that, people usually see the categories or what the reports are being reported for. But then when you look at the categories, you realize that sometimes it doesn’t really jive with the stories that they’re actually trying to bring in in the unstructured data… And so that’s the part where we felt like it’s going to bring an additional insight to the unstructured data that we see.

[00:12:22.25] Yeah. And so there’s this – we’ve talked a little bit about the data that was there, the potential insights in that around safety and maybe other insights… As you kind of came into this industry, and kind of were getting an understanding of – let’s say that you built the coolest data science app that was out there, and had a cool model in it to do some analysis… What is the reality of how that would have to run? What does “in production” mean in your context?

Yeah. So many things. I mean, just start from the data… We don’t even have labeled data to start with. Say you want a classification app or model - you need to have some sort of a label, right? We don’t have that, and so you have to figure out ways to bootstrap your labeling process to start off, and stuff. And then all the way down to – of course, training the model is pretty easy these days… But then you have to think about things like “Hey, what does that look like to put it on the server?” Most of our servers are running Windows server… And so I’ve had experiences putting in production some apps on Windows server and it was painful. So we have to figure out ways to work with the ITs and stuff and say “Hey, can you deploy a Linux server for us instead?” and just work it out from there, and set it up from there.

Like we said, that’s just the overall picture of it. And then you get into details of “How do you actually store your model?” We’ve got to have some sort of infrastructure to kind of hold that… Which in our case felt like MLFlow is a pretty good model registry, experimentation tracking and stuff, to keep track of what type of models that I’m using, and stuff like that. And then so many things, honestly… And then how do you actually put it on an orchestration kind of service? You could use cron jobs, but then it may not be so flexible. Then you kind of need to work something out, and so you have to get some sort of orchestrator to spin it up, and make that a service for your infrastructure as well.

I love this discussion, because I think it fits the theme of the show so well, around being practical, and the fact that - yeah, I’m sure that there’s actually a good number of listeners out there who are really wanting to do machine learning, AI, data science type of things, and they are sort of in a similar situation in their company, because actually I think probably more of the majority of companies are in this sort of situation than sort of infrastructure-wise. Extremely modern, and just cranking everything out on Kubernetes in the cloud, and that sort of thing.

[laughs] Yeah.

So yeah, I love this. So you were talking a little bit about the sort of problem of – so you have all of this tabular data, with extra comments, and unstructured data, and certain things you want to do, like extract insights, or classify maybe some of the unstructured data… But then also nothing has been labeled over time, it’s just unstructured data. So talk a little bit about that bootstrapping problem, and how you’ve thought about that in terms of “I’ve got all this stuff, I want to create a model, but I have no starting point.”

[00:16:02.20] When we were going through that whole labeling process or data preparation process, it was pretty interesting, because we really didn’t have anything; no labelers, or anything. And I didn’t have a budget to get an external labeler for me.

To just hire thousands of people online?

Right. I could just do that, maybe. But then again, even that - I’ve thought of it, and our data is sensitive to start with… But at the same time, it’s so nuanced, and it’s so nuanced to the context of our company… And so a lot of data that every company has is just so nuanced to their own context. And at the same time, one of the nuances is the way that these texts have been written. And so a lot of code switching happening… Code switching, which means in Asia a lot of times we speak in English, but we will kind of put in some native languages that we know, or that we grew up with… And so it’s just kind of like, you go back and forth, back and forth, and it’s just kind of a common thing, especially in Southeast Asia…

So you can’t just hire somebody online to just kind of label it for you, because you just can’t – I don’t even know how to bring in the context to these guys. But thankfully – I mean, I would say that the part that really helped was I happen to have a really good sponsor for the project. And these sponsors, they were super on board. They were not technical. They were SMEs, in their own departments, and they knew this stuff, but they know enough of machine learning and data and AI that, that “Hey, it’s kind of a model that predicts, not necessarily making a hundred percent accuracy at get go…” And so they understand those nuances, in essence, and so they kind of supported that, and understood the kind of things that we have to go through as practitioners; they kind of understand that part of it. So that was really helpful for my part, because having a good sponsor means you get really good support for the project. But that also means that there’s a product that I’m working on, the app that I’m working on is for their people, their subordinates, and the people that are reporting to them. And what happens is they said to them “Hey guys, this is your app. I want you to help Bengsoon out with building this app.”

And so having the users themselves on board from the foundational bootstrap level really helped us. So because we didn’t have any labels, they had the guys to actually be the ones to label for us. They were the labelers. And so the users themselves were the labelers. Honestly, I was really blessed to even just have that worked out together. I think that worked so much in my favor.

Yeah. And in that labeling process, how did you develop your set of instructions for explaining the problem to them, or helping them define the problem and the categories, for example, in a classification model? How was it for you? Because I’ve also had the experience personally probably, and been burned a couple of times where I’m like “Oh, this problem makes sense to me. I set up the labeling thing, I release a bunch of labelers in there”, and either the instructions don’t make sense, or I’ve biased this in some way… Likely because, like you had mentioned, I wasn’t super-close maybe to the users in that situation… But yeah, any learnings from that experience?

[00:19:55.24] It was a lot of iteration with them. I had so many times – like, I would travel to see them. They work in our operations, so I would literally travel there to see them in person… And I said – we would just go through “Hey, these are the labels that we want to label.” We kind of get a general idea of what they are, but when you get into the weeds of it, when you get into the details of it, you would think “In this situation, that label should be here. It should be number one.” But then no, somebody else said “No, it’s two.”

So the way I worked it out was - there will always be contention. I know this. No matter how tightly knit your labelers or your team are, there will always be contention. And you’ve just got to work around with it, at least in my experience.

I just had to work around with it. And the way I worked around with it was I just kind of had a voting system, and I set up an account… So the technical side of this is I could have just given them Excel sheets, and they could just label them on an Excel sheet. But I find that they are doing me a favor kind of thing, but I want them to have a really good user experience, instead of just going through Excel sheets, and stuff… And so I used Argilla back in the days, when they started, and…

Pre-Hugging Face days.

Yeah, yeah, exactly. And I noticed that Argilla is amazing, in the sense that it allows you to set up different users… I mean, even other kind interfaces I’ve used labels do as well, but Argilla was able to – I could use the API and just kind of like set up each user. And then for each user, I would just kind of like sample the same data for them. So I had to actually go through the first round, the same set of data for them to label, say 500 of them, I think I remember… And they would all label within two weeks, and at the end of that two weeks, I’ll collect them and I’ll find which are the most contentious ones.

And so the ones that are the most contentious, the ones that have the least percentage of the majority, I would pull it up and say “Hey guys, what do you think about this? This is contentious for you guys. Why is it contentious?” And you work it out from there. Because chances are you’re going to see the same kind of a label again, the same kind of data again… And if you talk it out, hopefully when you see a similar thing and you said “Hey, we already talked about this. We all agreed that we’re going to go with this.”

And so that’s the first round. We had the same labels, same dataset for everyone. It’s a bit inefficient to a certain degree, but I think it’s important to actually get into that space, so you understand the contentions of each person.

And then the second round, I have everyone just kind of like label at scale pretty much. And yeah, we collect that from there, pretty much.

Interesting. Yeah, so in this process you did a initial sort of offline bootstrapping of labels, right? And you did that… So scale-wise, what sort of scale, when you’re solving… So here we’re talking about an NLP problem, creating a classification model on some labels of this unstructured data… Of course, this would vary by domain, but what scale of labels did you shoot for when you were doing that initial, trying to get to that place where you could start up and train your first model?

We did just about 1,800 to 2,000 labels, rows of data, basically… And we started training our first model.

That probably means you’re not training like a 400-billion parameter model with 1,800 samples… [laughter]

We don’t even have the infrastructure to be able to train that… [laughs] So no, we don’t.

So what does a Broccoli AI model look like?

[00:24:02.03] I mean, these are all text, right? And so we were going with the simplest you can find on Hugging Face at that time.

I think at the time Sentence Transformers were really making it big, for different reasons, whether it’s topic modeling or classification… And at the same time too, I remember they came up with the SetFit model, which is fine-tuning the sentence transformer, which was honestly revolutionary for me. Amazing. And I thought it was amazing that you’re able to do something that was meant for similarity, but then you could actually fine-tune it for classification, and with pretty good performance. And it’s supposed to be something that is a few shot classification model, a few shot kind of fine-tuning. And so I thought 2,000 should be enough for me to start somewhere.

And in fact, when I trained that, I tried some other models, but I think Sentence Transformers were the ones that actually gave the best performance out of all. It still wasn’t that good; I’m talking about like 60 something, 70% kind of thing in terms of F1 score… But when I talked to my sponsors about this, I said “Hey guys, you okay with me deploying this at like 60%, 70%?” And they said, “No, actually that’s fine.” Because the objective for this was number one to bring visibility of these reports to the users… Because one of the pain points that they said was for us to be able to know what people have been reporting, at least in the past 24 hours, they had to [unintelligible 00:25:35.00] get on SharePoint, and just different hoops and loops to try to find out, filtering and stuff. But to be able to get that sent out in an email, with a classification, was already a win. And so I thought, “Okay, let’s do that, but let’s not stop there.” I mean, we should actually create a pipeline, and that’s where the active learning comes in. And it really helped, because I’m glad that I actually used Argilla to start with the bootstrapping of our dataset… And having that Argilla, means our users are already used to the interface, and they already have an account.

And so I was able to kind of hack around with Argilla’s Python API, and basically, I was able to create a loop where pretty much what this model does every day - it will bring in the new data that people have been reporting for the last 24 hours, and make some prediction on it at about 60%, 70% F1 score, accuracy, whatever it is, and then send it out to the users. And these users will see it… And at the end of that email – they will say “Hey, I don’t think this is that signal. It should be this signal. I want to give a feedback.” And at the end of it, they’ll able to click on a link that brings them to their profile in Argilla, that will allow them to give a feedback for the particular day’s dataset. And so over time – now it’s in production every day. From time to time, I’ll get people giving their feedback, and we’ve gotten close to 4,000 datasets now labeled from this active learning… And so we will train the model periodically. I could have done it automated, but I didn’t feel the need for it yet to put it on automation. But then at the same time, we’re just collecting [unintelligible 00:27:29.25] and we’re just training it from time to time, basically.

Break: [00:27:38.01]

So Bengsoon, it’s super-interesting to hear kind of you were able to engage the users of the application through this reporting process, essentially, that they had some of the right incentives in place to respond, and to give you updated labels… And you mentioned also the model repository, saving models, getting them out with MLFlow, in the context of you deploying your model on-prem, you updating the model… You just mentioned kind of retraining the model… What does that look like for you right now, in terms of that cycle of when you would want to push out a new model after gathering this data? …how you would judge that to be worthwhile or useful in any sort of testing that is relevant to that cycle of getting in new labels, retraining, evaluating, that sort of thing? What does that look like for you, and how do you kind of put in the right – or how have you thought about the right metrics to understand when to update the model?

At this point, honestly, we keep it simple. We just kind of like periodically do it, at a cadence, a couple months or two. But I did think about what does it look like to actually measure the drift of the data and stuff like that, of the model predictions… That could be one of the ways that we could do it, too. But what I’m seeing is actually the model is doing its job fairly well, well enough to actually solve the business problem. And so we don’t see a need to actually implement more sophisticated monitoring, unless we need to. That’s where we’re at with it.

Yeah. And when you push your model, sort of like you update it… You had mentioned the model repository. How are you shipping your model out to the application? Because I think you had mentioned you only have so many resources… I think there’s also a lot of people out there in your situation where – I think it was [unintelligible 00:32:00.17] on a previous episode, she had talked about kind of that data scientist out there that is maybe one of very few, or the only data scientists in potentially a large organization, and having to do all of these things. They’re not like an ML Ops person, they’re not a model trainer, they’re not an observability person… They’re doing all of that. So there are limitations to how much sophistication you can put in place. And I think that some people go way too far, and they’re like “Oh, I’m going to implement all this stuff”, and it actually makes their life as a practitioner less happy than otherwise. So yeah, how have you found that balance, and what does it look like for you to do these cycles in terms of tooling, and the things maybe that you’ve – like you say, you’ve mentioned you’ve thought at some point maybe it’s relevant to implement some of this observability stuff… But maybe not yet, or there’s other priorities? So what does that look like for you in terms of how you decide what level of sophistication is right, and how you push things out, and…?

I’m 100% with that, honestly, because it is a matter of priority. My customers are happy, I’m happy. And I’m not gonna change what’s good. I don’t want to break what’s been working right, so to speak. But that being said, when it comes to all of that, I think I have a general idea of what would be the minimum thing… So now I’m working on some other things as well, like anomaly detection and stuff like that, which needs to be deployed. So having gone through that, that kind of setup, a pseudo infrastructure for me to know what kind of infrastructure that I’m going to be looking for, for whatever else that I’m working on… And at the bare minimum, I think model registry is super-important. And being able to call the different versions that you’ve been training, and being able to track that, and being able to call it through an API, through a function… You know, MLFlow has this great Python connection with it, and so being able to do that is amazing. It keeps my life sane. I don’t have to figure out where I store my model, pretty much.

[00:34:16.14] So I would be doing exactly the same things with whatever I’m working on next, which - I’ve since moved on from that project, and I’m just kind of like maintaining it. That project’s now in maintenance, and now I’ve moved on to a different projects, to solve a different part of the business, in a sense. But that project, like I said, set the foundation, and knowing what kind of things that needs to be done.

Sorry, I’m kind of going ahead of myself… So one is MLflow being the most important thing for me in terms of this sort of scenario. The other one is – an orchestrator is also really important. Having a really robust orchestrator. So for me, I think Prefect was perfect for me, you know… And I was able to do different things, and stuff.

The types of things that you’re orchestrating are what types of things?

You could do it real-time, with Prefect… You could also be listening, and stuff, but at the same time, you could also just do running on schedule, calling different functions, subfunctions, and things like that. So that was really cool, to be able to have that. That’s pretty much what we do right now. We’re not really going into real-time monitoring yet… Until we do that. Then I’ll have to figure out something else, more sophisticated.

Yeah. And are you just shipping your models sort of as part of Docker container, or something like that?

Pretty much, yeah. We do use Docker containers, just so that we can keep it contained, in that sense.

Yeah, that’s awesome. I think you had mentioned in one of our conversations something about DuckDB… Where does that fit into some of this?

So the data, the raw data that we get is from SharePoint… But if anyone has any experience with SharePoint in terms of wrangling data stuff, it’s so painful… So I thought it would be good to actually have some sort of a middle layer, a mini lake house, or data lake kind of thing… And I didn’t want to bother my IT guys too much, so I thought DuckDB is a great thing for it. I don’t need a VM for it, and I’m going to have an embedded SQL service that you can use. So that’s being pulled every day, pulling that data into DuckDB, and DuckDB will be the one that actually cleans up the data, preparing the data to send it to the model, and that becomes like a pipeline for me to be able to work around the whole complexity of SharePoint, really.

Yeah, I’ve personally found a lot of use for DuckDB, even in the past year, even on the more Gen AI stuff, where you’re doing sort of like text-to-SQL, or like queries, and that sort of thing… And every company we’re working with has different, crazy sets of data, or different configurations of this or that, and that layer of having a kind of unified analytics layer, but also not – that’s sort of easy to pull into Python, easy to spin up, easy to test with locally, and then deploy with… Yeah, that’s been really useful.

I remember you talked about LanceDB, for RAG, and things like that… And it’s the same thing. I love embedded databases. I think it just works well, and it’s kind of scalable, eventually. I really liked that.

[00:37:45.26] I think there was one blog post I’ve always referred back to… Because I also went through the – you know, you and I were at Mines at the same time, and then there was like data science hype, and then there was the big data period, where everybody was in Hadoop and Spark and all this stuff… Which I know a good number of people still use Spark. But there’s a blog post by the Mother Docker company…

Oh, yeah…

I think the title is “Big data is dead”, or something. It basically goes through some of the discussion around “Hey, we all thought we had big data, but like the actual query problem, the types of queries that we need to run - these aren’t like big data problems. What’s needed is different.” So yeah, for those – shout-out to whoever wrote that blog post, because it was really, really good. If you ever want to come on the show and talk about it, that would be awesome.

Yeah, well, as you kind of look back on this process, and some of the things that you’ve learned, what are you looking forward to in terms of like the future of the process of your own work, or of the things you’re learning? Or maybe as you go into this next phase, it sounds like you’re working on some new things… You’ll want to reuse some of the tooling and kind of process that you have used. But what’s different, or what are you excited about, kind of, for this next phase, in light of what you’ve kind of learned over the past years?

Generally speaking, I think MLOps is just so nuanced, in different contexts… Everyone has a say of what should be done, and I think if I learned something from this was nobody really knows everything… So you’re gonna have to figure it out from there, and you kind of take a risk on certain things that you decide in terms of your system design.

What I’m excited for is actually to be able to take this and see what it looks like for other things, and other applications, whether it’s anomaly detection, or whatever it is. In a broader sense, I think I’m excited to see things like embedded database getting more and more mainstream… Especially in a context of LLM and Gen AI and stuff. I’d love to see that getting more and more mainstream as well.

One of the things I’m always thinking about is - scale is one thing, because a lot of the applications that we talk about today, especially in the context of Gen AI, we always talk about like the bigger compute, and bigger scale… I would love to see that getting smaller, which - it is happening now; getting more accessible on different devices, and stuff. Being able to do more cool stuff on device, [unintelligible 00:40:33.11] I’m excited for that too, yeah.

Yeah, I think there’s a lot of people excited for that, and sort of this new phase of AI where people talk about AI everywhere, or this sort of thing… Which - in reality, there’s been machine learning and data science sort of everywhere, for some time, but that sort of wave of these newer generation of models kind of being runnable in more practical scenarios is exciting.

But yeah, thanks for joining, Bengsoon, to talk about a little bit of your Broccoli AI. It’s been fun…

I love it. Thanks for indulging me.

Yeah, yeah. You and I can hype the Broccoli AI, and I’m sure we can get Demetrius to help us hype it, too. [laughter] I don’t know if he trademarked that term. He’s got it in his hype cycle now, so…

I love it.

Yeah. Thanks so much for joining, and I hope to talk to you again soon.

Thanks. Thanks very much.