Sam Ramji:

Hi, I'm Sam Ramji, and you're listening to Open||Source||Data. Today I'm talking with Simba Khadder, an entrepreneur and machine learning researcher. He began his career as an engineer at Google, where he worked directly on both the search and cloud products. Following his career at Google, he was the CEO at Triton, a UC Berkeley SkyDeck backed company, which is accelerating subscription growth for media companies around the world. Simba is now the CEO of Featureform, a feature store startup that accelerates the ML process by standardizing how features are defined, managed and shared. Simba, welcome.

Simba Khadder:

Thanks for having me, Sam.

Sam Ramji:

It's great to have you. We'd like to start each episode asking our guests what open source data means to them. So what does open source data mean to you?

Simba Khadder:

Yeah, it's an interesting question. I mean, it's funny, because we kind of take for granted in a way that mostly the process we use are open source. And it's funny to ask, why is everything open source? I guess my first thought is, why is that the case? Why is everything not? And I think it's just the problems to be solved are so hard and data, and just the uses of data, it's growing so quickly, it's almost for- it's impossible for anyone to work in their own little bubble, and try to solve them. So it's a perfect example where open source is useful.

Simba Khadder:

And anytime you think about data, in general, you kind of want to have some control into it, there's so many different ways you might want to use it. Like obviously, machine learning has become a big use case, a lot of, uh, data systems that were designed in the past have, were not built with machine learning use cases in mind at all. And even, you know, if you look back, like why does Cassandra and all these NoSQL, like why did that become a thing? And why is s- Spanner and a new SQL becoming a thing? Things change so quickly, and the algorithmic stuff you have to build is so difficult, that it's kind of, is the only way that things move fast enough to keep up.

Sam Ramji:

Yeah, and the ability to, to, not just to see inside of the code and share the code, but to tweak it and to integrate that whole box of Legos as opposed to starting up with some giant DUPLO block that you can't modify.

Simba Khadder:

Yeah, and also just like the ability to collaborate with different companies, it's like all these major companies realize that it's actually in everyone's benefit to just work on that kind of-

Sam Ramji:

Mm-hmm.

Simba Khadder:

... code base together, and just ex- expand it, it's kind of like all those who arise. If we can do more interesting things with the data, then, you know, in theory we can provide much more value to our users, it works out.

Sam Ramji:

Data pipeline architecture seems to have opened up a lot of opportunities for lots of different best of breed open source pieces to be kind of built out into a constellation, right? And then you can swap out or augment, you know, prior to that constellation. What are the things you've done a lot of work in of late, because you also built stream sequel, and now turning your attention into feature stores and building Featureform is, what's the role of data movements and data storage to improve MLOps? And we'll talk about feature stories in a bit, but I'd love it if you could share with our audience your perspective on the future of MLOps, like what is the world going to look like five years from now?

Simba Khadder:

I like to keep in mind the North Star of, what is MLOps? What's the point? Why are we even doing this? Why does this category exist? And it's, you know, as engineers, we like to build and sometimes we forget why we're building and what the point is, because we're just, we're, we're building. We just keep adding, adding stuff on. And I think it's really important, especially when a new category has been created to make sure that as a category, we're keeping track of what's the point here. What are we actually trying to solve? Why do we exist? What's the North Star of MLOps?

And it really comes down to making data science teams more productive, operationalizing machine learning. So we want our ML teams to be able to iterate quickly. Machine learning, like software engineering, is very iterative. Arguably more so because it's, it's lots of throwing stuff away, because it just, you have to keep trying stuff. And so much time is wasted doing things that aren't actually pushing forward our understanding of machine learning for whatever project we're working on. Like, every time a data scientist has to go and fiddle around with some, you know, big open source data package, or instead of Python, they have to learn Scala so they can write this transformation. But it's wasted cycles.

And so the point of MLOps, and I think the future of MLOps, is to fit so cleanly into the data scientists workflow. I mean, you're trying to get the closest thing to like, I have this idea to it happened, as possible. You can look at DevOps, I mean, MLops and DevOps have a lot of the parallels, there's so much in common. And with DevOps, the idea was, hey, I wanna write my code, I wanna make a PR, which is still kind of part of the DevOps thing, and someone will look at it and then it gets merged, and then everything that needs to happen happens. So I'd have to spend time mainly writing tests, make sure things are working, I have to go push it to some server, so I make sure that the dependencies are the same and spend all this time on risk braking systems.

I can get there as quickly as possible. I know how I want to write this code, I can write it out and boom, it's in progress, we're going. And for machine learning law, things don't make it into production and this way, to do that is, hey, I have this idea for a new feature. I think that for my music recommender, if I know, uh, the top genre the user listened to in the last 30 days, it's gonna perform better. So I know how to write that query and SQL and Python, whatever I'm comfortable with, I do that and boom, it, it happens, it automatically it's tested, it's trained, maybe even gets thrown to an AB test in production. That's kind of the North Star of MLOps.

And I think we're, in five years we'll get there. MLOps will be pretty well understood and, and a solid category, same as DevOps. I remember when that cloud was like new, you know, it was just kind of like cloud, DevOps, people would throw around these terms, but no one knew what it was. And now it's just like, if you're not doing DevOps, you're doing it wrong. And MLOps will be the same way, I- in my opinion.

Sam Ramji:

That's awesome. When I look back at DevOps, I think about two overriding changes that made it different from what it was before. And those two things are shared artifacts, like when we had containers. And of course, you know, YAML files, right? Where you have the containers and configuration, that could be really consistent shared artifacts, then that brought different people's expertise together, because maybe I couldn't reproduce the container but I could tweak and modify the configuration, all that stuff was versioned, it was easy to branch, it was easy to merge, everyone could see it.

Sam Ramji:

So shared artifacts is a big deal. And then the second thing is replayability. Because the fact that you could push rewind on your production infrastructure and go back to the last known good means, wow, what a relief, I'm not creating huge risk in creating this small incremental change, which is amazing, because now I don't feel like I've got to do super heavyweight testing, and making really large batch sizes. So between those two, we end up with really nice automation, uh, good sharing of competencies, everybody's looking at the same source of truth and the whole system becomes a lot safer through replays, so it gets a lot faster. When you look at MLOps, does shared artifacts play a strong role? And where do you see the enabling elements or the re- constraints right now in replayability?

Simba Khadder:

Yeah, I think shared elements play a huge role. I think that the goal is to make sure that an organization, a whole machine learning team, this compounding value, and software engineering is compounding value. The bigger your team is, it's not like a thousand engineers, it's a thousand extra work, it's much more than that. Because people reuse things. I'll use Google again, there's a team that built Spanner, Spanner is used like everywhere, and that team had an out-sized impact, there's compounding value in them building Spanner.

Simba Khadder:

And machine learning, a lot of things have been so ad hoc, it's almost like a ton of lone wolves. Like I had a thousand data scientists and it's pretty much either teams of one up to three, and they're all working like lone wolves, they're doing their own thing, they have ad hoc stuff on their local system. There's no compounding value. There's a lot of waste. And also, there's a lot of confusion, it gets really bad when people switch teams, or leave the company, or it's just things are lost, there's no place where things are stored, you. We talked about rewind, but there's also kind of understanding history, what's changed over time, how have things been affected?

I mean, we talk a lot about reproducibility in machine learning, it comes up in DevOps, but it's huge in machine learning. There's very few companies in the world who could go and retrain the exact same model, like exactly the same, like, it's almost like you create that model and based on how everything was aligned in the world that day, that top model came out, we could never make it exactly the same again, which sounds kind of nuts, to me. The ability to not be able to actually regenerate the exact same thing. You know, you have all the code in theory, and you have all this stuff.

So to answer the question, I think that the ability to share artifacts, maintain some sort of organizational workflow, like what we have in DevOps, and have these primitives, like in, in machine learning, that it models, that features, if it's multimodal, maybe it's a few [miles 00:08:52] tied together, all this stuff needs to exist at that level of abstraction. Once you have that level of abstraction, you need to add versioning, you need to have workflows, you need to have a way that works for your, um, organization. And every organization is different.

Like DevOps, most of the companies that used to promise DevOps in the box where we do everything, don't really exist anymore. The things that exist are kind of almost like the HashiCorp core Basque approach of we have a lot of small point solutions. And you can fit them together to fit how your organization works. And I think MLOps is, have lots of point solutions that fit together nicely, but they're oriented at Phoenix Philosophy, each tool is aiming to fix one problem.

Sam Ramji:

Yeah, that makes a ton of sense. And the sharing determined by the field of use and what the artifacts are. And DevOps will find you guide code, containers, and config, and they're all in a repo. And you look at those lone wolves that you described in ML teams. And what's the repo? often it's like some data that I got crunched that I got my hands on, or I got other people to help me get together. Now I'm training things on it. One of the things that inspired me when I got to meet you a little over a year ago, you had given some talks on pulsar, and you're building out a feature store, and it really captured my imagination to get the sense of the ability to drive shared artifacts and reuse, and to be able to bring a team together around a common repository for where you are storing features.

Within that, there are embeddings, and you recently launched embedding hub on GitHub. That seems like it's taking off. So there's something happening in the market, right? I think it was, in the first few days you had trending on GitHub, and like you know, hundreds of stars. And obviously, the stars and utilization are distinct things. But it shows for those of us who are watching, like, oh, people are paying attention to this, it solves a problem, the markets going people want reusability, they want to get more efficient. That's a really good moment, I was wondering if you could tell us a little bit about the difference between embeddings and what an embedding hub is, versus a feature store. And then maybe we can kind of bring the audience through a little bit more of this, thinking on how reusability and some of the elements of MLOps infrastructure are going to change the game for iteration time, latency, your ability to fail fast and make progress as an MLOps team.

Simba Khadder:

I think it first helps to understand when embedding is for listeners who maybe aren't as familiar with the term, specially in recommender systems, natural language processing, even some clear vision tasks. Embeddings are used widely, I mean pretty much all modern NLP, dramatically, is leveraging embeddings. An embedding is literally just a vector. It's just like an array of numbers. That's literally what it is. But it's usually a vector of that describes something, that something can be a user, could be a word, it could be a document. An embedding is a holistic view of something. So with a user, there is always things that you listen all this different music, it changes all the time, some days, someone's randomly in the mood to listen Katy Perry all morning. But you know, the model has to be able to build a holistic view of who this person is not going to focus on one day, or actually also zoom out to why they're like, we don't know anything about this person.

So it's really hard to build a holistic view. And embedding is a machine learning generated vector that somehow describes the user. And the interesting thing about embeddings is, if you have two users who are similar, similar being that they're close together and vector space, their embeddings, are considered similar. If you have two items that are similar, they're considered similar. So when you see recommendations on Amazon, it's literally true that the embeddings of each of those items are actually close together in vector space. So amazing news ever, essentially vectors, you need to have a nearest neighbor operation. All this stuff has existed for a while now. But what's changed recently, is people need to get this stuff out of the lab. It's no longer Hey, who can beat SEDAR anymore? Now it's like, Hey, we're Coca Cola. And we need to use our embeddings to be able to optimize our eCommerce experience, or we want to decide where to put ads or million different things, you can use embeddings especially for user embeddings.

Once that happened, we ended up in this place where we realized that all the stuff that we were doing in the lab doesn't really work in production. You have all these factors, you have this kind of hacky algorithm to Junior's neighbor. But that doesn't really scale. One and two, it's messy, like, What do I do? Do I upload a file to S3 and then create some like in memory index to try the nearest neighbor, that's actually what most people did, the most common thing I saw, and we did it, too, was that we would actually put a CSV or a similar type of file into the Docker file with the model and generate the embedding index right there. And if anyone has worked with embeddings, and put them in production, in this listening, I'm sure we got a laugh out of it, because everyone does something like that. It's kind of wild that we have to accept that this is how it's done.

And so it made sense to me that you needed a piece of infrastructure. And we don't do that in our databases, right? We don't upload the CSVs of our data onto our Docker file, we have a database. So we need the database for embeddings to it made sense. So the weird thing was it didn't really exist. So this idea of a vector database, we actually built my last company trying to solve a problem for us. And we kind of took it into embedding hub, we built that the core of such a database around embeddings. And we added all the things we just talked about workflows, like the ability to version things, rollback, share, search, and discover even some things around access control and governance.

We layered that on top. And that's what makes us kind of unique, the vector database space. Those capabilities are MLOps capabilities. Those are things you do in general in MLOps systems, and embeddings are a type of feature, a model feature. You can literally use embeddings in models like it's very common to actually take a user embedding and plug in place of user ID to a model to make, to do whatever with. So, in a sense, an embedding hub provides all the value of a feature store for embeddings. And there's an obvious missing thing, the great vector database. So that's kind of one of the problems that gets solved.

Sam Ramji:

Yeah, that was one of the questions I was gonna ask. Vector databases are also things that have been around for a while, there's some computer science part, right? You've got companies like Kinetica, or pine cone that are record databases for- for different reasons. So how does, how do embeddings, and has embedding have different from traditional vector database?

Simba Khadder:

Yeah, the main thing is that we're thinking about the workflow, data scientists, a lot of those products that we see out there are some open source wants to bui- trying to build in another piece of the infrastructure, which is part of the problem. I mean, it's- there's a missing- I mean, vector database needs to exists. That's, that's, that's a given. But we're looking at it from the perspective of a machine learning team. How do we not just make this another piece of infrastructure that has all the other problems of using just like any old piece of data infrastructure into your ml teams? And how do we build it from the ground up, to be optimized for the ML workflow, because the vector database, you know, it's funny that we use like vector databases, kind of, you know, generic term.

But if you look at any of these companies, and what they talk about, their use cases are so machine learning. So it's essentially an embedding database. So we just realized that and thought, cool. Well if I'm using embeddings, I need to generate training data, I sometimes want to locally cache all these embeddings, when I'm training, it's just a completely different workflow. There's all these things that become very obvious if you realize that the people using this are machine learning practitioners, and we're using it for machine learning use cases. So we built around that. So that's a workflow built around embeddings. It's not just like an, technically a database that can perform nearest neighbor and other operations natively.

Sam Ramji:

And what's an example of one novel capability that you built into embedding hub to make it easier to be part of the ML workflow?

Simba Khadder:

I mentioned two of them. The one that I think is very simple, but it's actually kind of unique to us as it's versioning. Versioning immutability, those two things together allow you to scale out to many different teams, if you have an immutable user table of embeddings. That means that anyone in the organization can safely use your user embedding. That's one thing. The other thing, which is more kind of a, that's, that's more of an organizational value, for a single data scientists value, when you're creating embeddings, or when you're training a model that uses embeddings, you really don't want to be doing a database call every single time, especially if you have one database, which is your production database, because you might actually be hampering latency for your users, because you're training a model.

So you could either build your own system, or you run two of them, and you create kind of a read replica, and you can do all of that yourself. Or we have it built into the API. So you just as a data scientist, you just say, hey, I want this embedding table, we call a space, like a vector space throughout this space, and I want to get the training version of it. So then you don't even really care what we do, if all you care about is that it's optimized for training, and it should not affect how, you know your production database does much-

Sam Ramji:

That's very cool. So let's, let's go from there and pop back level and look at the feature store. Problem space, right? There was a time before we used feature stores as an isolated sort of infrastructure to solve for some of these issues. So lets- I'll be talking about the issues a little bit, and then look a little bit about the difference between like, let's say a monolithic feature store, right? That's, that's sort of intended to be the one stop shop that holds everything in it, versus maybe a virtual feature store overlay?

Simba Khadder:

Yeah, there's three types of feature store architecture in a team. One thing I guess I'll start with is that even the term feature store isn't well defined yet. When I talk to people and they say feature store, I typically start with what does that mean to you? Because it's not well understood, hopefully, you know, someone listening to this four years from now will be like, “Oh, no, it's well understood now,” but that's part of our job is to make that happen. So the three architectures that are currently the most common. One is what I call the literal feature store, where all you're storing is the actual scalar, or the value, whatever the feature is, there's no transformations, it's actually literally where features are stored.

So almost like if the features are stored in Cassandra, you wrap Cassandra and like I put some lipstick on it called machine learning and boom, you have a feature store, technically. This is actually the approach at the moment like Google and Amazon's feature store kind of follow that approach to transformation layers quite differently. The value is actually truly simple. Like it's just the revenue storing directly into whatever databases you're using, you to store it into the feature store. Typically, we'll also have an offline online store, offline for training and online for serving. So there's value there.

The disadvantage, and I think it's a pretty major one, if you think in terms of the data scientist is workflow, is a data scientist thinks as a feature as a logical entity. This feature is defined as the last 10 songs these are listened to, it's that logic, is the features not the actual current, last 10 songs that user listen to. So because of this, if I change it from 10, to 15, I have to change transformation wherever I do transformation, after rewrite all of my data myself into the feature store. And so it's almost like the feature store, that version of feature store makes a ton of sense if nothing ever changes. And you're super happy with how your data pipelines work. Now, if you think about a typical data scientist's workflow, the majority of our time is spent on feature engineering. Anyway, so it's kind of not really solving the problem there, right the, the MLOps problem again, which is to allow data scientists and ML teams iterate more quickly.

Sam Ramji:

It seems like it creates a cyclical data engineering problem, in order to make progress on any iterations of the ML work that you're doing.

Simba Khadder:

Yes, it's, it's, it's technically solving the feature storage problem, but it's not actually solving the ML problem which is to make things smarter, and faster. So that's the common one I see, there's two others, and one is virtual, which I'll do last. The next one, which I think is kind of become common with, with some of the, the players in lots of the vendors in the space like Tecton have taken the approach of kind of badge included all in one feature store. This is also very common in the hidden house features stores like Airbnb, they have Zip line, Uber has Michelangelo's and Palette. And the approach they've taken is that their features are actually define their features or definitions. That was like the most logical choice a ton of sense. It can do things like backfill a feature, which means that if I change the the, you know, the last 10 songs to the last 15 songs, I need to actually update all my train data sets, which is asked go back in time, if it's streaming data, it's especially hard, I have to go back in time, and I have to regenerate all my train data, the new logical feature definition.

So this all included feature will do that. So that makes it many times better, in my opinion, than the kind of the feature storage architecture I talked about before. The problem I see movies, they work well, if it's kind of an in house, thing, only, because then you have a team in place, and, and, and you kind of can't commit really hard to this thing, coming to a vendor with all of your machine learning, you're actually it's an in house team. So the problem with a vendor doing this in particular, is that you have to rewrite all of your features to fit their definition language. And we talked to banks who have 1000s and 1000s of features, some of which they don't even know are being used. So they have to go through all of those and rewrite them, and hope that they don't screw anything up into the DSL that this all-inclusive feature store has.

It'll typically replace something for you. So if you're using, let's say, Spark and Cassandra, and all these tools, you would have to replace them with the feature store because it does its own storage, it does its own transformations, the adoption cost is extremely high, the lock in is very high. And you're kind of at the will of the feature store. If you own the team, and you have a ton of resources, and you can afford to do all that. And you can kind of afford to build a better version of Spark for you and a better version of Cassandra for you or whatever. But it can work. And you know, maybe in Uber's case and Airbnb's case, they feel they can do that for routines, for very specific use case. But for the majority of companies, it's just not feasible to build one, all inclusive feature service that just does everything for machine learning teams.

So the five architecture I'll talk about is the virtual feature store. And this is the approach we've taken, LinkedIn kind of pioneered it. And it was from when I was talking to that team. But a lot of things start to click with me about why it was the right approach. The idea of a virtual feature store is, data scientist wants to work with this feature store, almost like the API that the all inclusive features store has, like I want to be able to change feature definitions, how my training data set be automatically updated, have those features to be right in production, I want all of that. I want to talk in terms of features, on overtime training data sets. I don't want to talk in terms of S3 files and rows and Cassandra. That makes a ton of sense. But same time, I've already written all these features. I'm very comfortable on Spark. Some of our features are in Flink and some of our features are in Dask and we have all these different things we use.

And we don't want to just commit to one processing engine. The idea of a feature as an abstraction is actually agnostic in a data scientist's mind to how it's processed and how it's stored. Sure, they care about latency, they care about freshness, they care about the logic, they care about all of that stuff. But they don't actually care if it's built via Spark or Dask and, and they just care about what the logic is and what the plugin latency freshness is. So the virtual features are really commits that heavily and says, Hey, you give us infrastructure, as providers will sit on top of it, we will turn your current data infrastructure into a feature store for you. I think of HashiCorp's product Terraform.

In theory, HashiCorp could have built the entire cloud solution and made the interesting part of it that has its own definition language. Now that sounds silly to say right now, right? Because it's just like no, it should just be a definition language to be an API on top of the providers. Well, I think in the future we'll look back at it, yeah it's silly to think about features versus not being just an abstraction and a definite language on top of the providers. Because you know, are we as a SERB really going to build a better Cassandra or better spark? No, there's no way. But we can build a, a, a better workflow for machine learning on top of all these systems, definitely think that, that's, that's, that's doable for a startup. And that's, again, entirely our focus. And that's, I think, the focus of the virtual feature stories. How can we just do that, and not worry about everything else.

Sam Ramji:

That makes a ton of sense to me, it's almost feature store as middleware, or feature store as choreography, or orchestration overlay, which it's a pretty good classical pattern for these very complex, you know, high end heterogeneous, especially enterprise environments, where not only do you have to satisfy a lot of constituencies like right now today, but you're gonna have to solve for a lot more in the future. So you know MuleSoft I think is a really nice example of a company that did a beautiful job of creating the right tools, and then creating the right extensibility. And opening up the ability to have an unopinionated way to get messages through the system.

So for an old middleware person like me, what you're saying makes a lot of sense. So it's been a fantastic time to spend with you. Hopefully, our listeners will feel substantially advanced on their thinking about embeddings and vector databases and feature stores and how, what it might mean to get closer to the user's workflow. I would love it if you could leave our audience with one resource or one piece of advice as they sort of advance into the world of MLOps and help produce the kind of five year future that you described at the beginning of the show.

Simba Khadder:

It's important to just make sure that you don't lose sight of your North star, make sure you understand what it is. And it's not as obvious as people make it. Like, it's obvious in retrospect, in retrospect, we understand that DevOps, this is about DevOps, and all these different categories we can understand now, it's obvious now. But it was not obvious when it started. In fact, quite the opposite. Most people are non-believers, we don't need this. We don't need DevOps. We already have a system admins that do all the stuff for us. So keep your North star and don't lose sight of that and build towards that. And that will help you whether it's- you're building a company, you're building a team locally, whoever you're managing a large team, just making sure that you know what the goal is, and costly checking back and treating everything you're doing all the tasks, all the priorities, everything should be done in terms of North star.

Sam Ramji:

A little bit. You know, all the great products and great companies I can think of have shared one thing in common which is a relentless focus on their outcome.

Simba Khadder:

Yeah, and then the end that's all that matters is kind of making sure that you are solving a problem at hand. It's- no one cares how, everyone cares that it's just, it happened.

Sam Ramji:

That's awesome. Simba thank you so much for spending time with us today. I'm really grateful for the experience and the wisdom that you bring.

Simba Khadder:

Thanks for having me. It was a lot of fun to chat with you.

Open Source Data

A Podcast Series With Charna Parkey

Share

Season 2 · Episode 10

Embeddings, Feature stores, and MLOps with Simba Khadder

Episode Guest

Simba Khadder

Episode Transcript