How have Data Engineers been living without an equivalent to DevOps for so long? This is just one of the points Sam Ramji and Barr Moses discuss on the Open Source Data podcast. Moses co-founded the data observability platform Monte Carlo to give data teams the same data insights as software engineers have had for years.
In this episode of the Open Source Data podcast I talked with Barr Moses from Monte Carlo. We touched on data observability, being truly customer-led in your business growth and what made her leave Gainsight to co-found Monte Carlo with Lior Gavish, the former Senior VP of Engineering at Barracuda.
As always we started the conversation with my guest’s take on open source data. To Moses, the biggest value comes from the strong and energetic community in open source and data. She’s excited to be part of this community facing challenges like data trust, data discoverability, data ownership and new roles in data. Together with the team at Monte Carlo she’s honed in on the challenge of data observability.
Observability is a concept in DevOps, where software developers and IT operations work together to provide higher quality work faster. With observability they can better understand the health of the system they're working on. With real-time data and decision-making at the same speed, there is a need for a similar approach to the quality of data. And this is where Monte Carlo comes in.
“Being able to trust your data is in my opinion, the number one challenge to actually becoming data-driven and adopting data and fulfilling the promise of data.” – Barr Moses
Letting data catch up to software development
The issues with data management became apparent to Moses, when she was working at Gainsight. Every week something would break, and her team would be tasked with fixing the data pipeline. The problem was that there were too many ways for the data to be corrupted, and the team was always in a hurry to find the cause of the problem.
So they’d manually map out the entire data pipeline on a whiteboard. While this helped them all understand what could go wrong and where, it never really got them any closer to a solution. Moses and her colleagues felt at times like they were going crazy. So she talked to several hundred other data teams to see if she and her team were alone in that experience. They were not.
“Engineers have tools like AppDynamics, Datadog and PagerDuty. And then for some reason data teams are like, ‘Let me manually map out everything and try to figure it out.’” – Barr Moses
Moses’ first step to improving general data quality was to blog about data observability as the next frontier of data engineering.
Data observability is built on five pillars:
- freshness
- distribution
- volume
- schema
- lineage
Together, these five pillars document that the data is recent, properly formatted, and complete as well as clarifying schema history, sources and assets for the data. The Monte Carlo platform helps with all five, so data teams can trust their data and work as well as software development teams running DevOps.
How to achieve truly customer-led growth
Moses has been known to have a somewhat controversial take on growing tech companies and using phrases like customer-lead growth. She believes that the specific needs of the customer and their businesses should be the start and end of your product development.
When she co-founded Monte Carlo (named after the Monte Carlo method), the team had no more than an idea. They knew there was a problem that affected hundreds of teams. The idea was that the solution would be something like the cloud monitoring service DataDog or the full-stack observability service AppDynamics. But since this was a solution in a new category of its own, they decided to work with customers to build the app with the help of customer data.
“We started working with customers on their problems before we even wrote a single line of code.” – Barr Moses
This approach made the development of Monte Carlo products very concrete, since the team knew the detailed use issue and use case before they built the solution.
This is not a new way of approaching tasks or projects for Moses. She grew up in Israel and moved to the San Francisco Bay Area at a young age. She grew up with parents that were world views apart. Her father is a physics teacher, and her mother is a spiritual dance teacher. In her world, there was always going to be opposing answers to her questions, so she’d figure things out on her own. This prepared her well for the task of finding the solution to data issues even if the customer is a startup where almost every employee has a different opinion.
Moses is not an engineer herself. However, having served as Commander of a data analyst unit in the Israeli Air Force, she has a natural inclination to make content very approachable for engineers and data-interested people alike. For her, the most important stories are found in the people that create and use the data.
At Gainsight she helped create the category of prescriptive data guidance with a team using Gainsight data internally to make decisions. Now, she’s doing it all over again with Monte Carlo. Her advice for newly graduated “listeners” – whether they be engineers, computer scientists or something completely different – is:
"The magic happens when you find a strong fit between what people love working on and the problem that needs to be solved. Have fun. Life is too short for anything else." – Barr Moses
Learn more about Monte Carlo and their work on Barr Moses’ Medium. If you like the Open Source Data Podcast, please pass it on to a friend or leave a review to help us reach more listeners.
About Barr Moses
Barr Moses served as a Commander in the Israeli Air Force before moving to the San Francisco Bay Area where she worked at Gainsight for a number of years. In January of 2019 she co-founded Monte Carlo and currently serves as its CEO.
Resources
More Company
View All
London Called. RAG++, The AI Event, Answered!
DataStax Named a Leader in The Forrester Wave™: Vector Databases, Q3 2024
Live From New York! It’s RAG++, The AI Event
