Lior Gavish is CTO and Co-Founder of Monte Carlo, a data reliability company backed by Accel, Redpoint Ventures, GGV, ICONIQ Growth, and Salesforce Ventures. Prior to Monte Carlo, Lior co-founded cybersecurity startup Sookasa, which was acquired by Barracuda in 2016. At Barracuda, Lior was SVP of Engineering, launching award-winning ML products for fraud prevention. Lior holds an MBA from Stanford and an MSC in Computer Science from Tel-Aviv University.
The shift to treating data like production software: "Like software, data systems are becoming increasingly complex, with multiple upstream and downstream dependencies. Ten or even five years ago, it was normal and accepted to manage your data in silos, but now, teams and even entire companies are working with data, facilitating a more collaborative and fault-resistant approach to data management. Over the past few years, we’ve witnessed the widespread adoption of software engineering best practices by data engineering and analytics teams to address this gap, from adopting open source tools like dbt and Apache Airflow for easier data transformation and orchestration to cloud-based data warehouses and lakes like Snowflake and Databricks." - Lior Gavish, co-founder and CTO, Monte Carlo4 March 2022
Why it can be hard to measure the impact of data engineering / analytics work: "Most companies struggle with measuring the impact of the work done by data engineers, given that it is one step removed from the data products that scientists create. I hate to say it, but all-too-frequently, data engineers get attention only when things break and are overlooked when data systems work well. Because of this, I always recommend that teams track how much time, money, and resources data engineers save and directly tie their work to the impact-driven by data scientists and analysts." - Lior Gavish, co-founder and CTO, Monte Carlo4 March 2022
Whether or not to build vs. buy big data technology: "The first thing I recommend you do is to evaluate whether it makes sense to build an in-house solution for data cataloging or invest in a third-party vendor to provide that solution for you. There are pros and cons to each of these solutions. I have seen B2C companies such as Airbnb, Netflix, and Uber build their own tools to ensure that their particular needs and stack are supported. However, you have to remember that these organizations handle insanely large amounts of data and have the engineering resources available to invest in building and maintaining the solution. Also, keep in mind that oftentimes custom-built solutions can lead to limited visibility and collaboration, given they may not be able to fully support all use cases." - Lior Gavish, co-founder and CTO, Monte Carlo4 March 2022