Hello everyone, in this issue, Jesse wants to break away from the TSDB field and talk about the topic of data lakes and data warehouses. As an outsider, Jesse also wants to give a general introduction to the two.
This article only represents personal opinions, if there is any bias, please forgive me~
20 years ago, data warehousing wasn't the hottest technology in the industry. There have been barriers between data for a long time, and isolated data workflows are common. Most enterprises are local computing clusters, and tasks between businesses are only limitedly associated. Today, with the rise of data-driven analytics, cross-functional data teams, and the cloud, the terms "modern data warehouse" and data lakes are coined. In many ways, the cloud makes data easier to manage, accessible to a wider range of users, and faster to process. Without a data lake versus data warehouse discussion, companies cannot use data in a meaningful way. However, when it comes to choosing between a data lake or a data warehouse, the answer is not easy. With the release of Amazon Redshift in 2013, and the release of Snowflake, Google BigQuery, etc. in the following years, the market has become more and more popular. Add a data lake like S3 or Databricks to the mix and the decision between data lake vs. data warehouse becomes more difficult.
What are data warehouses and data lakes
A data warehouse is a data repository that provides data storage and computation, typically utilizing SQL queries for data analysis use cases. A data lake is a data repository that provides storage and computation for structured and unstructured data, typically for streaming, machine learning, or data science use cases.
Similarities and differences between data lake and data warehouse
Both data lakes and data warehouses are data repositories. The 3 main differences between data warehouses and data lakes are how they provide storage, metadata, and computing power.
Storage: Storage refers to the way data warehouses and data lakes store all records present in all tables. By utilizing various storage technologies and data formats, data warehouses and data lakes can serve a wide range of use cases with the required cost and performance characteristics. Traditionally, data lakes store raw structured, semi-structured, and unstructured data indefinitely, while data warehouses store data and its corresponding metadata in an orderly fashion. These differences converged over time as Databricks enabled users to add structure and metadata through the Unity Catalog and Delta Lake, while Snowflake introduced Apache Iceberg tables to bring the reliability and simplicity of SQL tables, while enabling Apache Engines such as Spark, Trino, Apache Flink, Presto, and Hive can safely use the same tables at the same time.
Metadata: Data warehouses and data lakes often provide a way to manage and track all the databases, schemas and tables we create. These objects are often accompanied by additional information such as schema, data type, user-generated description, and even freshness and other statistics about the data.
Compute: Compute refers to the way a data warehouse or data lake performs calculations on the data records it stores. This engine allows users to "query" data, ingest data, and transform data. Typically, these calculations are expressed through SQL. This is another area where data lakes overlap with data warehouses. Snowflake's Snowpark supports multiple programming languages, such as Java, Python, or Scala, which are then executed as SQL functions. Later they also launched Snowpark Python, a native Python experience with pandas and a PySpark-like API for data manipulation without writing lengthy SQL. On the other hand, Spark SQL can help convert languages like Python, R, and Scala into SQL commands.
Why do you need a data warehouse
Data warehouses are fully integrated and managed solutions, making them easy to build and use out of the box. When using a data warehouse, enterprises typically use metadata, storage, and compute in a single solution built and operated by a single vendor. In the data lake vs. data warehouse discussion, consider that data warehouses generally require more structure and schema, which often mandates better data hygiene and reduced complexity when reading and using data. With its prepackaged functionality and strong support for SQL, data warehouses facilitate fast, actionable queries, making them ideal for data analytics teams.
Why you need a data lake
In the data lake vs. data warehouse debate, a data lake is the DIY version of a data warehouse, allowing data engineering teams to choose the various metadata, storage, and compute technologies they want to use with their systems. Data lakes are great for data teams and data scientists looking to build a more custom platform, often supported by a handful (or more) of data engineers.
Some common characteristics of data lakes include:
(1) Decoupling storage and computation: This feature not only saves a lot of cost, but also helps in parsing and enriching data for real-time streaming and querying.
(2) Support for distributed computing: Distributed computing helps support the performance of large-scale data processing because it allows for better segmented query performance, more fault-tolerant designs, and superior parallel data processing.
(3) Customization and interoperability: Due to its "plug-and-play" nature, data lakes support the scalability of data platforms, and different elements of the stack can easily work together as the company's data needs develop and mature.
(4) Mainly based on open source technology: This helps reduce vendor lock-in and provides excellent customization, which is very effective for companies with large data engineering teams.
(5) The ability to process unstructured or weakly structured data: Data lakes can support raw data, which means you have more flexibility in processing data, which is very suitable for data scientists and data engineers. Working with raw data gives you more control over aggregations and calculations.
(6) Support complex non-SQL programming models: This is a difference between data lakes and data warehouses. Unlike most data warehouses, data lakes support Apache Hadoop, Apache Spark, PySpark, and other frameworks for advanced data science and machine learning.
What is the integration of lake and warehouse
The decision between a data lake or a data warehouse is hard enough, but an alternative has emerged, especially among data engineering teams. It's a solution that combines data warehouse and data lake capabilities, combining traditional data analytics techniques with techniques built for more advanced computing, such as machine learning. Lake-warehouse integration first emerged when cloud warehouse providers started adding features that offered lake-style benefits, such as Redshift Spectrum or Delta Lake. Likewise, data lakes have been adding technologies that provide warehouse-like capabilities, such as SQL functions and schemas. Today, the difference between a data lake and a warehouse is narrowing.
how to choose
The choice of data lake and data warehouse is not an easy answer. Regardless of what we choose in our data lake or data warehouse decision, here are some rules that should follow:
(1) Select the appropriate solution corresponding to the company's data goals. Building a data lake from scratch may not make sense in terms of time and resources if a company only regularly uses one or two key data sources in a few workflows. However, if the company is trying to use data to inform everything under the sun, the all-in-one solution may provide quick, actionable know-how for users across roles.
(2) Understand who the core users are. Is the primary user of the company's data platform the business intelligence team, spread across several different functions? How about a dedicated team of data engineers? Or groups of data scientists doing A/B testing on various datasets?
(3) Observability of data. Data warehouse, data lake, warehouse lake: All three solutions (and any combination of them) require a holistic approach to data governance and data quality. After all, it doesn't matter how advanced our pipeline is if the data is corrupted, lost, or otherwise inaccurate. Some of the best data teams are leveraging data observability, an end-to-end approach to monitoring and alerting to problems in data pipelines. In summary, the choice of data warehouse and data lake is not so much a matter of choosing one tool or the other as choosing the right tool for the job.
Having said so much, let's go back to the time-series database scenario. Will time-series data be entered into a data lake in the future, and then various data lakes will be aggregated and correlated for query in the data warehouse? For example: "Multi-system coexistence is a relatively common architecture in enterprises, such as a data lake, plus multiple data warehouses, and other specialized systems, such as streaming, time series, graph and image databases, etc." Everything will take time to give answer.
Introduction to CnosDB
CnosDB is an open source distributed time series database with high performance and high usability, which has been officially released and fully open sourced.
Welcome to pay attention to our code warehouse, one-click three links: https://github.com/cnosdb/cnosdb