A data lake is a central location that stores a huge volume of data in its original, unprocessed form. In contrast to a hierarchical data warehouse, which stores data in files or folders, a data lake stores data using a flat design and object storage. Object storage tags data and assigns it a unique identification, making it easier to identify and retrieve data across regions and improving speed. Data lakes enable numerous applications to take advantage of data by using low-cost object storage and open formats.
In reaction to the constraints of data warehouses, data lakes were created. While data warehouses provide businesses with highly performant and scalable analytics, they are costly, proprietary, and incapable of handling the modern use cases that most firms need to solve. Data lakes are frequently used to combine all of an organization's data in a single, centralized location where it can be kept "as is," without the need to impose a schema (i.e., a formal structure for how the data is arranged) up front, as a data warehouse does.
A data lake can hold data at all phases of the refinement process: raw data can be ingested and stored alongside an organization's structured, tabular data sources (such as database tables), as well as intermediate data tables generated during the refining process. Data lakes, unlike most databases and data warehouses, can process all data kinds, including unstructured and semi-structured data such as photos, video, audio, and documents, which are essential for today's machine learning and advanced analytics use cases.
A data lake is a huge container, analogous to a lake or a river. A data lake, like a lake, has various tributaries coming in, with structured data, unstructured data, machine-to-machine, and logs flowing through in real-time.
Elements of a Data Lake and Analytics Solution
When organisations are developing Data Lakes and Analytics platforms, they must consider a number of critical features, including:
Data Transfer
Data Lakes allow you to import any quantity of data in real-time. Data is acquired from many sources and placed into the data lake in its original format. This method allows you to scale to any size of data while saving time on building data structures, schema, and transformations.
Securely Store and Catalogue Data
Data Lakes allow you to store relational data, such as operational databases and data from line of business applications, as well as non-relational data, such as mobile apps, IoT devices, and social media. They also enable you to comprehend what data is in the lake by crawling, categorising, and indexing data. Finally, data must be secured to ensure the safety of your data assets.
Analytics
Data Lakes enable multiple roles in your organisation, such as data scientists, data developers, and business analysts, to access data using their preferred analytic tools and frameworks. This includes open-source frameworks like Apache Hadoop, Presto, and Apache Spark, as well as commercial options from data warehouse and business intelligence suppliers. Data Lakes enable you to do analytics without having to move your data to a separate analytics solution.
Machine Learning
Data Lakes enable enterprises to develop many types of insights, such as reporting on historical data and performing machine learning, in which models are built to foresee expected outcomes and suggest a range of prescribed actions to obtain the best possible result.
The Value of a Data Lake
The ability to collect more data from more sources in less time, as well as enable people to interact and analyse data in new ways, leads to better, faster decision making. Data Lakes can bring value in the following ways:
Improved Customer Interactions
A Data Lake can combine customer data from a CRM platform with social media analytics, a marketing platform that includes purchasing history, and incident tickets to enable the business to understand the most profitable customer cohort, the cause of customer churn, and the promotions or rewards that will increase loyalty.
Improve R&D Innovation Choices
A data lake can assist your R&D teams in testing hypotheses, refining assumptions, and assessing results, such as choosing the right materials in your product design to result in faster performance, conducting genomic research to result in more effective medication, or understanding customer willingness to pay for different attributes.
Increase Operational Efficiencies
The Internet of Things (IoT) brings new ways to collect data on processes such as manufacturing, with real-time data coming from internet-connected devices. A data lake makes it simple to store and run analytics on machine-generated IoT data in order to uncover methods to cut operational expenses and improve quality.
Planning a Data Lake
A data lake reduces the upfront effort of storing data because we are not obliged to arrange it initially. However, this does not imply that no planning is taking place. To avoid becoming a data swamp, several factors must be considered while constructing a data lake:
- Ingestion needs (push/pull via streaming or batch)
- Security around data access
- Data retention and archival policies
- Encryption requirements
- Governance
- Data quality
- Master data management
- Validity checks necessary
- Metadata management
- Organization of data for optimal data retrieval
- Scheduling and job management
- Logging and auditing
- Data federation utilization
- Enrichment, standardization, cleansing, and curation needs
- Technology choices comprising overall data lake architecture (HDFS, Hadoop components, NoSQL DBs, relational DBs, etc.)
- Modular approach to overall design
Cases for a Data Lake
Data lakes can be used in various ways:
- Ingestion of semi-structured and unstructured data (big data) such as equipment readings, telemetry data, logs, streaming data, and IoT data.
- Analyzing data experimentally before its value or purpose is fully defined, aiding in "proof of value" scenarios.
- Advanced analytics support for data scientists and analysts.
- Storage of archival and historical data, useful for active archiving strategies.
- Lambda architecture support, including speed, batch, and serving layers.
- Data warehousing preparation, using a data lake as a staging area.
- Enhance a data warehouse, including data difficult to store or rarely queried.
- Logical data warehouse with distributed processing capabilities.
- Application assistance, where a data lake serves as a source for front-end applications.
Data Lakes Compared to Data Warehouses
A typical organisation will require both a data warehouse and a data lake, depending on requirements, as they fulfil distinct needs:
- A data warehouse is designed to examine relational data from transactional systems. The data structure and format are predefined for rapid SQL queries and operational reporting.
- A data lake holds relational and non-relational data from multiple sources. No schema is imposed upfront, allowing broader analysis with SQL, big data analytics, real-time analytics, and machine learning.
- Organizations extend warehouses with data lakes to enable advanced query capabilities and data science use cases. Gartner calls this evolution the Data Management Solution for Analytics (DMSA).
Data Lake Concepts
Data Ingestion
Connectors extract data from different sources and load it into the data lake, supporting structured, semi-structured, and unstructured data, batch, real-time, or one-time load, from databases, web servers, emails, IoT, and FTP.
Data Storage
Should be scalable, cost-effective, fast-accessible, and support various data formats.
Data Governance
Manages availability, usability, security, and integrity of organizational data.
Security
Implemented at every layer (storage, retrieval, consumption), including authentication, accounting, authorization, and data protection.
Data Quality
Critical for business value; poor-quality data leads to poor insights.
Data Discovery
Tagging techniques help organize and interpret ingested data.
Data Auditing
Tracks changes to key datasets and evaluates risk and compliance.
Data Lineage
Tracks data origins, movement, and transformations, easing error corrections.
Data Exploration
Identifies the right dataset for analysis before starting deeper analytics.
Data Lake Architecture
Important tiers include:
- Ingestion Tier: Loads data from sources in batches or real-time.
- Insights Tier: Where research and analysis occur using SQL, NoSQL, or Excel.
- HDFS: Landing zone for all data at rest.
- Distillation Tier: Converts stored data to structured formats for easier analysis.
- Processing Tier: Runs analytical algorithms, queries, and batch or real-time processing.
- Unified Operations Tier: System management, monitoring, auditing, and workflow management.
What is a Data Mart?
A data mart is a subject-oriented subset of a data warehouse. It allows corporations to provide business divisions and product lines with access to relevant data. Data marts convert raw data into usable information, offering pre-built summaries and queries tailored to departmental needs.
Data Marts Defined
Data marts correspond to departments in larger organizations, providing faster, department-specific analytics. They preserve warehouse integrity while enabling smaller-scale, focused data access.
Types of Data Marts
Three main types exist: Dependent, Independent, and Hybrid, based on how they are populated. The ETT (Extraction, Transformation, and Transportation) process populates data marts.
1) Dependent Data Mart
- Sourced from an existing data warehouse (top-down approach).
- Can use logical views (virtual) or physical subsets.
2) Independent Data Mart
- Stand-alone systems not dependent on a central warehouse.
- Suitable for small departments; requires full ETT process.
3) Hybrid Data Mart
- Integrates data from both DW and operational systems.
- Flexible, can reference other data marts.
Implementation Steps of a Data Mart
- Designing: Requirements gathering, creating logical/physical structures, ER diagrams.
- Constructing: Design tables, views, indexes, etc.
- Populating: Extract, transform, load data along with metadata.
- Accessing: End-users query data for analysis and reports.
- Managing: User access, performance tuning, maintenance, recovery scenarios.
Data Mart Use Cases
- Subject-focused analytics: Specific to products, sales, customers, etc.
- Selective Data Access: Restricts access to sensitive information.
- Improved Resource Management: Balances resource use across departments.
- Time-limited Data Projects: Faster and cheaper setup than full DW.
Advantages of Data Marts
- Cost-effective deployment.
- Faster access due to smaller datasets.
- Efficient query execution; live and scheduled queries optimized.
- Independent operation prevents central DW outages from affecting access.
- Lower license costs for third-party data.
Disadvantages of Data Marts
- Limited visibility for broader company data.
- Independent marts may hinder cross-mart reporting.
- Managing multiple marts can be complex.
- Automated data propagation may incur extra costs.
- Field name syntax misalignment can cause reporting issues.
Structure of a Data Mart
Three schema-level architectures:
- Star: Central fact table connected to dimension tables.
- Snowflake: Dimensions further normalized into sub-dimensions.
- Denormalized Tables: Combines all required data into one table for faster queries; may increase redundancy.
Data Lake vs. Data Mart
- Data Lakes: Contain raw, unfiltered data; broad deep analysis.
- Data Marts: Small, structured subsets; fast, focused analytics.
- Data Lakes: All-in-one solution (warehouse, database, mart).
- Data Marts: Single-use, no ETL performed.
Database vs. Data Mart
- Database: Transactional repository (OLTP), raw data, first step in ETL.
- Data Mart: Analytical repository (OLAP), processed data, last step in ETL, end-user accessible.
