In this article, we'll explore Azure Data Lake analytics and query data using U-SQL.
Introduction to Azure Data Lake Analytics (ADLA)
The Microsoft Azure platform supports big data such as Hadoop, HDInsight, and data lakes. Typically, a traditional data warehouse stores data from various sources, transforms the data into a single format, and analyzes it to make decisions. Developers use complex queries that may take longer for data retrieval. Organizations are increasing their footprint in cloud infrastructure. It leverages cloud infrastructure warehouse solutions such as Amazon RedShift, Azure Synapse Analytics (Azure SQL Data Warehouse), or AWS Snowflake. Cloud solutions are highly scalable and reliable to support your data, query processing and storage needs.
The data warehouse follows the Extract-Transform-Load mechanism for data transmission.
Extract: extract data from different data sources
Transformation: converting data into a specific format
Load: Load data into predefined data warehouse schemas, tables
Data lakes do not require strict schemas and transform data into a single format prior to analysis. It stores data in raw formats such as binary, video, image, text, document, PDF, JSON. It transforms data only when needed. Data can be in structured, semi-structured and unstructured formats.
Some useful features of a data lake are:
It stores raw data (raw data format)
It does not have any predefined schema
where you can store unstructured, semi-structured and structured
It can handle data volumes of petabytes or even hundreds of petabytes
The data lake follows the schema (schema) in the reading method, and converts the data according to the requirements
In a nutshell, the Azure data platform architecture is as follows. Image reference: Microsoft Documentation
Ingestion: collect data from various data sources and store it in its original format into Azure Data Lake
Storage: Store data to Azure Data Lake Storage, AWS S3, or Google Cloud Storage
Processing: process the data in raw storage into a compatible format
Analysis: Perform data analysis using stored and processed data. You can use Azure Data Lake Analytics (ADLA), HDInsight or Azure Databricks
| This article: https://architect.pub/overview-azure-data-lake-analytics | ||
| Discussion: Knowledge Planet [Chief Architect Circle] or add WeChat trumpet [ca_cto] or add QQ group [792862318] | ||
| No public |
【jiagoushipro】 【Super Architect】 Brilliant graphic and detailed explanation of architecture methodology, architecture practice, technical principles, and technical trends. We are waiting for you, please scan and pay attention. |
 |
| WeChat trumpet |
[ca_cea] Community of 50,000 people, discussing: enterprise architecture, cloud computing, big data, data science, Internet of Things, artificial intelligence, security, full-stack development, DevOps, digitalization. |
|
| QQ group |
[285069459] In-depth exchange of enterprise architecture, business architecture, application architecture, data architecture, technical architecture, integration architecture, security architecture. And various emerging technologies such as big data, cloud computing, Internet of Things, artificial intelligence, etc. Join the QQ group to share valuable reports and dry goods. |
|
| video number | [Super Architect] Quickly understand the basic concepts, models, methods, and experiences related to architecture in 1 minute. 1 minute a day, the structure is familiar. |
|
| knowledge planet | [Chief Architect Circle] Ask big names, get in touch with them, or get private information sharing. |
|
| Himalayas | [Super Architect] Learn about the latest black technology information and architecture experience on the road or in the car. | [Intelligent moments, Mr. Architecture will talk to you about black technology] |
| knowledge planet | Meet more friends, workplace and technical chat. | Knowledge Planet【Workplace and Technology】 |
| Harry | https://www.linkedin.com/in/architect-harry/ | |
| LinkedIn group | LinkedIn Architecture Group |
https://www.linkedin.com/groups/14209750/ |
| Weibo | 【Super Architect】 | smart moment |
| Bilibili | 【Super Architect】 |
|
| Tik Tok | 【cea_cio】Super Architect |
|
| quick worker | 【cea_cio_cto】Super Architect |
|
| little red book | [cea_csa_cto] Super Architect |
|
| website | CIO (Chief Information Officer) | https://cio.ceo |
| website | CIOs, CTOs and CDOs | https://cioctocdo.com |
| website | Architect practical sharing | https://architect.pub |
| website | Programmer cloud development sharing | https://pgmr.cloud |
| website | Chief Architect Community | https://jiagoushi.pro |
| website | Application development and development platform | https://apaas.dev |
| website | Development Information Network | https://xinxi.dev |
| website | super architect | https://jiagou.dev |
| website | Enterprise technical training | https://peixun.dev |
| website | Programmer's Book | https://pgmr.pub |
| website | developer chat | https://blog.developer.chat |
| website | CPO Collection | https://cpo.work |
| website | chief security officer | https://cso.pub |
| website | CIO cool | https://cio.cool |
| website | CDO information | https://cdo.fyi |
| website | CXO information | https://cxo.pub |
Thank you for your attention, forwarding, likes and watching.