Data stack + AI: Data stack V6.2 is innovatively released to make data development smarter

Recently, the Kangaroo Cloud Spring Conference with the theme of "Data + AI, building new productivity" came to a successful conclusion. The conference brought a series of " +AI " digital products and the latest industry precipitation, aiming to closely integrate data and AI. , breaking the traditional boundaries of productivity and empowering enterprises to achieve higher quality and more efficient digital development. At the meeting, Tou Tian, ​​product manager of Kangaroo Cloud Data Stack, brought a new release of Data Stack V6.2 that integrates AI capabilities . This is not only a simple product upgrade, but also represents Kangaroo Cloud’s bold prediction of the future. .

Data Stack V6.2: Maximizing data value

In the data-driven era, data has become the lifeline of enterprises. How to effectively manage and utilize this data is a question that every enterprise is exploring. The release of Data Stack V6.2 is precisely to solve this challenge and help enterprises ride the waves in the ocean of data.file

The core concept of the newly released Data Stack V6.2 is Data+AI . The new version not only provides the basic functions of the big data platform, but also provides enterprises with intelligent data analysis and applications through deep integration with AI technology . This means that enterprises can use the data stack platform to achieve integration of industry content systems, flexible and convenient data insights, extremely fast analysis engine calculations, and comprehensive management and control of data security. In addition, Kangaroo Cloud product solutions also cover many aspects such as lightweight, data governance, and information creation. All of this is designed to help enterprises optimize computing storage costs, improve data quality, promote standards and specifications, and ultimately maximize the value of data.

1. Lightweight data center solution, data processing is more efficient

By introducing efficient computing engines Doris and StarRocks , a revolutionary reconstruction of platform performance is achieved. This innovative move not only greatly improves data processing speed, reduces storage costs and operation and maintenance costs, but also optimizes query efficiency and brings unprecedented data operation experience to enterprises. The ad hoc query capabilitiesfile and high-performance analytical processing capabilities of Doris and StarRocks jointly build a powerful and flexible data processing platform. Users can easily cope with the real-time analysis needs of massive data and achieve instant data insights and decision support. In this process, the accuracy and reliability of the data are fully guaranteed, providing solid data support for the company's key businesses such as fault prediction, precision marketing, and process optimization.

2. Comprehensive data governance system to maximize corporate value

In Data Stack V6.2, we have comprehensively upgraded and redefined data governance to meet the growing needs of enterprises in data management. The five dimensions of the data governance center : storage, computing, quality, specification and value, constitute a comprehensive data governance system to ensure the integrity, accuracy and availability of enterprise data.file

The governance workbench provides an intuitive operation interface, making it simple and efficient to initiate, record, assign and process data governance tasks. Through this platform, enterprises can display data governance status from a personal perspective, a project perspective to a panoramic perspective, thereby ensuring that the data quality of each link is effectively monitored and managed. The code inspection function standardizes SQL code through SQL inspection rules to prevent possible management problems in advance. Small file management aims at the problem of small files in Hadoop clusters, optimizing cluster performance and scalability through one-time or regular merging, and improving data processing efficiency .

The data governance of DataStack V6.2 is not only an upgrade of technology, but also a shaping of enterprise data culture. Through such a governance system, enterprises can establish a complete data governance framework , promote data standardization and standardization, and ultimately maximize the value of data assets.

3. Full-link Xinchuang adaptation supports comprehensive localization

In this era of informationization and information innovation, we are well aware of the needs of enterprises for data security and independent controllability. Therefore, our platform not only achieves full-link information innovation coverage in terms of servers, operating systems, chips, middleware, metadata databases, computing engines, etc., but also makes deep adaptations in privatized deployment and full-process security protection. . This is our firm support for the national security strategy and our active fulfillment of corporate social responsibilities.file

4. Innovate and break through the capabilities of Paimon data lake to realize a batch-stream integrated data processing model

In the traditional data processing model, enterprises are often faced with the dilemma of developing and maintaining two sets of code logic: one for batch processing and the other for real-time stream processing. This not only means doubling the development and maintenance workload, but also requires processing the data merging logic between the two to ensure that the two systems go online simultaneously. Such a model not only increases resource consumption, but may also lead to data ambiguity, making it difficult to guarantee data accuracy and reducing business personnel's trust in data results.file

The innovative breakthrough of the data stack realizes the batch-stream integrated data processing model through the capabilities of Paimon Data Lake , effectively solving the above problems. The platform provides real-time lake table development and ad hoc query functions , allowing data developers to process real-time and batch data simultaneously on a single platform without the need for additional resource investment and complex data synchronization processes. Such an integrated solution not only reduces the usage of computing and storage resources, but also ensures the consistency and accuracy of data, thereby improving business personnel's recognition of data analysis results. This innovation will provide strong support for the digital transformation and intelligent upgrade of enterprises.

5. EasyMR’s four major functions are deeply optimized to unlock a new big data processing and computing experience.

As an important product module in the data stack, EasyMR represents our in-depth understanding and continuous innovation of the big data ecosystem. It is based on open source Hadoop and iterates synchronously with the open source community. It is independently developed by our computing engine team and has optimized and enhanced features of core components such as Spark, Flink, and Paimon. These optimizations not only improve the performance and stability of data processing, but also give back to the community and promote the joint construction of the Hadoop ecosystem.file

EasyMR's improved capabilities are reflected in many aspects: it supports hot updates of Flink tasks , ensuring business continuity and flexibility; Spark's Z-Order index optimization and materialized view support improve data processing efficiency and response speed; Flink's Session class loading isolation ensures the security and reliability of the operating environment. In addition, EasyMR's automated migration function makes the migration of large-scale data clusters easy and simple, and monitors the status during the migration process in real time to ensure data security and reliability. Through these innovations and optimizations, EasyMR provides users with an efficient, intelligent, and easy-to-maintain big data platform, helping enterprises achieve a qualitative leap in data management and analysis.

Data+AI capabilities make data development smarter

AI technology has become the core driving force for enterprise innovation and efficiency improvement. By integrating generative AI technology , DataStack V6.2 has realized six major functions: intelligent development, intelligent tuning, intelligent diagnosis, intelligent retrieval, intelligent analysis, and intelligent verification, which greatly improves the efficiency and quality of data processing.file

Intelligent tuning can automatically optimize SQL code and improve execution performance; intelligent diagnosis uses AI to analyze logs to quickly locate problems and provide professional optimization suggestions; intelligent analysis helps to deeply understand data trends and provide strong support for decision-making. These functions not only improve development efficiency, but also ensure code quality and achieve business goals more accurately in a data-driven manner. The introduction of AI+ marks that we are entering a new era of more intelligent and efficient data management.

AI + intelligent tuning function can provide intelligent optimization suggestions when developers write code in the editor, allowing data development students to review and compare. This improves coding efficiency and code quality, allowing data development students to focus more on the implementation of business logic.file

AI + intelligent diagnosis function uses AI technology to intelligently analyze Spark SQL, Flink SQL and other task logs, identify error messages, and provide professional SQL optimization suggestions to help quickly locate the root cause of the problem and improve code development efficiency. fileThrough the integration with AI+, the data stack not only simplifies the data development process, but also improves the accuracy and reliability of data processing, providing solid technical support for enterprises' data-driven decision-making.

Products + services, a new upgrade of data stack product commercialization strategy

At this product launch, we have redefined the commercialization strategy of our products, aiming to provide flexible and diverse service solutions for enterprises with different needs. fileThe product series includes standard edition, professional edition, and ultimate edition, and provides application cloud deployment options to meet the data processing needs of enterprises of different sizes. In addition, we also provide value-added services such as Xinchuang adaptation and real-time lake warehouse, as well as advanced and top versions of systematic operation and maintenance services, ensuring that customers can enjoy all-round support from basic to advanced.

Datastack's product commercialization strategy not only focuses on the sale of products, but also focuses on the continuous optimization and upgrading of services. By providing two paths of product upgrade and version upgrade, it helps enterprises ensure the continuous adaptability and forward-looking nature of the data platform. Such a strategy not only enhances the customer experience, but also lays a solid foundation for the long-term development of data stack products.

Three major product practice cases to help enterprises digital transformation

1. A bank: implementation of AI-based performance appraisal

银行基于沉淀的绩效考核指标,结合企业自有的知识库,利用AI智能分析和数据处理能力,显著提升了绩效考核的管理效率和治理水平。 file

Our solution helped the bank realize the transformation from indicator reports to indicator dashboards, and then to indicator conversational BI, which greatly reduced the cost for employees to obtain and use data, made the assessment standards more scientific and rigorous, and the assessment content more complete, ensuring that the bank There is a close connection between overall performance and individual employee performance. Through AI intelligent attribution and intelligent suggestions, banks can track employee performance results in real time, identify problems in a timely manner and make adjustments, thereby promoting the consistency of employees and organizational goals and the continuous improvement of performance. This transformation not only optimizes the bank's human resource management, but also brings higher operational efficiency and business results to the entire organization.

2. A Chinese liquor brand: lightweight data center

Through Data Stack, the brand has established a unified marketing platform, helping companies realize multi-dimensional analysis capabilities such as data sharing , smart tags, and indicator management, and providing strong data support for companies' precision marketing, process optimization, etc. fileThe platform adopts a lightweight data middle-end solution and combines it with StarRocks' high-performance computing capabilities to enable liquor companies to achieve efficient data management and instant analysis. StarRocks' low-latency query and fast data loading features enable companies to quickly respond to market changes and achieve failure prediction and precision marketing. Compared with the traditional Hadoop ecosystem, such a lightweight data mid-end solution has excellent query performance, real-time data processing, high concurrency, and easy maintenance in scenarios where the data volume is small, making it an ideal solution. Ideal for rapid data analysis, it promotes the digital transformation of liquor companies.

3. Beijing Municipal State-owned Group Company: Full Link Xinchuang

为解决企业数字化转型以及信创要求等问题 , 此客户建立了“全链路信创的大数据平台”。 file 该平台深度适配信创生态,实现了从服务器、操作系统、芯片、应用元数据库、中间件和计算引擎的全流程安全防护和私有化部署。通过这样的全链路信创适配,集团不仅解决了数据孤岛问题,还满足了国家对信创的严格要求,确保了数据的安全性和可控性。这一举措显著提升了集团的数据治理能力,为企业的长远发展奠定了坚实的数据基础,同时也为其他国有企业提供了宝贵的信创实践经验。

The above is the introduction to the release of DataStack V6.2. It is not only a product, but also a summary of our deep understanding and practice of big data governance and intelligent analysis. We believe that Data Stack V6.2 can help more companies maximize the value of data and promote their digital transformation.

"Industry Indicator System White Paper" download address: https://www.dtstack.com/resources/1057?src=szsm

"Dutstack Product White Paper" download address: https://www.dtstack.com/resources/1004?src=szsm

"Data Governance Industry Practice White Paper" download address: https://www.dtstack.com/resources/1001?src=szsm

For those who want to know or consult more about big data products, industry solutions, and customer cases, visit the Kangaroo Cloud official website: https://www.dtstack.com/?src=szkyzg

Linus took it upon himself to prevent kernel developers from replacing tabs with spaces. His father is one of the few leaders who can write code, his second son is the director of the open source technology department, and his youngest son is an open source core contributor. Robin Li: Natural language will become a new universal programming language. The open source model will fall further and further behind Huawei: It will take 1 year to fully migrate 5,000 commonly used mobile applications to Hongmeng. Java is the language most prone to third-party vulnerabilities. Rich text editor Quill 2.0 has been released with features, reliability and developers. The experience has been greatly improved. Ma Huateng and Zhou Hongyi shook hands to "eliminate grudges." Meta Llama 3 is officially released. Although the open source of Laoxiangji is not the code, the reasons behind it are very heart-warming. Google announced a large-scale restructuring
{{o.name}}
{{m.name}}

Guess you like

Origin my.oschina.net/u/3869098/blog/11054027