Guide: The original "129 pages and 40,000 words of a smart energy group digital management platform project proposal WORD" (see the end of the article for the source), this article selects the essence and structure part, the logic is clear, the content is complete, and it is provided for the rapid formation of pre-sales solutions refer to.
directory
Digital management and control platform related project proposal
Table of contents:
1. Related project background
2. Demand understanding
2.1 Requirements Understanding
3. Scheme design
3.1 Overall scheme design
3.3.1 Overall Architecture
3.3.2 Solution Description
3.3.3 Response to demand
3.2 Data Warehouse
3.2.1 Data Warehouse Architecture
3.2.2 Data Warehouse Product Description
3.2.3 Response to demand
3.3 Data Integration and Governance
3.3.1 Solution Architecture
3.3.2 Product Description
3.3.3 Response to demand
3.4 Data display
3.4.1 Data presentation architecture
3.4.2 Product Description
3.4.3 Demand response
3.5 Mobile Application
3.5.1 Mobile Application Architecture
3.5.2 Product Description
3.5.3 Response to demand
3.6 Big Data Platform
3.6.1 Hadoop platform
3.6.2 Big data storage
3.6.3 Big Data Collection
4. Response to business needs
4.1 Collect business data types
4.2 Data Control
4.3 Business transparency
4.4 Business Theme Scenario Analysis
4.5 Business sharing platform
5. Integration instructions
6. Implementation and delivery of relevant projects
6.1 General idea
6.1.1 Data basis
6.1.2 KPI and report analysis system
6.1.3system planning and structure
6.1.4 Big data system management and control mechanism
6.2 Scope and implementation methods of relevant projects in this period
6.2.1 Scope of relevant projects in this period
6.2.2 Relevant project implementation methodology
6.2.3 Related project management and control
6.3 Deliverables at each stage of the system
6.4 Related Project Issues and Risks
6.5 Problem Management and Service Guarantee
6.6 Relevant project assumptions
7. Related project cost and value
8. Case
Overall scheme design
3.3.1 Overall Architecture
The overall architecture design of XX big data proposed by SAP adopts layered implementation, vertical data governance and control structure. As shown in Figure 3.3 the overall architecture of XX big data platform, the overall architecture is divided from bottom to top from the data source: data source layer; data integration and storage layer; data modeling layer; data analysis service layer; data application innovation service layer; Service portal layer; data governance and configuration control are vertically connected.
Figure 3.3 XX big data platform overall architecture
3.3.2 Solution description
The data source layer covers all existing business types and external data of XX.
Automation and informatization in the field of production and operation include: the SCADA centralized monitoring platform independently developed by XX, and the IBM wind farm management and control EAM system. For this part of data, the big data platform can be used as event stream processing, or as traditional static data batch extraction, transformation and loading (ETL).
The information systems in the management and control field include: SAP ERPsystem, Dassault PLMsystem; the big data platform can synchronize SAP ERP data in real time through SAP SLT, and integrate PLM data through Data Service ETL.
External public data include: weather, climate, hydrology, geographic information, environmental data, etc.; the big data platform processes this part mainly through the centralized storage of the Hadoop platform, and unified modeling and access through the HANA memory database.
The data integration and storage layer is the core of the entire big data platform. With the SAP HANA in-memory computing platform as the core, it adopts the distributed system architecture of ShareNothing; it provides Hadoop data storage services based on the distributed file system; the system architecture is inherently capable of large-scale parallel processing and horizontal and vertical expansion capabilities. The big data core platform has the capability of one-time construction and step-by-step expansion; while the platform scale is expanding, there is no need to add additional technologies or products technically. In terms of mass data storage, a layered data temperature zone design is adopted. Commonly used structured data is stored in the memory as hot data, and the usual data size is TB level; historical structured data is stored as warm data in the disk columnar data engine. The scale is 10TB-100TB; unstructured semi-structured data or data that cannot be directly analyzed is used as cold data, and the scale can reach PB level when stored in Hadoop. Unified data access is implemented by the HANA memory platform.
The data modeling layer is a virtual layer based on the data storage layer, which is implemented by using the HANA in-memory view modeling function of the SAP big data platform; in this solution, it is divided into two data model areas: data integration services for wind turbines and wind farms, and operation management and control data mart.
Wind farm data integration services include: wind resource data model, product data model, wind farm data model, unit operation data model, etc.
The business management and control data mart is oriented to business management and control, covering: business profit analysis, budget analysis, procurement plan, master production plan, master demand plan, market analysis, operating cost, settlement and payment collection, inventory turnover Category, on-site planning category, logistics distribution planning category, sales price category and other enterprise management and control indicators and reports. The business management control cockpit and the digital board board are typical applications of this model set.
Data analysis service layer: The analysis service is to use big data technology to mine the value from the data and realize the rapid realization of the data. This part of the work can be carried out jointly with partners, SAP data science services, and customer expert teams in the subsequent stage after the big data platform is built.
Data application innovation services: covering delivery and implementation services, intelligent operation and maintenance services, remote early warning services, and SaaS applications for owners.
Service portal: Provide integration and collaboration portals, covering manufacturer application portals, wind farm application portals, and customer application portals.
Data governance and configuration control: The big data platform provides metadata control, data-related quality control, data model control, data configuration control, data standard control, and data security control vertically to each layer.
3.3.3 Demand Response
1. Provide a distributed , and improve the parallel of the overall system by easily expanding server hardware resources .
Answer: The SAP big data platform is naturally a distributed system architecture; server hardware resources can be scaled up or scaled out to provide parallel computing capabilities of the overall system.
2. Provide a distributed file system , which can increase the storage capacity and parallel I /O performance of the entire system by easily expanding the server storage space , and can efficiently process massive data (PB level).
Answer: The SAP HANA big data platform can integrate the Hadoop distributed file system through Vora; there is no IO performance bottleneck, and it can handle massive amounts of data.
3. Provide distributed memory computing capabilities , increase the memory capacity of the entire system by simply expanding the server memory, and improve the rapid response capability of the entire system.
Answer: SAP HANA only needs to increase the memory to increase the computing power of the system and linearly improve the rapid response capability of the system.
Summary of core advantages of SAP big data platform:
1. Real-time analysis, real-time prediction : Real-time synchronization of business data; all hot data is stored in memory and calculated in memory, realizing real-time reporting and real-time analysis;
2. Unique data, reducing data redundancy: In the hierarchical architecture of SAP HANA, the data of each layer does not land, reducing data redundancy, reducing the difficulty of data maintenance, and technically avoiding the risk of data inconsistency as much as possible;
3. Open platform and flexible logic: SAP HANA supports multiple data source access, integrates multiple data collection tools, and provides different tools and means for different data source types; the model is divided into layers, and the logic is clear , the structure stability is very good;
4. Cross-platform access, full data coverage: Realize cross-platform data access in a unified interface, seamless integration of hot data and historical data, greatly reducing the threshold for users;
5. Data authority, high information security: core data and applications are kept in sync, traceable and auditable;
6. Multiple styles of display forms and access methods: Various forms of reports, interactive analysis, ad hoc query, data exploration, dashboards and mobile applications can be realized to meet the needs of different personnel for convenient, intuitive and visual analysis need;
3.2 Data Warehouse
3.2.1 Data Warehouse Architecture
This solution provides a scalable data warehouse architecture centered on SAP HANA, which is a future-oriented data management and control platform, a data warehouse architecture that seamlessly supports analysis applications, and an integrated architecture that supports big data and data lakes. As shown in Figure 3.4 SAP HANA data warehouse platform.
As shown in Figure 3.4 SAP HANA data warehouse platform
A data warehouse based on SQL and BW style that meets the standards to realize...
• Meet future needs: logical data warehouse; support dynamic system layout; support cloud deployment and hybrid deployment; integrate all data types and big data technologies; horizontal expansion, support massive data and data lakes.
• Beyond other data warehouse solutions: the best out-of-the-box integration with SAP solutions - local and cloud environments; HANA real-time data processing capabilities ; integration of Hadoop through SAP HANA Vora ; HANA-based analysis business Services ; reusable business-related content optimized for HANA.
• Seamless integration of XX New Energy's existing SAP ERP system data and other business system data; and the ability to provide stream data integration for IoT data such as wind farm extensions. Store and process external environment data. Complete modeling service, IOT integration and big data predictive analysis.
3.2.2 Data Warehouse Product Description
The main products in the data warehouse structure include: SAP HANA in-memory data warehouse and big data platform; SAP EIM data integration and master data control (details related content is introduced in 3.3 data integration and governance); SAP BO data mining display and visualization, SAP PA predictive analysis (3.4 data display part description); big data and data lake (3.6.2 big data storage part description); IoT streaming data integration (detailed relevant content is described in 3.6.3 big data collection part);
SAP High-Performance Analytic Appliance (HANA for short) is a set of flexible, multi-purpose, and data source-independent brand-new applications based on memory computing. By integrating hardware (by SAP's hardware partner: Huawei , HP, IBM, Fujitsu, Cisco, Dell, etc.) and an optimized set of applications based on memory computing technology.
The core of the SAP HANA in-memory computing platform is the next-generation database and data warehouse technology, and it is also a scalable big data platform. Its architecture is shown in Figure 3.5 SAP HANA memory computing platform:
3.5 SAP HANA内存计算平台
SAP HANA is based on full memory computing, adopts ShareNothing architecture, supports multi-service node parallel computing, and its certified HANA all-in-one hardware can support up to 94 nodes. Support comprehensive data integration, all data sources currently considered by Yangming New Energy can be supported; provide unified massive data storage and data modeling; support access to any device, including mobile. The features of SAP HANA are:
§ High-performance memory computing;
§ Software and hardware all-in-one machine solution;
§ MVCC large-scale concurrent access technology;
§ Row-column combination computing technology, column data compression technology;
§ Data parallel partition technology;
§ Data persistence and insert optimization technology;
§ View-based modeling;
§ Predictive analysis library;
§ Data display integration;
§ Text analysis technology;
§ Geographic information support;
§ Data layering and big data support;
§ Data integration platform components, IoT Foundation, SDI, SLT.
The following related content is an introduction to the above features.
High Performance In-Memory Computing:
Traditional database technology is based on disk computing, and performance optimization is performed through index and cache technologies. However, modern computer technology is characterized by large-scale parallel CPU, large-scale commercial memory, and disk is the bottleneck of high-performance computing. SAP HANA uses modern hardware technology architecture to develop a full-memory column computing database with complete independent innovation. As shown in Figure 3.6:
Figure 3.6 SAP HANA based on modern computer hardware architecture
This is the core database and data warehouse product designed for the next 15 years. The read and write performance of memory is 1 million times that of disk. The query and analysis capability of SAP HANA is 1000 times higher than that of traditional databases. The billions of records and full table analysis and query responses in seconds make the transactional operation OLTP and analytical operation OLAP can be integrated. It is an indispensable core platform in the process of digital transformation of enterprises to truly realize the requirements of enterprises for real-time business and analysis. Figure 3.7 shows the HANA internal architecture.
图3.7 HANA内部架构
Essentially, SAP HANA is still based on a relational database system (RDBMS). All database product knowledge such as ANSI92-SQL, Table, View, Procedure, Index (basically unnecessary), Lock, User, Role, Session, Schema and other objects are consistent with the original database concept. And the disk is only used for data persistence, so that data will never be lost after restart or power failure. There is no technical threshold for XX New Energy to adopt HANA.
Software and hardware all-in-one machine solution:
The hardware running SAP HANA is a server certified by SAP HANA. Almost all mainstream server manufacturers such as: Huawei, Lenovo, Dell, Cisco, HP, Hitachi, Fujitsu, etc. have servers certified by SAP HANA, which are usually called SAP HANA all-in-one machine. As shown in Figure 3.8 HANA hardware architecture. You can find all certified HANA appliances from the following URL, (https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/appliances.html). At the same time, SAP HANA also supports certified storage architecture servers and Cloud IaaS environments. SAP HANA server supports X86 architecture Intel Zhiqiang processor, SuSE Linux, RedHat Linux operating system, IBM Power7 for linux processor has also been certified; supports local deployment, VMWare virtualization and Cloud deployment.
图 3.8 HANA 硬件架构
MVCC large-scale concurrent access technology:
SAP HANA adopts MVCC (Multiple Version Consistency Contral) for concurrent access and data consistency guarantee. Reads and writes do not block reading, use row-level locks to control transactions, and improve the access capabilities of large-scale concurrent users; SAP HANA uses large-scale parallel CPUs, and each core can independently process a single column, making large-scale parallel computing capabilities extremely large The upgrade fully complies with the multi-channel CPU and large-scale memory computing technology of modern hardware.
Row-column combination computing technology, column data compression technology:
SAP HANA adopts row and column methods to organize the data structure. The row table is mainly used for OLTP operations with frequent update and delete, and the list is mainly used for OLAP in analytical scenarios. And SQL supports row and list associated queries. The XX big data platform basically uses columnar mode calculations. Column computing uses data compression technology that can be directly accessed to obtain a huge data compression rate. As shown in Figure 3.9 Data Compression, the SAP HANA dictionary compression principle.
图3.9 数据字典压缩
As shown in Figure 3.10, SAP HANA ERPsystem running on HANA can achieve a data compression rate of more than 5 times compared with traditional databases, and the query response time can be increased by dozens to hundreds of times. Reduce TCO.
Figure 3.10 Data compression ratio and query response time
Data parallel partitioning technology:
SAP HANA realizes parallel processing, realizes the linear expansion of the big data platform through the cluster distributed architecture, and flexibly responds to changes in enterprise business; as shown in Figure 3.11: Distribute the large amount of data and calculation to multiple processors and memory for parallel processing deal with. When performance expansion is required, the linear growth of cluster performance can be easily achieved by adding processing nodes. With business changes, enterprises can flexibly adjust the performance of the core of the big data platform to support massive data.
Figure 3.11 HANA parallel partition
Data persistence and insert optimization technology;
SAP HANA is a pure memory database, all data is stored in memory, and there is a mirror image of the disk in the disk for data persistence. To make up for disk write performance bottlenecks in the future, HANA has designed a set of insert optimization mechanisms, using SSD solid state disks as log volumes. All data is first written to log volumes and merged into data volumes asynchronously. As shown in Figure 3.12 HANA read and write mechanism:
图3.12 HANA读写机制
1) Data is always written directly into memory;
2) Write the log while updating the memory data;
3) Data is periodically written to disk and a savepoint is created;
4) Records are read from the disk and log area during data recovery.
View-based modeling;
SAP HANA models based on virtual views directly in memory. It is an advanced and mature data warehouse modeling platform; the HANA model can be directly invoked by BO data display. As shown in Question 3.13, HANA Studio can be used as a modeling tool;
• SAP HANA Studio provides direct in-memory virtual modeling tools;
• The model realized by dimension view and calculation view has no data redundancy;
• HANA Studio also integrates BW modeling tools;
• HANA modeling is an advanced and mature modeling tool.
图3.13 HANA Studio 建模
The SAP HANA view model can be understood as a CUBE, as shown in Figure 3.14. Because it uses memory view modeling, it has a very large improvement in performance and analysis dimensions compared with traditional CUBE. It belongs to the progress of technological qualitative change.
图3.14 HANA 视图建模
Additionally, SAP HANA Data Warehouse provides BW/4HANA modeling. As shown in Figure 3.15 BW/4 HANA modeling:
图3.15 BW/4 HANA 建模
• Quickly and flexibly generate reports based on all layers of the data warehouse
• Virtually combine data across different data layers
• Business and service level drivers
• A combination of bottom-up and top-up approaches - supporting agile and flexible development
Predictive analytics library;
SAP HANA comes with a function library: it includes the predictive analysis function library (PAL), which provides strong support for big data prediction and analysis, as shown in Figure 3.16 HANA PAL library:
Make full use of the predictive analysis capabilities of the SAP HANA computing engine:
• Advanced visual design using native modeling tools
• It can import PMML-compliant models and support interactive embedded calls directly with SAP BI client tools
• Embed third-party applications to provide real-time predictive analysis capabilities for PDMS and other applications
如图3.16 HANA PAL库所示
At the same time, SAP HANA also provides R language statistical function library: SAP HANA + R language, which utilizes abundant big data analysis resources on the Internet to provide more ideas for enterprise big data analysis applications. Integration of R and HANA
• Can use R's open environment, providing up to 5000 function libraries for in-memory computing
• R's functions are processed in parallel through high-performance in-memory computing
• R scripts can be embedded in SQL statements to complete the data model of HANA
Data display integration;
The SAP HANA data warehouse platform seamlessly supports the so-called data display platform of SAP BO. Specific instructions are introduced in the data display section.
text analysis techniques;
SAP HANA supports unstructured text search and analysis, helping enterprises to better mine information gold mines. Inherent text search and analysis capabilities built on a unified, flexible, and robust data platform for both structured and unstructured relevant content.
Customer complaints, call centers, maintenance, machines, accidents, etc. can be analyzed through the text search and analysis capabilities of HANA.
characteristic:
• Inherent full-text search;
• Integrated text analysis capabilities;
• Extraction of entities and semantics;
• Graphical modeling and search model;
• Dedicated graphical toolbox for building search applications.
benefit:
• Sort out unstructured related content in SAP HANA;
• Integrate business analysis search and text search work in OLAP and OLTP use cases in a unified system;
• Reduce repetitive, latency and operational overhead;
• Easy to model - already built into SAP HANA's modeling tools;
• Reusable graphical building blocks enable rapid development of search-based applications.
geographic information support;
SAP HANA spatial data processing provides new innovative capabilities for big data analysis and prediction; supports real-time spatial data processing, spatial data analysis optimization, spatial data types and functions, and provides geographic information and services. As shown in Figure 3.17 HANA spatial data support:
Figure 3.17 HANA spatial data support
Data layering and big data support;
The SAP HANA big data platform supports massive storage and processing, and adopts a data temperature zone layered design. Structured analysis hot data is resident in memory, and structured historical data is defined as warm data stored in a disk columnar database, that is, HANA Dynamic Tiering Technology (HANA DT); unstructured data and cold data are stored in Hadoop, passed through HANA Vora is accessed based on the spark framework or Smart Data Access. A detailed introduction is in the big data storage section.
Data integration platform components, IoT Foundation, SDI, SLT
The SAP HANA data warehouse platform integrates data integration tools; including ETL batch extraction, transformation and loading, SLT real-time loading from SAP system, real-time capture of transaction SQL from data source CDC and transmission to the data warehouse, and wind farm sensor data IoT event data streaming integrated. ETL and SLT integration are explained in 3.3 Data Integration and Governance; IoT is explained in 3.6.3 Big Data Integration.
How to get the full version
If you like the article, you can follow + comment + forward this article. For more information, please private message: plan