The source of this information is open to the Internet, for personal study only, please do not commercialize it, if there is any infringement, please contact to delete
1. Data center construction plan
1.1. Overall construction plan
Based on the understanding of the customer's big data application platform service requirements, according to the consideration of construction goals and design principles, it is recommended to adopt the architecture plan of Transwarp Data Hub (TDH) big data basic platform, It is deployed and constructed based on the Transwarp Operating System (TOS for short) cloud platform.
By establishing a big data integration platform, a big data computing platform, a big data development platform and a big data operation and maintenance platform to meet the customer's big data application platform service construction requirements.
1.1.1. Big Data Platform-TDH
Transwarp Data Hub (TDH for short) is based on the open source Spark technology, and has optimized performance in terms of query engine, computing framework, storage engine, and resource scheduling. , providing better performance. The improved distributed computing engine Inceptor solves the stability problem of open source Spark and has been tested in many successful cases. At the same time, Ipcetor greatly improves the computing performance of Spark, which is 2-10 times that of open source. TDH Inceptor has greatly improved the stability of Spark's functions and performance. It can run in the production environment of enterprises 7*24 hours, and can efficiently perform various stable statistical analyzes on TB-level data.
Transwarp big data platform TDH adopts a Hadoop-based data platform architecture. Massive data query and analysis service clusters can process not only structured data, but also unstructured and semi-structured data to meet the needs of configuration, logs, web pages, audio and video. , social network and other multi-source heterogeneous data loading and storage, providing data query, full-text retrieval, data offline batch analysis, interactive analysis, graph analysis, data mining, machine learning and other data processing modes. At the same time, based on the real-time stream processing cluster provided by the platform, it can meet the needs of real-time data research and analysis services. The entire platform provides complete multi-tenant functions, unified control and management of computing resources, storage resources, and data access resources, efficient scheduling management and usage control of computing resources; quota management of storage resources; strict data access permissions rights management. In terms of installation, configuration, monitoring, and alarming, the unified Transwarp Manager is used for operation and maintenance management.
Transwarp big data platform TDH application scope covers enterprises of various scales and different data volumes. Through in-memory computing, efficient indexing, execution optimization and highly fault-tolerant technologies, a platform can handle 10GB to 100PB of data, and in every order of magnitude In terms of performance, it can provide faster performance than existing technologies; enterprise customers no longer need a hybrid architecture, and TDH can dynamically expand capacity without stopping to accompany the data growth of enterprise customers, avoiding the thorny problem of MPP or hybrid architecture data migration.
1.1.2. Cloud Operating System-TOS
Transwarp Operating System (TOS for short) is a cloud operating system tailored for big data applications, developed based on Docker and Kubernetes. TOS supports one-click deployment of TDH, priority-based preemptive resource scheduling and fine-grained resource allocation, allowing big data applications to easily embrace cloud services.
In the future, enterprises will have a strong demand for building a unified enterprise big data platform (Data Hub) to drive various businesses. A unified enterprise big data platform needs to provide the following functions:
l Resource elastic sharing - improve resource utilization
Flexible deployment: supports flexible deployment of big data applications and other conventional applications
Resource scheduling: with automatic expansion and automatic repair functions
Service discovery: with a centralized warehouse
l Isolation — guarantee service quality and security
Data isolation: including data source, access mode, etc.
Computing isolation: isolate CPU, memory, network, disk IO, etc.
TOS meets the needs of the above enterprise big data platforms, supports one-click deployment, expansion, and shrinkage of TDH, and also allows other services and big data services to share the cluster, thereby improving resource utilization. The innovative preemptive resource scheduling model of TOS can guarantee real-time business while improving resource occupation when the cluster is idle, allowing batch jobs and real-time business to share computing resources in time without interfering with each other.
In addition, TDH running on TOS also introduces a micro-service architecture, which significantly reduces the impact of the user deployment environment on the stability of TDH, improves the availability of deployment, and allows users to Under the premise of , enjoy the upgrade and repair of future update versions.
TOS consists of the following parts:
container layer
The container layer mainly includes a series of containers—docker container. Docker is an open source engine that makes it easy to create a lightweight, portable, self-sufficient container for any application. Developers can compile and test containers that pass the test on the test machine and deploy them in batches in the production environment.
Docker containers are created through Docker images, and the relationship between containers and images is similar to objects and classes in object-oriented programming. Each container is allocated certain computing and storage resources and can run its own application. By loading application services in Docker containers, the decoupling of the application environment and the underlying environment can be achieved, and the containers are isolated from each other without affecting each other.
Scheduling module
Developed based on Kubernetes, it includes various scheduling modules and corresponding scheduling strategies. Supports CPU and memory scheduling, and embeds conventional scheduling strategies such as FIFO scheduling and fair scheduling. In addition, based on the fast startup of Docker containers, TOS also includes a preemptive priority scheduling strategy. By designing different priorities, containers with higher priorities can preempt the resources of containers with lower priorities to start.
System service layer
The system service layer includes the system services required for the normal operation of the container, such as service etcd, name service, etc. This layer is used to support the containerization of the platform.
Centralized service warehouse
Similar to Maven's centralized warehouse, TOS also comes with a centralized application service warehouse. When a user needs to start an application that already exists in the warehouse, he only needs to pull down the Docker image of the application and start it.
Commonly used application services are preset in the warehouse, such as complete Dockerized components of Transwarp Data Hub and commonly used services such as Tomcat and MySQL. If users want to use other application services, they can also use the interface provided by TOS to make the application service as a DockerImage in the test environment, upload it to TOSmarket, and then download and use it in the production environment.
The server virtualization resource pool based on the distributed container cluster management system (TOS) can provide multi-tenant isolated container resource allocation management, application packaging deployment and SLA management, job scheduling management, and unified operation and maintenance monitoring management.
The system has the following advantages:
l Convenient deployment
Based on TOS, users can instantly install and deploy TDH clusters with one click through the Web UI, REST API or command line, and can automatically install other required service components according to service dependencies. Before virtualization technology, it took several days to deploy hardware resources to meet new application requirements. Virtualization technology has reduced this time to minutes. However, the current Docker-based TOS cloud platform has reduced the time to seconds. As a container for loading processes, Docker does not need to restart the operating system, can be shut down within a few seconds, and can be created or destroyed in the data center without additional consumption. A typical data center utilization rate is 30%. Through more active resource allocation, a more reasonable resource allocation can be achieved for new instances at low cost, thereby improving the utilization efficiency of the data center.
l Complete resource isolation
TOS realizes better isolation of CPU, memory, hard disk and network based on Docker container by optimizing the Kubernetes resource management framework. The isolation of Docker containers in TOS is currently provided by the Linux kernel with six isolations, including isolation of host names and domain names, isolation of semaphores, message queues, and shared memory, isolation of process numbers, isolation of network devices, network stacks, and ports , isolation of mount points (file systems), isolation of users and user groups. These isolations ensure that the operating environments of different containers are basically unaffected. For example, the isolation of mount points ensures that processes in one container cannot arbitrarily access files in another container.
Compared with the traditional Apache Yarn management framework and the open source Kubernetes resource management framework, the TOS platform can control disks and networks in terms of resource granularity, while the traditional resource scheduling framework can only manage CPU and memory; in terms of isolation, containers Technology has natural advantages; in terms of dependency and versatility, it does not depend on Hadoop components and technologies, which means that cloud-based development, testing, upgrading, and management scheduling of all upper-layer applications can be realized.
l Flexible resource scheduling
TOS supports automatic expansion or contraction of TDH components, and also allows other application services and big data services to share the cluster, thereby improving resource utilization. TOS innovatively supports the preemptive resource scheduling model, which can improve the resource occupation of the cluster when the cluster is idle while ensuring real-time services, allowing batch jobs and real-time services to share computing resources in time without interfering with each other. By supporting dynamic expansion and contraction of clusters, the dynamic allocation and scheduling of resources has been realized, and the hot-swappable business and services do not need to be restarted during the whole process.
l Auto Repair
The Replicator module of TOS is responsible for detecting the size of the cluster. When a service problem occurs in the cluster, another service instance can be created to realize the self-healing function of the cluster. For example, if a HyperbaseRegionServer service stops due to hardware reasons, the TOS platform can sense it in real time and start another Docker container of Region Server within the scope of managed resources to replace the stopped container, dynamically ensuring the overall stability of the service cluster sex.
l Application Isolation
TDH and application services running on TOS also introduce a micro-service architecture, which significantly reduces the impact of the user deployment environment on the stability of TDH and application services, improves the availability of deployment, and allows users to not stop business and services to a certain extent Under the premise of fast TDH and application service update version rolling upgrade.
1.1.3. Big data platform product advantages
1.1.3.1. Complete SQL support
Transwarp big data platform supports the complete SQL 99 standard and SQL 2003 core extension, and can run all 99 test items of the TPC-DS standard test set;
The only SQL on Hadoop engine that supports stored procedures; it is compatible with more than 98% of Oracle PL/SQL and more than 80% of DB2 SQL PL syntax, and supports various syntaxes such as stored procedures, functions, control flow, cursors, and exception handling.
The only big data SQL engine that supports ACID distributed transactions; positioned in the data warehouse and data mart market, it can be used to supplement or replace Oracle, DB2 and other analytical data warehouses.
TDH provides ANSI SQL2003 syntax support and PL/SQL procedural language extension, and can automatically recognize HiveQL, SQL2003 and PL/SQL syntax, and provide more powerful SQL support while maintaining compatibility with Hive. Support standard SQL form to access data. TDH's more complete SQL support enables users to easily migrate original data analysis applications to the Transwarp big data platform, while processing larger amounts of data.
In order to reduce the difficulty of developing stream applications, TDH also supports the Stream SQL standard, which includes the stream-extended SQL 99. Therefore, developers can directly use SQL on TDH instead of writing stream computing programs through various APIs, and do not need to consider any packaging or deployment work. In order to better provide full-text search services, Search in TDH also supports SQL search extension syntax (compatible with Oracle standards). Since it supports standard JDBC 4.0 and ODBC 3.5, TDH is compatible with mainstream database software and middleware.
1.1.1.1.1. Memory/SSD/disk hybrid storage acceleration
Interactive analysis means that users expect second-level analysis responses. In order to support online transaction processing, interactive analysis, near real-time mining, or direct complex and ad hoc analytical application requirements for operational data, traditional data marts, data warehouses, etc. are pre-calculated for predefined analysis service types. The original data is extracted, converted, and loaded, and finally a materialized view is generated to achieve relevant analysis; at the same time, the incremental data in the data storage is periodically checked to optimize the analysis results, so it cannot meet the real-time and ad hoc complex analysis requirements. In addition, with the change of data storage mode and the surge of data volume, the limitations of traditional online analysis become more and more obvious, such as the decline of data access performance and the complexity of connection processing.
In order to accelerate the speed of interactive analysis, Inceptor launched Holodesk, a columnar storage engine based on memory or SSD. Holodesk stores data in memory or SSD in columnar form, supplemented by a memory-based execution engine, which can completely avoid the delay caused by IO and greatly improve the data scanning speed. In addition to columnar storage to speed up statistical analysis, Holodesk supports building distributed indexes for data fields. By using intelligent index technology to build the best query plan for the query, Inceptor can reduce the SQL query delay to the second level.
Holodesk in Inceptor supports distributed hybrid columnar storage across memory/flash media, which can be used to cache data for high-speed access by Spark. Holodesk uses the high IOPS feature of SSD to optimize the storage structure. Through columnar storage, built-in index and other technologies, the analysis performance on SSD is about 10%~20% lower than that of pure memory cache, and provides interaction with similar performance. Formula SQL analysis capability. Since the price of memory is nearly 10 times that of SSD, SSD can be used instead of memory as a cache. On the one hand, it can increase the storage capacity of the distributed memory database Holodesk, and on the other hand, it can reduce costs without significant performance loss.
Inceptor can load data from Hyperbase/HBase and HDFS into Inceptor distributed memory column storage Holodesk through SQL. Transwarp's Inceptor supports interactive data analysis of massive data, capable of scanning and analyzing billions of records in seconds data capabilities. In the future, interactive analysis capabilities can be opened to different business departments, providing interactive in-memory analysis capabilities that cannot be met by existing data warehouses.
Holodesk allows users to build OLAP-Cubes for multi-field combinations, and store the cubes directly in memory or SSD without additional BI tools to build Cubes. grade response. In addition to performance advantages, Holodesk also excels in usability. Both Holodesk's metadata and storage natively support high availability, and support exception handling and disaster recovery through consistency protocols and multiple versions. Under abnormal circumstances, Holodesk can automatically restore and rebuild all table information and data without manual restoration, thereby reducing development and operation and maintenance costs and ensuring system stability.
The platform supports columnar storage optimized for high IO features based on memory or solid-state drives, avoiding the delay caused by IO to improve data scanning speed.
Holodesk is a product independently developed by Holodesk to solve the problem of OLAP high-performance analysis and query of massive data. It is a distributed hybrid columnar storage across memory/flash memory/disk and other media, and is often used to cache data for high-speed access by Inceptor. Holodesk's built-in index, combined with the Inceptor computing engine, can provide higher interactive statistical performance than the open source Spark, and can flexibly analyze hundreds of millions of records in seconds; combined with the low-cost memory/flash hybrid storage solution, it can approach the full Analytical performance for in-memory storage.
It focuses on the improvement of real-time query efficiency in interactive analysis and can ensure scalability and stability. Transwarp Holodesk manages metadata through Zookeeper, so as to avoid data loss caused by single point of failure, and the data checkpoint is in HDFS. After the service recovers from a failure, Holodesk can automatically rebuild data and indexes through the information in Zookeeper, so it has high reliability.
1.1.1.1.2. Global index
The platform can use a global index to quickly find accurate query records. The index uses bucket technology to speed up the search for the index and quickly and accurately locate the data.
Holodesk optimizes the underlying storage by creating a global index, which has a good optimization effect on the case of high filtering rate. Holodesk provides an optimized way to organize tables into buckets, which can be achieved through bucketing technology:
1) Helps in sampling
Sampling is the random sampling of a portion of the data. When there is a large amount of data, it is inconvenient to use all the data verification system functions. In this case, it is necessary to use sample data for testing. If the table is divided into buckets, the content of each bucket is the result of discretizing the data, which meets the requirements for samples, so all the data in any bucket can be directly extracted as samples when sampling.
2) Reduce the amount of operations and increase the query rate
When conditional filtering, if the filtering field is consistent with the bucketing field, you can directly know the relevant bucket number of the record according to the hash result, and only find records that meet the conditions in these buckets, instead of searching all files, which is very high query efficiency.
3) Reduce the amount of Shuffle data
The bucketing operation enables such as GROUP BY and JOIN in specific scenarios (multiple JOIN tables have the same number of buckets) to be completed in one stage, avoiding the shuffle process. For example, there are two tables that bucket the columns of the Join Key. Now do a JOIN on the two tables. Since the records with the same column values in the two tables are in the same numbered bucket of the corresponding table, because the Inceptor implements the same numbered bucket The co-location feature of the bucket on the same node, so JOIN can be implemented directly on a Stage without Shuffle.
1.1.1.1.3. Local index
The platform supports the establishment of local indexes on distributed memory columnar storage, provides OLAP capabilities, creates indexes for large tables, supports flexible analysis of multidimensional data, and does not require pre-materialized calculations.
The establishment of indexes and cubes will improve the query rate and efficiency in the case of high filtering and high aggregation rates, and make it possible to study data from multiple dimensions and levels in a more intuitive way. Holodesk's index is established by considering each unit of columnar storage as a whole. The index is created using Dictionary Encoding technology (Dictionary Encoding). Compared with the means of conditional filtering by traversing each record, using the index greatly shortens the filtering time. You can choose to create an index for one field or multiple fields according to your needs. Holodesk's index supports all data types.
1.1.1.1. TDH resource scheduling optimization
Transwarp's big data platform uses fine-grained resource scheduling optimization to make full use of material resources, so that it can run tasks at full capacity per unit of time. Faced with high-concurrency ETL or interactive analysis The scene has very good processing ability.
1.1.1.1.1. Computing resource management
Transwarp Yarn supports the ability to manage computing resources (CPU core) and memory resources at the same time. Yarn and distributed storage are in the same physical cluster to achieve the purpose of data priority localization in distributed computing , to avoid the need to obtain all data from the network during the calculation process. After applying for a resource quota, if the current user's resources are tight or limited, the idle resources of other users can also be dynamically allocated to join, and returned when other users use it. Based on the YARN resource scheduling framework, the TDH big data platform starts Spark and Map/Reduce computing clusters, and dynamically creates and destroys computing clusters on demand.
1.1.1.1.2. Long-term occupied resource management
TDH platform can also realize the sharing and isolation of computing resources in different resource pools (Pools) within an Inceptor/Spark computing cluster, and ensure that high-priority Pools get idle resources first through fair scheduling algorithms , and at the same time, each Pool has a specified amount of resources to avoid high-load batch processing business from occupying all computing resources. At the same time, this mechanism can also support service-type businesses that occupy a certain amount of resources for a long time.
1.1.1.1.3. Resource allocation and recycling
On Transwarp Yarn, it is very convenient to dynamically create and destroy Spark or Map/Reduce clusters. For MapReduce and Spark jobs and Inceptor clusters submitted by users, they need to be submitted to their authorized queues and apply for resources from Yarn. When the user has enough resources on the nodes in the cluster to meet the requirements of the task in the job and does not meet the user's resources When using online, Yarn will encapsulate this part of CPU and memory resources into containers and provide them to the task to run; if the remaining resources are not enough to meet the resources requested by the calculation, the task needs to be queued. After the job task is completed, it is dynamically destroyed to release the occupied CPU and memory resources. Therefore, for platform analysis applications, the characteristics of Transwarp Yarn can be fully utilized to realize on-demand creation and destruction of analysis clusters, thereby helping to achieve unified scheduling and planning of resources and computing capabilities.
The space is limited and cannot be fully displayed. If you like the information, you can forward + comment, and private message for more information.