Cloud native, elasticity, flow batch of one - Lambda architecture is based on Tablestore big data analytics

background

Tablestore inspiration from Google's Bigtable paper, from 2009, in Ali cloud flying team, began to germinate. After 10 years of temper, now in the group, we have accumulated a wide range of customers and the scene on the cloud. In this article we will introduce the architecture and user scenarios in the case under Tablestore big data scene. Look at self-development of distributed storage platform to understand how personal customer scenarios, bringing to simplify and upgrade large data architecture, and ultimately help the business side of fast landing their needs.

Large data system construction pain

The data generated is multimodal diverse, we will choose according to the needs of different storage products to store data, such as order data, user information and data that we will use an OLTP database, the log data will choose SLS log service, or the self ELK Program. There are some data we want to do real-time computing, and during long-term storage in the ETL, data will then double wrote Kafka, the ability of the engine to achieve the above calculation using the Kafka demand streaming docking. Some of unstructured data will use HDFS / OSS to long-term storage. The data from the memory to the compute engine to be analyzed need to go through a long link. For example, if you selected an OLTP database, you want to calculate in real time or periodic batch calculations. Avoid taking into account the impact of TP online business, we will generally deliver data in real time or on a regular basis out into Kafka / AP database and then were calculated in real time and batch calculations. Delivery process may involve many modules, such as DTS, Kafka, Spark batch computing, Spark Streaming, AP data warehouse. Build a large data set architecture (for example, a typical large data Lambda architecture in the figure below), both for the students to develop the database, or operation and maintenance of the students have brought a higher threshold. After more than a module, whether or development costs and maintenance costs will be increased accordingly. While storing each module, computing resources may not be able to share the good, such as writing the peak comes, we might not large batch computing needs, but this time batch computing cluster resources is difficult to make direct real-time writing layer shared. And vice versa, batch computing peak, Kafka's resources, ETL resources and may not be granted to the calculated share. That there is no way to solve these problems? Reduce the access threshold for large data infrastructure, cost, enhance the development efficiency, resource utilization it? Our next chapter is.

Lambda architecture typical of large data

Tablestore cloud big data solutions

Before getting on the big data cloud architecture, we look at, what specific requirements need to double XI promote large data storage architecture will propose it:

1. High throughput, high availability of large data architecture while meeting the flexibility and low cost

   1. As big data architecture, data analysis is usually massive, large-throughput data analysis determines the delay, if not timely processing of data will lead to business decision-making lag, thereby allowing large data solutions greatly reduced.
   2. 高可用同样是一个很重要的指标，往往业务上云就是希望可以告别线下自建系统的不稳定性。
   3. 做为双十一大促的主旋律，业务峰值是不可避免的，一套弹性的架构是帮助我们真正实现业务在大促下依然可以高可用，高吞吐的的利器。

2. 资源成本

   1. 抛开成本谈弹性都是没有意义的，既然选择了云上的方案，我们自然希望可以获得较好弹性的同时，控制成本。
   2. Serverless 云原生的产品形态通常可以比较好的在具备弹性的同时控制我们的成本开销。架构无需在业务低峰期预留很多不必要的资源来防御业务峰值的到来。

3. 生态完善

   1. 大数据架构中涉及到的概念和需求非常多，这些需求也会对应不同的计算引擎，例如 Spark，Flink，Presto 或者云上的自研组建。存储需要和这些组建，开源产品进行无缝打通，这样可以降低使用的门槛。
   2. 除了计算引擎，数据通道链路的完善也很重要，例如日志类的数据和 OLTP 的数据能方便汇总，数据打宽等。

Tablestore 是一款 Serverless 云原生存储引擎，Serverless 相比实例售卖类型的产品，在业务有波峰波谷时天生就有较大的优势，基于 bigTable 的主存储采用行的方式进行存储，可以支撑单表亿级别的QPS。下面列了一些 Tablestore的核心特性：

Tablestore 除了有强大的主存储满足海量业务的实时读写外，基于主存储的分布式日志提供了完整的数据派生能力（详情参考），海量实时写入 Tablestore 的数据，可以实时订阅进行消费。这样就满足了我们的实时计算需求。
Lambda 架构中除了实时数据写入，实时计算之前，全量数据需要提供高性能扫描能力，Tablestore 采用行列混合，双引擎的架构，在主存储之外内部通过通道服务实时构建一个列存储，支撑 PB 级别数据的高吞吐扫描。同时在海量的数据场景下，我们相信数据是需要分层存储，所以在构建自身列存的同时，我们会帮助用户构建推送云上数据湖的链路，通过全托管的数据湖投递，降低用户的存储成本。

基于 Tablestore 的 Lambda 架构


Tablestore 在专注于打造一款极致性能和成本的存储引擎同时，更加关注完整的计算生态，伴随产品核心功能迭代的过程中，我们和阿里云的几大核心计算引擎做了完善的对接具体包括：

• MaxCompute 的对接，支持 MaxCompute 计算引擎通过外表的方式直读写 Tablestore
• EMR Spark 对接，支持流批源表读，流批结果表写，集团内第一款全 Connector 支持的 kv 存储引擎
• Blink 对接，支持流批源表读，流批结果表写，维表读，集团内第一款全 Connector 支持的 kv 存储引擎
• DLA 对接，支持 SQL 直接读写 Tablestore 的数据
• FC 对接，支持流式增量触发器

计算的结果集合需要提供丰富灵活的高并发查询，支撑用户生成实时大屏，报表的场景，Tablestore 通过主存储结合丰富的索引能力以及 MPP 类型计算引擎来实现。

Tablestore Lambda架构结果集合存储和报表展示


对 Tablestore 大数据架构感兴趣的同学还可以参考之前的一些架构文章：

1. 结构化大数据分析平台设计
2. 数据中台之结构化大数据存储设计
3. Lambda plus: 云上大数据解决方案
4. 基于大数据的舆情分析系统架构 - 架构篇
5. 通过EMR Spark Streaming实时读取Tablestore数据

Tablestore 助力双十一集团业务的落地

介绍完 Tablestore 结构化大数据存储引擎的基础能力后，来看看具体的一些双十一集团业务架构：

实时酒店推荐

场景描述：

商家会实时的更新酒店房型库存和报价，酒店的属性等字段。实时选品过程针对同一个酒店不同供应商的库存和报价进行计算和筛选，最终展示出有库存且按照报价质量排序的列表给平台用户，进行酒店搜索的结果返回

架构优势：

• 在线和分析业务流量隔离
• 基于 Tablestore 对接流计算的便利性实现源表和维表的存储统一
• 全链路延时要求高，商品价格变动后，5秒内可以反馈至搜索库

基于 Tablestore 的酒店选品实时推荐架构

双十一监控作战大屏

场景描述：

海量的机器，业务监控秒级别数据，实时千万级别每秒写入流量，写入 Tablestore 数据实时查询和展示。跨时间的实时聚合，形成分钟级别数据。关键指标的实时聚合和展示，离线批计算提供报表分析能力。

架构优势：

• 核心单表数据规模达 10 PB，可自定义数据生命周期
• 核心单表持续每秒写入进 5000 万个数据点
• 数据实时写入，大大提升数据可见时效性
• 毫秒级实时查询展示趋势图和报表，查询性能不受单表规模约束

基于 Tablestore 时序监控大屏场景

全网爬虫系统

场景描述：

Massive reptile high concurrent high-throughput data is written, the full amount of data stored, the data capacity of PB level, many types of data storage, the raw data included semi-structured, structured tag data and data analysis, the data is written to require immediate trigger real-time data process requires the subsequent docking offline processing.

Architectural advantages:

• LSM engine distributed data storage, providing high throughput high concurrent write, PB-level data storage
• By updating data capture, real-time logic trigger custom subsequent processing of the data
• Big Data platform with real-time data synchronization, the analysis results are written to the results table, the supply of real-time query with a layer

Based Tablestore reptile big data architecture

to sum up

As the amount of data processing, real-time processing changes, upgrading the physical hardware processing power, big data architecture evolution is very rapid. With gradual service data on the cloud, big data processing platform will inevitably migrate to the cloud, compared to TP business class, business class AP peak scene processing data bandwidth could be bigger. We want reasonable big data cloud computing architecture can make use of separate storage, to provide a more flexible data processing capabilities in a controlled cost, to address the business challenges in big promotion coming encountered. Let big data platform developers completely released from the base out of operation and maintenance, can devote all our energies to meet the business requirements for data analysis platform. If interested students are welcome to communicate with us to explore (nails exchange group: 23,307,953).