With the development of Xianyu's business, the scale of users has reached hundreds of millions, and the number of user-dimensional data indicators has reached hundreds.
How to quickly screen out the user population that meets expectations from the billion-level data and conduct refined population operations is a problem that technology needs to solve. Many solutions in the industry often require minutes or even hours to generate query results.
This article provides an efficient data screening, statistics and analysis method for solving big data scenarios. From billions of data, arbitrarily combine query conditions, filter the required data, and return in milliseconds.
Technology selection analysis
From a technical point of view, our business scenario has the following characteristics:
Need to support any combination of dimensions (and/or) nested queries, and require low latency.
The data scale is large, at least 100 million level, and needs to support continuous expansion.
There are many dimensions of a single data index, at least hundreds, and it needs support to increase continuously.
Comprehensive analysis, this is a typical OLAP scenario.
OLTP 与 OLAP
Here is a brief comparison between OLTP and OLAP:
The most common databases, such as MySQL, Oracle, etc., all use row storage, which is more suitable for OLTP.
If you use a row database such as MySQL to implement OLAP, you will generally encounter two bottlenecks:
Data volume bottleneck: MySQL is more suitable for data volume in the millions. If there is more, query and write performance will be significantly reduced. Therefore, the method of sub-database and sub-table is generally used to control the data scale at the million level.
Query efficiency bottleneck: MySQL needs to build an index or a combined index for common conditional queries. The query of non-indexed fields needs to scan the entire table, and the performance drops significantly.
The foregoing analysis, our scenario, the line is not suitable for data storage library, so we keep a database column key consideration.
Row storage and column storage
Let's briefly compare the characteristics of row storage and column storage:
Row storage is suitable for near-line data analysis, such as a scenario that requires querying all fields of a few qualified records in the table.
Column storage is suitable for statistical analysis of data. Consider the following scenario: There are 20 fields in a table for storing users, and we want to count the average age of users. If it is row storage, we need to scan the entire table and traverse all rows.
But if it is column storage, the database only needs to locate the age column, and then only scan the data in this column to get all the ages. Calculating the average value, the performance will be 20 times faster than the row storage theory.
In the column storage database, HBase is more common. The core design focus of HBase application is the design of rowkey. Generally, commonly used filter conditions are combined and designed into rowkey, and query through rowkey's get (single record) or scan (range).
Therefore, HBase is more suitable for unstructured data storage under limited query conditions. In our scenario, since all fields need to be used as filter conditions, structured storage is essentially required, and low-latency queries are required, so HBase cannot be used.
We comprehensively consider a variety of column-based storage DB products (ADS/PostgreSQL/HBase/HybridDB) within the group.
Comprehensive evaluation of read and write performance, stability, completeness of grammar, and development and deployment costs, and finally selected HybridDB for MySQL computing specifications to build a crowd selection engine.
Introduction to HybridDB for MySQL
HybridDB for MySQL computing specifications For our scenario, the core capabilities mainly include:
Any dimension intelligent combination index (users do not need to build their own indexes).
Millisecond response to queries of tens of billions of tables.
MySQL BI is ecologically compatible with complete SQL support.
Support for spatial retrieval, full-text retrieval, and complex data types (multi-value columns, JSON).
So, how does the HybridDB for MySQL calculation specification achieve the millisecond-level response to any combination of dimensions in the big data scenario?
Proceed as follows:
The first is HybridDB's high-performance columnar storage engine, built-in storage predicate computing capabilities, you can use various statistical information to quickly skip data blocks to achieve rapid screening.
The second is HybridDB's intelligent indexing technology, which automatically performs full indexing on a wide table with one key and intelligently combines various predicate conditions for filtering based on column indexes.
The third is the high-performance MPP+DAG fusion computing engine, which takes into account the two modes of high concurrency and high throughput to realize pipeline-based high-performance vector computing, and the computing engine and storage work closely together to make computing faster.
The fourth is that HybridDB supports various data modeling technologies such as star model, snowflake model, clustering and sorting, etc. Moderate business data modeling can achieve better performance indicators.
In summary, the HybridDB for MySQL computing specification is a SQL-centric multi-functional online real-time warehouse system, which is very suitable for our business scenarios, so we built our crowd selection bottom engine on this.
Business realization
After building the crowd circle selection engine, we focused on transforming our message push system as an important place for crowd refined operations.
Introduction to Xianyu Message Push
Push message (PUSH) is the fastest way for information to reach users. The more commonly used PUSH method of Xianyu is to first calculate the PUSH crowd offline, prepare the corresponding PUSH copy, and then push it at the specified time the next day.
Generally, it is a periodic PUSH task. However, temporary PUSH tasks that need to be sent immediately and urgently require BI students to intervene. Each PUSH task takes about half a day of development time of BI students on average, and the operation is also troublesome.
This time, we have integrated the crowd circle selection system with the original PUSH system, which greatly improved the efficiency of data preparation and transmission of such PUSH, and liberated development resources.
system structure
Offline data layer: User dimension data is scattered in offline tables of various business systems. We use offline T+1 timing tasks to summarize the data and import it into the user wide table of the real-time computing layer.
Real-time computing layer: According to the screening conditions of the crowd, query the number of users and the list of user IDs from the user wide table to provide services for the application system.
Crowd circle selection front desk system: Provides a visual operation interface. Operating students select the filter criteria, save it as a crowd, and use it for analysis or sending PUSH.
Each group corresponds to a SQL storage, similar to:
select count(*) from user_big_table where column1> 1 and column2 in ('a','b') and ( column31=1 or column32=2)
At the same time, SQL can support multiple and/or nested combinations of any field. Using SQL to save the crowd, when the data in the user table changes, you can execute SQL at any time to get the latest crowd users to update the crowd.
闲鱼 PUSH 系统：从人群圈选前台系统中获取人群对应的 where 条件，再从实时计算层，分页获取用户列表，给用户发送 PUSH。在实现过程中，我们重点解决了分页查询的性能问题。
分页查询性能优化方案：在分页时，当人群的规模很大(千万级别)时，页码越往后，查询的性能会有明显下降。
因此，我们采用把人群数据增加行号、导出到 MySQL 的方式，来提升性能。
表结构如下：
批次号：人群每导出一次，就新加一个批次号，批次号为时间戳，递增。
行号：从 1 开始递增，每一个批次号对应的行号都是从 1 到 N。
我们为"人群 ID"+"批次号"+"行号"建组合索引，分页查询时，用索引查询的方式替换分页的方式，从而保证大页码时的查询效率。
另外，为此额外付出的导出数据的开销，得益于 HybridDB 强大的数据导出能力，数据量在万级别至百万级别，耗时在秒级至几十秒级别。综合权衡之后，采用了本方案。
PUSH 系统改造收益
人群圈选系统为闲鱼精细化用户运营提供了强有力的底层能力支撑。同时，圈选人群，也可以应用到其他的业务场景，比如首页焦点图定投等需要分层用户运营的场景，为闲鱼业务提供了很大的优化空间。
本文实现了海量多维度数据中组合查询的秒级返回结果，是一种 OLAP 场景下的通用技术实现方案。
同时介绍了用该技术方案改造原有业务系统的一个应用案例，取得了很好的业务结果，可供类似需求或场景的参考。
未来
人群圈选引擎中的用户数据，我们目前是 T+1 导入的。这是考虑到人群相关的指标，变化频率不是很快，且很多指标(比如用户标签)都是离线 T+1 计算的，因此 T+1 的数据更新频度是可以接受的。
Later, we built a more powerful product circle selection engine based on HybridDB. Xianyu product data changes faster than user data.
On the one hand, users will update their products at any time; on the other hand, due to the characteristics of Xianyu's product list inventory (when it is sold out) and other reasons, the status of the products will change at any time.
Therefore, our product selection engine should sense the changes in these data as soon as possible and make real-time adjustments at the delivery level.
Based on HybridDB (storage) and real-time calculation engine, we have built a more powerful "Mach" real-time selection system.