Tencent Big Data has done a few things recently. It launched an official website http://data.qq.com/ and made TDW ( Tencent Big Database Warehouse) open source. It is rare for the closed Penguin to open up. There is some information on the big data website. I saw a very interesting system called Hermes . I will introduce it today.
Regarding the real-time analysis system, I wrote several articles to analyze, including " Real-time Analysis System (HIVE/HBASE/IMPALA) Analysis " and " Is MPP DB the future choice of big data real-time analysis system? " One set of data, multiple engines (impala/Hive/kylin) " " One set of data, multiple engines continued---Analysis of two data formats (Parquet/ORCfile) " If you are interested, you can take a look.
In order to solve the problem of real-time analysis, the software of the MPP architecture is mainly analyzed , and the core idea is to speed up the query through the MPP scanning technology. Tencent's Hermes system is an evolution of the open source lucene, which mainly uses search and indexing technology, so hermes is also called a real-time retrieval and analysis platform. This is another way of thinking that is worth looking at.
For a detailed introduction to Hermes , you can see the references at the end of this article. Mainly talk about the characteristics of Hermes :
1. The core is the design of storage. Through the reorganization of the data structure and the characteristics of the analysis system, nested column storage is realized, and random reading is fully avoided. The block reading + bitmap calculation is used to greatly reduce the time-consuming drawbacks, so that the The statistical analysis and calculation time of big data is shortened to the second level; dictionary sorting is used in the entry file, and prefix compression is implemented on this basis; increasing sorting is used in the sequence file, and the serial number is of variable length type, which is effective Compress storage space for easy calculation of bitmap construction;
2. Column storage .
3. Based on the analysis and processing of single instance data, datasource mainly contains two types of data: data imported by users (bitmap files) and source data (index files). The kernel mainly processes index files and bitmap files according to user request logic. The kernel is shown as follows:
4. The entire data corresponds to multiple copies, which are evenly distributed in each analysis instance according to different rules. The merge service of the data is performed in one of the shards. For each request, the light-loaded merge server will be selected according to the machine load.
What is the difference between Hermes and open source solr, elasticsearch ?
The features of solr and es are as follows:
1. From search engines, focusing on search and full-text retrieval.
2. The scale of data ranges from several million to tens of millions, and there are very few clusters with data volume exceeding 100 million. Ps: There may be more than 100 million data in individual systems, but this is not a common phenomenon (just like the data size in oracle tables may exceed that in hive , but a minicomputer is required).
The usage characteristics of hermes: are as follows:
1. A real-time retrieval and analysis platform for massive data based on search engine technology. Focus on data analysis.
2. The scale of data ranges from hundreds of millions to trillions. The smallest table is also in the tens of millions.
In Tencent , 12 machines can process 35 billion data per day ( each piece of data is about 1kb ) , each about 30T , and the data can be stored for a month.
solr\es is more focused on providing full-text retrieval services for small-scale data; hermes provides indexing support for large-scale data warehouses, provides ad hoc analysis solutions for large-scale data warehouses, and reduces the cost of data warehouses. Hermes data volume more "big".
The difference in positioning and data scale leads to the essential difference in the way of using indexes between hermes , solr and es . The following explains from the perspective of big data, why hermes is more suitable for large indexes.
The indexes of solr and es rely heavily on physical memory:
1. The first-level jump table is completely loaded in the memory. In addition to consuming a lot of memory, the loading speed of the index when it is opened for the first time will be very slow. The index in solr\es is always open, and it will not be opened frequently. Off; this mode will restrict the number of indexes and the scale of indexes on a machine. Usually, a machine is fixedly responsible for the indexes of a certain business.
2. Sorting and statistics ( sum , max , min ) is to traverse the inverted table, load all the values of a certain column into the memory, and then perform statistics based on the memory data even if only one of the records is used for a query, It will also load all the values of the entire column into the memory, which wastes resources and the performance of the first query is too poor. The data size is greatly limited by physical memory, and OOM is a common occurrence after the index scale reaches tens of millions.
3. The index is stored on the local hard disk. After an exception occurs, the copying time will take too long because the data needs to be restored.
4. Support master/slave mode, but like the traditional MySQL database, the cluster scale is not particularly large.
In addition to the limited scale of the processing cluster, the data migration for each expansion will be a very painful thing, and the data migration time is too long.
5. 倒排检索即使某个词语存在数据倾斜，因数据量比较小，也可以将全部的doclist都读取过来（比如说男、女），这个doclist会占用较大的内存进行cache，当然在数据规模较小的情况下占用内存不是特别多，查询命中率很高，会提升检索速度，但是数据规模上来后，这里的内存问题越来越严重。
6. Merger server只能是一个，制约了查询的节点数量；数据不能进行动态分区，数据规模上来后单个索引太大。
Hermes的索引特点如下：
1. 大部分的索引处于关闭状态，只有真正用到索引才会去打开；一级跳跃表采用按需load，并不会load整个跳跃表，用来节省内存和提高打开索引的速度。Hermes经常会根据业务的不同去动态的打开不同的索引，关闭那些不经常使用的索引，这样同样一台机器，可以被多种不同的业务所使用，机器利用率高。
2. 排序和统计并不会使用数据的真实值，而是通过标签技术将大数据转换成占用内存很小的数据标签，占用内存是原先的几十分之一。另外不会将这个列的全部值都load到内存里，而是用到哪些数据load哪些数据，依然是按需load。不用了的数据会从内存里移除。
3. 索引存储在hdfs中，理论上只要hdfs有空间，就可以不断的添加索引，索引规模不在严重受机器的物理内存和物理磁盘的限制。
4. 采用yarn进行进程管理，数据在hdfs中，集群规模和扩容都是一件很easy的事情。
5. 如果某个词语存在数据倾斜，则会与其他条件组合进行跳跃合并（参考doclist的skiplist资料）。
6. 采用多级的merger server；数据可以根据业务的不同，采用不同的分区方式。
参考资料：
1、关于hermes与solr，es的定位与区别 http://user.qzone.qq.com/165162897/2
2、Hermes实时检索分析平台 http://data.qq.com/article?id=817