Index in hive

Use pre-index configuration

Before using Hive indexes, some configuration is required to ensure that the indexes can work properly. Here are some common configuration steps:

Hive configuration

To enable the indexing function in Hive, you need to set the following properties in the Hive configuration file (hive-site.xml):

<property>
    <name>hive.index.compact.file.uris</name>
    <value>/user/hive/warehouse/myindex</value>
</property>
<property>
    <name>hive.input.format</name>
    <value>org.apache.hadoop.hive.ql.index.compact.CompactIndexInputFormat</value>
</property>

These configurations are used to enable Hive indexes and specify the location where the indexes are stored.

HDFS storage index

You need to select an HDFS directory for the Hive index to store the index data. In the above example, /user/hive/warehouse/myindexthis is the directory used to store index data. Make sure this directory exists and has sufficient permissions for Hive to use.

Table level index configuration

Table-level indexes can optionally be enabled when creating a table. Use TBLPROPERTIESto specify the type of index and other configuration. For example:

CREATE TABLE my_table (
    ...
)
TBLPROPERTIES (
    'orc.create.index'='true',
    'orc.bloom.filter.columns'='column1,column2'
);

This example enables indexing in the ORC file format and specifies which columns to create bloom filter indexes on.

Compactor configuration

If a Compact Index is used, Compactor needs to be configured to merge and optimize the index regularly. Compactor is a stand-alone tool for managing index merging and cleanup. The relevant Compactor properties need to be set to control its behavior and scheduling.

<property>
    <name>hive.compactor.initiator.on</name>
    <value>true</value>
</property>
<property>
    <name>hive.compactor.worker.threads</name>
    <value>1</value>
</property>

These configurations are used to enable and configure Compactor.

Using indexes in hive

Create index

The syntax for creating an index in Hive is as follows:

CREATE INDEX index_name ON table_name (column_name);

For example, to create an index_nameindex named on a column table_nameof a table column_name, you would use the following statement:

CREATE INDEX index_name ON table_name (column_name);

After using the index, Hive will use the index when scanning the table. This can improve the efficiency of queries.

Here are some considerations for using Hive indexes:

Indexes only take effect when the query uses the indexed column.
Indexes increase the size of the table.
Indexes need to be updated regularly to ensure they are consistent with table data.

When deciding whether to use Hive indexes, you need to weigh the performance improvements and costs brought by indexes.

Here are some examples of creating Hive indexes:

-- 创建一个名为`index_name`的索引，该索引在表`table_name`的`column_name`列上。
CREATE INDEX index_name ON table_name (column_name);

-- 创建一个名为`index_name`的索引，该索引在表`table_name`的`column_name`列上，并使用`COMPACT`索引处理器。
CREATE INDEX index_name ON table_name (column_name) AS 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler';

-- 创建一个名为`index_name`的索引，该索引在表`table_name`的`column_name`列上，并延迟重建索引。
CREATE INDEX index_name ON table_name (column_name) WITH DELAYED REBUILD;

When are indexes used?

Equality query : When you need to filter data by equality conditions, indexes can greatly improve query performance. For example, when you use WHEREa clause to find a specific value, Hive can quickly locate matching rows if you have the appropriate index.
Range queries : Indexes can also be used to speed up range queries. For example, when you need to retrieve data within a range, you can use the index to quickly locate matching rows.
Sorting and grouping : Indexes can improve performance when performing sorting and grouping operations. Indexes help Hive access and organize data more efficiently when performing these operations.
Connection operations : When you perform connection operations (such as INNER JOIN, LEFT JOIN, etc.), if the connected columns have indexes, query performance can be significantly improved, because indexes can reduce the number of data scans and comparisons.
Unique constraints : Indexes can be used to enforce unique constraints on columns to prevent the insertion of duplicate data.
Accelerating subqueries : If you use subqueries in a query, indexes can improve the performance of the subquery, thereby speeding up the execution of the entire query.

What are the structures of indexes in hive

The indexes in Hive mainly have the following structures:

Index based on Compact Index

Compact Index is the default index structure in Hive, which can greatly speed up query speed. It stores index data in HDFS files and uses MapReduce to build it.

BitMap-based indexing

BitMap index uses bitmap to represent which data meets the conditions, and the query performance is very high. But it is only suitable for columns with a small number of different values, and is suitable for low cardinality data.

Bitmap indexes are indexes based on bit operations that can be used to quickly filter columns with limited value ranges. The implementation principle of bitmap index is as follows:

Maps all values in the column into a bitmap.
Each bit in the bitmap indicates whether a certain value in the column exists.
When querying, use bit operations to quickly filter rows that meet the criteria.

For example, suppose you have a column named genderand whose values are maleand female. We can gendermap into two bitmaps, maleeach bit in the bitmap represents malewhether exists, and femaleeach bit in the bitmap represents femalewhether exists.

When querying, if the query condition is gender = 'male', you can use bit operations &to quickly filter rows that meet the conditions. &The operation will perform an AND operation on the corresponding bits in the two bitmaps. If the result is 1, it means that the row meets the criteria.

Bitmap indexes are suitable for the following scenarios:

The value range of the column is limited, such as gender, marital_statusetc.
Multiple columns are involved in the query conditions, and the value range of these columns is limited, for example gender = 'male' AND age >= 18.

Bitmap indexes can increase query speed, but also increase storage space. Therefore, when using bitmap indexes, you need to make trade-offs based on actual needs.

The following are the pros and cons of bitmap indexes:

advantage:

Can improve query speed, especially for queries containing multiple conditions.
Bit operations can be used to perform fast logical operations.

shortcoming:

Will increase storage space.
Not suitable for columns with a large number of values.
Does not work for columns with consecutive values.

Lucene-based indexing

Lucene index is built based on the full-text search engine Lucene, which can perform full-text indexing of text. The query speed is fast and it supports fuzzy query, etc.

HBase based index

Store Hive table data in HBase and use HBase's fast random access to optimize queries. However, it is necessary to ensure that the data in HBase and the data in Hive are synchronized in real time.

Druid based indexing

Druid is a real-time analysis database that can be used for real-time analysis of massive data in Hive. It features high-performance aggregation and near real-time OLAP analysis capabilities.

In addition, Hive also supports the use of self-built indexes, and users can customize the structure and working mechanism of the index as needed. Choosing an appropriate index structure is crucial to optimizing Hive query performance.