Lucene full-text search

Based on lucene 8

1 Lucene Introduction

Lucene is an open source under the apache full-text search engine tool kit.

1.1 full-text search (Full-text Search)

Full text search is the first word to create an index, and then perform the search process. The word is a piece of text into a word. Full-text search will be a piece of text into a word to query data

1.2 Lucene full-text search process

Full-text retrieval process is divided into two parts: the indexing process, the search process.

• Indexing process: data collection ---> Build Document Object ---> create index (write documents to index library).
• Search Process: Create a query ---> --- perform search> render search results.

2 start example

2.1 Requirements

Lucene project using electrical business books in commodity indexing and search capabilities.

2.2 Configuration step described

1. Built environment
2. Create an index Library
3. Search index Library

2.3 Configuration Step

2.3.1 Part I: build environment (creating a project, import the package)

2.3.2 Part II: Creating an index

Step Description:

1. Data collection
2. Convert the data into Lucene document
3. Write documents to index library, create an index

2.3.2.1 The first step: collecting data

Lucene full-text search, not directly query the database, you need to first collect the data out.

package jdbc.dao;

import jdbc.pojo.Book;
import jdbc.util.JdbcUtils;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;

public class BookDao {
    public List<Book> listAll() {
        //创建集合
        List<Book> books = new ArrayList<>();

        //获取数据库连接
        Connection conn = JdbcUtils.getConnection();

        String sql = "SELECT * FROM `BOOK`";
        PreparedStatement preparedStatement = null;
        ResultSet resultSet = null;
        try {
            //获取预编译语句
            preparedStatement = conn.prepareStatement(sql);

            //获取结果集
            resultSet = preparedStatement.executeQuery();

            //结果集解析
            while (resultSet.next()) {
                books.add(new Book(resultSet.getInt("id"),
                        resultSet.getString("name"),
                        resultSet.getFloat("price"),
                        resultSet.getString("pic"),
                        resultSet.getString("description")));
            }
        } catch (SQLException e) {
            e.printStackTrace();
        } finally {
            //关闭资源
            if (null != resultSet) {
                try {
                    resultSet.close();
                } catch (SQLException e) {
                    e.printStackTrace();
                } finally {
                    if (preparedStatement != null) {
                        try {
                            preparedStatement.close();
                        } catch (SQLException e) {
                            e.printStackTrace();
                        } finally {
                            if (null != conn) {
                                try {
                                    conn.close();
                                } catch (SQLException e) {
                                    e.printStackTrace();
                                }
                            }
                        }
                    }
                }
            }
        }
        return books;
    }
}

2.3.2.2 The second step: convert the data into Lucene document

Lucene is used to encapsulate the data type of the document, all data collected needs to be converted into first document type. The format is:

Modify BookDao, a new method, the conversion data

public List<Document> getDocuments(List<Book> books) {
    //创建集合
    List<Document> documents = new ArrayList<>();
    
    //循环操作 books 集合
    books.forEach(book -> {
        //创建 Document 对象，Document 内需要设置一个个 Field 对象
        Document doc = new Document();
        //创建各个 Field
        Field id = new TextField("id", book.getId().toString(), Field.Store.YES);
        Field name = new TextField("name", book.getName(), Field.Store.YES);
        Field price = new TextField("price", book.getPrice().toString(), Field.Store.YES);
        Field pic = new TextField("id", book.getPic(), Field.Store.YES);
        Field description = new TextField("description", book.getDescription(), Field.Store.YES);
        //将 Field 添加到文档中
        doc.add(id);
        doc.add(name);
        doc.add(price);
        doc.add(pic);
        doc.add(description);
        
        documents.add(doc);
    });
    return documents;
}

2.3.2.3 Step 3: Create an index Library

Lucene is in the process of a document written in the index library, auto-complete word, index creation. So create an index database from the formal point of view, it is to write a document index database!

package jdbc.test;

import jdbc.dao.BookDao;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.junit.Test;

import java.io.File;
import java.io.IOException;

public class LuceneTest {

    /**
     * 创建索引库
     */
    @Test
    public void createIndex() {
        BookDao dao = new BookDao();
        //该分词器用于逐个字符分词
        StandardAnalyzer standardAnalyzer = new StandardAnalyzer();
        //创建索引
        //1. 创建索引库存储目录
        try (Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath())) {
            //2. 创建 IndexWriterConfig 对象
            IndexWriterConfig ifc = new IndexWriterConfig(standardAnalyzer);
            //3. 创建 IndexWriter 对象
            IndexWriter indexWriter = new IndexWriter(directory, ifc);
            //4. 通过 IndexWriter 对象添加文档
            indexWriter.addDocuments(dao.getDocuments(dao.listAll()));
            //5. 关闭 IndexWriter
            indexWriter.close();

            System.out.println("完成索引库创建");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

You can view the results by luke tool

2.3.3 Part III: Search Index

2.3.3.1 Description

When searching, you need to specify a search which domain (ie, the field), and we need to do word processing on the keyword search.

2.3.3.2 perform a search

@Test
public void searchTest() {
    //1. 创建查询（Query 对象）
    StandardAnalyzer standardAnalyzer = new StandardAnalyzer();
    // 参数 1 指定搜索的 Field
    QueryParser queryParser = new QueryParser("name", standardAnalyzer);
    try {
        Query query = queryParser.parse("java book");
        //2. 执行搜索
        //a. 指定索引库目录
        Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
        //b. 创建 IndexReader 对象
        IndexReader reader = DirectoryReader.open(directory);
        //c. 创建 IndexSearcher 对象
        IndexSearcher searcher = new IndexSearcher(reader);
        /**
         * d. 通过 IndexSearcher 对象查询索引库，返回 TopDocs 对象
         * 参数 1：查询对象（Query）
         * 参数 2：前 n 条数据
         */
        TopDocs topDocs = searcher.search(query, 10);
        //e. 提取 TopDocs 对象中的的查询结果
        ScoreDoc[] scoreDocs = topDocs.scoreDocs;

        System.out.println("查询结果个数为：" + topDocs.totalHits);

        //循环输出数据对象
        for (ScoreDoc scoreDoc : scoreDocs) {
            //获得文档对象 id
            int docId = scoreDoc.doc;
            //通过 id 获得具体对象
            Document document = searcher.doc(docId);
            //输出图书的书名
            System.out.println(document.get("name"));
        }

        //关闭 IndexReader
        reader.close();
    } catch (ParseException | IOException e) {
        e.printStackTrace();
    }
}

result

3 word

Lucene word of the process, we can make the following summary:

1. When word, is the domain of the unit. Different domains independently of each other. The same domain, split out the same word, as the same term (Term). Different domains, split out the same word, is not the same word. Wherein, Term Lucene is the smallest unit of vocabulary can not be subdivided.
2. When word went through a series of filters. Such as case conversion, removal of stop words and so on.

From the figure above, we find:

1. Index library has two areas: the index area, the document area.
2. Document is a document storage area. Lucene automatically added to each document a document number docID.
3. Index area is stored in the index. note:
   • Index is based on field units of different domains, independent of each other.
   • Indexes are created out of the word according to the rules, you can find the corresponding documentation from the index.

4 Field Domain

We already know, Lucene is written in the document, complete word, index. That is how Lucene know how to word it? Lucene is to determine whether to word, whether to create an index based on the attributes of the document domain. So, we have to figure out what domain property.

4.1 attribute domain

4.1.1 Three Properties

Are 4.1.1.1 participle (tokenized)

Only the word attribute set to true, lucene this domain will be word processing.

In the actual development, some fields are not required word, such as product id, product images and so on. While some fields are required word, such as product names, descriptions and other information.

4.1.1.2 whether the index (indexed)

Only set the Index property true, lucene index was created for the Term of this domain is the word.

In the actual development, some fields are not required to create the index, such as pictures and other commodities. We just need to do the indexing process for participating in the search field.

4.1.1.3 is stored (stored)

Only set the storage property is true, when looking to get the value of this field from the document.

In the actual development, there are some fields that do not need to be stored. For example: product description. Because the product description information, usually large text data read time can cause great IO overhead. The description of the fields is not often query, so to wasted resources of the cpu. Therefore, as this does not require frequent queries, but also a large text fields, usually not stored in the index database.

4.1.2 Features

1. Three properties independent of each other.
2. Usually segmentation to create an index.
3. This field does not store the text content as well as the first word of this domain, create the index.

4.2 Field common type

There are many common types of domains, each class has its own default three properties. as follows:

Getting example 4.3 Rebuilding domain type

public List<Document> getDocuments(List<Book> books) {
    //创建集合
    List<Document> documents = new ArrayList<>();

    //循环操作 books 集合
    books.forEach(book -> {
        //创建 Document 对象，Document 内需要设置一个个 Field 对象
        Document doc = new Document();
        //创建各个 Field
        //存储但不分词、不索引
        Field id = new StoredField("id", book.getId());
        //存储、分词、索引
        Field name = new TextField("name", book.getName(), Field.Store.YES);
        //存储但不分词、不索引
        Field price = new StoredField("price", book.getPrice());
        //存储但不分词、不索引
        Field pic = new StoredField("pic", book.getPic());
        //分词、索引，但不存储
        Field description = new TextField("description", book.getDescription(), Field.Store.NO);
        //将 Field 添加到文档中
        doc.add(id);
        doc.add(name);
        doc.add(price);
        doc.add(pic);
        doc.add(description);

        documents.add(doc);
    });
    return documents;
}

result

5 index Library Maintenance]

5.1 Add index (document)

5.1.1 Demand

The new database shelves of books, these books must also be added to the index database, or to not search the shelves of new books up.

5.1.2 code implementation

Call indexWriter.addDocument (doc) add indexes. (Refer to create an index entry in the example)

5.2 Delete Index (document)

5.2.1 Demand

Some books out of print sales, we need to remove the book from the library index.

5.2.2 code implementation

@Test
public void deleteIndex() throws IOException {
    //1.指定索引库目录
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //2.创建 IndexWriterConfig
    IndexWriterConfig indexWriterConfig = new IndexWriterConfig(new StandardAnalyzer());
    //3.创建 IndexWriter
    IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);
    //4.删除指定索引
    indexWriter.deleteDocuments(new Term("name", "java"));
    //5.关闭 IndexWriter
    indexWriter.close();
}

5.2.3 Empty index code implementation

@Test
public void deleteAllIndex() throws IOException {
    //1.指定索引库目录
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //2.创建 IndexWriterConfig
    IndexWriterConfig indexWriterConfig = new IndexWriterConfig(new StandardAnalyzer());
    //3.创建 IndexWriter
    IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);
    //4.删除所有索引
    indexWriter.deleteAll();
    //5.关闭 IndexWriter
    indexWriter.close();
}

5.3 Update Index (document)

5.3.1 Description

Lucene index update is rather special, is to delete the document to meet the conditions, and then add the new document.

5.3.2 code implementation

@Test
public void updateIndex() throws IOException {
    //1.指定索引库目录
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //2.创建 IndexWriterConfig
    IndexWriterConfig indexWriterConfig = new IndexWriterConfig(new StandardAnalyzer());
    //3.创建 IndexWriter
    IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);
    //4.创建新加的文档对象
    Document document = new Document();
    document.add(new TextField("name", "testUpdate", Field.Store.YES));
    //5.修改指定索引为新的索引
    indexWriter.updateDocument(new Term("name", "java"), document);
    //6.关闭 IndexWriter
    indexWriter.close();
}

6 Search

Question: We start example, has been known by IndexSearcher Lucene is an object to perform a search. In the actual development, our business is relatively complex queries, such as when we find keyword, tend to filter price, product categories. The Lucene provides a set of query plan for us to achieve complex queries.

6.1 two ways to create a query

Prior to executing the query, you must create a query Query query object. Query itself is an abstract class that can not be instantiated, initialized must be achieved in other manners. Here, Lucene Query provides two ways to initialize the query object.

6.1.1 Using Lucene Query provides subclass

Query is an abstract class, lucene provides a lot of query object, such as TermQuery precise query terms , NumericRangeQuery numeric range queries and so on.

6.1.2 QueryParse parse the query expression

QueryParser will be entered by the user query expression Query resolved to object instances. The following code:

QueryParser queryParser = new QueryParser("name", new StandardAnalyzer());
Query query = queryParser.parse("name:lucene");

6.2 common subclasses Search Query

6.2.1 TermQuery

Features: keyword query will not do word processing, as a whole search. code show as below:

@Test
public void queryByTermQuery() throws IOException {
    Query query = new TermQuery(new Term("name", "java"));
    doQuery(query);
}

private void doQuery(Query query) throws IOException {
    //指定索引库
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //创建读取流
    DirectoryReader reader = DirectoryReader.open(directory);
    //创建执行搜索对象
    IndexSearcher searcher = new IndexSearcher(reader);

    //执行搜索
    TopDocs topDocs = searcher.search(query, 10);
    System.out.println("共搜索结果：" + topDocs.totalHits);

    //提取文档信息
    //score即相关度。即搜索的关键词和 图书名称的相关度，用来做排序处理
    ScoreDoc[] scoreDocs = topDocs.scoreDocs;

    for (ScoreDoc scoreDoc : scoreDocs) {
        int docId = scoreDoc.doc;
        System.out.println("索引库编号：" + docId);

        //提取文档信息
        Document doc = searcher.doc(docId);
        System.out.println(doc.get("name"));
        System.out.println(doc.get("id"));
        System.out.println(doc.get("priceValue"));
        System.out.println(doc.get("pic"));
        System.out.println(doc.get("description"));

        //关闭读取流
        reader.close();
    }
}

6.2.2 WildCardQuery

Use wildcard queries

/**
 * 通过通配符查询所有文档
 * @throws IOException
 */
@Test
public void queryByWildcardQuery() throws IOException {
    Query query = new WildcardQuery(new Term("name", "*"));
    doQuery(query);
}

private void doQuery(Query query) throws IOException {
    //指定索引库
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //创建读取流
    DirectoryReader reader = DirectoryReader.open(directory);
    //创建执行搜索对象
    IndexSearcher searcher = new IndexSearcher(reader);

    //执行搜索
    TopDocs topDocs = searcher.search(query, 10);
    System.out.println("共搜索结果：" + topDocs.totalHits);

    //提取文档信息
    //score即相关度。即搜索的关键词和 图书名称的相关度，用来做排序处理
    ScoreDoc[] scoreDocs = topDocs.scoreDocs;

    for (ScoreDoc scoreDoc : scoreDocs) {
        int docId = scoreDoc.doc;
        System.out.println("索引库编号：" + docId);

        //提取文档信息
        Document doc = searcher.doc(docId);
        System.out.println(doc.get("name"));
        System.out.println(doc.get("id"));
        System.out.println(doc.get("priceValue"));
        System.out.println(doc.get("pic"));
        System.out.println(doc.get("description"));

    }
    //关闭读取流
    reader.close();
}

6.2.3 Types of digital RangeQuery

Specified numeric range queries. (When you create a field type, the corresponding note), modify the price at the time of indexing

/**
 * 将 Book 集合封装成 Document 集合
 * @param books Book集合
 * @return Document 集合
 */
public List<Document> getDocuments(List<Book> books) {
    //创建集合
    List<Document> documents = new ArrayList<>();

    //循环操作 books 集合
    books.forEach(book -> {
        //创建 Document 对象，Document 内需要设置一个个 Field 对象
        Document doc = new Document();
        //创建各个 Field
        //存储但不分词、不索引
        Field id = new StoredField("id", book.getId());
        //存储、分词、索引
        Field name = new TextField("name", book.getName(), Field.Store.YES);
        //Float 数字存储、索引
        Field price = new FloatPoint("price", book.getPrice()); //用于数字的区间查询，不会存储，需要额外的 StoredField
        Field priceValue = new StoredField("priceValue", book.getPrice());//用于存储具体价格
        //存储但不分词、不索引
        Field pic = new StoredField("pic", book.getPic());
        //分词、索引，但不存储
        Field description = new TextField("description", book.getDescription(), Field.Store.NO);
        //将 Field 添加到文档中
        doc.add(id);
        doc.add(name);
        doc.add(price);
        doc.add(priceValue);
        doc.add(pic);
        doc.add(description);

        documents.add(doc);
    });
    return documents;
}

FloatPoint static method corresponding to obtain RangeQuery

/**
 * Float 类型的范围查询
 * @throws IOException
 */
@Test
public void queryByNumricRangeQuery() throws IOException {
    Query query = FloatPoint.newRangeQuery("price", 60, 80);
    doQuery(query);
}

private void doQuery(Query query) throws IOException {
    //指定索引库
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //创建读取流
    DirectoryReader reader = DirectoryReader.open(directory);
    //创建执行搜索对象
    IndexSearcher searcher = new IndexSearcher(reader);

    //执行搜索
    TopDocs topDocs = searcher.search(query, 10);
    System.out.println("共搜索结果：" + topDocs.totalHits);

    //提取文档信息
    //score即相关度。即搜索的关键词和 图书名称的相关度，用来做排序处理
    ScoreDoc[] scoreDocs = topDocs.scoreDocs;

    for (ScoreDoc scoreDoc : scoreDocs) {
        int docId = scoreDoc.doc;
        System.out.println("索引库编号：" + docId);

        //提取文档信息
        Document doc = searcher.doc(docId);
        System.out.println(doc.get("name"));
        System.out.println(doc.get("id"));
        System.out.println(doc.get("priceValue"));
        System.out.println(doc.get("pic"));
        System.out.println(doc.get("description"));

    }
    //关闭读取流
    reader.close();
}

6.2.4 BooleanQuery

BooleanQuery, Boolean queries, to achieve a combination of query conditions.

@Test
public void queryByBooleanQuery() throws IOException {
    Query priceQuery = FloatPoint.newRangeQuery("price", 60, 80);
    Query nameQuery = new TermQuery(new Term("name", "java"));

    //通过 Builder 创建 query
    BooleanQuery.Builder booleanQueryBuilder = new BooleanQuery.Builder();
    //至少有一个时 Occur.MUST，不然结果为空
    booleanQueryBuilder.add(nameQuery, BooleanClause.Occur.MUST_NOT);
    booleanQueryBuilder.add(priceQuery, BooleanClause.Occur.MUST);
    BooleanQuery query = booleanQueryBuilder.build();

    doQuery(query);
}

private void doQuery(Query query) throws IOException {
    //指定索引库
    Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath());
    //创建读取流
    DirectoryReader reader = DirectoryReader.open(directory);
    //创建执行搜索对象
    IndexSearcher searcher = new IndexSearcher(reader);

    //执行搜索
    TopDocs topDocs = searcher.search(query, 10);
    System.out.println("共搜索结果：" + topDocs.totalHits);

    //提取文档信息
    //score即相关度。即搜索的关键词和 图书名称的相关度，用来做排序处理
    ScoreDoc[] scoreDocs = topDocs.scoreDocs;

    for (ScoreDoc scoreDoc : scoreDocs) {
        int docId = scoreDoc.doc;
        System.out.println("索引库编号：" + docId);

        //提取文档信息
        Document doc = searcher.doc(docId);
        System.out.println(doc.get("name"));
        System.out.println(doc.get("id"));
        System.out.println(doc.get("priceValue"));
        System.out.println(doc.get("pic"));
        System.out.println(doc.get("description"));

    }
    //关闭读取流
    reader.close();
}

6.3 Search by QueryParser

6.3.1 Features

Search for keywords, do word processing.

6.3.2 Syntax

6.3.2.1 basic grammar

域名:关键字 Such as: name:java

6.3.2.2 combination of conditions grammar

• Condition Condition 1 AND 2
  
• Condition 2 Condition 1 OR
• Condition 2 Condition 1 NOT

E.g: Query query = queryParser.parse("java NOT 编");

6.3.3 QueryParser

@Test
public void queryByQueryParser() throws IOException, ParseException {
    //创建分词器
    StandardAnalyzer standardAnalyzer = new StandardAnalyzer();
    /**
     * 创建查询解析器
     * 参数一： 默认搜索的域。
     *         如果在搜索的时候，没有特别指定搜索的域，则按照默认的域进行搜索
     *         指定搜索的域的方式：   域名:关键词  如：  name:java
     * 参数二： 分词器，对关键词做分词处理
     */
    QueryParser queryParser = new QueryParser("description", standardAnalyzer);
    Query query = queryParser.parse("java 教程");
    doQuery(query);
}

6.3.4 MultiFieldQueryParser

Query for multiple domains by MulitFieldQueryParse.

@Test
public void queryByMultiFieldQueryParser() throws ParseException, IOException {
    //1.定义多个搜索的域
    String[] fields = {"name", "description"};
    //2.加载分词器
    StandardAnalyzer standardAnalyzer = new StandardAnalyzer();
    //3.创建 MultiFieldQueryParser 实例对象
    MultiFieldQueryParser multiFieldQueryParser = new MultiFieldQueryParser(fields, standardAnalyzer);
    Query query = multiFieldQueryParser.parse("java");

    doQuery(query);
}

7 Chinese word breaker

7.1 What is the Chinese word breaker

I learned English all know that, between words separated by spaces or commas period is the English word for the units. Standard word, a word by word did not like the English, only a character a character to divide. It is necessary to automatically identify a semantic word is Chinese.

7.2 Lucene comes with the Chinese word breaker

7.2.1 StandardAnalyzer：

Word word: word is carried out in accordance with the Chinese word for word. Such as: "I love China"

Effect: "I", "love", "medium" and "country."

7.2.2 CJKAnalyzer

Dichotomy word: be segmented according to the word. Such as: "I am Chinese"

Effect: "I am", "it is in", "China", "the people."

7.2.3 SmartChineseAnalyzer

Chinese intelligence official recognition, the need to import a new jar package

@Test
public void createIndexByChinese () {
    BookDao dao = new BookDao();
    //该分词器用于中文分词
    SmartChineseAnalyzer smartChineseAnalyzer = new SmartChineseAnalyzer();
    //创建索引
    //1. 创建索引库存储目录
    try (Directory directory = FSDirectory.open(new File("C:\\Users\\carlo\\OneDrive\\Workspace\\IdeaProjects\\lucene-demo01-start\\lucene").toPath())) {
        //2. 创建 IndexWriterConfig 对象
        IndexWriterConfig ifc = new IndexWriterConfig(smartChineseAnalyzer);
        //3. 创建 IndexWriter 对象
        IndexWriter indexWriter = new IndexWriter(directory, ifc);
        //4. 通过 IndexWriter 对象添加文档
        indexWriter.addDocuments(dao.getDocuments(dao.listAll()));
        //5. 关闭 IndexWriter
        indexWriter.close();

        System.out.println("完成索引库创建");
    } catch (IOException e) {
        e.printStackTrace();
    }
}

The effect is as: