To achieve real-time search tips and full text search capability to synchronize data from MySQL

Recently spent a few days with integrated full-text search engine for the company's project, the initial goal of the project is the immediate prompt for the search box. Need to synchronize data coming from MySQL, since the data is not small, it is necessary to consider the initial ongoing incremental synchronization after synchronization. Here used open source service is ElasticSearch.

ElasticSearch is a very easy to use open source full-text search engine service, before a colleague recommended I did not understand before, but to see examples of Amazon specializes in providing the service, then guess not previously know more about Redis and should be the same as the reputation of the product, the estimated but also stand the test can be used in a production environment. After the discovery of the Internet to understand something is really the case:

Full-text search are among the most common needs, open source Elasticsearch is now the preferred full-text search engine. It can quickly store, search and analyze massive amounts of data. Wikipedia, Stack Overflow, Github have adopted it.

Ado, as usual record of my build process.

Installation ElasticSearch

There are several ways to install, I personally still prefer yum CentOS install from source.

Yum installed the CentOS

First to enter the /etc/yum.repos.ddirectory, create a file named elasticsearch.reposource, fill in the following contents:

1. [elasticsearch-6.x]
2. name=Elasticsearch repository for 6.x packages
3. baseurl=https://artifacts.elastic.co/packages/6.x/yum
4. gpgcheck=1
5. gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
6. enabled=1
7. autorefresh=1
8. type=rpm-md

Here Elastic latest version is 6.2, but the corresponding Elasticsearch-PHP requires PHP version 7.0 or higher. As the company's PHP version 5.x, so only second best, choose a little bit of old versions 5.6.9, 5.x version installed, this step will only need all 6 above the source file content. x 5.x can be replaced.

The next execution

1. yum install elasticsearch

After installation, the default is the only local service can be accessed, if the need to access the network from another server, also need to open listening range. Enter the installation directory /usr/share/elasticsearch, edit the elasticsearch.ymlfile, modify the following sections:

1. network.host: 0.0.0.0
2. path.data: /var/lib/elasticsearch
3. path.logs: /var/log/elasticsearch
4. http.host: 0.0.0.0
5. transport.host: 127.0.0.1

Which network.hostis on the external network access, while path.dataand path.logsbecause the default path is not set correctly, there need to manually set it. After the setup is complete path need to check these two directory exists, if there is a residual content in the previous installation directory, the need to back up empty, otherwise it will cause problems.

Then restart the service:

1. service elasticsearch restart

Installation test

After the reboot is complete, enter in your browser

1. http://127.0.0.1:9200/?pretty

If you can see the corresponding information indicating a successful installation

Installation LogStash

Then install LogStash service, which is used to summarize all types of log log information to a unified local administration, and here we use this service, because of the need to use it to synchronize data from MySQL to the Elastic.

YUM installed LogStash

This is also the home of Elastic product, and therefore included in the source set up earlier, and now only need to perform the installation:

1. yum install logstash

This completes the installation. Then do not worry, you also need to install a plugin.

Installation logstash-input-jdbc plug

First to enter the /usr/share/logstash/bindirectory, execute:

1. ./logstash-plugin install logstash-input-jdbc

After the plugin is installed, logstash installation is currently regarded as complete. There are many plug-ins can achieve a variety of feature-rich, but here we say no more.

MySQL data to configure the synchronization Elastic

Then the focus is where the comparison of the configuration data from a MySQL database synchronization to Elastic. First established in any directory synchronization profiles, synchronization scripts I have here is not much, and therefore directly put them perform directory logstash's:

1. cd /usr/share/logstash/bin
2. mkdir ktsee
3. cd ktsee

Then create two files jdbc.confand jdbc.sqlwhich jdbc.confare synchronized configuration file, jdbc.sqlMySQL script synchronization. First edit jdbc.conf, fill in the content:

1. input {
2. stdin {
3. }
4. jdbc {
5. # mysql jdbc connection string to our backup databse 后面的ktsee对应mysql中的test数据库
6. jdbc_connection_string => "jdbc:mysql://192.168.1.1:3306/ktsee"
7. # the user we wish to excute our statement as
8. jdbc_user => "root"
9. jdbc_password => "password"
10. # the path to our downloaded jdbc driver 这里需要设置正确的mysql-connector-java-5.1.38.jar路径，找不到可以从网上下载后放在配置路径中
11. jdbc_driver_library => "/elasticsearch-jdbc-2.3.2.0/lib/mysql-connector-java-5.1.38.jar"
12. # the name of the driver class for mysql
13. jdbc_driver_class => "com.mysql.jdbc.Driver"
14. jdbc_paging_enabled => "true"
15. jdbc_page_size => "50000"
16. # 以下对应着要执行的sql的绝对路径
17. statement_filepath => "/usr/local/logstash/bin/logstash_jdbc_test/jdbc.sql"
18. # 定时字段 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
19. schedule => "* * * * *"
20. # 设定ES索引类型
21. type => "ktsee_type"
22. }
23. }
25. filter {
26. json {
27. source => "message"
28. remove_field => ["message"]
29. }
30. }
32. output {
33. elasticsearch {
34. #ESIP地址与端口
35. hosts => "192.168.1.1:9200"
36. #ES索引名称（自己定义的）
37. index => "ktsee_index"
38. #自增ID编号
39. document_id => "%{id}"
40. }
41. stdout {
42. #以JSON格式输出
43. codec => json_lines
44. }
45. }

It should be noted that where, there is a corresponding note in the above configuration file.

Use Elasticsearch-PHP library integrated into the project

Here choose to use Elasticsearch official PHP library Elasticsearch-PHP, if the project using the composer for packet management, then it is simple, direct mounting to the corresponding version, composer automatically download other dependent libraries. Add code in the project:

1. $client = \Elasticsearch\ClientBuilder::create()
2. ->setHosts(['192.168.1.1:9200'])
3. ->allowBadJSONSerialization()
4. ->build();
5. $params = [
6. 'index' => 'ktsee_index',
7. '_source' => [
8. "id",
9. "product_name",
10. "product_type"
11. ],
12. 'body' => [
13. 'query' => [
14. 'match_phrase_prefix' => [
15. 'product_name' => [
16. "query" => $post['keyword'],
17. "slop" => 10
18. ]
19. ],
20. ]
21. ]
22. ];
23. $response = $client->search($params);

Thus achieving a simple call to ElasticSearch according to the search keyword.

Search Instant achieve prompt code

HTML part:

1. <form method="get" action="/search" id="header_search">
2. <input type="text" id="keyword" name="keyword" value="" autocomplete="off" />
3. <input type="submit" value="" />
4. </form>
5. <ul id="header_search_suggest"></ul>

It is worth noting that the search box input control coupled with autocomplete="off"close native tips drop box, avoid and we are going to do the smart prompt conflict.

CSS part:

1. #header_search_suggest{
2. position: absolute;
3. width: calc(100% - 10px);
4. left: 4px;
5. border: solid 1px #ccc;
6. background-color: white;
7. text-align: left;
8. z-index: 101;
9. display: none;
10. }
11. #header_search_suggest li{
12. font-size: 14px;
13. border-bottom: 1px solid #eeeeee;
14. }
15. #header_search_suggest li a{
16. padding:0.5em 1em;
17. color:#333333;
18. display: block;
19. text-decoration: none;
20. }
21. #header_search_suggest li a:hover{
22. background-color: #EDF0F2;
23. color:#2F7EC4;
24. }
25. #header_search_suggest li a em{
26. font-style: italic;
27. color:#999;
28. font-size:0.8em;
29. }

JS part:

1. var xhr = null;
2. $('#keyword').bind('input propertychange', function () {
3. if (xhr) {
4. xhr.abort();//如果存在ajax的请求，就放弃请求
5. }
6. var inputText = $.trim(this.value);
7. if (inputText != "") { //检测键盘输入的内容是否为空，为空就不发出请求
8. xhr = $.ajax({
9. type: 'POST',
10. url: '/search/suggest',
11. cache: false,//不从浏览器缓存中加载请求信息
12. // data: "keyword=" + inputText,
13. data: {keyword: inputText},
14. dataType: 'json',
15. success: function (json) {
16. //console.log(json);
17. if (json.count != 0) {
18. //检测返回的结果是否为空
19. var lists = "";
20. $.each(json.data, function (index, obj) {
21. //处理高亮关键词
22. var searchContent = obj.product_name;
23. var suggestItem = '';
24. if (searchContent.toLowerCase().indexOf(inputText.toLowerCase()) > -1) {
25. var searchRegExp = new RegExp('(' + inputText + ')', "gi");
26. suggestItem = searchContent.replace(searchRegExp, ("<strong>$1</strong>"));
27. }
28. suggestItem = suggestItem + "<em> - " + obj.product_type + "</em>";
30. //遍历出每一条返回的数据
31. lists += "<li class='listName' ><a href='/search/suggest?id=" + obj.id + "&key=" + encodeURI(searchContent + ' - ' + obj.product_type) + "'>" + suggestItem + "</a></li>";
32. });
34. $("#header_search_suggest").html(lists).show();//将搜索到的结果展示出来
35. } else {
36. $("#header_search_suggest").hide();
37. }
39. //记录搜索历史记录
40. $.post('/search/savesearchlog',{keyword: inputText,count: json.count});
41. }
42. });
43. } else {
44. $("#header_search_suggest").hide();//没有查询结果就隐藏搜索框
45. }
46. }).blur(function () {
47. setTimeout('$("#header_search_suggest").hide()',500);//输入框失去焦点的时候就隐藏搜索框，为了防止隐藏过快无法点击，设置延迟0.5秒隐藏
48. });

Demonstration effect

Figure:

This article from the  surenkid  creation, using  the Creative Commons Attribution 3.0 can be freely reproduced, quoted with named authors and indicate the source of the article.