One,
1, the query Product Details: cache because commodity Detailed
2, the query product pages: ES, because too much data, fuzzy query slow
3. Product Details page to make static html page, in order to better indexed by search engines
4, ES How to synchronize data with the database?
5. Why is it found out whether the keyword is included in the product name or product detail page during ES fuzzy search? Participant
aggregation query
6, kibana: a bit like postman test
7, PDMan:
8, ELK, logstash plugin, log collection
input (read local file or connect to database)
output (json) when
adding data will automatically map type
PUT should be modified , Modification found that no data will automatically add
POST is to add
lucene language
es to remove the type
pansoso! !
Highlight
The difference between inverted and participle
The principle of the network disk search engine: ES is used to store the captured data and crawl the server.
Logs cannot be stored in the database. The database is for persistent data. The logs can be placed in mongdb, es, redis (expired deletion)
9,
10.
11,
12,
13,
14,
15,
16
pansoso
2. Why use the Elasticsearch framework? Version 6.4.3
By default, IGB memory
can be used to process PB-level data, 1PB = 1024TB
large-scale distributed log analysis system ELK:
Elasticsearch (storage log) + logstash (collect log) + kibana (display data)
Elasticsearch storage structure: JSON
relational database database table row column
ES index type document field
9300 (HTTP protocol): used for communication between ES nodes, http protocol port number
9200 (interface form): port number for ES and external communication, 192.168.212.180:9200/index/type/id to access ES
Three, kibana visualization plugin
Default 5601 port
Create index
PUT /mymayikt
GET /mymayikt
DELETE /mymayikt
Create / Modify Document (Line) : Version will +1, optimistic locking
PUT /索引/类型/id
PUT /mymayikt/user/1?version=6
{
"name":"lh",
"age":21,
"sex":0
}
GET /mymayikt/user/1
Assign default ID
POST /mymayikt/user/
{
"name":"lh",
"age":21,
"sex":0
}
Query all data
GET /mymayikt/user/_search
Query mapping
The default mapping is long type
String type is divided into: text (word segmentation) and keyword (no word segmentation)
GET /mymayikt/_mapping
Query settings
GET /mymayikt/_setting
number_of_shards: 5 five main shards
number_of_replicas: 1 each main shard has several sub-shards for
a total of 10 shards
After the number of primary shards of the index is defined, it cannot be modified. The secondary ones can be changed. Why?
Routing algorithm: hash (routing document id)% number of main shards
The secondary shard corresponding to the primary shard cannot be placed on the same node (server)
Document mapping
POST /mymayikt/_mapping/user
{
"user":{
"properties":{
"age":{
"type":"integer"
},
"sex":{
"type":"integer"
},
"name":{
"type":"text",
"analyzer":"ik_smart",
},
"car":{
"type":"keyword"
}
}
}
}
Query by multiple IDs
GET /mymayikt/user/_search
{
"id":["1","2"]
}
Conditional query
GET /mymayikt/user/_search?q=age:21
Range query
GET /mymayikt/user/_search?q=age[30 TO 60]
descending sort
GET /mymayikt/user/_search?q=age[30 TO 60]&sort=age:desc
Paging query
GET /mymayikt/user/_search?q=age[30 TO 60]&sort=age:desc&from=0&size=2
Starting from line 0, two pieces of data
DSL language query: POST request, JSON format query
Exact query, no word segmentation query
GET mymayikt/user/_search
{
"query": {
=="term"==: {
"name": "xiaoming"
}
}
}
Support fuzzy query based on car name
Search the Audi a61 participle, the Chinese will be divided into a single, the English will be connected, the foreign framework
GET /mymayikt/user/_search
{
"from": 0,
"size": 2,
"query": {
=="match"==: ｛
"car": "奥迪a61"
}
}
}
filter
GET /mymayikt/user/_search
{
"query": {
"bool": {
"must": [{
"match_all": {} #查询所有
}],
"filter": {
"range": {
"age": {
"gt": 21, #age>21
"lte": 51 #age<21
}
}
}
}
},
"from": 0,
"size": 10,
"_source": ["name", "age"]#显示的字段
}
Four, SpringBoot integrated ES
Step 1: Introduce ES dependencies
Step 2: Specify the cluster name in the yml configuration file and linux
Step 3: UserEntity, ID must be of type String
Step 4: UserDao inheritance interface
Step 5: ESController
Five, ES inverted index principle
The full text search uses the inverted row at the bottom.
Why is the inverted index faster than the B + tree in the database?
Five documents, the traditional query is one by one, the reverse is the use of keywords
Six, ik tokenizer
Chinese word breaker, thesaurus
http://192.168.212.181:9200/_analyze
{
"analyzer": "ik_smart",
"text": "奥迪"
}
Manually add
a custom thesaurus file in the config directory of the hot word ik
Will match the longest word first,
such as the glory of the king.
Seven, ES cluster construction
Master-slave replication
forward
Configure different node ids and
configure the same cluster name node.name = node-1
SpringBoot and ES integrate ES to be a cluster, otherwise it will report an error
Eight, traditional log collection
The log is stored in each node
Nine, ELK
Logstash: log porter
1, first ES
2, LOG
3, k
Each node is installed with L, and L outputs the log file to ES
ANY + Kafka
Ten, ES and MySQL to achieve data synchronization
When the sql request is sent for the first time, the modification time parameter is the current time.
Determine whether update_time is updated every minute, and update it to ES synchronously
Incremental update
Mysql master-slave replication to determine whether the data is updated?
L synchronize multiple Mysql tables to ES, write multiple mysql.conf files, one file synchronizes one table
Some infrequent changes, such as the specification table, just use the cache, do not need to be saved in the ES
eleven,
The difference between distributed infrastructure and public Common
twelve,
thirteen,
fourteen,
fifteen,
sixteen,
17.
eighteen,
19.
twenty,
twenty one,
twenty two,
twenty-three,
twenty four,
twenty five,
Twenty-six,
Twenty-seven,
Twenty-eight,
Twenty-nine,
thirty,
