Introduction to HTTP protocol and system commands curl, AIP call (used in ELK)

http request consists of three parts

• They are: request line, message header, request body
• The URI and protocol version of the request line, the format is as follows:
  Method Request-URI HTTP-Version CRLF
• http request methods
  -common methods GET, POST, HEAD
  -other methods OPTIONS, PUT, DELETE, TRACE and CONNECT
• ES commonly used
  -PUT-add
  -
  DELETE- delete- POST-change-
  GET-check

  System command curl

  • In linux, curl is a file transfer tool that uses URL rules to work under the command line. It can be said to be a very powerful http command line tool. It supports multiple request modes, custom request headers and other powerful functions. It is a comprehensive tool
  • Introduction to common curl parameters
    --A modify request agent
    --X set request method
    --i display return header information

  RESTful API call

  • Elasticsearch provides a series of RESTful APIs
    -check the health, status and statistics of
    clusters, nodes, indexes-manage data and metadata of clusters, nodes, and
    indexes-perform CRUD operations and query operations on indexes
    -perform other advanced operations such as paging , Sorting, transition, etc.

POST or PUT data uses json format

• JSON
  – JSON (JavaScript Object Notation), which means JavaScript Object Notation, is a lightweight data exchange format based on text and independent of language.
  – JSON format
  – Array ["aa","bb","cc" ]
  -Key-value pair {key:value}

Simple use of RESTful API

– _Cat API query cluster status, node information
– V parameter shows detailed information

]# http://192.168.1.11:9200/_cat/health?v

-Help display help information

]# http://192.168.1.11:9200/_cat/health?help

• Simple use of Rest API

-Nodes query node status information

  ]# http://192.168.1.11:9200/_cat/nodes?v

– Index information

  ]# http://192.168.1.11:9200/_cat/indices?v

• Rest API increase
  -create an index, and set the number of shards and the number of replicas

  ]# curl -XPUT 'http://192.168.1.11:9200/home/' -d  '{
    
    
  "settings":{
    
    
  	"index":{
    
    
  		"number_of_shards":5,
  		"number_of_replicas":1
  		}
  	}
  }'

RESTful API insert data

– (Increase) add data, use PUT method
– call method: database address/index/type/id value

]#[root@es5 ~]# curl -X PUT "http://192.168.1.11:9200/home/tan/1" -d '{
    
    
"职业":"诗人",
"名字":"李白",
"称号":"诗仙",
"年代":"唐"
}'

POST modification

– To modify (revise) the data, use the POST method
– The _update keyword must be called when modifying the data
– Method of calling: database address/index/type/id value/_update

]#[root@es1 ~]# curl -XPUT "http://192.168.1.11:9200/home/tan/3/_update" -d '{
    
    
 "doc":{
    
    
 "年代": "唐代"
 }
 }'

(Check) Query

– The query uses the GET method, and GET is the default method
– The pretty standard display format can be used when the query displays the results
– Multiple queries need to use the _mget keyword with json invocation

]#[root@es1 ~]# curl -XGET 'http://192.168.1.11:9200/home/tan/1/'

(Deleted)

-When deleting, it can be a document or a library, but not a type

]#[root@es1 ~]# curl -XDELETE 'http://192.168.1.11:9200/home/tan/1/'
]#[root@es1 ~]# curl -XDELETE 'http://192.168.1.11:9200/home'

Import data in bulk

• Use _bulk to import data in
  batches-use POST to import data in batches, the data format is json, and the URL encoding uses data-binary
  -to import json files with index configuration
  #gzip -d logs.jsonl.gz
  #curl -XPORT'http:/ /192.168.1.11:9200/_bulk' [email protected]

map mapping

• Mapping:
  – Mapping: When creating an index, you can predefine the type and related attributes of the field
  – Function: This will make the index more detailed and complete
  – Classification: Static mapping and dynamic mapping
  – Dynamic mapping: Automatically correspond to the data Mapping
  -static mapping: custom field mapping data type

Case:

Use the curl command to import data in batches for the cluster, and view the
test file to obtain:

[root@es1 ~]# wget https://github.com/remembertr/elasticsearch--/blob/master/logs.jsonl.gz

[root@es1 ~]# gzip  -d logs.jsonl.gz 
[root@es1 ~]# curl -X POST "http://192.168.1.11:9200/_bulk"  \ 
--data-binary @logs.jsonl

Use GET to query results

[root@es1 ~]# curl -XGET 'http://192.168.1.11:9200/_mget?pretty' -d '{
    
    
> "docs":[
>      {
    
    
>         "_index":"shakespeare",
>         "_type:":"act",
>         "_id":0
> },
> {
    
    
>         "_index":"shakespeare",
>         "_type:":"line",
>         "_id":0
> },
> {
    
    
>         "_index":"tedu",
>         "_type:":"teacher",
>         "_id":25
> }
> ]
> }'

{
“docs” : [ {
“_index” : “shakespeare”,
“_type” : null,
“_id” : “0”,
“error” : {
“root_cause” : [ {
“type” : “index_not_found_exception”,
“reason” : “no such index”,
“resource.type” : “index_expression”,
“resource.id” : “shakespeare”,
“index” : “shakespeare”
} ],
“type” : “index_not_found_exception”,
“reason” : “no such index”,
“resource.type” : “index_expression”,
“resource.id” : “shakespeare”,
“index” : “shakespeare”
}
}, {
“_index” : “shakespeare”,
“_type” : null,
“_id” : “0”,
“error” : {
“root_cause” : [ {
“type” : “index_not_found_exception”,
“reason” : “no such index”,
“resource.type” : “index_expression”,
“resource.id” : “shakespeare”,
“index” : “shakespeare”
} ],
“type” : “index_not_found_exception”,
“reason” : “no such index”,
“resource.type” : “index_expression”,
“resource.id” : “shakespeare”,
“index” : “shakespeare”
}
}, {
“_index” : “tedu”,
“_type” : null,
“_id” : “25”,
“error” : {
“root_cause” : [ {
“type” : “index_not_found_exception”,
“reason” : “no such index”,
“resource.type” : “index_expression”,
“resource.id” : “tedu”,
“index” : “tedu”
} ],
“type” : “index_not_found_exception”,
“reason” : “no such index”,
“resource.type” : “index_expression”,
“resource.id” : “tedu”,
“index” : “tedu”
}
} ]
}