django-haystack-and-es

https://github.com/Einsteinish/Django-Haystack-Elasticsearch/tree/master/search_app

https://zhuanlan.zhihu.com/p/34094103

elasticsearch module: http://www.openskill.cn/article/47

Elasticsearch is a CS architecture software. It is a distributed and scalable real-time search and analysis engine. It is also a framework. After the elasticsearch search engine is installed on the server in the backend, the client also needs to use elasticsearch-client to interact with the client.

The realization idea is: First, there must be a MySQL database, and then the ES server builds an index database.

Start es in the background

[May@zanhu2 elasticsearch-7.9.0]$ touch running.log
[May@zanhu2 elasticsearch-7.9.0]$ ./bin/elasticsearch > running.log 2>&1 &
[1] 15508
[May@zanhu2 elasticsearch-7.9.0]$ ps -ef | grep java
May      15508 14580 42 11:15 pts/0    00:00:25 /usr/java/jdk-14.0.2/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms512m -Xmx512m -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-10984539820480802946 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=268435456 -Des.path.home=/tmp/elasticsearch-7.9.0 -Des.path.conf=/tmp/elasticsearch-7.9.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /tmp/elasticsearch-7.9.0/lib/* org.elasticsearch.bootstrap.Elasticsearch
May      15759 14580  0 11:16 pts/0    00:00:00 grep --color=auto java
[May@zanhu2 elasticsearch-7.9.0]$ exit
exit
(py37) [root@zanhu2 allauthdemo]# 
(py37) [root@zanhu2 allauthdemo]# 
(py37) [root@zanhu2 allauthdemo]# exit

note_text.txt is the result that needs to be indexed

free -m

When a user publishes an article, it is impossible for us to run the python manage.py rebuild_index command on the server at any time to make it rebuild the index. We must configure when new data is added or modified or deleted in the database. It can update the es index database in real time

, So you need to use the semaphore mechanism in haystack

Install the pit that es stepped on:

https://www.cnblogs.com/ghw0501/p/12940039.html

Automatically killed: https://blog.csdn.net/chunmiao3032/article/details/107427407

https://blog.csdn.net/weixin_43093501/article/details/89920675

The error is as follows:

Unrecognized VM option 'UseParNewGC
error：Could not create the Java Virtual Machine. Error:A Fatal exception has occurred,程序将退出

It may be because the version of java is too high, and then install a suitable java and specify the java environment variable, that is to say, you can install multiple versions of java!

[May@zanhu2 bin]$ ./elasticsearch
[2020-08-24T17:57:46,879][INFO ][o.e.n.Node               ] [zanhu2] version[7.9.0], pid[506605], build[default/tar/a479a2a7fce0389512d6a9361301708b92dff667/2020-08-11T21:36:48.204330Z], OS[Linux/3.10.0-1127.13.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/14.0.2/14.0.2+12-46]
[2020-08-24T17:57:46,908][INFO ][o.e.n.Node               ] [zanhu2] JVM home [/usr/java/jdk-14.0.2]
[2020-08-24T17:57:46,909][INFO ][o.e.n.Node               ] [zanhu2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-13774167505103428687, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/tmp/elasticsearch-7.9.0, -Des.path.conf=/tmp/elasticsearch-7.9.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-08-24T17:57:53,310][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [aggs-matrix-stats]
[2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [analysis-common]
[2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [constant-keyword]
[2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [flattened]
[2020-08-24T17:57:53,311][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [frozen-indices]
[2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [ingest-common]
[2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [ingest-geoip]
[2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [ingest-user-agent]
[2020-08-24T17:57:53,312][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [kibana]
[2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [lang-expression]
[2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [lang-mustache]
[2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [lang-painless]
[2020-08-24T17:57:53,315][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [mapper-extras]
[2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [parent-join]
[2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [percolator]
[2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [rank-eval]
[2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [reindex]
[2020-08-24T17:57:53,316][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [repository-url]
[2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [search-business-rules]
[2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [searchable-snapshots]
[2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [spatial]
[2020-08-24T17:57:53,317][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [tasks]
[2020-08-24T17:57:53,322][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [transform]
[2020-08-24T17:57:53,322][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [transport-netty4]
[2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [vectors]
[2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [wildcard]
[2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-analytics]
[2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-async]
[2020-08-24T17:57:53,323][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-async-search]
[2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-autoscaling]
[2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-ccr]
[2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-core]
[2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-data-streams]
[2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-deprecation]
[2020-08-24T17:57:53,324][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-enrich]
[2020-08-24T17:57:53,325][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-eql]
[2020-08-24T17:57:53,325][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-graph]
[2020-08-24T17:57:53,325][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-identity-provider]
[2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-ilm]
[2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-logstash]
[2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-ml]
[2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-monitoring]
[2020-08-24T17:57:53,327][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-ql]
[2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-rollup]
[2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-security]
[2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-sql]
[2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-stack]
[2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-voting-only-node]
[2020-08-24T17:57:53,328][INFO ][o.e.p.PluginsService     ] [zanhu2] loaded module [x-pack-watcher]
[2020-08-24T17:57:53,331][INFO ][o.e.p.PluginsService     ] [zanhu2] no plugins loaded
[2020-08-24T17:57:53,522][INFO ][o.e.e.NodeEnvironment    ] [zanhu2] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [21.4gb], net total_space [39.2gb], types [rootfs]
[2020-08-24T17:57:53,523][INFO ][o.e.e.NodeEnvironment    ] [zanhu2] heap size [1gb], compressed ordinary object pointers [true]
[2020-08-24T17:57:53,740][INFO ][o.e.n.Node               ] [zanhu2] node name [zanhu2], node ID [4mDodbjUSJmimuA9XVAuhw], cluster name [elasticsearch]
Killed

Install jdk before installing es

Download URL: http://apache.apooloo.cn/#/down/f1b54d77df533af9d9de55cf3f4e2cea

​ https://blog.csdn.net/baidu_20457783/article/details/89604867

Installation steps: https://www.php.cn/centos/445507.html

​ https://blog.csdn.net/u010590120/article/details/94736800

vim /etc/profile   # 配置java的环境变量

JAVA_HOME=/usr/java/jdk1.8.0_202
CLASSPATH=$JAVA_HOME/lib/
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME CLASSPATH


source /etc/profile 
java -version

/tmp/elasticsearch-7.9.0/config/elasticsearch.yml

Alibaba Cloud opens Elasticsearch: 9200 port is still inaccessible solution

https://blog.csdn.net/yang958554999/article/details/105334505

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Error encountered when elasticsearch is started

The problem is translated: the memory permissions owned by the elasticsearch user are too small, at least 262144 is required;

解决：

切换到root用户

执行命令：

sysctl -w vm.max_map_count=262144

查看结果：

sysctl -a|grep vm.max_map_count

显示：

vm.max_map_count = 262144



上述方法修改之后，如果重启虚拟机将失效，所以：

解决办法：

在  /etc/sysctl.conf文件最后添加一行

vm.max_map_count=262144

即可永久修改

the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

https://blog.csdn.net/qq_43655835/article/details/104637625

Access Alibaba Cloud Elasticsearch through the client

https://help.aliyun.com/document_detail/69194.html

from elasticsearch import Elasticsearch, RequestsHttpConnection
import certifi
es = Elasticsearch(
    ['<HOST>'],
    http_auth=('username', 'password'),
    port=9200,
    use_ssl=False
)
res = es.index(index="my_index", doc_type="my_type", id=1, body={
    
    "title": "One", "tags": ["ruby"]})
res = es.get(index="my_index", doc_type="my_type", id=1)
print(res['_source'])

django-contrib-comments

https://blog.csdn.net/weixin_30670151/article/details/96889561

https://github.com/xiaosongshine/djangoWebs