ElasticSearch Lecture 4: ES Detailed Explanation: ElasticSearch and Kibana Installation

ElasticSearch Lecture 4: ES Detailed Explanation: ElasticSearch and Kibana Installation

This article is the fourth lecture of ElasticSearch: ElasticSearch and Kibana installation. It mainly introduces the installation of ElasticSearch and Kibana. After understanding the basics of ElasticSearch and the Elastic Stack ecosystem, we can start learning to use ElastiSearch.

1. Official website related tutorials

To install ElasticSearch, you still need to check the official website first.

• Official website
• Installation tutorial in the official 2.x Chinese tutorial
• Official ElasticSearch download address
• Official Kibana download address

This series of tutorials is based on ElasticSearch version 7.x.

2. Install ElasticSearch

ElasticSearch is based on the Java platform, so you need to install Java first

• Platform confirmation

Here I have prepared a Centos7 virtual machine. To facilitate the selection of the version to be installed later, I need to check the system version information.

[root@qwj-centos ~]# uname -a
Linux qwj-centos 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Install Java

Before installing Elasticsearch, you need to install a newer version of Java. The best option is that you can get the latest version of Java officially provided from www.java.com . After installation, confirm whether the installation is successful:

[root@qwj-centos ~]# java --version
openjdk 11.0.2 2019-01-15
OpenJDK Runtime Environment 20.3 (slowdebug build 11.0.2+12)
OpenJDK 64-Bit Server VM 20.3 (slowdebug build 11.0.2+12, mixed mode, sharing)

• Download ElasticSearch

Download ElasticSearch from here

For example, you can download it through curl

[root@qwj-centos opt]# curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

• Unzip

[root@qwj-centos opt]# tar zxvf /opt/elasticsearch-7.12.0-linux-x86_64.tar.gz 
...
[root@qwj-centos opt]# ll | grep elasticsearch
drwxr-xr-x  9 root root      4096 Mar 18 14:21 elasticsearch-7.12.0
-rw-r--r--  1 root root 327497331 Apr  5 21:05 elasticsearch-7.12.0-linux-x86_64.tar.gz

• Add elasticSearch user

A non-root user must be created to run ElasticSearch (ElasticSearch5 and above, due to security considerations, it is mandatory to not run as root.)

If you use the root user to start ElasticSearch, the following error message will appear:

[root@qwj-centos opt]# cd elasticsearch-7.12.0/
[root@qwj-centos elasticsearch-7.12.0]# ./bin/elasticsearch
[2021-04-05T21:36:46,510][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [qwj-centos] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.0.jar:7.12.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.0.jar:7.12.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.0.jar:7.12.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:101) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:168) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.12.0.jar:7.12.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.0.jar:7.12.0]
        ... 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:101)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:168)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.Command.main(Command.java:79)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
For complete error details, refer to the log at /opt/elasticsearch-7.12.0/logs/elasticsearch.log
2021-04-05 13:36:46,979269 UTC [8846] INFO  Main.cc@106 Parent process died - ML controller exiting

So we add an independent elasticsearch user to run

# 增加elasticsearch用户
[root@qwj-centos elasticsearch-7.12.0]# useradd elasticsearch
[root@qwj-centos elasticsearch-7.12.0]# passwd elasticsearch
Changing password for user elasticsearch.
New password: 
BAD PASSWORD: The password contains the user name in some form
Retype new password: 
passwd: all authentication tokens updated successfully.

# 修改目录权限至新增的elasticsearch用户
[root@qwj-centos elasticsearch-7.12.0]# chown -R elasticsearch /opt/elasticsearch-7.12.0
# 增加data和log存放区，并赋予elasticsearch用户权限
[root@qwj-centos elasticsearch-7.12.0]# mkdir -p /data/es
[root@qwj-centos elasticsearch-7.12.0]# chown -R elasticsearch /data/es
[root@qwj-centos elasticsearch-7.12.0]# mkdir -p /var/log/es
[root@qwj-centos elasticsearch-7.12.0]# chown -R elasticsearch /var/log/es

Then modify the above data and log paths,vi /opt/elasticsearch-7.12.0/config/elasticsearch.yml

# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/es
#
# Path to log files:
#
path.logs: /var/log/es

• Modify the restriction configuration of the Linux system

1. Modify the limit permissions in the system such as the maximum number of files an application is allowed to create. By default, Linux generally limits the maximum number of files created by an application to 65535 . But ES requires at least 65536 file creation permissions.
2. Modify how many threads a user-initiated process is allowed to open in the system. The default Linux limit is that processes started by the root user can start any number of threads, and processes started by other users can start 1024 threads. The limit number must be modified to 4096+. Because ES requires at least 4096 thread pool preparations . After ES version 5.x, it is mandatory that the root user cannot be used to start the ES process in Linux. Therefore, you must use another user to start the ES process.
3. The memory allocated for threads by the Linux lower version kernel is 128K. The 4.x version of the kernel allocates larger memory. If the memory of the virtual machine is 1G, only 3000+ threads can be enabled at most. Allocate at least 1.5G of memory to the virtual machine.

Modify the following configuration

[root@qwj-centos elasticsearch-7.12.0]# vi /etc/security/limits.conf

elasticsearch soft nofile 65536
elasticsearch hard nofile 65536
elasticsearch soft nproc 4096
elasticsearch hard nproc 4096

• Start ElasticSearch

[root@qwj-centos elasticsearch-7.12.0]# su elasticsearch
[elasticsearch@qwj-centos elasticsearch-7.12.0]$ ./bin/elasticsearch -d
[2021-04-05T22:03:38,332][INFO ][o.e.n.Node               ] [qwj-centos] version[7.12.0], pid[13197], build[default/tar/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z], OS[Linux/3.10.0-862.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-04-05T22:03:38,348][INFO ][o.e.n.Node               ] [qwj-centos] JVM home [/opt/elasticsearch-7.12.0/jdk], using bundled JDK [true]
[2021-04-05T22:03:38,348][INFO ][o.e.n.Node               ] [qwj-centos] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-17264135248464897093, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1894m, -Xmx1894m, -XX:MaxDirectMemorySize=993001472, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch-7.12.0, -Des.path.conf=/opt/elasticsearch-7.12.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]

• Check if the installation is successful

[root@qwj-centos ~]# netstat -ntlp | grep 9200
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      13549/java          
tcp6       0      0 ::1:9200                :::*                    LISTEN      13549/java          
[root@qwj-centos ~]# curl 127.0.0.1:9200
{
    
    
  "name" : "qwj-centos",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ihttW8b2TfWSkwf_YgPH2Q",
  "version" : {
    
    
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

3. Install Kibana

Kibana is an interface-based data query tool. When downloading, try to download the same version as ElasicSearch.

• DownloadKibana

Download Kibana from here

• Unzip

[root@qwj-centos opt]# tar -vxzf kibana-7.12.0-linux-x86_64.tar.gz

• Use elasticsearch user permissions

[root@qwj-centos opt]# chown -R elasticsearch /opt/kibana-7.12.0-linux-x86_64
#配置Kibana的远程访问
[root@qwj-centos opt]# vi /opt/kibana-7.12.0-linux-x86_64/config/kibana.yml
server.host: 0.0.0.0

• start up

Need to switch to elasticsearch user

[root@qwj-centos opt]# su elasticsearch
[elasticsearch@qwj-centos opt]$ cd /opt/kibana-7.12.0-linux-x86_64/
[elasticsearch@qwj-centos kibana-7.12.0-linux-x86_64]$ ./bin/kibana
  log   [22:30:22.185] [info][plugins-service] Plugin "osquery" is disabled.
  log   [22:30:22.283] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
  log   [22:30:22.482] [info][plugins-system] Setting up [100] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,dataEnhanced,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]
  log   [22:30:22.483] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: xxxxxx
  ...

If it is started in the background:

[elasticsearch@qwj-centos kibana-7.12.0-linux-x86_64]$ nohup ./bin/kibana &

• Interface access

Simple data can be imported

View data

4. Configure password access

When using a base license, Elasticsearch security features are disabled by default. Since my test environment is on the public network, I need to set a password for access. Related documents can be found here

1. Stop kibana and elasticsearch services
2. Add xpack.security.enabledthe setting to the ES_PATH_CONF/elasticsearch.yml file and set the value to true
3. start elasticsearch( ./bin/elasticsearch -d)
4. Execute the following password setter ./bin/elasticsearch-setup-passwords interactiveto set the password for each component
5. Add the elasticsearch.username setting to the KIB_PATH_CONF/kibana.yml file and set the value to the elastic user:elasticsearch.username: "elastic"
6. Create kibana keystore,./bin/kibana-keystore create
7. Add password in kibana keystore./bin/kibana-keystore add elasticsearch.password
8. Just restart the kibana servicenohup ./bin/kibana &

Then you can log in using your password: