Elasticsearch certification and security

360DBA team 360 cloud computing

Heroine declaration

Some time ago, we posted an article "Elasticsearch Security Policy", introducing some of Elasticsearch's security policies, and mentioned in the article Elasticsearch's permission system-SearchGuard. In the follow-up, many friends asked about SearchGuard related content, the editor specially sorted it out Share some of the content installed and used by SearchGuard for everyone, hoping to help friends who use Elasticsearch.
PS: Rich first-line technology and diversified expressions are all in the "HULK first-line technology talk", please pay attention!

Search Guard function

Elasticsearch certification and security
A brief summary of the issues that everyone cares about the most (here only the functions provided by the free version are translated)

Search Guard supports SSL/TLS encrypted transmission of Transport Layer (Node-to-node) and REST Layer (HTTP/HTTPS), and both Transport Layer and REST Layer can be individually configured to enable SSL/TLS encryption.
Search Guard provides a complete "user-role-permission" control system. The permissions of the free version can be controlled to indice/type and host levels.
If you need Document level security (DLS) and Field level security (FLS) level access control, or Audit logging audit function, or need to support third-party user authentication systems such as LDAP, Kerberos, etc., then obediently purchase Enterprise License (per One license for each cluster, it doesn’t matter the size of the cluster)

To add, Search Guard can achieve a perfect combination with Logstash and Kibana (this article is based on Elasticsearch 2.3.5). For users who use ELK, there is no need to worry, and the modification is easy.

Version correspondence

Search Guard needs to rely on Search Guard SSL to provide services. Before Elasticsearch 5.x, Search Guard and Search Guard SSL need to be installed separately; after Elasticsearch 5.x, Search Guard has integrated the corresponding version of Search Guard SSL by default, and there is no need to install it separately.
By the way, after Elasticsearch 5.x, the version numbers of Search Guard, Search Guard SSL (of course, Logstash, Kibana) and other plugins have been unified, which has become more simple and intuitive.
Take Elasticsearch version 2.3.5 as an example. First, we need to obtain the corresponding versions of Search Guard and Search Guard SSL. For the version correspondence, please refer to here:
https://github.com/floragunncom/search-guard/wiki
Elasticsearch certification and security

Download and install

Because Elasticsearch's default plug-in installation method is online, each node needs to be connected to the Internet (to connect to a foreign network, as you know), so it is recommended to download the plug-in installation package first and use the local installation method; and it is easy to make automatic deployment.

Here, we introduce the deployment process of the entire system and the details that need attention

1 Download Elasticsearch 2.3.5

Download link: https://www.elastic.co/downloads/past-releases/elasticsearch-2-3-5
Choose rpm or zip/tar package according to your own habits

2 Download Search Guard SSL 2.3.5.19

Download link: https://github.com/floragunncom/search-guard-ssl/tree/v2.3.5.19

3 Download Search Guard 2.3.5.10

Download link: https://github.com/floragunncom/search-guard/tree/v2.3.5.10

4 Install Elasticsearch

This article does not specifically explain how to install Elasticsearch for the time being. In subsequent articles, we will introduce the installation and deployment of Elasticsearch in detail.

5 Install Search Guard SSL

安装方式：
/path-to-es-install-dir/bin/plugin install -b file:///your-path/search-guard-ssl-2.3.5.19.zip

6 Install Search Guard

安装方式：
/path-to-es-install-dir/bin/plugin install -b file:///your-path/search-guard-2-2.3.5.10.zip

Tips: For
local installation, you must specify the keyword "file://" (two slashes followed by an absolute path).
The plug-in installation mechanism of Elasticsearch 5.X has changed. Please pay attention to it. For details, please follow the official documentation.
Search Guard and Search Guard SSL installation does not matter in order, as long as they are installed before starting.
Generate SSL/TLS certificate

Whether it is Transport Layer or REST Layer, if you want to use SSL/TLS, you must rely on certificates.

For the sake of simplicity, we can use the Example PKI Scripts officially provided by Search Guard SSL with minor modifications to generate our own certificates.

Because the content here is too much and does not affect the content of this article, the follow-up can be written separately, so I won't expand it here for the time being. Interested children's shoes can refer to
https://github.com/floragunncom/search-guard-ssl-docs/blob/master/certificates.md
As long as we know, this tool has generated the truststore.jks certificate common to all nodes in the cluster for us File, and the unique NODE_NAME-keystore.jks certificate file for each node.


这里涉及到的三个密码，一定要牢记
ROOT_CA_PASS
YOUR_KEYSTORE_PASS
YOUR_TRUSTSTORE_PASS

Next, we need to distribute truststore.jks and the NODE_NAME-keystore.jks certificate file of the corresponding node to the Elasticsearch configuration file directory of each node, and the user who starts the Elasticsearch service (non-root users are strongly recommended) need to have readable permissions.

If it is zip/tgz installation, the final directory structure is as follows:
Elasticsearch certification and security

Placement Search Guard

1 Corresponding parameters in the Elasticsearch configuration file

To load Elasticsearch and enable Search Guard, you need to add the following content to elasticsearch.yml:
Elasticsearch certification and security

Here are a few things to note:
1. In my example, I did not use openssl as the SSL/TLS provider, but LVM, so searchguard.ssl.transport.enable_openssl_if_available: false must be configured, and NSS needs to be installed.

Those interested in how to use openssl with Search Guard can refer to here:
https://github.com/floragunncom/search-guard-ssl-docs/blob/master/openssl.md

2. The searchguard.ssl.http.enabled: true|false part, this can be turned on according to actual needs; there are also three corresponding client authentication methods:

The searchguard.ssl.http.clientauth_mode parameter has three options: OPTIONAL|REQUIRE|NONE The
default is OPTIONAL

3. The admin certificates part is to configure where the sgadmin tool can be invoked to import the search guard authority configuration to ELasticsearch. Because Search Guard stores permissions and other configurations as an index in the Elasticsearch cluster.

2 Configure Search Guard authority authentication system

Search Guard authority authentication system includes 5 configuration files and two shell scripts

sgconfig/sg_config.yml: Configure Search Guard authentication and permission verification methods
sgconfig/sg_roles.yml: Configure Search Guard roles and the permissions list corresponding to the roles
sgconfig/sg_roles_mapping.yml: Configure the mapping relationship between users and Search Guard roles
sgconfig/sg_internal_users.yml: Configure the users recognized by the Search Guard internal authority system (hash value of user name and password)
sgconfig/sg_action_groups.yml: Configure permission groups for sg_roles.yml, which can reduce the length of sg_roles.yml
tools/hasher.sh: The user generates the hash value of the password of the search Guard internal user, avoiding the configuration of plain text
tools/sgadmin.sh: Used to import/update Search Guard configuration to Elasticsearch cluster. The
specific configuration is not complicated, you can refer to here:
https://github.com/floragunncom/search-guard-docs/blob/master/configuration.md
As for why the "users" and "roles" are not merged into one here, the advantage of this is that a series of "users" with similar permissions can be configured, and only the "roles" need to be configured once.
3

Import permission configuration to Elasticsearch through sgadmin

The related configuration of Search Guard permission authentication will be stored in Elasticsearch in the form of index. The default index name is searchguard, and this index cannot be modified directly through Elasticsearch API like other Elasticsearch indexes. Use sgadmin to complete all permissions modification as much as possible.

sgadmin can be executed on any node in the cluster, or it can be made standalone.

For example: remote execution on a separate management machine, the installation package can be found at the following address, but the corresponding version of jdk needs to be installed.
https://github.com/floragunncom/search-guard/wiki

The specific implementation method refers to the following methods:


/path-to-es-install-dir/plugins/search-guard-2/tools/sgadmin.sh -h NODE -p PORT -cd /path-to-your-sgconfig/ -ks /path-to-your/NODE_NAME-keystore.jks -kspass 
YOUR_KEYSTORE_PASS  -ts /path-to-your/truststore.jks -tspass YOUR_TRUSTSTORE_PASS -nhnv -cn YOUR_CLUSTERNAME

Please note that before Elasticsearch 5.x, after the index of searchguard is initialized for the first time, the number of replica shards will not automatically increase with the increase of the number of nodes. You need to use sgadmin to set it to achieve auto-expand.

Elasticsearch certification and security

end

At this point, Elasticsearch has integrated Search Guard. It needs to be explained again that this configuration can perfectly solve the situation of REST API accessing Elasticsearch. But if you use various clients to access Elasticsearch, because you are using the transport layer port, you need to configure a certificate to use it, and the management complexity of certificates and permissions is relatively high.