Native apache hadoop3.3.1 cluster installation and configuration Kerberos

To install kerberos, you can read my other article. Let’s start configuring hadoop directly.

Detailed explanation of CDH configuration Kerberos and Sentry_cdh kerberos configuration_Mumunu-'s blog-CSDN blog

 After deploying kerberos, first add users and generate authentication files

Adding users who need to be authenticated in KDC depends on the specific user (hadoop cluster is mainly managed by hdfs, so in addition to creating hdfs account and HTTP account, hive, hbase, and dwetl will also access the hadoop cluster. If there is anything else Users can add additional items in this way, as shown below:

The format is: username/host [email protected]

kadmin.local -q "addprinc -randkey HTTP/[email protected]"
所有需要加入集群的节点都需要一个对应的账密和keytab
kadmin.local -q "addprinc -randkey HTTP/[email protected]"
kadmin.local -q "addprinc -randkey HTTP/[email protected]"
像这样 下面省略

kadmin.local -q "addprinc -randkey hive/[email protected]"

kadmin.local -q "addprinc -randkey hbase/[email protected]"

kadmin.local -q "addprinc -randkey hdfs/[email protected]"

kadmin.local -q "addprinc -randkey presto/[email protected]"

kadmin.local -q "addprinc -randkey dwetl/[email protected]"

2、按用户批量生成 keytab

kadmin.local -q "xst -k /export/common/kerberos5/hdfs.keytab HTTP/[email protected]"
kadmin.local -q "xst -k /export/common/kerberos5/hdfs.keytab HTTP/[email protected]"
kadmin.local -q "xst -k /export/common/kerberos5/hdfs.keytab HTTP/[email protected]"

kadmin.local -q "xst -k /export/common/kerberos5/hive.keytab hive/[email protected]"

kadmin.local -q "xst -k /export/common/kerberos5/hbase.keytab hbase/[email protected]"

kadmin.local -q "xst -k /export/common/kerberos5/hdfs.keytab hdfs/[email protected]"

kadmin.local -q "xst -k /export/common/kerberos5/presto-server.keytab presto/[email protected]"

kadmin.local -q "xst -k /export/common/kerberos5/dwetl.keytab dwetl/[email protected]"

The keytab file of the corresponding user will be generated in the current /export/common/kerberos5 directory. Then distribute hadoop.keytab to each machine, including from kdc and client (note: after distribution, since different users access the keytab, corresponding permissions must be assigned to the keytab file).

Configure HDFS to use HTTPS secure transfer protocol

A. Generate key pair

Keytool is a Java data certificate management tool that enables users to manage their own public/private key pairs and related certificates.

• -keystore Specify the name and location of the keystore (the various information generated will be stored in the .keystore file)
• - genkey(or -genkeypair) generate a key pair
• -alias Specify an alias for the generated key pair. If not, the default is mykey.
• -keyalg Specify the key algorithm RSA/DSA. The default is DSA.

Generate the keystore password and corresponding information for the keystore

[root@hadoop102 ~]# keytool -keystore /etc/security/keytab/keystore -alias jetty -genkey -keyalg RSA输入密钥库口令:  
再次输入新口令: 
您的名字与姓氏是什么?[Unknown]:  
您的组织单位名称是什么?[Unknown]:  
您的组织名称是什么?[Unknown]:  
您所在的城市或区域名称是什么?[Unknown]:  
您所在的省/市/自治区名称是什么?[Unknown]:  
该单位的双字母国家/地区代码是什么?[Unknown]:  
CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown是否正确?[否]:  y
输入 <jetty> 的密钥口令
(如果和密钥库口令相同, 按回车):  
再次输入新口令:

B. Modify the owner and access permissions of the keystore file

[root@hadoop102 ~]# chown -R root:hadoop /etc/security/keytab/keystore
[root@hadoop102 ~]# chmod 660 /etc/security/keytab/keystore

Notice:

• The password of the keystore must be at least 6 characters, and can be pure numbers or letters or a combination of numbers and letters, etc.
• Ensure that the hdfs user (HDFS startup user) has read permissions on the generated keystore file

C. Distribute the certificate to the same path of each node in the cluster

[root@hadoop102 ~]# xsync /etc/security/keytab/keystore

2. Modify the cluster configuration file
1.hdfs and add the following configuration
core-site.xml

<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
 
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>

hdfs-site.xml

<!-- kerberos start -->
<!-- namenode -->
 
<property>
<name>dfs.namenode.keytab.file</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/[email protected]</value>
</property>
 
<property>
<name>dfs.namenode.kerberos.internal.spnego.keytab</name>
<value>/export/common/kerberos5/HTTP.keytab</value>
</property>

<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/[email protected]</value>
</property>
 
<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>/export/common/kerberos5/HTTP.keytab</value>
</property>
 
<!-- datanode -->
<property>
<name>dfs.datanode.keytab.file</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>dfs.http.policy</name>
<value>HTTPS_ONLY</value>
</property>
 
<!-- <property>
<name>dfs.https.port</name>
<value>50470</value>
</property> -->
 
<property>
<name>dfs.data.transfer.protection</name>
<value>integrity</value>
</property>
 
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
 
<property> 
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
 
<!--
<property>
<name>dfs.datanode.https.address</name>
<value>0.0.0.0:50475</value>
</property> -->
 
 
 
<!-- journalnode -->
<property>
<name>dfs.journalnode.keytab.file</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<property>
<name>dfs.journalnode.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>
<value>HTTP/[email protected]</value>
</property>

<property>
<name>dfs.journalnode.kerberos.internal.spnego.keytab</name>
<value>/export/common/kerberos5/HTTP.keytab</value>
</property>
 
<!-- kerberos end-->

 hadoop_env.sh

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=${JAVA_HOME}/lib -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49:88"

ssl-server.xml (placed in the hadoop configuration directory: /export/common/hadoop/conf, empower hdfs:hadoop)

         

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at
       http://www.apache.org/licenses/LICENSE-2.0
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
 
<configuration>
 
 
 
<property>
  <name>ssl.server.truststore.location</name>
  <value>/etc/security/keytab/keystore</value>
  <description>Truststore to be used by NN and DN. Must be specified.
  </description>
</property>
 
<property>
  <name>ssl.server.truststore.password</name>
  <value>123456</value>
  <description>Optional. Default value is "".
  </description>
</property>
 
<property>
  <name>ssl.server.truststore.type</name>
  <value>jks</value>
  <description>Optional. The keystore file format, default value is "jks".
  </description>
</property>
 
<property>
  <name>ssl.server.truststore.reload.interval</name>
  <value>10000</value>
  <description>Truststore reload check interval, in milliseconds.
  Default value is 10000 (10 seconds).
  </description>
</property>
 
<property>
  <name>ssl.server.keystore.location</name>
  <value>/etc/security/keytab/keystore</value>
  <description>Keystore to be used by NN and DN. Must be specified.
  </description>
</property>
 
<property>
  <name>ssl.server.keystore.password</name>
  <value>123456</value>
  <description>Must be specified.
  </description>
</property>
 
<property>
  <name>ssl.server.keystore.keypassword</name>
  <value>123456</value>
  <description>Must be specified.
  </description>
</property>
 
<property>
  <name>ssl.server.keystore.type</name>
  <value>jks</value>
  <description>Optional. The keystore file format, default value is "jks".
  </description>
</property>
 
<property>
  <name>ssl.server.exclude.cipher.list</name>
  <value>TLS_ECDHE_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,
  SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
  SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,
  SSL_RSA_WITH_RC4_128_MD5</value>
  <description>Optional. The weak security cipher suites that you want excluded
  from SSL communication.</description>
</property>
 
</configuration>

 
 ssl-client.xml (placed in the hadoop configuration directory: /export/common/hadoop/conf, empower hdfs:hadoop)

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at
       http://www.apache.org/licenses/LICENSE-2.0
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
 
<configuration>
 
<property>
  <name>ssl.client.truststore.location</name>
  <value>/etc/security/keytab/keystore</value>
  <description>Truststore to be used by clients like distcp. Must be
  specified.
  </description>
</property>
 
<property>
  <name>ssl.client.truststore.password</name>
  <value>123456</value>
  <description>Optional. Default value is "".
  </description>
</property>
 
 
 
<property>
  <name>ssl.client.truststore.type</name>
  <value>jks</value>
  <description>Optional. The keystore file format, default value is "jks".
  </description>
</property>
 
 
 
<property>
  <name>ssl.client.truststore.reload.interval</name>
  <value>10000</value>
  <description>Truststore reload check interval, in milliseconds.
  Default value is 10000 (10 seconds).
  </description>
</property>
 
<property>
  <name>ssl.client.keystore.location</name>
  <value>/etc/security/keytab/keystore</value>
  <description>Keystore to be used by clients like distcp. Must be
  specified.
  </description>
</property>
 
<property>
  <name>ssl.client.keystore.password</name>
  <value>123456</value>
  <description>Optional. Default value is "".
  </description>
</property>
 
<property>
  <name>ssl.client.keystore.keypassword</name>
  <value>123456</value>
  <description>Optional. Default value is "".
  </description>
</property>
 
<property>
  <name>ssl.client.keystore.type</name>
  <value>jks</value>
  <description>Optional. The keystore file format, default value is "jks".
  </description>
</property>


</configuration>

 2.yarn add the following configuration
yarn-site.xml

<!-- resourcemanager -->
<property>
<name>yarn.web-proxy.principal</name>
<value>HTTP/[email protected]</value>
</property>
 
<property>
<name>yarn.web-proxy.keytab</name>
<value>/export/common/kerberos5/HTTP.keytab</value>
</property>
 
<property>
<name>yarn.resourcemanager.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>yarn.resourcemanager.keytab</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<!-- nodemanager -->
<property>
<name>yarn.nodemanager.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>yarn.nodemanager.keytab</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
 
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>hdfs</value>
</property>
 
<!-- timeline kerberos -->
<property>
<name>yarn.timeline-service.http-authentication.type</name>
<value>kerberos</value>
<description>Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#</description>
 
</property>
<property>
<name>yarn.timeline-service.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>yarn.timeline-service.keytab</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<property>
<name>yarn.timeline-service.http-authentication.kerberos.principal</name>
<value>HTTP/[email protected]</value>
</property>
 
<property> 
  <name>yarn.timeline-service.http-authentication.kerberos.keytab</name>
  <value>/export/common/kerberos5/HTTP.keytab</value>
</property>
 
<property>
<name>yarn.nodemanager.container-localizer.java.opts</name>
<value>-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49 :88</value>
</property>
 
<property>
<name>yarn.nodemanager.health-checker.script.opts</name>
<value>-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49:88</value>
</property>

mapred-site.xml

<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1638M -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49:88</value>
</property>
 
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx3276M -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49:88</value>
</property>
 
<property>
<name>mapreduce.jobhistory.keytab</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>
 
<property>
<name>mapreduce.jobhistory.principal</name>
<value>hdfs/[email protected]</value>
</property>
 
<property>
<name>mapreduce.jobhistory.webapp.spnego-keytab-file</name>
<value>/export/common/kerberos5/HTTP.keytab</value>
</property>
 
<property>
<name>mapreduce.jobhistory.webapp.spnego-principal</name>
<value>HTTP/[email protected]</value>
</property>
 
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49:88</value>
</property>
 
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx3276m -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=HADOOP.COM -Djava.security.krb5.kdc=192.168.0.49:88</value>
</property>

3.hive add the following configuration
hive-site.xml

<!--hiveserver2-->
<property>
<name>hive.server2.authentication</name>
<value>KERBEROS</value>
</property>

<property>
<name>hive.server2.authentication.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>

<property>
<name>hive.server2.authentication.kerberos.keytab</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>

<!-- metastore -->

<property>
<name>hive.metastore.sasl.enabled</name>
<value>true</value>
</property>

<property>
<name>hive.metastore.kerberos.keytab.file</name>
<value>/export/common/kerberos5/hdfs.keytab</value>
</property>

<property>
<name>hive.metastore.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>

4.hbase add the following configuration 
hbase-site.xml

<!-- hbase配置kerberos安全认证start -->
 
    <property>
        <name>hbase.security.authentication</name>
        <value>kerberos</value>
    </property>
    <!-- 配置hbase rpc安全通信 -->
 
    <property>
        <name>hbase.rpc.engine</name>
        <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
    </property>
    <!-- hmaster配置kerberos安全凭据认证 -->
 
    <property>
        <name>hbase.master.kerberos.principal</name>
        <value>hdfs/[email protected]</value>
    </property> -->
    <!-- hmaster配置kerberos安全证书keytab文件位置 -->
 
    <property>
        <name>hbase.master.keytab.file</name>
        <value>/export/common/kerberos5/hdfs.keytab</value>
    </property> -->
    <!-- regionserver配置kerberos安全凭据认证 -->
 
    <property>
        <name>hbase.regionserver.kerberos.principal</name>
        <value>hdfs/[email protected]</value>
    </property> -->
    <!-- regionserver配置kerberos安全证书keytab文件位置 -->
 
    <property>
        <name>hbase.regionserver.keytab.file</name>
        <value>/export/common/kerberos5/hdfs.keytab</value>
    </property>
 
<!--
    <property>
        <name>hbase.thrift.keytab.file</name>
        <value>/soft/conf/hadoop/hdfs.keytab</value>
    </property>

     <property>
         <name>hbase.thrift.kerberos.principal</name>
         <value>hdfs/[email protected]</value>
     </property>

     <property>
         <name>hbase.rest.keytab.file</name>
         <value>/soft/conf/hadoop/hdfs.keytab</value>
     </property>

     <property>
         <name>hbase.rest.kerberos.principal</name>
         <value>hdfs/[email protected]</value>
     </property>

     <property>
         <name>hbase.rest.authentication.type</name>
         <value>kerberos</value>
     </property>

     <property>
         <name>hbase.rest.authentication.kerberos.principal</name>
         <value>HTTP/[email protected]</value>
     </property>

     <property>
         <name>hbase.rest.authentication.kerberos.keytab</name>
         <value>/soft/conf/hadoop/HTTP.keytab</value>
     </property>
-->
<!-- hbase配置kerberos安全认证end -->

 

3. Kerberos related commands
: Exit authorization: kdestroy

The main kdc opens kadmin management: kadmin.local

View the Kerberos accounts currently used by the system: klist

Use keytab to get user credentials:

kinit -kt /export/common/kerberos5/kadm5.keytab admin/[email protected]

View keytab content:

klist -k -e /export/common/kerberos5/hdfs.keytab

Generate keytab file:

kadmin.local -q "xst -k /export/common/kerberos5/hdfs.keytab admin/[email protected]"

Extend the Kerberos authentication time: kinit -R

Delete the kdc database: rm -rf /export/common/kerberos5/principal (this path is the newly created database path when create)

4. Quick test
Test hdfs: switch to hdfs user, type the command: hdfs dfs -ls /, authentication is required. Type the command again

“kinit -kt /export/common/kerberos5/hdfs.keytab hdfs/`hostname | awk '{print tolower($0)}'`”

After you can check the result, it means the integration with hdfs is successful.

Also note some significant changes: 1. Tasks can be started and run by the operating system account of the user who submitted the job, and not necessarily by the user who runs the node management. This means that the operating system can be used to isolate running tasks so that they cannot send instructions to each other. In this way, the privacy of local information such as task data can be protected through the security of the local file system.

(Need to set yarn.nodemanager.container-executor.class to org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.)