Official documents:
Compile Ranger source code
First prepare the Java and Maven environment:
[root@hadoop01 ~]# java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
[root@hadoop01 ~]# mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/maven
Java version: 1.8.0_261, vendor: Oracle Corporation, runtime: /usr/local/jdk/1.8/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix"
[root@hadoop01 ~]# - Tips: Maven needs to configure the domestic mirror source, otherwise the dependency will not be downloaded the next day
Pull the source code of Ranger directly from GitHub, or download the corresponding version of the source code package from the official website :
[root@hadoop01 ~]# cd /usr/local/src
[root@hadoop01 /usr/local/src]# git clone https://github.com/apache/rangerEnter the source directory:, cd rangermodify the pomfiles in this directory , mainly modify two places, the first is to comment out the warehouse related configuration:
<!--
<repositories>
<repository>
<id>apache.snapshots.https</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/snapshots</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>apache.public.https</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>repo</id>
<url>file://${basedir}/local-repo</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
-->The second is to modify the version of Hadoop related components, change the version to the version you installed:
<hadoop.version>3.3.0</hadoop.version>
<hbase.version>2.2.6</hbase.version>
<hive.version>3.1.2</hive.version>- Tips: If you need to use Ranger to manage other components, you also need to find its version definition and modify it to a consistent version to avoid version incompatibility during integration.
Then modify the security-admin/pom.xmlconfiguration related to nodejs in the file. Search within the file <id>install node and npm</id>, the configurationcontents of the label amended as follows:
<configuration>
<nodeVersion>v10.13.0</nodeVersion>
<!--<npmVersion>6.4.1</npmVersion>-->
</configuration>Then continue to search within the file <id>npm install for packaging</id>, the configurationcontents of the label amended as follows:
<configuration>
<workingDirectory>${project.build.directory}/jsmain</workingDirectory>
<arguments>install -registry=https://registry.npm.taobao.org --cache-max=0 --no-save</arguments>
</configuration>Continue to search within the file <id>npm install for tests</id>, the configurationcontents of the label amended as follows:
<configuration>
<skip>${skipJSTests}</skip>
<workingDirectory>${project.build.directory}/jstest</workingDirectory>
<arguments>install -registry=https://registry.npm.taobao.org --cache-max=0 --no-save</arguments>
</configuration>After completing the above modifications, use the maven command to compile and package:
[root@hadoop01 /usr/local/src]# cd ranger/
[root@hadoop01 /usr/local/src/ranger]# mvn -DskipTests=true clean packageAfter a long wait, the following information will be output when the compilation and packaging are completed:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 3.0.0-SNAPSHOT:
[INFO]
[INFO] ranger ............................................. SUCCESS [ 0.763 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [ 0.903 s]
[INFO] Credential Support ................................. SUCCESS [ 35.119 s]
[INFO] Audit Component .................................... SUCCESS [ 24.206 s]
[INFO] ranger-plugin-classloader .......................... SUCCESS [ 0.823 s]
[INFO] Common library for Plugins ......................... SUCCESS [ 4.650 s]
[INFO] ranger-intg ........................................ SUCCESS [ 1.672 s]
[INFO] Installer Support Component ........................ SUCCESS [ 0.494 s]
[INFO] Credential Builder ................................. SUCCESS [ 2.276 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 4.777 s]
[INFO] Key Management Service ............................. SUCCESS [ 27.430 s]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [01:47 min]
[INFO] HBase Security Plugin .............................. SUCCESS [ 25.536 s]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 13.548 s]
[INFO] Hive Security Plugin ............................... SUCCESS [01:41 min]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [ 12.290 s]
[INFO] Knox Security Plugin ............................... SUCCESS [02:12 min]
[INFO] Storm Security Plugin .............................. SUCCESS [ 3.999 s]
[INFO] YARN Security Plugin ............................... SUCCESS [ 1.452 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 16.509 s]
[INFO] Ranger Util ........................................ SUCCESS [ 1.000 s]
[INFO] Unix Authentication Client ......................... SUCCESS [ 0.590 s]
[INFO] User Group Synchronizer Util ....................... SUCCESS [ 0.457 s]
[INFO] Security Admin Web Application ..................... SUCCESS [01:15 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [ 13.393 s]
[INFO] SOLR Security Plugin ............................... SUCCESS [ 19.696 s]
[INFO] NiFi Security Plugin ............................... SUCCESS [ 1.556 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 1.586 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [ 0.809 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [ 34.854 s]
[INFO] Ldap Config Check Tool ............................. SUCCESS [ 0.643 s]
[INFO] Unix Authentication Service ........................ SUCCESS [ 0.917 s]
[INFO] Unix Native Authenticator .......................... SUCCESS [ 0.475 s]
[INFO] KMS Security Plugin ................................ SUCCESS [ 7.668 s]
[INFO] Tag Synchronizer ................................... SUCCESS [02:24 min]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [ 0.906 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [ 5.423 s]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 0.914 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 0.944 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [ 0.961 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 0.881 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 1.096 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 12.065 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 7.139 s]
[INFO] ranger-examples .................................... SUCCESS [ 0.017 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 1.479 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [ 0.384 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 0.831 s]
[INFO] sample-client ...................................... SUCCESS [ 0.865 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [ 1.262 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 3.747 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 1.149 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [ 32.873 s]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 6.273 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 0.810 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [03:13 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [ 9.244 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 21.863 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [01:42 min]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 3.510 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 1.047 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [03:07 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------At this time target, you can see the packaged plug-in installation package in the directory:
[root@hadoop01 /usr/local/src/ranger]# ls target/
antrun ranger-3.0.0-SNAPSHOT-hive-plugin.tar.gz ranger-3.0.0-SNAPSHOT-presto-plugin.tar.gz ranger-3.0.0-SNAPSHOT-storm-plugin.tar.gz
maven-shared-archive-resources ranger-3.0.0-SNAPSHOT-kafka-plugin.tar.gz ranger-3.0.0-SNAPSHOT-ranger-tools.tar.gz ranger-3.0.0-SNAPSHOT-tagsync.tar.gz
ranger-3.0.0-SNAPSHOT-admin.tar.gz ranger-3.0.0-SNAPSHOT-kms.tar.gz ranger-3.0.0-SNAPSHOT-schema-registry-plugin.jar ranger-3.0.0-SNAPSHOT-usersync.tar.gz
ranger-3.0.0-SNAPSHOT-atlas-plugin.tar.gz ranger-3.0.0-SNAPSHOT-knox-plugin.tar.gz ranger-3.0.0-SNAPSHOT-solr_audit_conf.tar.gz ranger-3.0.0-SNAPSHOT-yarn-plugin.tar.gz
ranger-3.0.0-SNAPSHOT-elasticsearch-plugin.tar.gz ranger-3.0.0-SNAPSHOT-kylin-plugin.tar.gz ranger-3.0.0-SNAPSHOT-solr-plugin.tar.gz version
ranger-3.0.0-SNAPSHOT-hbase-plugin.tar.gz ranger-3.0.0-SNAPSHOT-migration-util.tar.gz ranger-3.0.0-SNAPSHOT-sqoop-plugin.tar.gz
ranger-3.0.0-SNAPSHOT-hdfs-plugin.tar.gz ranger-3.0.0-SNAPSHOT-ozone-plugin.tar.gz ranger-3.0.0-SNAPSHOT-src.tar.gz
[root@hadoop01 /usr/local/src/ranger]# Deploy Ranger Admin
Unzip the ranger admin installation package to a suitable directory. I am used to putting it here /usr/local:
[root@hadoop01 /usr/local/src/ranger]# tar -zxvf target/ranger-3.0.0-SNAPSHOT-admin.tar.gz -C /usr/local/Enter the decompressed directory, the directory structure is as follows:
[root@hadoop01 /usr/local/src/ranger]# cd /usr/local/ranger-3.0.0-SNAPSHOT-admin/
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# ls
bin contrib dba_script.py ews ranger_credential_helper.py set_globals.sh templates-upgrade upgrade_admin.py
changepasswordutil.py cred db_setup.py install.properties restrict_permissions.py setup_authentication.sh update_property.py upgrade.sh
changeusernameutil.py db deleteUserGroupUtil.py jisql rolebasedusersearchutil.py setup.sh updateUserAndGroupNamesInJson.py version
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# Configure installation options:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# vim install.properties
# 指定MySQL驱动包所在的路径
SQL_CONNECTOR_JAR=/usr/local/src/mysql-connector-java-8.0.21.jar
# 配置root用户名密码以及MySQL实例的连接地址
db_root_user=root
db_root_password=123456a.
db_host=192.168.1.11
# 配置操作ranger库的用户名密码
db_name=ranger
db_user=root
db_password=123456a.
# 指定审计日志的存储方式
audit_store=db
audit_db_user=root
audit_db_name=ranger
audit_db_password=123456a.Create a ranger database in MySQL:
create database ranger;Since I am using MySQL 8.x here, I need to modify the database-related scripts. Open dba_script.pyand db_setup.pyfile, search for the following:
-cstring jdbc:mysql://%s/%s%sModify it all as follows, mainly adding JDBC serverTimezoneconnection parameters:
-cstring jdbc:mysql://%s/%s%s?serverTimezone=Asia/ShanghaiThen execute the following command to start installing ranger admin:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# ./setup.shError resolution
If the following error is reported during installation:
SQLException : SQL state: HY000 java.sql.SQLException: Operation CREATE USER failed for 'root'@'localhost' ErrorCode: 1396
SQLException : SQL state: 42000 java.sql.SQLSyntaxErrorException: Access denied for user 'root'@'192.168.1.11' to database 'mysql' ErrorCode: 1044The solution is to execute the following statement in MySQL:
flush privileges;
grant system_user on *.* to 'root';
drop user'root'@'localhost';
create user 'root'@'localhost' identified by '123456a.';
grant all privileges on *.* to 'root'@'localhost' with grant option;
drop user'root'@'192.168.1.11';
create user 'root'@'192.168.1.11' identified by '123456a.';
grant all privileges on *.* to 'root'@'192.168.1.11' with grant option;
flush privileges;If the following error is reported:
SQLException : SQL state: HY000 java.sql.SQLException: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable) ErrorCode: 1418Solution:
set global log_bin_trust_function_creators=TRUE;
flush privileges;If the following error is reported:
SQLException : SQL state: HY000 java.sql.SQLException: Cannot drop table 'x_policy' referenced by a foreign key constraint 'x_policy_ref_role_FK_policy_id' on table 'x_policy_ref_role'. ErrorCode: 3730Solution: delete all the tables in the ranger library and execute again./setup.sh
After the installation is complete, it will finally output:
Installation of Ranger PolicyManager Web Application is completed.Start Ranger Admin
Modify the configuration file, configure the database connection password and jdbc url time zone parameters:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# vim conf/ranger-admin-site.xml
...
<property>
<name>ranger.jpa.jdbc.url</name>
<value>jdbc:log4jdbc:mysql://192.168.1.11/ranger?serverTimezone=Asia/Shanghai</value>
<description />
</property>
<property>
<name>ranger.jpa.jdbc.password</name>
<value>123456a.</value>
<description />
</property>
...The startup command is as follows:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# ranger-admin start
Starting Apache Ranger Admin Service
Apache Ranger Admin Service failed to start!
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]#You can see that the startup failed. For the specific reason, you need to check the log information. The log directory of ranger admin is configured in the conf/ranger-admin-env-logdir.shfile, the default is $RANGER_ADMIN_HOME/ews/logs/. Check the log file and learn that the key error messages are as follows:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# cat /usr/local/ranger-3.0.0-SNAPSHOT-admin/ews/logs/catalina.out
...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/compress/archivers/tar/TarArchiveInputStream
...Obviously, TarArchiveInputStreamthis class cannot be found . This class is in the Apache commons-compresspackage, and it is easy to solve. First, download the jar package from the central warehouse:
Then put it in the ews/lib/directory:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# ls ews/lib/ |grep commons-compress
commons-compress-1.20.jar
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# Restart ranger admin, this time it starts successfully:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# ranger-admin start
Starting Apache Ranger Admin Service
Apache Ranger Admin Service with pid 52505 has started.
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# Check whether the port and process are normal:
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# jps
52626 Jps
52505 EmbeddedServer
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# netstat -lntp |grep 52505
tcp6 0 0 :::6080 :::* LISTEN 52505/java
tcp6 0 0 127.0.0.1:6085 :::* LISTEN 52505/java
[root@hadoop01 /usr/local/ranger-3.0.0-SNAPSHOT-admin]# Use a browser to access port 6080 and enter the login page. The default user name and password are both admin :
After successfully logging in, enter the home page as follows:
Ranger HDFS Plugin installation
Unzip the hdfs plugin installation package to a suitable directory:
[root@hadoop01 ~]# mkdir /usr/local/ranger-plugin
[root@hadoop01 ~]# tar -zxvf /usr/local/src/ranger/target/ranger-3.0.0-SNAPSHOT-hdfs-plugin.tar.gz -C /usr/local/ranger-plugin
[root@hadoop01 ~]# cd /usr/local/ranger-plugin/
[root@hadoop01 /usr/local/ranger-plugin]# mv ranger-3.0.0-SNAPSHOT-hdfs-plugin/ hdfs-pluginEnter the decompressed directory, the directory structure is as follows:
[root@hadoop01 /usr/local/ranger-plugin/hdfs-plugin]# ls
disable-hdfs-plugin.sh enable-hdfs-plugin.sh install install.properties lib ranger_credential_helper.py upgrade-hdfs-plugin.sh upgrade-plugin.py
[root@hadoop01 /usr/local/ranger-plugin/hdfs-plugin]# Configure installation options:
[root@hadoop01 /usr/local/ranger-plugin/hdfs-plugin]# vim install.properties
# 指定ranger admin服务的访问地址
POLICY_MGR_URL=http://192.168.243.142:6080
# 配置仓库配置，可自定义
REPOSITORY_NAME=dev_hdfs
# 配置hadoop的安装目录
COMPONENT_INSTALL_DIR_NAME=/usr/local/hadoop-2.6.0-cdh5.16.2
# 配置hdfs的目录
XAAUDIT.HDFS.HDFS_DIR=hdfs://__REPLACE__NAME_NODE_HOST:8020/ranger/audit
XAAUDIT.HDFS.DESTINATION_DIRECTORY=hdfs://__REPLACE__NAME_NODE_HOST:8020/ranger/audit/%app-type%/%time:yyyyMMdd%
# 配置用户和用户组
CUSTOM_USER=root
CUSTOM_GROUP=rootExecute the following script to open hdfs-plugin :
[root@hadoop01 /usr/local/ranger-plugin/hdfs-plugin]# ./enable-hdfs-plugin.sh If the following error is reported:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/compress/archivers/tar/TarArchiveInputStreamCopy the missing jar package to the install/lib/directory:
[root@hadoop01 /usr/local/ranger-plugin/hdfs-plugin]# cp /usr/local/ranger-3.0.0-SNAPSHOT-admin/ews/lib/commons-lang3-3.3.2.jar ./install/lib/
[root@hadoop01 /usr/local/ranger-plugin/hdfs-plugin]# cp /usr/local/ranger-3.0.0-SNAPSHOT-admin/ews/lib/commons-compress-1.20.jar ./install/lib/After the script is successfully executed, the following content will be output:
Ranger Plugin for hadoop has been enabled. Please restart hadoop to ensure that changes are effective.Restart Hadoop:
[root@hadoop01 ~]# stop-all.sh
[root@hadoop01 ~]# start-all.shAuthentication permission control
Add hdfs service to Ranger Admin. The Service Name here needs to correspond to the configuration in the configuration file:
Fill in the corresponding information:
After filling in, go to the bottom of the page and click "Test Connection" to test whether the connection is normal, and then click "Add" to complete the addition:
Create some test directories and files in hdfs:
[root@hadoop01 ~]# hdfs dfs -mkdir /rangertest1
[root@hadoop01 ~]# hdfs dfs -mkdir /rangertest2
[root@hadoop01 ~]# echo "ranger test" > testfile
[root@hadoop01 ~]# hdfs dfs -put testfile /rangertest1
[root@hadoop01 ~]# hdfs dfs -put testfile /rangertest2Then add Ranger's internal users on Ranger Admin, "Settings" -> "Add New User", and fill in user information:
Then add the permission policy, "Access Manager" -> "dev_hdfs" -> "Add New Policy", configure the user, directory and other information for the permission policy:
Pull to the bottom and click "Add" to complete the addition, you can see that a new policy configuration has been added:
Back to the operating system, switch to the hiveuser, test whether the directory and file can be read normally:
[root@hadoop01 ~]# sudo su - hive
上一次登录：一 11月 9 21:08:34 CST 2020pts/3 上
[hive@hadoop01 ~]$ hdfs dfs -ls /rangertest1
Found 1 items
-rw-r--r-- 1 root supergroup 12 2020-11-11 16:26 /rangertest1/testfile
[hive@hadoop01 ~]$ hdfs dfs -cat /rangertest1/testfile
ranger test
[hive@hadoop01 ~]$ Test the write operation. At this time, you will find that you can rangertest1add files to the directory normally , but rangertest2adding files to the directory will report an error, because we only give the rangertest1directory read and write permissions:
[hive@hadoop01 ~]$ hdfs dfs -put testfile2 /rangertest1
[hive@hadoop01 ~]$ hdfs dfs -put testfile2 /rangertest2
put: Permission denied: user=hive, access=WRITE, inode="/rangertest2":root:supergroup:drwxr-xr-x
[hive@hadoop01 ~]$ So far, Ranger's permission control on HDFS has also been verified. In addition, you can also perform other tests. The ranger plugins of other components are similar, so I won't demonstrate them in this article.