Big data and rights management -Hue case ldap configuration and operation (5)

A, LDAP configuration and CDH

1) LDAP Hive integration and
2) LDAP Hive and test
3) LDAP integration and impala
4) LDAP and impala test
5) LDAP integration and HUE (search binding)
   A) arranged Hue
   B) Create ldap group
   c ) create ldap user
   d) shows the foreground group
   e) the front desk to show users

Two, Ladp operation using

1) Basic operation
2) Ldap establish user groups and establishing

Three, Error

1) Ldap Start Error
2) ldapmodify execution error
3) Ldap and hive, impala integrated error
4) Hue connection hive and impala error
5) Hue add users
6) Hue user can not synchronize
7) Hue group can not synchronize
8) HDFS Error
9) Impala error

Four, Hue metadata table described

1) auth_group group name table
2) auth_user user table
3) auth_user_groups user and group mapping table
4) useradmin_grouppermission hue group permission table
5) useradmin_huepermission hue permission Table
6) useradmin_ldapgroup ldap group associations and hue group Table
7) useradmin_userprofile user hdfs path table information

Fifth, see the rest of the document

A, LDAP configuration and CDH

1) LDAP integration and Hive
http://cxy7.com/articles/2018/07/17/1531808242972.html

ldap://silentwolfyh.com
ou=people,dc=silentwolfyh,dc=com
hadoop.proxyuser.hue.hosts  (hue是代理用户)
hadoop.proxyuser.hue.groups (hue是代理用户)

Here Insert Picture Description Account test, password 123456

[yuhui@hadoop11 ~]$ beeline -u "jdbc:hive2://hadoop11:10000" -n yuhui1 -p 123456 -d org.apache.hive.jdbc.HiveDriver

Here Insert Picture Description
2) LDAP and Hive test

!connect jdbc:hive2://hadoop11:10000/
chenxiaolong34
123456

Here Insert Picture Description
3) LDAP integration and impala
http://cxy7.com/articles/2018/07/17/1531827540376.html (focus)

enable_ldap_auth
ldap_uri=ldap://silentwolfyh.com:389
LDAP BaseDN=ou=people,dc=silentwolfyh,dc=com
--ldap_passwords_in_clear_ok=true
--authorized_proxy_user_config=hue=*
hadoop.proxyuser.hue.hosts （hue代理用户）
hadoop.proxyuser.hue.groups（hue代理用户）

Here Insert Picture Description
4) LDAP and impala test

impala-shell -l -u chenxiaolong34 --auth_creds_ok_in_clear

Here Insert Picture Description
5) LDAP integration and HUE (search bindings)

https://blog.csdn.net/u011026329/article/details/79171890

a) hue configuration
interface configuration as follows, the picture is as follows:

backend=desktop.auth.backend.LdapBackend
ldap_url= ldap://172.25.33.131:389
ldap_cert= /path_to_certificate/cert.pem
use_start_tls= false
ldap_username_pattern="uid=<username>,ou=people,dc=silentwolfyh,dc=com "
使用搜索绑定身份验证
search_bind_authentication = true
create_users_on_login=true
LDAP 搜索基础
base_dn="dc=silentwolfyh,dc=com "
LDAP 绑定用户可分辨名称
bind_dn="uid=hue,ou=people,dc=silentwolfyh,dc=com "
LDAP 绑定密码
bind_password=hue

Here Insert Picture Description

LDAP 用户名属性(LDAP Username Attribute)：user_name_attr = uid 
LDAP 用户筛选（LDAP User Filter）：user_filter = (objectClass=posixAccount)
LDAP 组筛选（LDAP Group Filter：group_filter = (|(objectClass=organizationalUnit)(objectClass=posixGroup)) 
LDAP 组名称属性（LDAP Group Name Attribute）：group_name_attr = cn 
LDAP 组成员身份属性（LDAP Group Membership Attribute）：group_member_attr = memberUID # 非常重要，否则无法识别映射

Here Insert Picture Description hue_safety_valve.ini

[desktop]
[[ldap]]
  sync_groups_on_login=true

Here Insert Picture Description

b) Create a group ldap

useradd songxitang
sudo groupadd silentwolfyh_ops
usermod -g silentwolfyh_ops songxitang
passwd songxitang (123456)

[root@hadoop11 ~]# id songxitang
uid=810(songxitang) gid=811(silentwolfyh_ops) groups=811(silentwolfyh_ops)

[root@ hadoop11 ~]# cat  /tmp/create_group_silentwolfyh_ops.ldif
dn: cn=silentwolfyh_ops,ou=silentwolfyh,dc=silentwolfyh,dc=com
objectClass: top
objectClass: posixGroup
gidNumber: 810
memberUID: ${MEMBER_UID}
description: ${DESCRIPTION_STIRNG}

ldapadd -x -D "uid=hue,ou=people,dc=silentwolfyh,dc=com" -w hue -f /tmp/create_group_silentwolfyh_ops.ldif

Here Insert Picture Description c) Create a user ldap

[root@hadoop11 ~]# cat /tmp/create_user_songxitang.ldif
dn: uid=songxitang,cn=silentwolfyh_ops,ou=silentwolfyh,dc=silentwolfyh,dc=com
objectClass: top
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
givenName: xitang
sn: song
cn: songxitang
mail: [email protected]
telephoneNumber: 0101-3333333
mobile: 13111111111
uid: songxitang
uidNumber: 1005
gidNumber: 1006
homeDirectory: /home/songxitang
loginShell: /bin/bash

ldapadd -x -D "uid=hue,ou=people,dc=silentwolfyh,dc=com" -w hue -f /tmp/create_user_songxitang.ldif

Here Insert Picture Description Change password

ldappasswd -x -D "uid=hue,ou=people,dc=silentwolfyh,dc=com" -w hue "uid=songxitang,cn=silentwolfyh_ops,ou=silentwolfyh,dc=example,dc=com" -S

d) show front group
Here Insert Picture Description e) show front user

Two, Ladp operation using

1) Basic operation
https://blog.csdn.net/u011026329/article/details/79171996
https://www.zhukun.net/archives/7980

Add files ldap

ldapadd -x -D "uid=hue,ou=people,dc=javachen,dc=com" -w hue -f xxx.ldif

Modify ldap password

ldappasswd -x -D "uid=hue,ou=people,dc=javachen,dc=com" -w hue "uid=people1,cn=silentwolfyh_ops,ou=silentwolfyh,dc=javachen,dc=com" -S

Inquire

ldapsearch -x -D "uid=hue,ou=people,dc=javachen,dc=com" -w hue -b "uid=user1,cn=silentwolfyh_ops,ou=silentwolfyh,dc=javachen,dc=com"

Delete: The user or group entry

$ ldapdelete -x -w root -D 'uid=ldapadmin,ou=people,dc=example,dc=com' "uid=test,ou=people,dc=example,dc=com" 

$ ldapdelete -x -w root -D 'uid=ldapadmin,ou=people,dc=example,dc=com' "cn=test,ou=group,dc=example,dc=com"

2) Ldap establish user groups and establish
establishing organizational units: organizationalUnit

dn: ou=language,dc=javachen,dc=com
objectClass: organizationalUnit
ou: language

Establish groups: group established ml in silentwolfyh department organizationalUnit below

dn: cn=ml,ou=silentwolfyh,dc=javachen,dc=com
objectClass: top
objectClass: posixGroup
cn: ml
gidNumber: 823
description: 机器学习

1,2,3 establish user (multiple users can be built into a ldif file, batch add users)

dn: uid=people1,cn=ml,ou=silentwolfyh,dc=javachen,dc=com
objectClass: top
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
sn:people1
cn:people1
uid: people1
uidNumber: 817
gidNumber: 823
homeDirectory: /home/people1
loginShell: /bin/bash

# User and group mapping add_user_to_group.ldif

dn: cn=ml,ou=silentwolfyh,dc=javachen,dc=com
changetype: modify
add: memberUid
memberUid: people1

Here Insert Picture Description

Three, Error

1) Ldap startup error
to start the LDAP service:

chkconfig --add slapd
chkconfig --level 345 slapd on

/etc/init.d/slapd start

Authorize

chown -R ldap.ldap /var/lib/ldap/

Here Insert Picture Description
2) ldapmodify execution error

[yuhui@hadoop11 ldapshell]# ldapmodify -Y EXTERNAL -H ldapi:/// -f m1.ldif
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "olcDatabase={2}bdb,cn=config"
ldap_modify: Inappropriate matching (18)
	additional info: modify/add: olcRootPW: no equality matching rule

Solution:
Modify modify.ldif corresponding options "add" to "replace" to

3) Ldap and hive, impala integrated error
https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hiveserver2_security.html#concept_vxf_pgx_nm

User 'hue' is not authorized to delegate to ‘hue’

solve:

hadoop.proxyuser.hue.hosts
hadoop.proxyuser.hue.groups

Here Insert Picture Description
4) Hue connection error hive and impala

https://stackoverflow.com/questions/51854958/user-impersonation-for-access-to-hive
http://cxy7.com/articles/2018/07/17/1531827540376.html

Only configure the hive and ladp. hue error
Here Insert Picture Description

TSocket read 0 bytes
TSocket read 0 bytes (code THRIFTTRANSPORT): TTransportException('TSocket read 0 bytes',)
Bad status: 3 (Error validating the login)
Bad status: 3 (Error validating the login) (code THRIFTTRANSPORT): TTransportException('Bad status: 3 (Error validating the login)',)

10:10:25 上午	WARNING	thrift_util	Out of retries for thrift call: GetSchemas
10:10:25 上午	INFO	thrift_util	Thrift saw a transport exception: TSocket read 0 bytes

Failed to validate proxy privilege of ldapadmin for yuhui1
 
Bad status for request TOpenSessionReq(username=u'yuhui1', password=None, client_protocol=6, configuration={'hive.server2.proxy.user': u'yuhui1'}): TOpenSessionResp(status=TStatus(errorCode=0, errorMessage='Failed to validate proxy privilege of ldapadmin for yuhui1', sqlState='08S01', infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Failed to validate proxy privilege of ldapadmin for yuhui1:14:13', 'org.apache.hive.service.auth.HiveAuthFactory:verifyProxyAccess:HiveAuthFactory.java:326', 'org.apache.hive.service.cli.thrift.ThriftCLIService:getProxyUser:ThriftCLIService.java:753', 'org.apache.hive.service.cli.thrift.ThriftCLIService:getUserName:ThriftCLIService.java:384', 'org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:411', 'org.apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:316', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1253', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1238', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617', 'java.lang.Thread:run:Thread.java:745', '*org.apache.hadoop.security.authorize.AuthorizationException:User: ldapadmin is not allowed to impersonate yuhui1:0:-1'], statusCode=3), sessionHandle=None, configuration=None, serverProtocolVersion=6)

Solve: hue modified as follows
hue_safety_valve_server.ini of Hue Server Advanced Configuration code segment (safety valve)
Here Insert Picture Description

[impala]
server_host=hadoop11
server_interface=hiveserver2
server_port=21050
query_timeout_s=100
impersonation_enabled=True
auth_username=hue
auth_password=hue

[beeswax]
close_queries=True
use_sasl=False
auth_username=hue
auth_password=hue

5) Hue add users

Could not get LDAP details for users in pattern yuhui1.

Here Insert Picture Description

Could not get LDAP details for users with pattern 123

Failed to find LDAP user: Distinguished Name provided does not contain configured Base DN. Base DN: ou=people,dc=silentwolfyh,dc=com, DN: bjjizhou
Could not get LDAP details for users with pattern bjjizhou

Solution: You can only add users in ldap
Here Insert Picture Description
6) Hue Users can not synchronize
https://community.cloudera.com/t5/Web-UI-Hue-Beeswax/Failed-to-validate-proxy-privilege-of-hue-hive -for / td-p / 49686

Could not create home directory at login for hue.

Hue will configure all proxy users to other users and groups, which means that it can represent any other user submits a request. Add core site.xml:

hadoop.proxyuser.hue.groups
hadoop.proxyuser.hue.hosts

Here Insert Picture Description 7) Hue can not synchronize group

Here Insert Picture Description Could not get LDAP details for group pattern abc

Synchronization ldap group can only be in the group, to see through phpLDAPadmin interface

8) HDFS error

Resetting dropped connection: YZ-25-33-141.h.chinabank.com.cn 50070
Potential detail: StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error (error 403)

Resolution: 141 hdfs will become active

9) Impala error

LDAP authentication specified, but without TLS. Passwords would go over the network in the clear. Enable TLS with --ldap_tls or use an ldaps:// URI. To override this is non-production environments, specify --ldap_passwords_in_clear_ok
. Impalad exiting.
Wrote minidump to /var/log/impala-minidumps/impalad/e50eb300-de19-4486-380a6c8f-0b3aa37c.dmp

solve:

Impala 命令行参数高级配置代码段（安全阀）
--ldap_passwords_in_clear_ok=true
--auth_creds_ok_in_clear
--authorized_proxy_user_config=hue=*

Here Insert Picture Description

Four, Hue metadata table described

1) auth_group group name table
Note: Table hue when creating groups in the name of the new group
Here Insert Picture Description

2) auth_user user table
Note: create a user login or hue hue created when the user
Here Insert Picture Description 3) auth_user_groups user and group mapping table
Note: when the user logs in and automatically associating group
4) useradmin_grouppermission hue group permission table
5) useradmin_huepermission hue permission table

Here Insert Picture Description 6) useradmin_ldapgroup ldap group and hue Association Table
NOTE: Group Name ldap query appear in this table
7) useradmin_userprofile user hdfs path information table