A preliminary exploration of Ranger Api based on Feign

User Management of Ranger Api

In the article on the authority management component of the big data platform-Aapche Ranger , we learned about Ranger, the installation and deployment process, and the use of the Admin visual interface.

In addition to managing permissions and users on the visual Ranger Admin interface, Ranger also supports these operations through the REST API. Because if we want to develop our own big data platform, we may not use the visual interface of Ranger Admin, but hope to operate Ranger on our own big data platform interface. With the support of Ranger Api, we can easily achieve this.

The official documents about Ranger Api are as follows:

• https://ranger.apache.org/apidocs/index.html

This section briefly demonstrates the use of User Api. User Api is used to manage users, and to add, delete, modify and check users. First create an empty Maven project. Since APIs are requested through http, http request tools are needed. What is used here is feignthat the complete pomfile content is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>ranger-client</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.4</version>
        </dependency>
        <dependency>
            <groupId>com.netflix.feign</groupId>
            <artifactId>feign-core</artifactId>
            <version>8.18.0</version>
        </dependency>
        <dependency>
            <groupId>com.netflix.feign</groupId>
            <artifactId>feign-jackson</artifactId>
            <version>8.18.0</version>
        </dependency>
        <dependency>
            <groupId>com.netflix.feign</groupId>
            <artifactId>feign-okhttp</artifactId>
            <version>8.18.0</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.9.8</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>21.0</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.25</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

First define a configuration class to configure the username and password used to access the ranger api:

package com.example.ranger.config;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class RangerAuthConfig {
    private String username = "admin";
    private String password = "admin";
}

Define another configuration class to provide http client configuration:

package com.example.ranger.config;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import feign.Logger;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class RangerClientConfig {
    private int connectionTimeoutMills = 5 * 1000;
    private int readTimeoutMills = 30 * 1000;
    private Logger.Level level = Logger.Level.BASIC;
    private String url = "http://192.168.243.161:6080";
    private RangerAuthConfig authConfig = new RangerAuthConfig();
}

Declare a request interceptor to add some request headers before initiating the request:

package com.example.ranger.interceptor;

import feign.RequestInterceptor;
import feign.RequestTemplate;

public class RangerHeadersInterceptor implements RequestInterceptor {

    @Override
    public void apply(RequestTemplate template) {
        template.header("Accept", "application/json");
        template.header("X-XSRF_HEADER", "\"\"");
        template.header("Content-Type", "application/json");
    }
}

Usually in actual development, we will define a business exception to customize the encapsulation of the exception information:

package com.example.ranger.exception;

import java.io.Serializable;

public class RangerClientException extends RuntimeException implements Serializable {
    private static final long serialVersionUID = -4441189815976639860L;
    private Throwable cause;
    private int status;
    private String message;

    public RangerClientException(int status, String message) {
        this.status = status;
        this.message = message;
    }

    @Override
    public String getMessage() {
        return String.format("%s http status = %s", message, status);
    }

    @Override
    public String toString() {
        return String.format("%s http status = %s", message, status);
    }
}

Customize an exception information parser. When an exception occurs in the request, feign will call this method to return our custom exception class:

package com.example.ranger.decoder;

import com.example.ranger.exception.RangerClientException;
import feign.Response;
import feign.Util;
import feign.codec.ErrorDecoder;

import java.io.IOException;

public class RangerErrorDecoder implements ErrorDecoder {

    @Override
    public Exception decode(String methodKey, Response response) {
        return new RangerClientException(
                response.status(), errorMessage(methodKey, response)
        );
    }

    private String errorMessage(String methodKey, Response response) {
        String msg = String.format("status %s reading %s", response.status(), methodKey);
        if (response.body() != null) {
            try {
                msg += "content:\n" + Util.toString(response.body().asReader());
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        return msg;
    }
}

After completing the feign-related pre-preparation above, we can start to write the code to request the ranger api. First, define the request and response entities of the user interface:

package com.example.ranger.model;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.List;

/**
 * 用户信息实体类
 * https://ranger.apache.org/apidocs/json_VXUser.html
 *
 * @author 01
 * @date 2020-11-12
 **/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class User {
    private int id;
    private String name;
    private String createDate;
    private String updateDate;
    private String owner;
    private String updateBy;
    private String firstName;
    private String lastName;
    private String emailAddress;
    private String password;
    private String description;
    private int status;
    private int isVisible;
    private int userSource;
    private List<String> userRoleList;
}

Define user api-related interfaces, which is the approach of Feign's declarative http client:

package com.example.ranger.api;

import com.example.ranger.model.User;
import feign.Param;
import feign.RequestLine;

/**
 * 用户相关api
 * https://ranger.apache.org/apidocs/resource_XUserREST.html
 *
 * @author 01
 * @date 2020-11-12
 **/
public interface UserFeignClient {

    /**
     * 创建用户接口
     * https://ranger.apache.org/apidocs/resource_XUserREST.html#resource_XUserREST_secureCreateXUser_POST
     *
     * @param user user
     * @return 用户信息
     */
    @RequestLine("POST /service/xusers/secure/users")
    User createUser(User user);

    /**
     * 删除用户
     * https://ranger.apache.org/apidocs/resource_XUserREST.html#resource_XUserREST_deleteSingleUserByUserId_DELETE
     *
     * @param id          用户id
     * @param forceDelete 是否强制删除
     */
    @RequestLine("DELETE /service/xusers/secure/users/id/{id}?forceDelete={forceDelete}")
    void deleteUser(@Param("id") Integer id,
                    @Param("forceDelete") boolean forceDelete);

    /**
     * 获取用户信息
     * https://ranger.apache.org/apidocs/resource_XUserREST.html#resource_XUserREST_getXUserByUserName_GET
     *
     * @param name 用户名称
     * @return 用户信息
     */
    @RequestLine("GET /service/xusers/users/userName/{name} ")
    User getUserByName(@Param("name") String name);
}

Then we wrap a layer outside this, usually we will do some additional processing in this layer, such as parameter verification, result verification and so on:

package com.example.ranger.api;

import com.example.ranger.exception.RangerClientException;
import com.example.ranger.model.User;
import lombok.AllArgsConstructor;

@AllArgsConstructor
public class UserApi {

    private final UserFeignClient userClient;

    public User createUser(User user) throws RangerClientException {
        return userClient.createUser(user);
    }

    public void deleteUser(Integer id, boolean forceDelete) {
        userClient.deleteUser(id, forceDelete);
    }

    public User getUserByName(String name) throws RangerClientException {
        return userClient.getUserByName(name);
    }
}

Finally, define a client tool class to provide a unified operation entry for external use:

package com.example.ranger;

import com.example.ranger.api.PolicyApi;
import com.example.ranger.api.PolicyFeignClient;
import com.example.ranger.api.UserApi;
import com.example.ranger.api.UserFeignClient;
import com.example.ranger.config.RangerClientConfig;
import com.example.ranger.decoder.RangerErrorDecoder;
import com.example.ranger.interceptor.RangerHeadersInterceptor;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import feign.Feign;
import feign.Logger;
import feign.Request;
import feign.auth.BasicAuthRequestInterceptor;
import feign.jackson.JacksonDecoder;
import feign.jackson.JacksonEncoder;
import feign.okhttp.OkHttpClient;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.concurrent.atomic.AtomicBoolean;

@Slf4j
public class RangerClient {

    @Getter
    private UserApi userApi;

    @Getter
    private PolicyApi policyApi;

    private final RangerClientConfig rangerClientConfig;

    public RangerClient(RangerClientConfig rangerClientConfig) {
        this.rangerClientConfig = rangerClientConfig;
    }

    private static final ObjectMapper MAPPER = new ObjectMapper()
            .setSerializationInclusion(JsonInclude.Include.NON_NULL)
            .configure(SerializationFeature.INDENT_OUTPUT, true)
            .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    private static final JacksonEncoder ENCODER = new JacksonEncoder(MAPPER);
    private static final JacksonDecoder DECODER = new JacksonDecoder(MAPPER);

    /**
     * 标识client是否已启动
     */
    private final AtomicBoolean started = new AtomicBoolean(false);

    /**
     * 配置client的构建信息
     *
     * @return {@link Feign.Builder}
     */
    private Feign.Builder feignBuilder() {
        return Feign.builder()
                .logger(new Logger.JavaLogger())
                .logLevel(rangerClientConfig.getLevel())
                .options(new Request.Options(
                        rangerClientConfig.getConnectionTimeoutMills(),
                        rangerClientConfig.getReadTimeoutMills()
                )).encoder(ENCODER).decoder(DECODER)
                .client(new OkHttpClient())
                .errorDecoder(new RangerErrorDecoder())
                .requestInterceptor(new RangerHeadersInterceptor())
                .requestInterceptor(new BasicAuthRequestInterceptor(
                        rangerClientConfig.getAuthConfig().getUsername(),
                        rangerClientConfig.getAuthConfig().getPassword()
                ));
    }

    /**
     * 启动client
     */
    public void start() {
        if (started.get()) {
            log.info("ranger client is already started");
            return;
        }

        userApi = new UserApi(feignBuilder().target(
                UserFeignClient.class, rangerClientConfig.getUrl()
        ));
        policyApi = new PolicyApi(feignBuilder().target(
                PolicyFeignClient.class, rangerClientConfig.getUrl()
        ));
        started.set(true);
    }

    /**
     * 停止client
     */
    public void stop() {
        if (started.get()) {
            started.set(false);
        } else {
            log.info("ranger client is not started");
        }
    }
}

After completing the above function code, let's write some unit tests to verify whether the functions are implemented correctly:

package com.example.ranger.api;

import com.example.ranger.RangerClient;
import com.example.ranger.config.RangerClientConfig;
import com.example.ranger.model.User;
import org.junit.Before;
import org.junit.Test;

import java.util.Collections;

import static org.junit.Assert.assertNotNull;

public class UserApiTest {

    private static RangerClient rangerClient;

    @Before
    public void initRangerClient() {
        rangerClient = new RangerClient(new RangerClientConfig());
        rangerClient.start();
    }

    @Test
    public void testCreateUser() {
        User user = User.builder().name("test")
                .firstName("first").lastName("last").password("user@123")
                .isVisible(1).status(1).userSource(0)
                .userRoleList(Collections.singletonList("ROLE_USER"))
                .build();

        User result = rangerClient.getUserApi().createUser(user);
        assertNotNull(result);
        System.out.println(result);
    }

    @Test
    public void testDeleteUser() {
        User result = rangerClient.getUserApi().getUserByName("test");
        assertNotNull(result);
        rangerClient.getUserApi().deleteUser(result.getId(), true);
    }

    @Test
    public void testGetUserByName() {
        User result = rangerClient.getUserApi().getUserByName("test");
        assertNotNull(result);
        System.out.println(result);
    }
}

Run testCreateUserthis unit test, and then go to ranger admin to see if there are any new corresponding users:

Then run testDeleteUserthis unit test to see if the user can be deleted normally:

---

Policy management of Ranger Api

This section will introduce the use of Policy Api to manage permission policies on Ranger. First define the request/response entity class of the interface. Since the Policy is a little more complicated, there are more classes that need to be defined:

/**
 * 策略所作用的资源，即hdfs目录、hive的库/表/列等
 * https://ranger.apache.org/apidocs/json_RangerPolicyResource.html
 *
 * @author 01
 * @date 2020-11-12
 **/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class PolicyResource {
    private List<String> values = Lists.newArrayList();
    private Boolean isExcludes;
    private Boolean isRecursive;
}

/**
 * https://ranger.apache.org/apidocs/json_RangerPolicyItemCondition.html
 *
 * @author 01
 * @date 2020-11-12
 **/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class PolicyItemCondition {
    private String type;
    private List<String> value = Lists.newArrayList();
}

/**
 * 策略条件项中的权限信息，即在该项中拥有哪些权限，对应“Permissions”
 * https://ranger.apache.org/apidocs/json_RangerPolicyItemAccess.html
 *
 * @author 01
 * @date 2020-11-12
 **/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class PolicyItemAccess {
    private String type;
    private Boolean isAllowed;
}

/**
 * 策略中的条件项，对应“Allow Conditions”或“Deny Conditions”中的每一栏信息
 * https://ranger.apache.org/apidocs/json_RangerPolicyItem.html
 *
 * @author 01
 * @date 2020-11-12
 **/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class PolicyItem {
    private List<PolicyItemAccess> accesses = Lists.newArrayList();
    private Set<String> users = Sets.newHashSet();
    private List<String> groups = Lists.newArrayList();
    private List<PolicyItemCondition> conditions = Lists.newArrayList();
    private Boolean delegateAdmin;
}

/**
 * 策略实体
 * https://ranger.apache.org/apidocs/json_RangerPolicy.html
 *
 * @author 01
 * @date 2020-11-12
 **/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class Policy {
    private Map<String, PolicyResource> resources;
    private List<PolicyItem> policyItems = Lists.newArrayList();
    private List<PolicyItem> denyPolicyItems = Lists.newArrayList();
    private List<PolicyItem> allowExceptions = Lists.newArrayList();
    private List<PolicyItem> denyExceptions = Lists.newArrayList();
    private List<Object> dataMaskPolicyItems = Lists.newArrayList();
    private List<Object> rowFilterPolicyItems = Lists.newArrayList();

    private int id;
    private String guid;
    private boolean isEnabled;
    private int version;
    private String service;
    private String name;
    private int policyType;
    private String description;
    private boolean isAuditEnabled;
}

Define the API interface related to the permission policy:

package com.example.ranger.api;

import com.example.ranger.model.Policy;
import feign.Param;
import feign.RequestLine;

import java.util.List;

/**
 * 权限策略相关api
 * https://ranger.apache.org/apidocs/resource_PublicAPIsv2.html
 * https://ranger.apache.org/apidocs/resource_ServiceREST.html
 *
 * @author 01
 * @date 2020-11-12
 **/
public interface PolicyFeignClient {

    /**
     * 创建策略
     * https://ranger.apache.org/apidocs/resource_PublicAPIsv2.html#resource_PublicAPIsv2_createPolicy_POST
     *
     * @param policy 策略信息
     * @return 策略信息
     */
    @RequestLine("POST /service/public/v2/api/policy")
    Policy createPolicy(Policy policy);

    /**
     * 删除策略
     * https://ranger.apache.org/apidocs/resource_ServiceREST.html#resource_ServiceREST_deletePolicy_DELETE
     *
     * @param id 策略id
     */
    @RequestLine("DELETE /service/plugins/policies/{id}")
    void deletePolicy(@Param("id") Integer id);

    /**
     * 通过服务和策略名称获取策略信息
     * https://ranger.apache.org/apidocs/resource_PublicAPIsv2.html#resource_PublicAPIsv2_getPolicyByName_GET
     *
     * @param serviceName 服务名称
     * @param policyName  策略名称
     * @return 策略信息
     */
    @RequestLine("GET /service/public/v2/api/service/{serviceName}/policy/{policyName}")
    Policy getPolicyByName(@Param("serviceName") String serviceName,
                           @Param("policyName") String policyName);

    /**
     * 获取指定服务下的策略信息列表
     *
     * @param serviceName 服务名称
     * @return 该服务下的策略信息列表
     */
    @RequestLine("GET /service/public/v2/api/service/{serviceName}/policy")
    List<Policy> getAllPoliciesByService(@Param("serviceName") String serviceName);
}

Similarly, wrap a layer on top of the interface:

package com.example.ranger.api;

import com.example.ranger.model.Policy;
import lombok.AllArgsConstructor;

import java.util.List;

@AllArgsConstructor
public class PolicyApi {

    private final PolicyFeignClient policyFeignClient;

    public Policy getPolicyByName(String serviceName, String policyName)  {
        return policyFeignClient.getPolicyByName(serviceName, policyName);
    }

    public List<Policy> getAllPoliciesByService(String serviceName)  {
        return policyFeignClient.getAllPoliciesByService(serviceName);
    }

    public Policy createPolicy(Policy policy)  {
        return policyFeignClient.createPolicy(policy);
    }

    public void deletePolicy(Integer id) {
        policyFeignClient.deletePolicy(id);
    }
}

Modify RangerClientand add PolicyApirelated code:

@Slf4j
public class RangerClient {

    @Getter
    private PolicyApi policyApi;

    ...

    /**
     * 启动client
     */
    public void start() {
        if (started.get()) {
            log.info("ranger client is already started");
            return;
        }

        userApi = new UserApi(feignBuilder().target(
                UserFeignClient.class, rangerClientConfig.getUrl()
        ));
        policyApi = new PolicyApi(feignBuilder().target(
                PolicyFeignClient.class, rangerClientConfig.getUrl()
        ));
        started.set(true);
    }

    ...
}

Write unit tests:

package com.example.ranger.api;

import com.example.ranger.RangerClient;
import com.example.ranger.config.RangerClientConfig;
import com.example.ranger.model.Policy;
import com.example.ranger.model.PolicyItem;
import com.example.ranger.model.PolicyItemAccess;
import com.example.ranger.model.PolicyResource;
import org.junit.Before;
import org.junit.Test;

import java.util.*;

import static org.junit.Assert.assertNotNull;

public class PolicyApiTest {

    private static RangerClient rangerClient;

    @Before
    public void initRangerClient() {
        rangerClient = new RangerClient(new RangerClientConfig());
        rangerClient.start();
    }

    @Test
    public void testCreatePolicy() {
        PolicyResource policyResource = PolicyResource.builder()
                .values(Collections.singletonList("/testdir2"))
                .isRecursive(true)
                .build();

        Map<String, PolicyResource> policyResourceMap = new HashMap<>();
        policyResourceMap.put("path", policyResource);

        Set<String> users = new HashSet<>();
        users.add("hive");

        List<PolicyItemAccess> policyItemAccessList = new ArrayList<>();
        policyItemAccessList.add(PolicyItemAccess.builder().type("read").build());
        policyItemAccessList.add(PolicyItemAccess.builder().type("write").build());
        policyItemAccessList.add(PolicyItemAccess.builder().type("execute").build());

        PolicyItem policyItem = PolicyItem.builder()
                .delegateAdmin(true).users(users)
                .accesses(policyItemAccessList)
                .build();

        Policy policy = Policy.builder()
                .service("dev_hdfs")
                .name("test_ranger_api")
                .isEnabled(true).policyType(0)
                .resources(policyResourceMap)
                .policyItems(Collections.singletonList(policyItem))
                .build();

        Policy result = rangerClient.getPolicyApi().createPolicy(policy);
        assertNotNull(result);
        System.out.println(result.getName());
    }

    @Test
    public void testGetPolicyByName() {
        Policy result = rangerClient.getPolicyApi()
                .getPolicyByName("dev_hdfs", "test_ranger_api");
        assertNotNull(result);
        System.out.println(result.getName());
    }

    @Test
    public void testGetAllPoliciesByService() {
        List<Policy> result = rangerClient.getPolicyApi()
                .getAllPoliciesByService("dev_hdfs");
        assertNotNull(result);
        System.out.println(result.size());
    }

    @Test
    public void testDeletePolicy() {
        Policy result = rangerClient.getPolicyApi()
                .getPolicyByName("dev_hdfs", "test_ranger_api");
        assertNotNull(result);
        rangerClient.getPolicyApi().deletePolicy(result.getId());
        System.out.println(result.getName());
    }
}

Perform a testCreatePolicyunit test to verify whether the corresponding strategy is created on the ranger admin:

Check whether the content of the policy is consistent with the one defined in the code:

---

Code repository for this article:

• https://gitee.com/Zero-One/ranger-client-demo