Sentinel 2.0 Microservice Zero Trust Exploration and Practice

Author: Yake, Shimian

Talking from the classical and simple safety philosophy

Current State of Cyber ​​Security

Now the most common enterprise network security architecture is to do security protection at the border of the enterprise network, but not to do security protection inside the enterprise network. This does save the cost of the security construction of the enterprise and also provides a certain protection capability for the enterprise. But this kind of community is comparable to the real situation. All the houses in this community have no doors. There is a security guard standing at the gate of the community. He will identify who can enter the community and who cannot enter the community. As long as the security allows a person to enter the community , this person can do whatever he wants in the community. So will everyone live in this community? I think everyone will not.

Why can't we, because such a community is too fragile, as long as there is a way to bypass the security at the gate, this kind of protection is useless. For example, a thief posing as a resident inside, following a user, or colluding with users inside can break through the security line of defense. Analogous to the network environment, a hacker can break through the security at the network boundary by stealing employee accounts, reusing replay links, and colluding with enterprise employees.

Zero Trust Requirements

Zero trust came into being in this context. Zero trust requires that all security protection be deposited at the application level, that is, each application needs to authenticate and authenticate requests.

a. Authentication is confirmation of who you are.

We still use the example of the community just now. Issuing identity means that you go to the neighborhood committee with the real estate certificate and other materials, and the neighborhood committee will give you a pass, which identifies you as a resident of this community and who you are. Of course, this pass is old, and you have to take the materials to the neighborhood committee to get a new one.

In the requirements of zero trust, a client or server needs to carry a token to obtain a certificate from a certificate authority. This certificate uniquely identifies the identity of the client or server. When requesting from the client, they need to hold this certificate to declare their identity. The certificate will expire after a certain period of time, so you need to re-execute to get the certificate.

b. Authentication is to confirm what rights you have.

Residents can obtain the key of the corresponding house with the pass, and this key can only open the door of the designated house.

In the requirements of zero trust, each certificate identity has corresponding rights, for example, you can only access service A, but not service B, and so on. Each service will execute corresponding rules, intercept requests that do not comply with the rules, and release requests that meet the rules.

In addition to the above-mentioned basic functions, Zero Trust still needs certain additional functions, such as:

a. Authentication can be changed: Authentication and authorization must be strictly enforced, and require dynamic change. For the granting of authorization, it is necessary to continuously evaluate resources according to network conditions, enterprise organizational structure, network threats, and identity changes. The granting of authorization should be driven by dynamic policies.

b. Visibility and Monitoring: Information about the current state of assets and their communications must be collected, analyzed, and used to improve the organization's security posture. The integrity and security of all assets must be continuously monitored, and deviations in the security posture must be mitigated in a timely manner. Furthermore, these findings should feed back into customer risk assessment and identity verification procedures to further improve their quality.

c. Auditing and compliance: Secure data retention and comprehensive compliance reporting are imperatives to address regulation.

Istio: an excellent builder of zero trust

Istio's Zero Trust Construction Solution

The service network Istio has a relatively complete construction of zero trust. In the service network, there are three roles:

• Control plane Istiod: There is one Istiod in each service network, which acts as a control plane to coordinate and manage all applications in the service network.
• User Applications: Applications run by the user.
• Data plane proxy: Each user application running in the service network has a proxy container, which acts as a proxy for all traffic of the user application. All outgoing and incoming traffic from user applications must pass through the proxy container.

In the service network, regarding the two core functions of zero trust, the specific behaviors of authentication and authentication are as follows:

• Authentication: When the proxy starts, the proxy will carry the token stuffed in the container by k8s to obtain the certificate from istiod. Any subsequent calls between services require two-way authentication. For example, when A initiates a request to B, A needs to verify B's certificate, and B also needs to verify A's certificate.
• Authentication: The control plane istiod will distribute authentication rules to all proxies, and each proxy will allow or reject requests according to the rules.

Some shortcomings of the Istio solution

Istio's zero-trust solution is already very good. If there are any deficiencies, we can consider the following aspects:

a. In terms of authentication strategy: Istio's trust comes from the fact that the k8s cluster will not be compromised, and the token in the pod will not be stolen. When verifying the token, it is based on trusting the apiserver; the authentication strategy of istio is single.

b. In terms of identity identification: In the certificate issued by Istio, the field object used to identify the identity uses the spiff standard. It needs to strongly rely on the namespace and serviceaccount of K8s; the identity is not universal enough.

c. In terms of authentication rules: Istio's authentication rules are very fine-grained, but there are many rules that strongly depend on the concept of K8s, such as requiring that requests must come from a certain namespace; secondly, there are still some rules missing, such as application deployment Environment level authentication and so on.

Sentinel 2.0: What do we expect the security foundation of microservices to look like?

What is Sentinel?

At the beginning of Sentinel's open source, it focused on the protection of the application running state, and focused on the flow control degradation, including hotspot flow control, circuit breaker degradation, adaptive overload protection, concurrent isolation, flow control and smoothing, etc.

In 2022, Sentinel announced a brand upgrade, from Sentinel 1.0 to Sentinel 2.0, and linked with OpenSergo, focusing on the protection of the entire application life cycle. Including standardization, flow control and self-healing, service fault tolerance, service isolation, traffic routing and coloring, zero trust, etc.

The zero trust function in Sentinel involves the two core functions of zero trust, namely certificate management and authentication rules.

Sentinel 2.0 overview

We also raised a related issue [ 1] in the Sentinel community , hoping to build the zero-trust security capability of Sentinel2.0.

With the development of cloud-native technologies, network boundaries are gradually disappearing, and the concept of zero trust therefore prevails. The most important functions of zero trust are certificate management and request authentication. As a generic, cloud-native traffic governance component, Sentinel 2.0 will support zero-trust capabilities for certificate management and request authentication:

• Obtain the certificate from the external data source and use it to configure https when the web service is enabled.
• Obtains authentication rules from the external data source and permits or blocks each request based on the authentication rules.

How Sentinel's Zero Trust works

Let's talk in detail how Sentinel 2.0's zero-trust capability is more suitable for the development of the community and can provide a secure base capability for our microservice ecosystem. Currently in the ecology of Sentinel and OpenSergo:

• OpenSergo: As the center of rule management and distribution, it is positioned similar to the control plane in the service network.
• Sentinel: It is introduced by the application as a dependency package, and its location is similar to a data plane in the service network.

For authentication and authentication, the two core functions of zero trust, Sentinel and OpenSergo will do the following:

• Authentication: Sentinel carries the token to obtain the corresponding certificate from OpenSergo. OpenSergo will judge the legitimacy of the token according to the corresponding policy, and issue the corresponding identity certificate, and Sentinel will automatically rotate the certificate when the certificate expires. Sentinel will first support authentication strategies with Istio's k8s-apiserver, and will continue to support authentication strategies such as OPA and IDaas in the future.
• Authentication: The user can configure the authentication CRD to OpenSergo, and OpenSergo will convert it into a standard XDS and send it to Sentinel. Sentinel will save all the authentication rules about the application, and automatically release or reject the request for the application according to the authentication rules . Authentication rules include JWT rules, deny rules and allow rules in total.

In order to better evolve zero trust capabilities, a unified zero trust CRD rule is required, and CRD is appropriately expanded on the basis of compatibility with Istio. The CRD involves a total of 3 aspects, namely:

a. TlsMode: authentication policy, whether the request needs to verify the identity of both parties.

b. JWT: JWT strategy, how to verify the token in the request that conforms to the JWT specification.

c. Auth: Authentication strategy, which determines which requests will pass and which requests will not pass.

An example CRD of Auth is as follows:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
  namespace: default
spec:
  action: DENY
  rules:
    - from:
        - source:
            # 需要匹配的身份
            principals: [ "principal1","principal2" ]
            # 需要不匹配的身份
            notPrincipals: [ "notPrincipal1","notPrincipal2" ]
            # 需要匹配的JWT中iss+"/"+sub
            requestPrincipals: [ "jwtp1","jwtp2" ]
            # 需要不匹配的JWT中iss+"/"+sub
            notRequestPrincipals: [ "notjwtp1","notjwtp2" ]
            # 需要匹配的命名空间
            namespaces: [ "namespace1","namespace2" ]
            # 需要不匹配的命名空间
            notNamespaces: [ "notNamespace1","notNamespace2" ]
            # 需要匹配的直接来源ip
            ipBlocks: [ "10.1.1.1","10.1.1.0/24" ]
            # 需要不匹配的直接来源ip
            notIpBlocks: [ "11.1.1.1","11.1.1.0/24" ]
            # 需要匹配的请求最初ip，最初请求的来源ip来自于header中X-Forwarded-For的值
            remoteIpBlocks: [ "12.1.1.1","12.1.1.0/24" ]
            # 需要不匹配的请求最初ip ，最初请求的来源ip来自于header中X-Forwarded-For的值
            notRemoteIpBlocks: [ "13.1.1.1","13.1.1.0/24" ]
        - source:
            principals: [ "principal3" ]

      to:
        - operation:
            # 需要匹配的到达域名
            hosts: [ "www.host1.com","www.host2.com" ]
            # 需要不匹配的到达域名
            notHosts: [ "www.nothost1.com","www.nothost2.com" ]
            # 需要匹配的到达端口
            ports: [ "8080","443" ]
            # 需要不匹配的到达端口
            notPorts: [ "18080","1443" ]
            # 需要匹配的请求方法
            methods: [ "GET","POST" ]
            # 需要不匹配的请求方法
            notMethods: [ "PUT","DELETE" ]
            # 需要匹配的请求path
            paths: [ "/info1*","/info2" ]
            # 需要不匹配的请求path
            notPaths: [ "/notinfo1*","/info2" ]
        - operation:
            hosts: [ "www.host3.com" ]

We also raised a related issue [ 2] in the OpenSergo community , hoping to build a standardized zero-trust CRD.

We want to add a standard CRD on the zero-trust direction to OpenSergo.

The CRD will be expanded to be compatible with istio.

The CRD involves three aspects in total, namely

1. TlsMode: Authentication policy, whether to authenticate both parties.

2. JWT: JWT policy, how to verify tokens in a request that comply with the JWT specification.

3. Auth: Authentication policy that determines which requests are approved and which requests are rejected.

The code structure of Sentinel

The part about zero trust in sentinel's code will be divided into three layers:

• extension: In the extension package of sentinel, certificate pulling, XDS reception, and authentication rule conversion will be completed.
• core: The entity that defines certificates and authentication rules in the core layer of sentinel, and stores certificates and authentication rules.
• adapter: The adapter layer of sentinel will provide various adapters for mvc, webflux, and dubbo.

Spring Cloud Alibaba, Dubbo, and Spring Boot applications can use adapters or directly use core entities to make certificates and rules take effect.

According to 3.2, Spring Cloud Alibaba, Dubbo, and Spring Boot need to change the certificates and rule entities stored in Sentinel.

Spring Cloud Alibaba adaptation

How to adapt TlsMode:

SpringCloud Alibaba cannot implement single-port dual-protocol, that is, it cannot implement the compatibility mode in TlsMode. Therefore, only dual modes are supported: PERMISSIVE and DISABLE are plaintext modes, and STRICT is strict mode.

Adapt JWT, RBAC:

SpringCloud Alibaba is used on the user's springcloud, so it handles http requests itself, so it is fully compatible with JWT and RBAC.

Dubbo adaptation

Adapt TlsMode:

Dubbo adapts to single-port dual-protocol, and adapts to this mode

Adapt JWT:

• There is no concept of header and params in Dubbo's transmission protocol, but there is a field of Attachments,
• For the fromHeaders mapping in the CRD it is taken from the Attachments field.
• Does not work for fromParams rules in CRDs.

Adapt to RBAC: There are no concepts of paths, methods, and headers in Dubbo.

• The rules for the header in CRD take effect in the Attachments field of Dubbo.
• Invalid rule for methods in CRD.
• For the paths rule in the CRD becomes /package.service/method

For Dubbo's strategy of adapting to CRD, we refer to istio's description of rpc style:

Optional. A list of paths as specified in the HTTP request. See the Authorization Policy Normalization for details of the path normalization. For gRPC service, this will be the fully-qualified name in the form of “/package.service/method”.

If not set, any path is allowed. Must be used only with HTTP.

Smooth upgrade: the road to zero trust for microservices still needs to be worked hard

The problem with smooth upgrades

Although Sentinel has built a basic framework for pulling certificates, obtaining authentication rules, and accessing to take effect, there are still problems of smooth compatibility when accessing. The most basic capability in zero trust is to upgrade all call links from plaintext transmission to mtls transmission, that is, two-way identity authentication.

Let's take the http protocol, the most common call protocol between applications, as an example. The requirement of zero trust is that all call links are set to https. Of course, the ideal situation is that the user stops all applications, upgrades the http ports of all applications to https ports, and then restarts them all. However, this is not realistic. The real situation is that the user will sequentially access all application instances to the zero trust function.

As shown in the figure, assume that the calling link is business gateway -> application A -> application B -> application C, and set application A and application B to have two instances, and application C to have one instance.

In the user's initial state, all calls between applications use the http protocol.

After that, it enters the intermediate state of user switching, and the user upgrades the application in turn and accesses the zero trust capability. During this process, the following requirements are required:

a. The HTTPS protocol is used when both the front and rear sides of the call link are connected to zero trust. For example, when the user switches the intermediate state in the figure, application A instance 1 accesses the zero trust function, and application B instance 1 accesses the zero trust function. Therefore Application A instance 1 -> application B instance 1 is https protocol.

b. If the zero trust function is not connected to the front side of the call link, and the zero trust function is connected to the back side, the http protocol should be used. For example, the application A instance 2 is not connected to the zero trust function, and the application B instance 1 is connected to the zero trust function , so application A instance 2 -> application B instance 1 is http protocol.

c. If the zero trust function is connected to the front side of the call link, and the zero trust function is not connected to the back side, then there should be no call link. For example, application A instance 1 is connected to the zero trust function, and application B instance 2 is not connected to the zero trust function Trust function, so there is no call link between application A instance 2 and application B instance 2.

Finally, in the user's final state, all applications have access to the zero-trust capability, and all calling links must use the https protocol.

Compatibility

From the above findings, when switching the intermediate state, an application instance must have the ability to handle both https and http after accessing zero trust. In the final state, the ability to handle http is turned off and only the ability to handle https is retained.

Based on the above situation and according to istio's definition of PeerAuthentication, the following modes can be set:

• STRICT (strict mode, final state): HTTPS communication must be used between applications.
• PERMISSIVE (compatibility mode, intermediate state): The https protocol is preferred for communication between applications. For example, for the call link application A -> application B (in the intermediate state of switching in the figure, instance 1 opens the https port, instance 2 has not yet opened the https port), application A chooses instance 1 to use the https port for communication, and selects instance 2 Use the http port for communication.
• DISABLE (plain text mode, initial state): http communication must be used between applications.

Then, if the user chooses to upgrade the application instance so that it will eventually change from http to https, and the traffic cannot be lost during the process, the application instance can be upgraded according to the following steps:

The user restarts the application instance and accesses the zero trust function. After startup, the default plaintext mode -> enable compatibility mode -> monitoring application instance traffic has all been transferred to https -> use strict mode.

istio's definition of PeerAuthentication includes 4 modes, namely:

UNSET：Inherit from parent, if has one. Otherwise treated as PERMISSIVE.DISABLE：Connection is not tunneled.PERMISSIVE：Connection can be either plaintext or mTLS tunnel.STRICT：Connection is an mTLS tunnel (TLS with client cert must be presented).

Citation: https://istio.io/latest/docs/reference/config/security/peer_authentication/#PeerAuthentication

Then according to the above requirements, an intermediate state of the application instance must be made. The intermediate state must ensure that the application instance can handle http and https requests at the same time. The following two solutions can be given:

a. The scheme of opening an additional https port: This scheme is to open a new https port that is not the same as the http port, and indicate 2 ports in the registration center. For example, the following information can be included in the metedata of the registration center to indicate dual ports:

The biggest problem with this solution is poor compatibility, and it cannot be well adapted to the service network and gateway. Second, compared with most users, there is a reporting requirement for new ports.

b. Single-port dual-protocol solution: The biggest problem with this solution is that it is difficult to implement. For the most commonly used tomcat, tomcat does not natively support single-port dual-protocol, but for the nio mode in tomcat, you can modify the source code or use Bytecode enhancement technology realizes single-port dual-protocol. Tomcat's arp mode can't implement aspect at all when using native mode.

It is one of the key conditions for whether the zero trust solution can be used on a large scale to make the microservice application feel free and smooth upgrade to the zero trust solution. We hope to help our enterprise microservices to achieve the default security without feeling.

Outlook

In order to make zero trust better and more convenient, we still have a lot of work to do. For example:

a. CI/CD integration: Integrate zero trust rules into CI/CD, and configure security policies more automatically

b. Adaptive adjustment: When the external environment of an application changes, such as changes in the deployment environment and application subordinate departments, the security policy is adaptively changed.

c. Security by default: How to ensure that no configuration is required when the application is released, and a security policy is automatically generated based on information such as the environment, developer identity, etc., so that the application is secure by default.

There is still a long way to go to enable microservices to truly realize simple and convenient zero trust, and this may be just the beginning. All like-minded students are also welcome to participate in the construction together.

Related Links:

[1] issue

https://github.com/alibaba/Sentinel/issues/3166

[2] OpenSergo issue

https://github.com/opensergo/opensergo-specification/issues/85