Project preparation and construction process
Typical CI/CD process-DevOps
GitOps continuous delivery process
- GitOps: a continuous delivery method for cluster management and application distribution
- GitOps is different from a typical CI/CD. The biggest difference lies in the use of Git as a source of trust and preservation of declarative infrastructure and applications
- With Git as the center of the delivery process (pipeline), configuration files such as k8s yaml files are stored in git for management
- Developers only need to complete application deployment and operation and maintenance tasks through pull request, and do not need to use other CI/CD tools
- Advantages: increased productivity, improved development experience, consistency and standardization, safety
Push vs pull process (pipeline):
Build and publish applications with Flux
Flux official definition:
- The GitOps operator for Kubernetes
- Automated deployment tool (based on GitOps)
- characteristic:
- Automatic synchronization, automatic deployment
- Declarative
- Based on code (Pull request), not container
Ready to work
First, we need to prepare a Kubernetes cluster:
- Build k8s high availability cluster based on kubeadm
- Build k8s high availability cluster-binary mode
- minikube start
And install the Istio environment in k8s:
As shown in the figure below, we want to deploy a Mesh consisting of two services, in addition to a gateway and an external service, which can be said to be streamlined and complete:
- It can be seen from the call link that sleep is the role of the client and httpbin is the role of the server
Prepare a Git repository:
Install Flux
Official documents:
- https://docs.fluxcd.io/en/latest/tutorials/get-started/
- https://docs.fluxcd.io/en/latest/guides/use-private-git-host/
First, install fluxctlcommand tool, to Github repository on the download executable file. Then put it into /usr/binthe directory, and give executable permissions:
[root@m1 /usr/local/src]# mv fluxctl_linux_amd64 /usr/bin/fluxctl
[root@m1 ~]# chmod a+x /usr/bin/fluxctl
[root@m1 ~]# fluxctl version
1.21.0
[root@m1 ~]# Create a namespace for Flux, and then deploy Flux Operator to the k8s cluster:
[root@m1 ~]# kubectl create ns flux
namespace/flux created
[root@m1 ~]# git clone https://github.com/fluxcd/flux.git
[root@m1 ~]# cd flux/Before deploying Flux, you need to modify several Git-related configurations to your Git warehouse username, email, url, etc.:
[root@m1 ~/flux]# vim deploy/flux-deployment.yaml # 修改如下几个配置项
...
# Replace the following URL to change the Git repository used by Flux.
# HTTP basic auth credentials can be supplied using environment variables:
# https://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@github.com/user/repository.git
- [email protected]:fluxcd/flux-get-started
- --git-branch=master
# Include this if you want to restrict the manifests considered by flux
# to those under the following relative paths in the git repository
# - --git-path=subdir1,subdir2
- --git-label=flux-sync
- --git-user=Flux automation
- [email protected]After the modification is completed, deploy:
[root@m1 ~/flux]# kubectl apply -f deploy
[root@m1 ~/flux]# kubectl get all -n flux
NAME READY STATUS RESTARTS AGE
pod/flux-65479fb87-k5zxb 1/1 Running 0 7m20s
pod/memcached-c86cd995d-5gl5p 1/1 Running 0 44m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/memcached ClusterIP 10.106.229.44 <none> 11211/TCP 44m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/flux 1/1 1 1 44m
deployment.apps/memcached 1/1 1 1 44m
NAME DESIRED CURRENT READY AGE
replicaset.apps/flux-65479fb87 1 1 1 7m20s
replicaset.apps/memcached-c86cd995d 1 1 1 44m
[root@m1 ~]# In addition to the above methods, you can also use the command line to deploy Flux:
fluxctl install \
--git-user=xxx \
--git-email=xxx@xxx \
[email protected]:xxx/smdemo \
--namespace=flux | kubectl apply -f -The use of the private warehouse, we need some additional work needs to be added to the Flux daemon host key container ~/.ssh/known_hostsfile. Specific steps are as follows:
[root@m1 ~]# kubectl exec -n flux flux-65479fb87-k5zxb -ti -- \
env GITHOST="gitee.com" GITREPO="[email protected]:demo_focus/service-mesh-demo.git" PS1="container$ " /bin/sh
container$ ssh-keyscan $GITHOST >> ~/.ssh/known_hosts # 添加host key
container$ git clone $GITREPO # 测试确保能正常对仓库进行克隆
Cloning into 'service-mesh-demo'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 10 (delta 2), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (10/10), done.
Resolving deltas: 100% (2/2), done.
container$ After completing the deployment of Flux, we need to add the deploy key generated by Flux to the git repository (read/write permission). The command to obtain the deploy key is as follows:
[root@m1 ~]# fluxctl identity --k8s-fwd-ns flux
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsyfN+x4jen+Ikpff8LszXLFTwXSQviFxCrIx7uMy7LJM5uUEsDdFs/DZL1g9h/YnkfLJlFrxOCJ+tuqPrXuj3ceEFfal4T3YWiDwf1RsGJvJd6ED5APjsxyu5gkj9LvkOB8OlYwPlS8Pygv997n93gtH7rFbocK5EQpbhhBlue3Or2ufI/KBxDCx6xLaH9U/16EEi+BDVSsCetGIQI+TSRqqpN30+Y8paS6iCYajKTubKv7x44WaVFgSDT9Y/OycUq1LupJoVoD8/5Y2leUMaF9dhMbQgoc8zjh8q2HF2n97mAvgYWJosjeIcAKS82C0zPlPupPevNedAhhEb82svPWh7BI4N4XziA06ypAEmfEz3JuUTTeABpF2hEoV4UEagkSyS8T3xhfdjigVcKiBW5AqRsRyx+ffW4WREHjARSC8CKl0Oj00a9FOGoNsDKkFuTbJePMcGdgvjs61UlgUUjdQFfHoZz2UVo2OEynnCpY7hj5SrEudkujRon4HEhJE= root@flux-7f5f7776df-l65lx
[root@m1 ~]# Copy the key content and add it to the Git repository:
Deploy the application
Applications created as a separate namespace, and to add istio-injection=enabledtags, so that the agent can be injected Istio:
[root@m1 ~]# kubectl create ns demo
namespace/demo created
[root@m1 ~]# kubectl label namespace demo istio-injection=enabled
namespace/demo labeled
[root@m1 ~]# The local Git repository to clone, created in the repository configdirectory:
[root@m1 ~]# git clone [email protected]:demo_focus/service-mesh-demo.git
[root@m1 ~]# cd service-mesh-demo/
[root@m1 ~/service-mesh-demo]# mkdir configCreate the service configuration file in this directory:
[root@m1 ~/service-mesh-demo]# vim config/httpbin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin
namespace: demo
---
apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: demo
labels:
app: httpbin
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin
namespace: demo
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v1
template:
metadata:
labels:
app: httpbin
version: v1
spec:
serviceAccountName: httpbin
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
ports:
- containerPort: 80
[root@m1 ~/service-mesh-demo]# vim config/sleep.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: sleep
namespace: demo
---
apiVersion: v1
kind: Service
metadata:
name: sleep
namespace: demo
labels:
app: sleep
spec:
ports:
- port: 80
name: http
selector:
app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sleep
namespace: demo
spec:
replicas: 1
selector:
matchLabels:
app: sleep
template:
metadata:
labels:
app: sleep
spec:
serviceAccountName: sleep
containers:
- name: sleep
image: governmentpaas/curl-ssl
command: ["/bin/sleep", "3650d"]
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /etc/sleep/tls
name: secret-volume
volumes:
- name: secret-volume
secret:
secretName: sleep-secret
optional: trueSubmit the configuration file to the remote warehouse and update the git repo:
[root@m1 ~/service-mesh-demo]# git add .
[root@m1 ~/service-mesh-demo]# git commit -m "commit yaml"
[root@m1 ~/service-mesh-demo]# git push origin masterExecute the following command to let Flux synchronize the changes in the warehouse and perform automatic deployment:
[root@m1 ~]# fluxctl sync --k8s-fwd-ns flux
Synchronizing with ssh://[email protected]/demo_focus/service-mesh-demo
Revision of master to apply is 49bc37e
Waiting for 49bc37e to be applied ...
Done.
[root@m1 ~]# - By default, Flux will automatically sync every 5 minutes and does not require us to manually operate
Check the resources under the demo namespace at this point, you can see that Flux automatically deployed all services for us:
[root@m1 ~]# kubectl get all -n demo
NAME READY STATUS RESTARTS AGE
pod/httpbin-74fb669cc6-v9lc5 2/2 Running 0 36s
pod/sleep-854565cb79-mcmnb 2/2 Running 0 40s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/httpbin ClusterIP 10.105.17.57 <none> 8000/TCP 36s
service/sleep ClusterIP 10.103.14.114 <none> 80/TCP 40s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/httpbin 1/1 1 1 36s
deployment.apps/sleep 1/1 1 1 40s
NAME DESIRED CURRENT READY AGE
replicaset.apps/httpbin-74fb669cc6 1 1 1 36s
replicaset.apps/sleep-854565cb79 1 1 1 40s
[root@m1 ~]# Test whether the connectivity between services is normal:
[root@m1 ~]# kubectl exec -it -n demo sleep-854565cb79-mcmnb -c sleep -- curl http://httpbin.demo:8000/ip
{
"origin": "127.0.0.1"
}
[root@m1 ~]# Realize automatic gray release
Grayscale publishing process
Automated Grayscale Release-Flagger
Gray release is a process of migrating a little bit of traffic for a rolling upgrade. Therefore, if you manually operate this process manually, it is obviously inefficient and error-prone, so we need to use an automatic gray release tool, such as Flagger:
- Flagger : Weaveworks open source automatic grayscale publishing tool
- Support multiple Service Mesh products: Istio, Linkerd, App AWS Mesh
- Indicator monitoring gray release status
- Notification (slack, Microsoft team)
Flagger workflow:
Flagger installation
Official documents:
Add Flagger's Helm repository:
[root@m1 ~]# helm repo add flagger https://flagger.app
"flagger" has been added to your repositories
[root@m1 ~]# Create Flagger's crd:
[root@m1 ~]# kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/main/artifacts/flagger/crd.yaml
[root@m1 ~]# kubectl get crd |grep flagger
alertproviders.flagger.app 2020-12-23T14:40:00Z
canaries.flagger.app 2020-12-23T14:40:00Z
metrictemplates.flagger.app 2020-12-23T14:40:00Z
[root@m1 ~]# Deploy Flagger to the istio-system namespace through Helm:
[root@m1 ~]# helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set meshProvider=istio \
--set metricsServer=http://prometheus.istio-system:9090Adding a slack hook to flagger allows flagger to send notifications to the slack channel. This step is optional:
[root@m1 ~]# helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set slack.url=https://hooks.slack.com/services/xxxxxx \
--set slack.channel=general \
--set slack.user=flagger- The usage document of slack webhooks: https://api.slack.com/messaging/webhooks
In addition to slack, we can also configure a grafana for flagger, which integrates a canary dashboard, which allows us to view the progress of the grayscale release:
[root@m1 ~]# helm upgrade -i flagger-grafana flagger/grafana \
--namespace=istio-system \
--set url=http://prometheus.istio-system:9090 \
--set user=admin \
--set password=adminAfter the above operations are completed, confirm the deployment of flagger:
[root@m1 ~]# kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
flagger-b68b578b-5f8bh 1/1 Running 0 7m50s
flagger-grafana-77b8c8df65-7vv89 1/1 Running 0 71s
...Create an ingress gateway for the grid:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: public-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
EOFIn addition, we can also deploy a load testing tool, of course this is also optional:
[root@m1 ~]# kubectl create ns test
namespace/test created
[root@m1 ~]# kubectl apply -k https://github.com/fluxcd/flagger/tree/main/kustomize/tester
[root@m1 ~]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
flagger-loadtester-64695f854f-5hsmg 1/1 Running 0 114s
[root@m1 ~]# If the above method is slow, you can clone the warehouse and deploy the tester:
[root@m1 ~]# cd /usr/local/src
[root@m1 /usr/local/src]# git clone https://github.com/fluxcd/flagger.git
[root@m1 /usr/local/src]# kubectl apply -k flagger/kustomize/tester/Grayscale release configuration
Configure HAP for the httpbin service so that it can support dynamic scaling. This is also optional, but it is usually recommended to configure HAP:
[root@m1 ~]# kubectl apply -n demo -f - <<EOF
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: httpbin
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: httpbin
minReplicas: 2
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
# scale up if usage is above
# 99% of the requested CPU (100m)
targetAverageUtilization: 99
EOFCreate a metric used to verify the grayscale release, and falgger will gradually migrate traffic based on this indicator:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: latency
namespace: istio-system
spec:
provider:
type: prometheus
address: http://prometheus.istio-system:9090
query: |
histogram_quantile(
0.99,
sum(
rate(
istio_request_duration_milliseconds_bucket{
reporter="destination",
destination_workload_namespace="{{ namespace }}",
destination_workload=~"{{ target }}"
}[{{ interval }}]
)
) by (le)
)
EOFCreate the flagger canary, the specific configuration content is as follows, the related configuration information published by Grayscale is defined here:
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: httpbin
namespace: demo
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: httpbin
# the maximum time in seconds for the canary deployment
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
# HPA reference (optional)
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: httpbin
service:
# service port number
port: 8000
# container port number or name (optional)
targetPort: 80
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
analysis:
# schedule interval (default 60s)
interval: 30s
# max number of failed metric checks before rollback
threshold: 5
# max traffic percentage routed to canary
# percentage (0-100)
maxWeight: 100
# canary increment step
# percentage (0-100)
stepWeight: 20
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)
# percentage (0-100)
thresholdRange:
min: 99
interval: 1m
- name: latency
templateRef:
name: latency
namespace: istio-system
# maximum req duration P99
# milliseconds
thresholdRange:
max: 500
interval: 30s
# testing (optional)
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://httpbin-canary.demo:8000/headers"After creating Canary, you will find that it automatically creates some resources named primary for httpbin in the cluster, and also creates a Virtual Service whose routing rules point to httpbin-primary and httpbin-canary services:
[root@m1 ~]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
httpbin-74fb669cc6-6ztkg 2/2 Running 0 50s
httpbin-74fb669cc6-vfs4h 2/2 Running 0 38s
httpbin-primary-9cb49747-94s4z 2/2 Running 0 3m3s
httpbin-primary-9cb49747-xhpcg 2/2 Running 0 3m3s
sleep-854565cb79-mcmnb 2/2 Running 0 94m
[root@m1 ~]# kubectl get svc -n demo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
httpbin ClusterIP 10.105.17.57 <none> 8000/TCP 86m
httpbin-canary ClusterIP 10.99.206.196 <none> 8000/TCP 3m14s
httpbin-primary ClusterIP 10.98.196.235 <none> 8000/TCP 3m14s
sleep ClusterIP 10.103.14.114 <none> 80/TCP 95m
[root@m1 ~]# kubectl get vs -n demo
NAME GATEWAYS HOSTS AGE
httpbin ["public-gateway.istio-system.svc.cluster.local"] ["httpbin"] 3m29s
[root@m1 ~]# Then we use the following command to trigger the grayscale:
[root@m1 ~]# kubectl -n demo set image deployment/httpbin httpbin=httpbin-v2
deployment.apps/httpbin image updated
[root@m1 ~]# - Tips: dep, configmap, secret will all be triggered
Check the canary event, you can see that the new version has been detected:
[root@m1 ~]# kubectl describe canary httpbin -n demo
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
...
Normal Synced 2m57s flagger New revision detected! Scaling up httpbin.demo
Warning Synced 27s (x5 over 2m27s) flagger canary deployment httpbin.demo not ready: waiting for rollout to finish: 1 out of 2 new replicas have been updatedChecking the Virtual Service of httpbin at this time, you will find that 20% of the traffic has been switched to the gray release version:
[root@m1 ~]# kubectl describe vs httpbin -n demo
...
Spec:
Gateways:
public-gateway.istio-system.svc.cluster.local
Hosts:
httpbin
Http:
Route:
Destination:
Host: httpbin-primary
Weight: 80
Destination:
Host: httpbin-canary
Weight: 20
Events: <none>Then enter the sleep service and use the script to access the httpbin service in a loop:
[root@m1 ~]# kubectl exec -it -n demo sleep-854565cb79-mcmnb -c sleep -- sh
/ # while [ 1 ]; do curl http://httpbin.demo:8000/headers;sleep 2s; doneLooking at the Virtual Service of httpbin again, you will find that 60% of the traffic has been switched to the gray release version:
[root@m1 ~]# kubectl describe vs httpbin -n demo
...
Spec:
Gateways:
public-gateway.istio-system.svc.cluster.local
Hosts:
httpbin
Http:
Route:
Destination:
Host: httpbin-primary
Weight: 40
Destination:
Host: httpbin-canary
Weight: 60
Events: <none>We can open Flagger's Grafana:
[root@m1 ~]# kubectl -n istio-system port-forward svc/flagger-grafana 3000:80 --address 192.168.243.138
Forwarding from 192.168.243.138:3000 -> 3000The following dashboard is built-in:
The release process can be viewed in Istio Canary Dashboard:
Finally, switching 100% of the traffic to the gray release version represents the completion of the release:
[root@m1 ~]# kubectl describe vs httpbin -n demo
...
Spec:
Gateways:
public-gateway.istio-system.svc.cluster.local
Hosts:
httpbin
Http:
Route:
Destination:
Host: httpbin-primary
Weight: 0
Destination:
Host: httpbin-canary
Weight: 100
Events: <none>You can also see the process of traffic migration from the event log of canary httpbin:
[root@m1 ~]# kubectl describe canary httpbin -n demo
...
Normal Synced 3m44s (x2 over 18m) flagger New revision detected! Restarting analysis for httpbin.demo
Normal Synced 3m14s (x2 over 18m) flagger Starting canary analysis for httpbin.demo
Normal Synced 3m14s (x2 over 18m) flagger Advance httpbin.demo canary weight 20
Warning Synced 2m44s (x2 over 17m) flagger Halt advancement no values found for istio metric request-success-rate probably httpbin.demo is not receiving traffic: running query failed: no values found
Normal Synced 2m14s flagger Advance httpbin.demo canary weight 40
Normal Synced 104s flagger Advance httpbin.demo canary weight 60
Normal Synced 74s flagger Advance httpbin.demo canary weight 80
Normal Synced 44s flagger Advance httpbin.demo canary weight 100When the release is complete, the status of canary httpbin will change to Succeeded:
[root@m1 ~]# kubectl get canary -n demo
NAME STATUS WEIGHT LASTTRANSITIONTIME
httpbin Succeeded 0 2020-12-23T16:03:04Z
[root@m1 ~]# Improve the flexibility of the system
Flexible design is currently popular in many fields. For example, flexibility in environmental landscape design refers to the ability to recover from a disaster, but the landscape can quickly restore its structure and function after a disaster. In product design, general flexibility refers to leaving a certain amount of leeway to facilitate modification when designing product morphological features.
Resilience in a distributed system generally means that the system has a certain fault tolerance and response ability, can quickly recover when a fault occurs, and be able to cope with a fault. In this section, we will add some flexibility to the previously deployed sample application.
System availability measurement
Let's first understand a concept: Service Level Agreement (SLA-Service Level Agreement). A service level agreement refers to an agreement or contract between the service provider and the customer on the quality, level, and performance of the service. For example, a service provider usually guarantees to customers what level of availability their services have, which is what we usually call a few 9 levels of availability.
The system's availability calculation formula:
Common availability levels are as follows:
Flexible design
- One way to deal with failures is to make the system fault-tolerant and adaptable
- Prevent fault (Fault) from turning into failure (Failure)
- mainly include:
- Fault tolerance: retry, idempotence
- Scalability: automatic horizontal expansion (autoscaling)
- Overload protection: timeout, fusing, degradation, current limit
- Resilience testing: fault injection
Flexibility provided by Istio:
- time out
- Retry
- Fuse
- Fault injection
Provide flexibility for demo applications
First, we create a Virtual Service for the demo application:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: demo
spec:
hosts:
- "*"
gateways:
- httpbin-gateway
http:
- route:
- destination:
host: httpbin
port:
number: 8000
EOFAdd the first elastic capability: configure timeout, the configuration is as follows:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: demo
spec:
hosts:
- "*"
gateways:
- httpbin-gateway
http:
- route:
- destination:
host: httpbin
port:
number: 8000
timeout: 1s # 配置超时
EOFTimeout configuration rules:
- timeout & retries.perTryTimout exist at the same time
- Timeout effective =
min (timeout, retry.perTryTimout * retry.attempts)
On the basis of the timeout, we can also configure the retry strategy:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: demo
spec:
hosts:
- "*"
gateways:
- httpbin-gateway
http:
- route:
- destination:
host: httpbin
port:
number: 8000
retry: # 配置重试策略
attempts: 3
perTryTimeout: 1s
timeout: 8sRetry configuration items:
x-envoy-retry-on：5xx, gateway-error, reset, connect-failure…x-envoy-retry-grpc-on：cancelled, deadline-exceeded, internal, unavailable…
Configuration fuse:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpbin
namespace: demo
spec:
host: httpbin
trafficPolicy:
connectionPool:
tcp:
maxConnections: 1
http:
http1MaxPendingRequests: 1
maxRequestsPerConnection: 1
outlierDetection:
consecutiveErrors: 1
interval: 1s
baseEjectionTime: 3m
maxEjectionPercent: 100Fuse configuration:
- TCP and HTTP connection pool size is 1
- Only allow 1 error
- 1 request count per second
- All pods can be removed from the load pool
- The failed pod can only be added again after 3m is removed
Configure security policy
Istio's security solutions
Istio security architecture
Actual combat
Create authorization for a specific service (httpbin). Note that no rule is configured, which means deny the current service:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: demo
spec:
selector:
matchLabels:
app: httpbin
EOFThe above configuration means that this service is completely inaccessible, we can test it:
# 请求被拒绝
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/get"
RBAC: access denied # 响应
# 其他版本可以正常访问
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin-v2.demo:8000/get"We can limit the request source through the following configuration. For example, the request source must be the demo namespace:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: demo
spec:
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/demo/sa/sleep"]
- source:
namespaces: ["demo"]
EOFtest:
# 请求通过
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/get"
# 请求被拒绝
$ kubectl exec -it -n ${other_namespace} ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/get"
# 修改service account为${other_namespace}后，通过In addition to limiting the source of the request, you can also limit that only specific interfaces are allowed to be accessed:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: demo
spec:
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/demo/sa/sleep"]
- source:
namespaces: ["demo"]
to:
- operation:
methods: ["GET"]
paths: ["/get"]
EOFtest:
# 请求通过
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/get"
# 请求被拒绝
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/ip"You can also configure other specific conditions, such as limiting request headers, which are usually used in scenarios where we need the client to carry specific request headers to allow access to the interface:
[root@m1 ~]# kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: demo
spec:
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/demo/sa/sleep"]
- source:
namespaces: ["demo"]
to:
- operation:
methods: ["GET"]
paths: ["/get"]
when:
- key: request.headers[x-rfma-token]
values: ["test*"]
EOFtest:
# 请求不通过
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/get"
# 加token后通过
$ kubectl exec -it -n demo ${sleep_pod_name} -c sleep -- curl "http://httpbin.demo:8000/get" -H x-rfma-token:test1