ACK One GitOps Best Practices

Author: Zhuang Yu, Liu Sheng

ACK One is a distributed cloud container platform launched by Alibaba Cloud for scenarios such as hybrid cloud, multi-cluster, and distributed computing. management interface. Through ACK One multi-cluster management, you can associate and manage various forms of Kubernetes clusters, provide a unified cluster control plane, and realize multi-cluster unified application distribution, traffic management, operation and maintenance management, security management, and GitOps capabilities. This article introduces how to use ACK One GitOps capabilities to release applications in multiple clusters, as well as version management, automatic updates, permission control, CI pipeline integration, etc., to help you get started with GitOps quickly.

GitOps overview

insert image description here

The core of GitOps for application distribution is to use the Git warehouse to manage the deployment template of the application, continuously deploy the application to the designated Kubernetes cluster, and use the Git warehouse as the only source of application deployment, continuously adjust the status of the application on the Kubernetes cluster, and finally integrate with the Git warehouse. expectations are consistent.

Advantages of GitOps:

easy to learn

Git is easily accepted by accepted developers, easy to integrate, and has no additional learning costs.

Strong reliability

As the only source of application deployment, the Git repository provides version control, fast rollback and audit capabilities.

high security

Developers do not need any Kubernetes cluster permissions to use GitOps, only Git repository permissions.

Application Continuous Deployment

The application state in the Kubernetes cluster and the Git warehouse is automatically synchronized and consistent.

CNCF pointed out in its 2023 Cloud Native forecast that Gitops has matured and entered a stable period of productivity. The CNCF Gitops open source project **Argo has officially become a CNCF graduation project in December 2022[1] **, marking the stability of the Argo project and maturity, and more and more users are using the Argo project to implement GitOps application distribution.

ArgoCD is a sub-project of the Argo project, a continuous delivery tool following the declarative GitOps concept, docking Git warehouses and Helm warehouses, and providing powerful visual pages. ArgoCD runs as a controller in the Kubernetes cluster, constantly monitoring the actual state of the application and keeping it in sync with the desired state declared in the Git repository.

ACK One GitOps

The ACK One multi-cluster management master instance hosts the open-source ArgoCD project to realize GitOps continuous delivery of applications, and at the same time integrates ACK One multi-cluster capabilities to achieve multi-cluster GitOps continuous delivery, meeting the requirements of high-availability deployment of applications and multi-cluster distribution of system components. .

insert image description here

The advantages of ACK One GitOps are as follows:

Hosts open source ArgoCD, providing ArgoCD native CLI and UI experience.
Exclusive ArgoCD console domain name, integrated Alibaba Cloud account SSO login, supports ArgoCD multi-user permission setting.
For multi-cluster distribution, ACK One associated sub-clusters are automatically added to ArgoCD and become the target cluster for application distribution GitOps.
Supports ArgoCD Applicationset to improve the experience of multi-cluster application distribution.
Out-of-the-box, O&M-free.

With ACK One GitOps, you can get the GitOps capabilities of CNCF graduation project ArgoCD out of the box, including ArgoCD native console and CLI integrated with Alibaba Cloud account, multi-user rights management, multi-cluster publishing capabilities, etc., to help you quickly build multiple Cluster release pipeline.

prerequisite

The multi-cluster management function has been enabled. For details, see **Enable multi-cluster management [ 2] **.
Multiple associated clusters have been added to the master control instance. The examples in this article are ackpro-cluster1 and ackpro-cluster2. For details, see **Adding an Associated Cluster [ 3] **.
The KubeConfig of the master instance has been obtained on the ACK One console****, and connected to the master instance through kubectl.
Install the latest version of Alibaba Cloud CLI ** [ 4] **** and configure credentials [ 5] **. If it is a sub-account operation, please grant the sub-account AliyunAdcpFullAccess permission in RAM.
Visit **ArgoCD [ 6] **, download and install the latest ArgoCD CLI.
Install the AMC command line tools. For details, see **AMC Command Line Help [ 7] **.

Deploy applications through the ACK One GitOps console

Step 1: Open ACK One GitOps (ArgoCD) with one click

Run the following command to enable GitOps, replacing clusterid with your master instance ID.

aliyun adcp UpdateHubClusterFeature --ClusterId <clusterid> --ArgoCDEnabled true

Through the above command, ACK One will create an ECI instance in your account to run the ArgoCD service, create an intranet SLB to expose the ArgoCD Server service, generate the ArgoCD Server domain name, and set DNS to resolve to the SLB intranet IP.

ArgoCD Server domain name format:

https://argocd. cluster id>…alicontainer.com，

You can run the following command to view the GitOps activation results:


#查看ArgoCD pod
kubectl get pod -n argocd
NAME                             READY   STATUS    RESTARTS   AGE
argocd-server-76c9b99f47-c8hwn   8/8     Running   0          41s

#查看ArgoCD Server域名
kubectl get cm argocd-cm -n argocd -o json | jq .data.url
"https://argocd.xxx.cn-xxx.alicontainer.com"

Step 2: Log in to the ACK One GitOps (ArgoCD) console and deploy the application

1. You can directly use a browser to access the ArgoCD Server domain name obtained in "Step 1", and click "LOGIN VIA ALIYUN" to log in to the ArgoCD console.

insert image description here

2. Add the release source Git Repo.

a. On the Settings > Repositories page, click + CONNECT REPO.

b. Enter the Git Repo URL: https://github.com/AliyunContainerService/gitops-demo.git and click "CONNECT".

insert image description here

c. After the addition is successful, the page will display Git's CONNECTION STATUS as Successful.

insert image description here

3. Create ArgoCD Application and publish the application to the target cluster.

a. On the "Application" page, click "+ NEW APP" and configure as follows. Application will pull the Helm Chart in GitRepo and deploy it to the default namespace of the target cluster.

insert image description here

b. After the creation is complete, you can view the application status of echo-server-app on the Applications page of ArgoCD.

insert image description here

c. Click SYNC under the target application to deploy the application to the target cluster synchronously.

After the synchronization is completed, Healthy and Synced appear on the right side of the status of the echo-server-app application, indicating that the application has been synchronized. Click echo-server to view the details of the application, and ArgoCD will display the topology of the application-related kubernetes resources and the corresponding status.

insert image description here

Step 3: View the actual deployment of the application

Connect to the Kubeconfig of the master instance, run the following command, the master instance will forward the request to a specific cluster.

//查看主控实例已关联的子集群
kubectl amc get managedcluster
Name       Alias             HubAccepted
cdxxx      ackpro-cluster2   true
cexxx      ackpro-cluster1   true    //GitOps部署目标集群

//通过-m参数指定集群，查看default命令空间下的Deployment，Service, Pod资源。
kubectl amc get all -n default -m cexxx
Run on ManagedCluster cexxx (ackpro-cluster1)
NAME                               READY   STATUS    RESTARTS   AGE
pod/echo-server-5cf54bdbcb-jv58k   1/1     Running   0          59m

NAME                  TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)          AGE
service/echo-server   LoadBalancer   192.xxx.xxx.xxx   39.xxx.xxx.xxx   8080:31769/TCP   59m        21d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echo-server   1/1     1            1           59m

//访问Loadbalance Service的External-IP，验证应用执行情况，显示版本为v1.0
 curl http://39.xxx.xxx.xxx:8080/version
"Hello Echo Server v1.0"

Step 4: Upgrade the application version

1. Update the image tag version in git from "v1.0" to "v2.0".

insert image description here

2. ArgoCD Application shows "OutOfSync" because of changes in GitRepo.

insert image description here

Click on the yellow "OutOfSync" icon to view the detailed differences:

insert image description here

3. Click "Sync" to initiate synchronization. After the synchronization is completed, the Pod Image Tag will become "v2.0", which is consistent with expectations.

insert image description here

Visit Service to see the actual effect, and the version becomes v2.0.

curl http://39.98.55.113:8080/version
"Hello Echo Server v2.0"

Step 5: Roll back the application version

You can find that after the release of version "v2.0", ArgoCD will keep the previous replicaset, which is convenient for quickly rolling back the previous version.

// 在ACK One主控实例上，通过amc -m参数，查看指定子集群的资源状态
kubectl amc get replicaset  -n default -m cexxx
Run on ManagedCluster cexxx (ackpro-cluster1)
NAME                     DESIRED   CURRENT   READY   AGE
echo-server-55664c4677   1         1         1       26m    // 版本v2.0
echo-server-5cf54bdbcb   0         0         0       109m   // 版本v1.0

Now we assume that there is a problem with "v2.0" and want to roll back to "v1.0". Click the "HISTORY AND ROLLBACK" of the application to view all historical versions of the application. You can select the corresponding version to roll back. In this example, select "v1.0", and click "Rollback".

insert image description here

Check the rollback situation, the Pod Image Tag has been rolled back to "v1.0". Because it is different from GitRepo, the application status becomes "OutOfSync". After the problem is fixed, you can click Sync to synchronize. Of course ArgoCD also supports automatic synchronization.

insert image description here

Deploy the application via the ArgoCD CLI

After enabling ACK One GitOps, you can also complete GitOps application management through ArgoCD CLI.

//登陆argocd server
argocd login argocd.<ackone cluster id>.<region>.alicontainer.com --sso
Opening browser for authentication
Performing authorization_code flow login: https://signin.aliyun.com/oauth2/v1/auth?xxx
Authentication successful
'root' logged in successfully
Context 'argocd.<ackone cluster id>.<region>.alicontainer.com' updated

Add and view Git Repo


//增加git repo
argocd repo add https://github.com/AliyunContainerService/gitops-demo.git --name echo-server
Repository 'https://github.com/AliyunContainerService/gitops-demo.git' added

//查看git repo
argocd repo list
TYPE  NAME  REPO                                                       INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git         https://github.com/AliyunContainerService/gitops-demo.git  false     false  false  false  Successful           default

view cluster

argocd cluster list
SERVER                          NAME                    VERSION  STATUS   MESSAGE                                                  PROJECT
https://10.xxx.xxx.xxx:xxx      cexxx-ackpro-cluster1            Unknown  Cluster has no applications and is not being monitored.
https://10.xxx.xxx.xxx:xxx      cdxxx-ackpro-cluster2            Unknown  Cluster has no applications and is not being monitored.
https://kubernetes.default.svc  in-cluster                       Unknown  Cluster has no applications and is not being monitored.

You can manage the master instance through ACK One multi-cluster, associate a new cluster, and ACK One will automatically synchronize the cluster to ArgoCD.

Create an application and synchronize the application.

//创建应用
argocd app create echo-server --repo https://github.com/AliyunContainerService/gitops-demo.git --path manifests/helm --revision one-demo --dest-namespace default --dest-server https://10.0.47.122:6443
application 'echo-server' created

//同步应用，从Git中拉取应用，并实际部署到目标集群中
argocd app sync echo-server

Manage ACK One GitOps user permissions

ArgoCD provides powerful user rights management functions. Based on the following rights policy templates, you can flexibly configure a user to have certain rights to a resource. Supported resource resource types include: applications, clusters, repositories, etc. You can also specify the project to which the specific resource name belongs. Permission actions include: create, update, delete, get, etc.

p, <user/group>, <resource>, <action>, <project>/<object>

Below we use an example to demonstrate user permission settings.

Define an ArgoCD administrator, who has all permissions to all resources, so can create ArgoCD Application deployment applications. Define a developer user who can submit changes to GitRepo, but can only view ArgoCD resources, for example, you can view the deployment results of ArgoCD Application.

The corresponding configuration is as follows: the administrator edits the ArgoCD configmap argocd-rbac-cm through the command "kubectl edit cm argocd-rbac-cm -n argocd".

data:
  policy.csv: |
    g, "26xxx", role:admin     //阿里云子账号，授权为ArgoCD管理员
    g, "27xxx", role:readonly     //阿里云子账号，授权为ArgoCD只读用户

role:admin and role:readonly are the permission names embedded in ArgoCD, corresponding to administrator permission and read-only permission respectively.

For specific definitions, you can refer to:

https://github.com/argoproj/argo-cd/blob/master/assets/builtin-policy.csv

You can also build permissions suitable for specific business scenarios and authorize specific users. In addition, it is recommended that you read ACK One GitOps security model, you can view more detailed examples and descriptions of user permissions.

Multi-cluster GitOps deployment with ArgoCD Applicationsset

Applicationset is built by ArgoCD for multi-cluster scenarios. Through Applicationset, multiple Application instances can be produced to support different GitRepos and different target clusters. Following the example below, we will define the Applicationset, read the deployment Yaml from different GitRepo directories, and complete the application deployment to two clusters.

insert image description here

Since different clusters contain different image versions and service configurations, we use two directories in GitRepo to store two cluster deployment Yaml files, corresponding to production and staging environments.

└── manifests
    └── directory
        ├── production
        │   ├── deployment.yaml  //image tag 为v1.0
        │   └── service.yaml
        └── staging
            ├── deployment.yaml  //image tag 为v2.0
            └── service.yaml

Create an Applicationset using the following command

//创建命名空间，在主控实例中创建命名空间，将自动同步到子集群
kubectl create ns one-demo

//创建Applicationset
cat << EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: echo-server
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: production            //集群1
        url: https://10.xxx.xxx.xxx:6xxx
      - cluster: staging               //集群2
        url: https://10.xxx.xxx.xxx:6xx
  template:
    metadata:
      name: '{
   
   {cluster}}-gitops-demo'
    spec:
      project: default
      source:
        repoURL: https://github.com/AliyunContainerService/gitops-demo.git
        targetRevision: one-demo
        path: manifests/directory/{
   
   {cluster}}  //对应集群目录名称
      destination:
        server: '{
   
   {url}}'                      //引用集群url
        namespace: one-demo
EOF

View the application deployment results, you can see that 2 Applications have been created, and the deployment Yaml is obtained from different GitRepo directories, and deployed to different target clusters.

insert image description here

Click "Sync" to get the actual running status of Kubernetes resources by running kubectl in the master instance of ACK One.

kubectl amc get deployment -m all -n one-demo   //amc -m all将获取所有关联集群的资源状态 
Run on ManagedCluster cdcxxx (ackpro-cluster2)
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
echo-server   1/1     1            1           18m
Run on ManagedCluster cedxxx (ackpro-cluster1)
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
echo-server   1/1     1            1           18m

kubectl amc get svc -m all -n one-demo    //amc -m all将获取所有关联集群的资源状态 
Run on ManagedCluster cdcxxx (ackpro-cluster2)
NAME          TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)    AGE
echo-server   ClusterIP      192.xxx.xxx.xxx   <none>         80/TCP     18m
Run on ManagedCluster cedxxx (ackpro-cluster1)
NAME          TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)    AGE
echo-server   ClusterIP      192.xxx.xxx.xxx   <none>         80/TCP     18m

CI pipeline integration through Image Updater

insert image description here

ACK One GitOps solves the problem of application deployment, but the E2E required by the development team, from code submission, image construction, to automatic deployment, is a fully automatic solution for the whole process. The fully automatic solution is especially important for the development and testing environment, which can help improve R&D efficiency. ACK One GitOps integrates the open source ArgoCD Image Updater project, and extends support for the ACR mirror repository. When the image built by the CI pipeline is released to the ACK mirror warehouse, the Image Updater can monitor the changes of the image and automatically update and deploy the Image Tag in the Git Repo. After that, the ArgoCD Application monitors the changes of the Git Repo and initiates Sync to deploy the application. Through the Image Updater, ACK One GitOps can be linked with the CI pipeline to provide a CI+CD automation solution for the entire E2E process.

For a detailed description and usage of Image Updater, please refer to the document: Practice of fully automated GitOps delivery of applications based on ACK One and ACR:

https://developer.aliyun.com/article/1122570