In the cloud-native era, how to build secure applications through KubeSphere x Jihu GitLab?

This article is compiled from a speech given by Ma Jinghe, GitLab DevOps technical evangelist, on the Cloud Native Meetup Hangzhou Station.

When you hear cloud native, what do you think of?

Many people may naturally think of Kubernetes, containers, microservices, open source, etc. These keywords are topics that we cannot avoid when we get in touch with cloud native. But there is one keyword missing from the above: security.

Cloud native appeared in 2013, and after it developed in 2015, security has gradually been paid attention to and valued.

Taking image security commonly used in cloud native as an example, the following figure is the result of scanning with Trivy by pulling commonly used images. Different colors correspond to different levels of vulnerabilities. For example, node.js, an open source project, has 544 high-risk vulnerabilities and 921 medium-risk vulnerabilities; there are also many vulnerabilities in Jenkins.

It can be seen that container image security issues are relatively serious. For more analysis reports on container image security, interested friends can refer to the 2021 and 2022 software supply chain security reports released by Anchore.

A complete software development life cycle includes source code development, construction, testing, deployment and other links, and potential security risks may exist in each step. We should embed security into every link, that is, apply DevSecOps to every link of cloud-native application development, and add K8s container image security scanning to create a complete cloud-native security ecosystem.

What is DevSecOps? How can it help us build a cloud-native security ecosystem?

DevSecOps is an in-depth security defense system with both depth and breadth . From Source code, Build, Test to Deploy, there are corresponding security measures at any stage. Secondly, DevSecOps is an in-depth combination of processes, tools, and culture. In a traditional R&D team, developers are only responsible for code development and do not pay attention to subsequent operation and maintenance processes. However, under the requirements of the DevSecOps standard, security is the responsibility of everyone in the team and needs to run through every link in the entire life cycle from development to operation and maintenance.

The left shift of security testing emphasized by DevSecOps is actually to allow R&D to join in earlier. Most of the security issues in our software development process are introduced during the development phase. Therefore, if we take security into account as early as possible from the source code stage and improve security capabilities from the source, we can discover and solve security issues more effectively and at a lower cost.

Continuous security automation is also a very important part, because a complete security process involves many security tools. If each tool is manually configured and manually triggers security tests, the workload will be greatly increased. So what DevSecOps expects is continuous automation of security. After the code is developed and submitted for change (MR or PR), it can automatically scan, generate a report and suggest how to fix it.

Of course, now we have also entered the ChatGPT era. I made a demo some time ago. ChatGPT will tell you what is a security risk in the Code Review stage, and even give recommended repair codes. Interested friends can also try it

Gartner releases a technology maturity curve for new technologies every year. The figure above shows the data for 2022. DevSecOps is on the far right, which means it is already mature. This is why many companies are starting to discuss DevSecOps since 2019.

The picture below is a schematic diagram I drew based on industry data. It can be seen that the term DevSecOps was proposed by Gartner in 2012, which attracted strong attention and reached its peak; it slowly went down in 2016, and reached the bubble burst period around 2018; finally reached maturity in 2022.

It has been 11 years since the concept was proposed in 2012. DevSecOps has gone through a complete technology maturity curve, and now it has reached the stage of taking root. This is why, now we can share the practice of DevSecOps instead of staying in the concept.

How to find the entry point for cloud-native DevSecOps?

Security is a relatively large concept. When we mention security, it may include host security, network security, application security... This article focuses on the field of application security.

Although the figure below is just a Yaml file, it already contains important information about how we should practice DevSecOps.

Mirroring is the key point, because in the cloud-native era, mirroring is the only way to deliver. Although it is a simple binary file, it contains many things, such as OS, code, etc., and most of the security issues arise from this. In addition, the Yaml file itself may also have security issues, such as the content shown in the red box at the bottom of the above picture, that is, sensitive information is written into the Yaml file.

Let's do a split:

The overall application deployment can be split into 3 layers:

The base is a K8s cluster
In the middle is the container image
The top layer is the application

When we implemented DevSecOps, we also built a security protection system based on these three layers.

The first layer: K8s security

First of all, the first layer, K8s-based security protection methods, may be known to many people, such as RBAC (authority management), Network Policy (network policy), Audit (audit log), etc.

The second layer: container image security

The second layer container image, when we write Dockerfile, there are some best practices to follow, such as how to choose the base image.

Different base images have different numbers of security vulnerabilities and different attack surface sizes. As shown above, the number of vulnerabilities in several container images is different. If we choose the right base image, then we can reduce the attack surface. I won’t go into details, you can search for Dockerfile best practice, there are many introductions on the Internet.

The other is mirror scanning, which is also a very important part.

There are many open source tools for image scanning, such as Trivy, Clair, Anchore, etc. Using these tools can help scan whether the basic image we use is safe, find vulnerabilities, and then repair it. Finally, use this image as a single trusted source image. The team uses this image to make other images later, so the security can be greatly improved.

In addition, in the past two years, we have been talking about software supply chain security. One of them is to improve the credibility of products through digital signatures. For example, the cosign project under the OpenSSF Foundation (Open Source Security Foundation) can sign images, but why do we need signatures?

When building the image, use the 1.0.0 tag to build, but we can re-use the 1.0.0 tag to build the image after modifying the code again, and push it to the product warehouse. If you still use 1.0.0 for deployment, then the content has actually been tampered with, which means that the image is not trustworthy, and it is another dimension of image security.

If the image is signed, this problem can be well avoided. After the image is built, it is signed, and then verified when it is deployed. If the verification fails, it means that the image has been tampered with, and subsequent deployment should not be performed.

The above are several methods for container image security. Next we look at the top layer of application security.

Layer 3: Application Security

Most companies have dedicated operation and maintenance personnel or security personnel to help us strengthen the security protection of K8s and container images, and the top layer is closest to the business, and it is also the layer that developers pay more attention to.

This layer has a lot of security protection methods, such as Secret Detection, Dependency scanning, SAST, DAST, etc., the above figure only lists a part.

Taking Secret Detection as an example, according to my experience, more than 70% of security problems come from the leakage of sensitive information. When we are debugging locally, it is easy to write the password to death, but when we test and submit the code, we may forget to delete it, and push the password directly to the code warehouse, causing a leak, so the role of Secret Detection in the application security part is very important.

Cloud-native DevSecOps is a defense-in-depth system with depth and breadth. From K8s, Docker to application security, each layer has a lot of security means. In summary, there are the following landing principles:

Layered implementation, step by step
From simple to complex, from complex to simple
Focus on data, simplify tools
Safe left shift, R&D closed loop
Continuous optimization, continuous security

For the above cloud-native development security protection system, Jihu GitLab provides out-of-the-box DevSecOps functions, including seven functions: container image scanning, static application security testing (SAST), dynamic application security scanning (DAST), key detection, license compliance, dependency scanning, and fuzz testing .

How to implement so many security functions?

In a word, it is to integrate these security functions into CI/CD (continuous integration/continuous deployment) to achieve the effect of continuous testing and continuous security . Jihu GitLab integrates all security functions and CI/CD into Workflow, which is integrated with R&D workflow and is very complete.

Moreover, only a few lines of code are needed to complete the security scan of the corresponding means, which is easy to use .

The figure below shows the CI/CD Pipeline code corresponding to each security function. In this process, developers do not need to know what tools are used for each security function, and do not need to learn how to configure the tools. They only need to enter these few lines of code to scan, which is very easy to use.

Different security measures are marked with different colors in the figure, which is a simple suggestion, that is, the blue security measures are very easy to practice and have good results. If you want to practice all of them, you can adopt this step-by-step method, first implement the blue part, and then implement the green part.

How to build Jihu GitLab x KubeSphere cloud-native security system?

Next, let's talk about how to combine KubeSphere and Jihu GitLab, and use CI/CD to integrate security protection methods into the software development life cycle.

Jifox GitLab Introduction

Jihu GitLab is an integrated DevOps platform, which provides integrated DevOps capabilities covering the entire software development life cycle (from planning to operation and maintenance), and has built-in security functions, which can build a DevSecOps system with out-of-the-box security capabilities.

Introduction to KubeSphere

Kubernetes is a very complex container orchestration platform, and the learning cost is very high. What KubeSphere does is highly productized and abstracts the underlying Kubernetes. It is a cloud-native operating system that solves the pain points of high barriers to use of K8s and the complexity of cloud-native ecological tools.

What are the advantages of using Jihu GitLab to build DevOps on KubeSphere?

The most typical advantage is that it can realize the cloud-native operation of Jihu GitLab & Runner and realize flexible scheduling.

You can see the example in the figure below. When a developer submits a code change, the CI/CD Pipeline will be triggered. If Jihu GitLab CI detects that there is a job that needs to be run, it will send a signal to the Runner to trigger the scheduling mechanism behind it. Executors help complete all the steps, such as Check out, Build, and Test for the Go source code.

How to install Jihu GitLab on KubeSphere?

There are two main ways to integrate Jihu GitLab and KubeSphere:

➤ Method 1

Directly install Jihu GitLab Runner as a single component on KubeSphere. When we submit the application code and start running CI, all construction operations will be automatically completed on KubeSphere and the data will be directly fed back.

The installation steps are as follows:

Under KubeSphere's App Management, add an App Repo through App Repositories; click the Add button, and enter the Repo name and URL in the interface that appears (the Helm Chart address of Jihu GitLab Runner is https://charts.gitlab.io) .

Select "From App Template" in Create App:

Select the gitlab runner:

Modify the content of the Yaml file, enter the Jihu GitLab instance address Runner Token, etc., and click Install.

After the installation is successful, you can see Pods running normally under the jh-runner namespace on KubeSphere.

➤ Method 2

Install the entire GitLab source code hosting platform on KubeSphere. Currently, it is very convenient to deploy GitLab on KubeSphere. You only need to use the KubeSphere app store to search for GitLab to deploy with one click. After the installation is complete, the team can directly use Jifox GitLab to collaborate on KubeSphere.

Detailed installation tutorial:

https://gitlab.cn/blog/2022/04/01/jihu-kubesphere/?jh=gu0%20%E4%BD%9C%E8%80%85%EF%BC%9ABender%E5%BC%80%E6%BA%90%E4%B8%8D%20https://www.bilibili.com/read/cv16647014/%20%E5%87%BA%E5%A4%84%EF%BC%9Abilibili

The above describes how to install Jihu GitLab/Runner on KubeSphere. Next, we will introduce how to perform DevSecOps detection on KubeSphere through a demo.

Jifox GitLab x KubeSphere Security Scanning Demo

After installation, find the Runner page through the path of Project → Settings → CI/CD. On the left is the project-specific Runner, that is, only this project can be used, and other projects cannot be used; the shared Runner on the right is the Runner that can be used by other projects under this instance.

Next, demonstrate the scanning process in detail:

First create an Issue, which is the first step in Workflow, and then assign this required Issue to yourself; then start to write local code, here simulate a scenario, write the sensitive information of Token into it, and see if it can be found during the security scanning process.
Next, we create an MR and submit this problematic code. When submitting, we can select an assignor or reviewer, and let other people help review the code.
After the MR is created, an automatic scan will be performed. At this time, the Token we deliberately put in will be scanned out, thus discovering this potential security problem.
Finally, resubmit after repairing, scan again and merge without any problems.

The above is a simple demo to use the security scanning function of GitLab.

Summarize

This article introduces the respective advantages of Jihu GitLab and KubeSphere, and discusses how to combine KubeSphere and Jihu GitLab to build a cloud-native application security system. Finally, an example is used to show the working principle of Jihu GiLab DevSecOps function. I hope it will be helpful to everyone~

This article is published by OpenWrite, a multi-post platform for blogging !