gitlab-runner docker implements project CI/CD automatic deployment based on sftp

Recently, I set up a CI/CD for my unit, and asked to build and deploy the developer's git code. This is the first time I have studied CI/CD from scratch, and I really stepped on a lot of pitfalls, especially the deploy part, which stuck me for three days. .

origin

environment:

The ubuntu physical machine is specially used as a CI/CD tool machine, and docker-compose has been used to install gitlab and gitlab-runner at the same time

gitlab is used for code hosting, and gitlab-runner is used for run gitlab-ci, which realizes automatic compilation, testing, and deployment in one step

A remote WEB server, production environment

question:

Since gitlab-runner runs under docker, and the execution environment used by gitlab-runner is also docker, this leads to a problem. The compiled file cannot be directly copied to the physical machine. Docker is a completely isolated sandbox, and docker will also Destroy it immediately after finishing CI/CD. Fortunately, we can get the build file before the docker CI/CD task is completed.

How to deploy the files that need to be deployed, such as jar and bin packages, to the server? I have considered many options:

Use scpcopy to remote, failure: account password is required, gitlab-ci will not give you a chance to enter
Using sshpass+ scp, you can pass the password, but the process is cumbersome, and there is still a permission problem, so I didn’t go deep
Use ftp, because the old thing of ftp opens random ports to both parties in passive mode or active mode, I think it is not safe, give up
To use rsync, it is really troublesome to configure the server and client at the same time, and I don’t want to see it
Finally decided to use it sftp, using the ssh user that comes with the server (need to create a new secure user, specify the directory)
jekenis , this is very powerful and heavy, and requires a long learning period to deploy various services, but our project is small, so we will not consider it for now, and give priority to lightweight sftp

The above are all I have referred to a lot of online documents, blogs, and finally summed up a set of my own sftp deploy method under docker

Here we use the vue project under node as an example. For example, the vue project after build will generate a dist directory. Now I want to publish all the files below this to the web server!

start

First of all, you need to add the runner to gitlab before it can run. This is described in the last two articles about building gitlab with docker-compose. Here is a reminder: open the project in gilab, go to Settings-"CI/CD on the left menu bar, Pull down the runner, find the url and token, and register it through gitlab-runner register. After registration, you can see that the new runner shows a green available status.
insert image description here
In the next step, you need to write gitlab-ci.yml and deploy.sh, of course you can also write them into gitlab-ci.yml together. I split because I love writing shells!

mine.gitlab-ci.yml

stages:
    - build
    - deploy

cache:
    key:
        files:
            - package.json
    paths:
        - node_modules

build:
    stage: build
    before_script:
        - npm config set registry https://registry.npm.taobao.org
    script:
        - npm install
        - npm run build
    artifacts:
        expire_in: 1 day
        paths:
            - dist

deploy:
    stage: deploy
    image: ubuntu:20.04
    before_script:
        - rm /etc/apt/sources.list
        - echo 'deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse' >> /etc/apt/sources.list
        - echo 'deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse' >> /etc/apt/sources.list
        - echo 'deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse' >> /etc/apt/sources.list
        - mkdir ~/.ssh
        - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" >~/.ssh/config
        - 'which lftp || ( apt-get update -y && apt-get install lftp -y )'
    script:
        - bash ./deploy.sh

Regarding the yml script of gitlab-ci, I must specifically explain that there are many essences in it that I wrote out after stepping on a lot of pitfalls.

Here is an official document:

Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD

The focus is in before_script:

First of all, you need to replace the apt source of ubuntu to China to increase the speed, otherwise CI/CD will be very slow!
Change npm to China, otherwise it will be very slow!
Then you need to turn off ssh verification, this is very complicated, the runner system under docker will ask every time (every run is a new environment), but you cannot enter yes, so you must turn off StrictHostKeyChecking, otherwise it will fail. Generally, you will not report an error locally, because you look at the knownhosts under .ssh! You have allowed it!
Install lftp, use this tool to upload to sftp! very convenient!

deply.sh

#!/bin/bash
#指定对端FTP服务器的用户名和密码
USER="xxxx"
PASSWD="xxxx"
IP='xxx.xxx.xxx.xxxx'
P='22'
DIR='example.com'
targetDay=$(date -d "-1 days" +"%Y-%m-%d")

lftp -u $USER,$PASSWD sftp://$IP:$P <<EOM
    mirror -R dist $DIR
    bye
EOM

[ $? -eq 0 ] && echo "Upload SFTP server successful [$targetDay]"

The deploy.sh here can actually be merged into yml, because gitlab-ci.yml is originally a simplification of the shell script. With gitlab-ci, it is not necessary to write the shell separately. It is clearer for me to separate it here , I explain the content inside:

First define the variables to connect to the sftp server, such as user name and password. It is safer to put these in the variables of gitlab-runner in the future , otherwise the person who submits git will know the account password for deploying the production server, which is absolutely not allowed in a large company.
Regarding the user in the previous article, it is definitely not recommended to use root. You can create a new user with useradd and assign the www directory to him. I created a new deploy user. It is best to add the deploy directory of the website, otherwise there may chmod 777be "permission denied" problem
The whole directory is uploaded in lftp, you need to use mirrorthe command! Note that it is not put, and you can’t use the so-called put -r, it’s all cheating, I’m crying because of the cheating.

Seeing that job succeeded , I was very excited and finally climbed out of the pit.

insert image description here