Basic instructions for use of Liquibase

table of Contents

Introduction

use

Local execution of changeLog

Jenkins execution steps

After the first execution is completed, the target database will have two more tables:

Common mistakes:

1: Modify the historical script file, the following error will be reported when executed

2: If the execution has not been completed, it is possible that the database is locked by abnormal liquibaselock.

ps: file package optimization

Introduction

Liquibase is an open source tool for database reconstruction and migration. It records database changes in the form of log files, and then executes the modifications in the log files to update or roll back the database to a consistent state. Its goal is to provide a database type-independent solution to achieve migration by executing schema-type files. The main advantages are as follows:

Support almost all mainstream databases, such as MySQL, PostgreSQL, Oracle, Sql Server, DB2, etc.;
Support the collaborative maintenance of multiple developers;
Log files support multiple formats, such as XML, YAML, JSON, SQL, etc.;
Support multiple operation modes, such as command line, Spring integration, Maven plug-in, Gradle plug-in, etc.

installation

Download and unzip Liquibase: https://download.liquibase.org
Install java (configure environment variables)
Download the database driver package and put it in the lib under the Liquibase directory

use

There are two use cases, one is used in a completely new project, and the other is used in an old project (that is, there are historical table structures and data in the database, and changeLog needs to be generated in reverse).

For old projects, use liquibase's two solutions:

One: You can sort out the current ddl in the project database (maybe not only ddl, some table data should also be maintained in liquibase, this depends on the specific project requirements), and maintain it in an initialized changeLog as an initialization script, so You can execute the initialization script in the new environment ( **In this case, pay special attention to specifying the context in the changeSet, and you must also specify the contexts when executing to avoid executing the initialization script in the old environment** ).

Two: liquibase has a special command to generate changeLog in reverse, generateChangeLog. (As to whether it is easy to use or not, I haven't tried it yet. I will add it when I try it...)

liquibase builds the directory in the project

src/main/db/sql/init-personal_bank_swift.sql

Define a sql file, this sql file can be directly referenced and executed by the changeSet in another changeLog

INSERT INTO personal_bank_swift (bank_code, clearing_code, swift_code, create_by, create_at, updated_by, updated_at) VALUES 
('bank_code2300', '012', 'BKCHHKHHXXX', null, null, null, null),
('bank_code2302', '009', 'CCBQHKAXXXX', null, null, null, null),
('bank_code2304', '041', 'LCHBHKHHXXX', null, null, null, null),
('bank_code2305', '040', 'DSBAHKHHXXX', null, null, null, null),
('bank_code2306', '032', null, null, null, null, null);

src/main/db/2019/08/2600-init-tables.xml

changeLog: xml (** is the changelog concept in liquibase, it can be nested in multiple layers to reference other changelogs**)
include : reference to the tag of the changelog
file : refer to the texture of the changelog file
changeSet : A changeLog can contain multiple changeSet tags. Each changeSet is uniquely identified by the id, author, and filepath attributes. When Liquibase executes the database changeLog, it reads the changeSet in sequence and checks the databasechangelog table for each changeSet. Check if the id/author/filepath combination is run. If it is already running, the changeSet will be skipped unless there is a real runAlways tag. After running all the changes in the changeSet, Liquibase inserts a new row with id/author/filepath and MD5Sum of the changeSet into the databasechangelog. Each changeSet is separate. The best practice is to ensure that each changeSet is changed as atomically as possible to avoid the result of failure and leave the remaining unprocessed statements in the database in an unknown state.
author : author
id : The id of the changeSet (**It is better to use the date format 2019082600-01 to make it clearer. If a target library may be executed by the liquibase of multiple projects, the id will be written to the same In the databasechangelog table, you can add the project name or the abbreviation of the project name in front of the date to distinguish, such as reference-2019082600-01)
runAlways : Perform the changes set at each run, even if it has been run before the change
context : It can be used to flexibly control the environment in which the script is executed. Our system generally defines the environment (such as team2, uat, prod), if there are more than one, you can use, or and split, and also support the inverted form of !prod (* *If context is defined, contexts must be specified when executing. If not specified, all sql will be executed and the context will be invalid)
comment: The description of the changeSet .
preConditions : The preconditions that must be passed before the changeSet will be executed. Can be used to perform data sanity checks for unrecoverable content
rollback : Describe how to roll back the SQL statement of the changeSet or refactor the label
createTable: Create table label
tableName : table name
column: the label of the table field
name : field name
type: field type
remarks: field remarks
constraints : field constraints
primaryKey : true represents the primary key
nullable: true,false
unique: true,false
createIndex: create the label of the index
indexName : index name
tableName : the name of the table where the index is created
column: the label of the index field (multiple representatives of the joint index)
name : field name
addColumn: add the label of the field
tableName : add the table name of the field
column: added field
name : field name
type: field type
sql: sql tag, the content can be directly sql statement
endDelimiter : The delimiter to be applied to the end of the statement. The default value is ;, can be set to''.
splitStatement: true,false
stripComments : Set to true to remove any comments in the SQL before execution, otherwise false. in case
If not set, the default value is false
comment : comment
sqlFile : the label that refers to the sql file
path : Refer to the address of the sql file

<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.6.xsd">
   <!--第一种标签建表方式-->
  <changeSet author="future_zwp (generated)" id="reference-2019082600-00" context="team2,uat,prod">
        <createTable tableName="personal_bank_swift">
            <column name="id" type="serial">
                <constraints primaryKey="true"/>
            </column>
            <column name="bank_code" type="text" remarks="银行编码"></column>
            <column name="clearing_code" type="text" ></column>
            <column name="swift_code" type="text" ></column>
            <column name="create_by" type="text" ></column>
            <column name="created_at" type="timestamp"></column>
            <column name="updated_by" type="text" ></column>
            <column name="updated_at" type="timestamp"></column>
            <column name="pt" type="text"></column>
        </createTable>
        <rollback>
            <dropTable tableName="personal_bank_swift"/>
        </rollback>
    </changeSet>

    <changeSet author="future_zwp (generated)" id="reference-2019082600-01" context="team2,uat,prod">
        <createIndex indexName="idx_bank_info_bank_clearing" tableName="personal_bank_swift">
            <column name="bank_code"/>
            <column name="clearing_code"/>
        </createIndex>
    </changeSet>

    <changeSet author="future_zwp (generated)" id="reference-2019082600-02" context="team2,uat,prod">
        <createIndex indexName="idx_personal_bank_swift_swift_code" tableName="personal_bank_swift">
            <column name="swift_code"/>
        </createIndex>
    </changeSet> 
  
   <!--第二种sql建表方式,所有的sql语句都支持，学习成本低，更灵活-->
  <changeSet author="zhaowenpeng" id="reference-2019082600-03" context="team2,uat,prod">
        <sql splitStatements="true">
            drop table if exists personal_bank_swift;

            create table personal_bank_swift
            (
              id serial primary key,
            bank_code     text,
            clearing_code text,
            swift_code    text,
            create_by     text,
            create_at     timestamp(6),
            updated_by    text,
            updated_at    timestamp(6)
            );

            comment on column personal_bank_swift.bank_code
            is '银行编码';

            create index idx_bank_info_bank_clearing on personal_bank_swift(bank_code,clearing_code);
            create index idx_personal_bank_swift_swift_code on personal_bank_swift(swift_code);
        </sql>
    </changeSet>
  
   <!--引用sql文件-->
   <changeSet author="zhaowenpeng" id="reference-2019082600-04" context="team2,uat,prod">
        <sqlFile  path="sql/init-personal_bank_swift.sql"></sqlFile>
    </changeSet>
</databaseChangeLog>

src/main/db/2019/driver.xml

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
                      http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd">


    <include file="2019/08/2600-init-tables.xml"/>


</databaseChangeLog>

src/main/db/driver.xml

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
                      http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd">

    <include file="2019/driver.xml"/>

    <include file="2020/driver.xml"/>

</databaseChangeLog>

Local execution of changeLog

src/main/db/update.sh (This file can only be executed locally, idea can install bashSupport, right-click the file can be executed)

The contexts value corresponds to the context value of the changeSet (if not satisfied, the changeSet will not be executed), and the changeLogFile will be read.

Driver.xml in the same package directory

#!/usr/bin/env bash
liquibase   \
    --driver=org.postgresql.Driver       \
    --changeLogFile=driver.xml     \
    --url=jdbc:postgresql://118.31.105.14:3546/team2_reference_data   \
    --username=team2_app    \
    --password=8sVG98uziKfAEqzM     \
    --contexts=team2 \
   update

Execution is complete:

Jenkins execution steps

src/main/db/maycur/update-db.sh

#!/bin/bash

DB_HOST="127.0.0.1"
DB_PORT="3306"
DB_TYPE="mysql"
DB_NAME="maycur-pro"
DB_USER="maycur"
DB_PASSWORD="Maycur@2018"

DB_DRIVER="com.mysql.jdbc.Driver"
OPERATOR="update"
CONTEXTS=''

function help_info() {
    echo "Please use the canonical setttings: "
    echo "./update-db.sh -h <DB_HOST> -p <DB_PORT> -d <DB_NAME> -u <DB_USER> -w <PASSWORD> -t <DB_TYPE> \
          -o <OPERATOR>"
    echo "ATTENTION! DB_TYPE alternatives would be *mysql* or *postgresql*, nothing more"
    exit 1
}

while [[ $# -gt 0 ]]
do 
    case "$1" in
        "-h")
            shift
            echo "DB_HOST: $1"
            DB_HOST=$1
            ;;
        "-p")
            shift
            echo "DB_PORT: $1"
            DB_PORT=$1
            ;;
        "-d")
            shift
            echo "DB_NAME: $1"
            DB_NAME=$1
            ;;
        "-u")
            shift
            echo "DB_USER: $1"
            DB_USER=$1
            ;;
        "-w")
            shift
            #echo "DB_PASSWORD: $1"
            DB_PASSWORD=$1
            ;;
        "-t")
            shift
            echo "DB_TYPE: $1"
            DB_TYPE=$1
            if [ "$DB_TYPE" = "mysql" ]; then
                DB_DRIVER="com.mysql.jdbc.Driver"  
            elif [ "$DB_TYPE" = "postgresql" ]; then
                DB_DRIVER="org.postgresql.Driver"
            else 
                help_info
            fi
            echo "DB_DRIVER: $DB_DRIVER"
            ;;
        "-c")
            shift
            echo "CONTEXTS: $1"
            CONTEXTS=$1
            ;;
        "-o")
            shift
            OPERATOR=$1
            ;;
        *)
            shift
            ;;
    esac
    shift
done

echo "NOW COMES the EXECUTION..."

/opt/liquibase/liquibase \
            --driver=$DB_DRIVER \
            --changeLogFile=driver.xml \
            --url="jdbc:$DB_TYPE://$DB_HOST:$DB_PORT/$DB_NAME?useSSL=false&useUnicode=yes" \
            --username=$DB_USER \
            --password=$DB_PASSWORD \
            --contexts=$CONTEXTS \
                    $OPERATOR

if [ $? -eq 0 ]; then
    echo "Congratulations! Things are all set, you are good to go!"
else 
    echo "Oops! Something just went wrong. You're gonna have to take a look at it"
    exit 1
fi

Create a project on jenkins, configure configure

cd src/main/db
/bin/bash update-db.sh -h maycurpg10-uat.pg.rds.aliyuncs.com -p 3433 -u team2_app -d team2_reference_data -w ${TEAM2_DB_PASSWORD} -t postgresql -c team2 -o update

Save and execute (before execution, make sure that the server is installed with liquibase and the required database driver jar package is placed in the lib folder under the liquibase directory)

Update-db.sh parameter description:

cd src/main/db This is to enter the directory where update-db.sh exists

-h database connection host

-p database connection port

-u database connection user name

-d database library name

-w database connection user password (referred to here is the name configured in jenkins)

-t database type

-c specifies contexts, need to cooperate with the context in the changeSet to conditionally execute sql

-o update、clearCheckSums

After the first execution is completed, the target database will have two more tables:

DATABASECHANGELOG 表

Liquibase uses the databasechangelog table to keep track of the changesets that have been run.

The table tracks each change setting as a row, identified by the combination of the id, author, and filename columns of the path where the changelog file is stored.

Column	Standard data type	description
`ID`	`VARCHAR(255)`	`changeSet`The `id`property value
`AUTHOR`	`VARCHAR(255)`	`changeSet`The `author`property value
`FILENAME`	`VARCHAR(255)`	`changelog`route of. This may be an absolute path or a relative path, depending on `changelog`how it is passed to `Liquibase`. For best results, it should be a relative path
`DATEEXECUTED`	`DATETIME`	`changeSet`Date/time of execution . And `ORDEREXECUTED`used together to determine the order rollback
`ORDEREXECUTED`	`INT`	`changeSet`The order of execution . In addition `DATE EXECUTED`, but also to ensure the correct order, even if the database supports the poor accuracy of the date and time as well. Note: The guaranteed value increase is only in a single update run. Sometimes they will restart at zero.
`EXECTYPE`	`VARCHAR(10)`	`changeSet`A description of how it is performed. Possible values are `EXECUTED`, `FAILED`, `SKIPPED`, `RERAN`, and`MARK_RAN`
`MD5SUM`	`VARCHAR(35)`	`changeSet`Check during execution . For each run to ensure that the `changelog`file `changSet`is not accidentally changed
`DESCRIPTION`	`VARCHAR(255)`	`changeSet`Generated readable description
`COMMENTS`	`VARCHAR(255)`	`changeSetcomment`The value of the label
`TAG`	`VARCHAR(255)`	`changeSet`The tracking corresponds to the label operation.
`LIQUIBASE`	`VARCHAR(20)`	For the execution `changeSet`of `Liquibase`version

**The table has no primary key. This is to avoid any database-specific restrictions on the key length.

DATABASECHANGELOGLOCK表

Liquibase uses the databasechangeloglock table to ensure that only one Liquibase instance is running at a time.

Because Liquibase only reads from the databasechangelog table to determine the changeSet that needs to be run, if multiple Liquibase instances are executed on the same database at the same time, conflicts will occur. This can happen if multiple developers use the same database instance, or if multiple servers in the cluster automatically run Liquibase at startup.

Column	Standard data type	description
`ID`	`INT`	The ID of the lock. Currently there is only one lock, but it will be useful in the future
`LOCKED`	`INT`	If you `Liquibase`'re running against this database is set to "1." Otherwise set to "0"
`LOCKGRANTED`	`DATETIME`	Date and time the lock was acquired
`LOCKEDBY`	`VARCHAR(255)`	Description of who was locked

**If Liquibase does not exit cleanly, the locked rows may remain locked. You can clear the current lock by running UPDATE DATABASECHANGELOGLOCK SET LOCKED=0

Common mistakes:

1: Modify the historical script file, the following error will be reported when executed

If you need to ignore the changes to this script at this time, you can change the update value in src/main/db/update.sh to clearCheckSums and execute it again, and then re-execute the script.

2: If the execution has not been completed, it is possible that the database is locked by abnormal liquibaselock.

At this time, you can connect to the database that executes the script, select * from databasechangeloglock;

If you find that locked is in a locked state, stop the script, change locked to 0 and then re-execute the script.

ps: file package optimization

1. Scene 1:

If you put all the xml files under one package, the file structure will not be very beautiful after a long time, and the content in driver.xml will be very long. Each time you add a script, you must add it to the bottom of the file. , The experience is extremely bad, so it is very important to define a sustainable file structure from the beginning. The solution is shown in the figure: just add a year's catalog after a year.

Create db directory
Create a directory of the year in the db directory
Create the driver.xml of the month directory and year level under the year directory
Create a script to be executed in the month directory
Create year-level driver.xml under the year directory (include the xml files in all months under the year directory)
Create db level driver.xml in the db directory (include driver.xml files in all years in the db directory)
When executing, execute the driver.xml in the db directory.

2. Scene two:

We have two product lines maycur and ng. Both product lines use the same project. However, because of the differences in some product lines, some SQL scripts will be inconsistent in the two product lines. At this time, they must be executed separately. Up. The strategy of this solution is to classify the executed files, put the files that can be executed by both product lines in the common folder, which will only be used in the bulletproof maycur folder of the files executed by maycur, and put the files that are executed only in ng To the ng folder. There are separate driver.xml and update.sh in common, maycur, ng, and three directories. In this way, you can deploy differently when deploying:

maycur: execute update.sh in the common and maycur directories
ng: execute update.sh in the common and ng directories

3. Scene three:

Sometimes the SQL maintained in liquibase may fail to execute due to the version of the database in a particular environment. At this time, the script maintained by liquibase will not be modified. Generally, the program of modifying the sql will be executed directly in the database to achieve the goal. But this solution will also be accompanied by some problems (the changSet where the sql is located will always be executed unsuccessfully and cannot be executed downwards). At this time, we must consider how to ignore this script.

Option one (not very recommended):

Manually insert an execution record into the databasechangelog table in the target library. Each changeSet is uniquely identified by the id, author, and filepath attributes, so these three values must be strictly corresponding to the script to be ignored (or from the successfully executed database Copy the record of the databasechangelog corresponding to sql to insert, pay attention to the unmatched values of contexts and liquibase versions to be changed). Because the value of md5 may be different, at this time, execute clearCheckSums before re-executing the script.

Option two (recommended):

A changeSet may have multiple sql scripts. Our scenario may only change one sql. At this time, we can change the corresponding sql to a meaningless sql (for example: select 1) to execute liquibase again. At this time The databasechangelog table has the corresponding execution record. At this time, we can change the sql back and re-clearCheckSums.

4. Scene four:

Our report service will have a lot of tables with a large amount of data, and every time a version is released, when liquibase is used to perform DDL operations on these tables, because of the characteristics of polardb, it will take a very long time to execute. Finally, after liquibase is executed, we have to go Run the full initialization data warehouse task (this operation will clear the table data first), which causes the release time to be further extended. In order to solve this problem, we have summarized a plan.

Release steps:

1. Execute update.sh after specifying 2020/0515/driver_1.xml in changeLogFile in update.sh

2. Data warehouse initialization tasks related to data warehouse running

3 Execute update.sh after specifying driver.xml in changeLogFile in update.sh

4. Deploy front-end and back-end services

(Because the first and second steps are all operating temporary tables, they can be executed in advance)

For each version, we create a package, such as 0515.

Create two files driver_1.xml and driver_2.xml under this package

1. In driver_1.xml, include the xml file to be executed in the first step, as shown in the figure below. When we want to add fields to the custom_fee_dynamic_detail table, we will create a temporary table of custom_fee_dynamic_detail_temp, and then add fields on the custom_fee_dynamic_detail_temp table. . The output of the initialization task of the data warehouse can be specified to be output to custom_fee_dynamic_detail_temp

2. The xml file to be executed in the second step can be included in driver_2.xml, as shown in the figure below. When the initialization task of the data warehouse is specified to be output to custom_fee_dynamic_detail_temp, we will drop the old data structure table custom_fee_dynamic_detail, and then we will add the custom_fee_dynamic_detail_temp temporary table Rename back to custom_fee_dynamic_detail, and then elsewhere you can refer to the custom_fee_dynamic_detail table of the new data structure

Driver.xml under the 2020 package

driver.xml under the maycur package