Tormented by relational database table building and upgrades? Because you don't use the simple Flyway

foreword

    Flyway has been an important part of my TDD development and continuous integration tool stack six years ago. As an early user, I should have made an "advertisement" for it. Unfortunately, time is too precious for entrepreneurs. With the epidemic, I have the opportunity to summarize something at home. Although Flyway is now a part of the Spring-Boot integration tool, I find that few people understand its power.

The Death of Relational Databases

    In the process of using a relational database, have you ever encountered the following situations, or even wanted to give up or have already given up on the relational database?

Scenario 1: Development environment, multiple people share a database

开发正调试着，忽然代码报错“XX字段不存在”：谁TMD又把表结构给改了…

Scenario 2: Development environment, everyone builds their own database

开发完一个功能，提交代码、更新，重启准备调试下，代码报错“XX表不存在”
吼一嗓子：谁又改表结构了？什么？每个人都要把xxx.sql执行一遍？
...
新员工：我要搭一套开发数据库，到底应该执行哪些SQL脚本？

Scenario 3: Development to Testing

Test: Do you see if there is a bug in this function? 
Development 1: Oh, you're going to execute this SQL script. 
Test: Well, it's fine now, but how can I make sure that this script has no bugs, can I reproduce and test it again? 
Development: Uh~, you can build the database again...

Scenario 4: Build a demo environment

Execute SQL script 1, SQL script 2, SQL script 3... Failed to start the service!
What? This script N is the test version, and the war package is the version that has been launched?
delete the library again...

Scenario 5: Abandoning the pit of relational databases

I can't stand the relational database anymore, let's switch to MongoDB... Well, the console is really quiet
... 
a few versions later:
Some data in the production environment cannot be found, and the types do not match. What?
The A field has been renamed, and the B field has been changed to a different type?

    If you have not encountered any of the above problems, either your project is too simple, or your development process is very standardized, or your change control is too good, you can skip this article.

If you are troubled by the     above problems, you may want to enter another pit: NoSQL, then congratulations you will encounter more pits, such as association query problems, data version problems...

Note: This does not negate the value of NoSQL, various NoSQLs are good complements to relational databases.
But if you want to use NoSQL as an alternative to relational databases, you're going to have more problems online than relational databases.

    If you see this, it means that you don't want to escape the problem, then let's get to know this relational database upgrade management tool - Flyway

 

Introduction to the principle of Flyway

What is Flyway? In a word, Flyway is a database version management component. Its principle is very simple :

1. Pull up Flyway when the project starts, first check whether there is a Flyway metadata table in the database, and create it if not;
2. Check the records in the Flyway metadata table, which scripts have been executed and what the current version is;
3. Find the database upgrade scripts (names that meet the rules) in the code, find the scripts whose version number is greater than the current version (Flyway metadata), execute them one by one and record the execution results to the Flyway metadata table.

    You read that right, the above three points are the core functions of Flyway. I am convinced that students who are proficient in my other blog " TDD Implementing Custom Expression Template Parser in Two Hours " can implement these three points by themselves in less than a day. Function, for non-JVM developers, I recommend developing a set on the basis of understanding the above ideas.

    The road is simple, and the simplest designs are often the most effective. With the above functions, we can easily do:

1. The code and database table building & upgrade script are put together for synchronous management, and the table structure can be understood through the code (SQL);
2. No need to manually execute any script, run the code or service to complete the environment construction (database table structure);
3. From any version of the environment (table structure), you can automatically upgrade to the specified new version by running the specified (new) version of the code or service;
4. (with in-memory database/Docker/cleanup script) The database build & upgrade script is easy to test iteratively with the code.

In short, after using Flyway, the "relationship" of relational database is no longer a bottleneck limiting your development efficiency , but becomes a necessary agreement for development & testing, and an important guarantee for improving version quality .

Get started quickly

No matter how powerful the tool is, how to use it is our primary concern. Let us take a look at how to use Flyway in the code with various Java project environments, and then let you know how to use Flyway for relational database version management. huge changes to come.

Note: All examples are based on Maven, and Gradle's own translation is used to translate dependencies. Both are not used... First study the dependencies of POM, and then download the jar package yourself.

1. Native Java project (without Spring, Spring-Boot)

Add flyway dependencies to the pom.xml file:

    <dependency>
      <groupId>org.flywaydb</groupId>
      <artifactId>flyway-core</artifactId>
      <!--通常情况下推荐最新发布版，此处是从旧代码中复制-->
      <version>3.2.1</version>
    </dependency>

The java code pulls up Flyway:

DataSource dataSource = ...
...
//在数据连接创建之后，其它代码运行之前，先调用Flyway升级
Flyway flyway = new Flyway();
flyway.setDataSource(dataSource);
flyway.migrate();
...

Write table creation scripts and data initialization scripts

src
|-main
  |-java
  |-resources
    |-db
      |-migration
        |-V0.0.1__init-schema.sql
        |-V0.0.2__init-data.sql

Note: The content in the script is the normal table building script, or the table structure change and data upgrade of the previous version.

2. Spring project

Add flyway dependencies to the pom.xml file (same as the native java project):

    <dependency>
      <groupId>org.flywaydb</groupId>
      <artifactId>flyway-core</artifactId>
      <!--通常情况下推荐最新发布版，此处是从旧代码中复制-->
      <version>3.2.1</version>
    </dependency>

The Spring configuration file pulls up Flyway:

<!-- 创建Flyway的bean，并调用其migrate方法 -->
<bean id="flyway" class="org.flywaydb.core.Flyway" init-method="migrate">
  <!-- 脚本文件校验和验证默认开启，可以防止脚本被修改。请视情况关闭校验 -->
  <property name="validateOnMigrate" value="false" />
  <property name="dataSource" ref="dataSource" />
</bean>

Write table building scripts and data initialization scripts (consistent with native java projects)

src
|-main
  |-java
  |-resources
    |-db
      |-migration
        |-V0.0.1__init-schema.sql
        |-V0.0.2__init-data.sql

Note: The content in the script is the normal table building script, or the table structure change and data upgrade of the previous version.

3. Spring-Boot project

Flyway has been integrated by Spring-Boot and has become Spring's standard database upgrade tool. It is easier to use Flyway in Spring-Boot. You only need to add dependencies and write database scripts, saving this step.

pom.xml add dependencies (Spring-Boot has been integrated, no version number is required)

    <dependency>
      <groupId>org.flywaydb</groupId>
      <artifactId>flyway-core</artifactId>
    </dependency>

If you use IDEA, it also provides you with the function of creating a Flyway upgrade script, which directly generates the SQL upgrade script for you with the current date and time:

The generated script names are as follows:

src
|-main
  |-java
  |-resources
    |-db
      |-migration
        |-V20190315174656__init-schema.sql
        |-V20190315201742__init-data.sql
        |-V20191225205157__update-userid-to-bitint.sql

4. Example upgrade script

The SQL script written for Flyway does not have any special requirements, it is the same as normal SQL, but each script is always considered to upgrade the table structure of the previous version, which is also a rigorous upgrade script in the traditional way requirements that should be met.

Example of table creation script:

BEGIN;

-- 新数据库建表
CREATE SCHEMA IF NOT EXISTS staff;

CREATE TABLE IF NOT EXISTS staff.staffs
(
    id             BIGINT AUTO_INCREMENT NOT NULL,
    staffId        VARCHAR(10)           NOT NULL,
    createTime     TIMESTAMP             NOT NULL,
    lastUpdateTime TIMESTAMP             NOT NULL,
    name           VARCHAR(50)           NOT NULL,
    PRIMARY KEY (id)
);

CREATE SCHEMA IF NOT EXISTS duty;

CREATE TABLE IF NOT EXISTS duty.onDutyDef
(
    id        BIGINT AUTO_INCREMENT NOT NULL,
    name      VARCHAR(50)           NOT NULL,
    startTime TIME                  NOT NULL,
    endTime   TIME                  NOT NULL,
    PRIMARY KEY (id)
);

CREATE TABLE IF NOT EXISTS duty.breakDef
(
    dutyId    BIGINT      NOT NULL,
    name      VARCHAR(10) NOT NULL,
    startTime TIME        NOT NULL,
    endTime   TIME        NOT NULL,
    PRIMARY KEY (dutyId, name)
);

-- 插入默认配置数据
INSERT INTO duty.onDutyDef
    (name, startTime, endTime)
VALUES ('普通班', '09:00:00', '18:30:00');

INSERT INTO duty.breakDef
    (dutyId, name, startTime, endTime)
VALUES (1, '午餐', '12:30:00', '14:00:00'),
       (1, '晚餐', '18:30:00', '19:30:00');

COMMIT;

Upgrade script example

BEGIN;
-- 已有表字段变更
ALTER TABLE duty.signrecords ADD COLUMN clientId VARCHAR(40);
ALTER TABLE staff.staffs ADD COLUMN supervisor VARCHAR(100);
ALTER TABLE staff.staffs ADD COLUMN password VARCHAR(64);

-- 数据升级
UPDATE staff.staffs SET supervisor='00001,00002';
UPDATE staff.staffs SET password='8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92';


-- 新增表
CREATE SCHEMA IF NOT EXISTS users;

CREATE TABLE IF NOT EXISTS users.userroles (
  staffId          VARCHAR(15) NOT NULL,
  rolename         VARCHAR(15) NOT NULL,
  PRIMARY KEY (staffId, rolename)
);

-- 新增表补充默认数据
INSERT INTO users.userroles(staffId,rolename)
VALUES
('00001','hr'),
('00001','supervisor'),
('00002','hr'),
('00002','supervisor'),
('00003','supervisor');

COMMIT;

Recommended partner

According to my experience of using so many years, there are the following usages, which can maximize the role of Flyway

1. H2+Flyway

H2 is a Derby-like database implemented in Java. Its biggest features are three:

• Pure Java implementation, just introduce a jar file in the code, and you have a database;
• Support memory, file, C/S three modes;
• Compatibility with the SQL standard, and compatibility mode with other databases.

The above three characteristics determine that it is especially suitable for continuous integration or one-click deployment projects:

• The development environment uses the memory mode, cooperates with Flyway to automatically build tables, and brings an extremely fast development experience together with TDD;
• The test environment uses memory mode or file mode, and it is not a dream to re-run automated tests a hundred times a day;
• The production environment uses file mode or C/S mode, and dynamic expansion of the (Web) server is not a problem.

If you cooperate with Spring-Boot's Profile mechanism, a set of codes can be switched flawlessly in the development environment, test environment, and production environment:

#application.properties:
#主配置文件中配置好所有默认参数
spring.profiles.active=dev
spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.initialize=false
flyway.baseline-version=0.0.0
flyway.baseline-on-migrate=true
flyway.validate-on-migrate=false

#application-dev.properties:
#开发环境使用内存模式，支持一键运行
spring.datasource.url=jdbc:h2:mem:kq

#application-test.properties:
#测试环境使用内存模式或文件模式，支持反复运行或数据持久化
spring.datasource.url=jdbc:h2:~/data/h2/kq
#保密要求高时，请使用JAVA虚拟机参数配置账号密码，如: -Dspring.datasource.username=test
spring.datasource.username=test
spring.datasource.password=123456

#application-prod.properties:
#生产环境建议所有数据库参数都使用JAVA虚拟机参数配置
spring.datasource.url=jdbc:h2:/var/lib/h2/kq
#尤其是账号密码，一定不要写死在配置文件中
#使用JAVA虚拟机参数配置账号密码，如: -Dspring.datasource.username=test

#开发环境运行项目：
mvn spring-boot:run

#测试环境运行项目：
mvn spring-boot:run -Dspring.profiles.active=test

#生产环境通过jar包运行项目：
java -jar kq.jar -Dspring.profiles.active=prod -Dspring.datasource.username=secret ...

#生产环境部署在tomcat下，在setenv.sh中配置参数：
#tomcat/bin/setenv.sh:
JAVA_OPTS="${JAVA_OPTS} -Dspring.profiles.active=prod"
JAVA_OPTS="${JAVA_OPTS} -Dspring.datasource.username=secret"
JAVA_OPTS="${JAVA_OPTS} -Dspring.datasource.password=secret"
...

 

2. H2+Postgresql+Flyway

    H2 is a very good choice in the development environment and some small projects, especially its memory or file mode has no network overhead, and it is very fast to start and run. But when the amount of data reaches millions, its performance is obviously inferior to several large C/S databases at the head of RDBMS.

    Due to the good compatibility of H2 and Postgresql for SQL standards, it is not difficult to switch from H2 to Postgresql, which enables us to enjoy the extremely fast development mode brought by H2+Flyway, and the stability of Postgresql and the support of large data volume in the same project There is no conflict. Also with Spring-Boot, we can configure it like this:

#application.properties:
#主配置文件中配置好所有默认参数
spring.profiles.active=dev
spring.datasource.initialize=false
flyway.baseline-version=0.0.0
flyway.baseline-on-migrate=true
flyway.validate-on-migrate=false

#application-dev.properties:
#开发环境使用内存模式，支持一键运行
spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.url=jdbc:h2:mem:mydb;MODEL=;MODE=PostgreSQL

#application-sit.properties:
#自动化测试环境使用内存模式，支持一键运行
spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.url=jdbc:h2:mem:mydb;MODEL=;MODE=PostgreSQL

#application-uat.properties:
#用户模拟测试使用postgresql数据库，保证代码与postgresql的兼容性
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localsrv:5432/mydb
#保密要求高时，请使用JAVA虚拟机参数配置账号密码，如: -Dspring.datasource.username=test
spring.datasource.username=test
spring.datasource.password=123456

#application-prod.properties:
#生产环境建议所有数据库参数都使用JAVA虚拟机参数配置
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localsrv:5432/mydb
#尤其是账号密码，一定不要写死在配置文件中

#开发环境运行项目：
mvn spring-boot:run

#自动化测试环境运行项目：
mvn spring-boot:run -Dspring.profiles.active=sit

#用户模拟测试环境运行项目：
mvn spring-boot:run -Dspring.profiles.active=uat

#生产环境通过jar包运行项目：
java -jar kq.jar -Dspring.profiles.active=prod -Dspring.datasource.username=secret ...

#生产环境部署在tomcat下，在setenv.sh中配置参数：
#tomcat/bin/setenv.sh:
JAVA_OPTS="${JAVA_OPTS} -Dspring.profiles.active=prod"
JAVA_OPTS="${JAVA_OPTS} -Dspring.datasource.username=secret"
JAVA_OPTS="${JAVA_OPTS} -Dspring.datasource.password=secret"
...

It should be noted that there are still differences between stored procedures, non-SQL standard types, and functions. If you don’t use these, you can switch between H2 and Postgresql smoothly without changing a line of code; if you use special functions, you can extend H2 functions through Java to ensure that SQL is consistent with Postgresql; stored procedures and special types, you need everyone Researched it myself.

3. Docker+Mysql/Postgresql+Flyway

    Since using Docker six years ago, I've loved its ability to run/stop/reset/"uninstall" a suite of open source software (eg databases, web services) with a single command. Through Docker, you can try or use most of the new open source things without leaving any traces without installing/uninstalling them. In addition, Docker now fully supports Windows, Mac, and Linux. What software to use now? There are no ready-made Docker images.

    Using Docker+Flyway for continuous integration (repetitive database reconstruction) is also a very good choice. Docker configures a script or two to achieve the effect of installing or resetting the database:

#删除已经存在的数据库容器，即使不存在也没什么影响
docker rm mydb
#启动数据库容器
docker run --name=mydb \
  -p 5432:5432 \
  -d --restart=unless-stopped \
  -e POSTGRES_USER=${dbuser} \
  -e POSTGRES_PASSWORD=${dbpwd} \
  -e POSTGRES_DB=${database} \
  postgres:alpine

Using Docker also brings another advantage - the development, testing, and production environments use the same image, which can ensure the consistency of the three environments, and will not encounter environment-related problems.

 

Issues to consider when using Flyway

Of course, if you want to use Flyway well, some issues need to be considered in advance:

• How to use Flyway on an existing project

    I also started using Flyway after a period of time since the project started. When I launched Flyway, I didn't encounter anything more difficult than manually executing scripts. Existing projects that want to use Flyway can follow these three steps:

1. Export the table building script of the existing database and the basic data necessary for the new project, and put it into the Flyway upgrade script, such as: src/main/resources/db/migration/V0.0.0__init.sql
2. According to the actual situation of the project, introduce Flyway according to the above method;
3. In the Flyway configuration, before calling migrate, set the baselineVersion to a value greater than the version in step 1, such as: V0.0.1.

The baselineVerion parameter setting method is also divided into three types: Java code, Spring Xml configuration, and Spring-Boot configuration file. Seeing that you must be able to set it yourself, I will not repeat it.

• What if there are more and more upgrade scripts?

After dozens of versions, there are more and more scripts under "src/main/resources/db/migration", which brings two problems:

1. Too many scripts and difficult to maintain;
2. The latest table structure cannot be seen intuitively through SQL scripts.

My experience and advice are:

1. Turn off Flyway's script file checksum check, that is, set validateOnMigrate=false;
2. Periodically merge historical scripts, such as merging V1.0.1, V1.0.2, and V1.0.3 scripts into V1.0, and delete the first three files.

Scripts incorporate skills that are not required for Flyway, I have some tricks to make the SQL more concise:

1. Incorporate the ALTER statement into the CREATE statement;
2. DROP and CTEATE+ALTER offset;
3. UPDATE is merged into the INSERT statement (basic data upgrade) as appropriate, or removed directly (business data upgrade).

Epilogue

    If you are using a relational database, use Flyway now and start your fast coding experience!

    If you switch to NoSQL because of RDBMS data upgrade problems, then quickly find an opportunity to switch back to RDBMS and use Flyway to manage the upgrade! Otherwise, more online bugs are waiting for you!

References

• Flyway official documentation: https://flywaydb.org/getstarted/how
• Practical case: One-click deployment of minimalist attendance system  https://gitee.com/chentao106/kq

 

{{o.name}}

{{m.name}}