Three black dragon continuous integration server, delivery, deployment road

Foreword

  Today, the continuous integration of software development has become standardized processes, agile development is also an important part. The so-called continuous integration refers to the software development team members on a regular basis (every day or even less) will merge their product development efforts to the project by an automated build (compile, publish, automated testing, etc.) to verify the results of this development, as soon as possible find errors and vulnerabilities (this may be the problem of the development of the content itself, it may be compatibility issues between the existing product development content and content).

  The continuous delivery, it is designated a new version of the product delivered to the team or the quality of the user review and testing to find defects as early as possible, risk assessment and troubleshooting before the product into the production environment. In the game development process, we continued to deliver the performance of the server / client latest code (resources) to update the test server, for planning / QA team to test.

  Finally continuous deployment, refer to the code delivered by the previous step after the assessment, it can be automatically deployed to the production environment. His goal is that the code can be deployed at any one time is, you can enter the production stage. In the game development process, the continued deployment of the performance of the server / client code (resources) to update the official service.

  Development team through continuous integration, delivery, deployment, can greatly reduce the functional integration problems and improve the efficiency of the development team, the team can develop high-quality software products.

  Since the continuous integration is so important, why "black dragon" This product has not been doing continuous integration and other related processes it? This history from the project start.

Project History

  Black dragon team project in early 2014, black dragon in early development, the forces are focused on the product itself, in order to quickly on-line products, highly valued (chóng) with the company's existing project developments. But in terms of the deployment of peacekeeping operation, it is still very primitive means: manually compile, package manually, manually update .... The good news is that black dragon because of the short development cycle, before the emergence of other similar games occupied the "realistic three cards" The emerging markets in Japan, Korea, Southeast Asia and have achieved good results. However, because of lack of respect, etc., including continuous integration process, but also brought a lot of questions and problems to the project team.

problem

1. Game Product value changes frequently, but the plan is no independent way to update the test server configuration table. The efficiency of its over-reliance on server programmers to help update the configuration table, increasing the configuration table update time consuming, but also seriously affected the server programmer.
2. Manually compile, package, updated for the server programmer is concerned, not only time consuming, and error-prone process. (Temporary maintenance updates to the accident many times because the code is not compiled but there have been a result of historically black dragon)
3. No automated testing process (unit testing), relying only when smoke testing, black box testing and acceptance update function when the function, and can not guarantee the quality of the code.
4. Update the official service will still rely on manual updates uploaded to the specified location, and operation and maintenance update operation, update the content does not guarantee exactly the same (although the increase later in formal clothes and md5 checksum test server, but still time-consuming process of uploading and updating laborious)
5. Update the official service personnel involved in the operation and maintenance needs, developers need at each docking maintenance and operation and maintenance personnel to maintain content, there will be inadvertent operation and maintenance personnel and drain update certain server.

Prototype: configuration table publishing tools

  So many questions need to be solved step by step. The most basic and most urgent problem is the question 1 (update planned independent testing service configuration table), because it seriously affects the efficiency of planning and server programmers. Time was 2016, when I came dragon team, received the first task is to develop a plan can be delivered to the table with updated test server configuration tools within a week. After combing the server update process, roughly handled about implementation steps: invoke a remote server script shut down the server -> Upload planning configuration table needs to be published -> remote boot server. Because of the time requirements are relatively tight, so that only one version of the console, to the following figure:

  Although relatively simple, the interface is not very friendly, but it has been able to basically solve the problem of planning to update the configuration table. Later also increased packaging code, release code and other functions.

  Postscript: It may be mind-set limit, only in accordance with the tasks required to do a tool to address update update problem, and did not think of using jenkins to introduce automated processes, but continues to move towards optimization iterations of the publishing tools, resulting in black dragon complete continuous integration, delivery local deployment delayed the work, which is more mistakes, should take warning.

Optimization: the graphical interface version

  Although the testing service has been publishing tools, but the console window unfriendly interface, to maneuver, even I can compare anything against it, let alone other people. Then there was the idea for the graphical interface version, in the investigation of the various needs of planning, procedures, I used to work in neutral to start the GUI version of the development. Because the desktop software focuses on streamlining, hundreds of m jre environment so I gave up the idea to continue to use java as a development language, but turned python. python GUI development framework wxpython, ssh library has paramiko, windows environment package has pyinstaller, fully in line with my expectations, so the graphical interface version was born.

  The version of the interface by selecting jdk path, automatic code-compile, package, upload, update, the basic realization of continuous integration, delivery workflow, and solves the problem 2 (test server update process is tedious, error-prone problem).

Code has been uploaded to GitLab, are interested can go to View

Continuous integration, delivery and deployment

  With the continuous integration process after the above, I was thinking about how to do continuous deployment, but discovery seems to have entered a dead end. Because using a custom tool mode continuous integration and delivery, the code did not submit, does not guarantee that the updated test server code is complete and up to date, so when the operation and maintenance update the official service can not use the test server code directly, Therefore, the existing custom update tool can not complete the follow-up process continued deployment.

Jenkins

Jenkins introduced

  Jenkins is an open source CI & CD (continuous build and deployment) software, used to automate a variety of tasks, including build, test and deploy software. You can use Maven to build Java applications with Node.js and npm React to build applications with PyInstaller to build python applications and so on. Jenkins also install and use very simple, the official website details the different platforms in a variety of installation methods, I chose the most convenient to use war package to run, because the server comes with JDK environment.

Use Jenkins

  jenkins的使用非常简单，只要按照提示一步步进行即可。这里选择最常用的“构建一个自由风格的软件项目”，如果后续步骤比较复杂，可以考虑使用流水线。

现有的构建方式 or Maven改造？

  这一步比较关键，Jenkins让我们选择使用何种方式来构建项目。这里我进行了一番斟酌，按照目前苍龙传统的构建方式，是使用javac来编译，然后使用zip来打包应该选择shell方式。但目前这种方式没有持续集成流程中“自动测试”这样一环，而自动测试对于持续集成来讲又非常重要。因此为了加入自动测试以及今后能更方便的构建苍龙服务器代码，最后决定，首先进行Maven的集成。

Maven集成

Maven介绍

  Maven是一款专门对Java应用进行依赖管理的工具，它采用了统一的标准（Pom文件）来构建Java应用，使得Java开发者对于项目中使用的依赖组件能够一目了然，很便捷地处理依赖冲突的问题，也能很高效地完成编译、打包等操作。除此之外，Maven还能在打包之前完成自动化单元测试，非常有利于开发人员的自我测试，在功能开发阶段就找出并解决一些Bug。但非常不幸的是，苍龙服务器代码因为一些历史原因，并没有使用Maven。没有条件就创造条件，下面就开始了Maven集成工作。

Maven集成步骤

1. 整理各服务器项目代码（游戏服、战斗服、世界服、日志服）所使用的jar包，区分出公有jar包与私有jar包。
2. 将公有jar包配置为直接从阿里云中央仓库下载，私有jar包上传到私有仓库，再从私有仓库下载。
3. 整理与解决依赖冲突问题

遇到的问题与解决

1. 大部分私有jar包来源未知，并且没有源码，也在开源平台中无法找到（应该是祖传jar包），无法通过编译源码方式deploy到私有仓库。解决方法：在Nexus Repository Manager后台登陆后直接上传jar包。
2. 某些jar包根据名字和内容判断，应该是公有jar包，但是却没有标注版本号（坑啊！），因此无法从阿里云中央仓库中下载。解决方法：作为私有jar包处理。
3. 原项目中jar包冲突却一直没被发现，例如activemq-all-5.10.2.jar与log4j-slf4j-impl-2.1.jar冲突，启动时会提示slf4j重复绑定。解决方法：将all包拆分为单个组件，并使用exclusion标签排除冲突的jar。
4. 原项目代码采用JDK1.6编译，但JDK1.6无法兼容最新版的Maven3.6.x。解决方法：使用最后能支持JDK1.6的Maven3.2.5版本。
5. （最棘手）项目路径问题。普通Java项目源码路径为ROOT/src，但maven项目的源码路径为ROOT/src/main/java。如果要更改源码路径，那么以前在SVN上建立的分支将无法识别新的源码路径（已经过测试验证）。这个问题引起了我的好奇，并且发现在Git上修改源项目码路径不会有该问题，但是在SVN上就会出现。经过一番比较深入的研究后发现，在Git中如果将某个文件A移动了路径B，git会记录该版本进行了rename A->B，在其他分支上对A的修改仍然可以合并到B。而SVN则不一样，如果文件A移动到了路径B，SVN会记录该版本将A删除，新建了B，在其他分支上对A的修改就无法合并到B。因此，受苍龙使用SVN的限制，就无法修改源码路径，那么如何使Maven能够识别原有的源码路径ROOT/src呢？好在Maven提供了对源码和资源指定的支持：

<sourceDirectory>src</sourceDirectory>
<testSourceDirectory>test/java</testSourceDirectory>
<resources>
    <resource>
        <directory>resources</directory>
    </resource>
</resources>
<testResources>
    <testResource>
        <directory>test/resources</directory>
    </testResource>
</testResources>

  至此，Maven集成完毕。

测试集成

  Maven已经有了，那么可以开始使用Jenkins进行持续集成了吗？不！不要忘了Maven还有一项重要使命：自动完成单元测试。苍龙服务器代码中虽然之前有引入JUnit包，但是需要手动运行，非常不方便（以前也从来没有手动运行过）。测试集成也比较简单，主要分为两类：

单元测试

• 静态公共方法(public static ...)：直接在测试类中调用即可。

• 私有方法(private ...)：这类方法无法在测试类中直接调用，这时候Java的反射调用就能排上用场了：

  Method method = xxx.getClass().getDeclaredMethod("xxx", xxx.class,xxx.class);
  method.setAccessible(true);
  method.invoke(xxx, xxx, xxx);

• Spring注入对象的成员方法（service等）：从Spring上下文中获取对象再调用即可。

Mock测试

  有时候编写测试代码时会出现这些情况，某个类可能过于复杂，可能因为依赖过多，甚至可能构造方法因为业务需要被设置成了私有（private）访问，导致我们无法直接new出这个对象，这时候怎么测试呢？这时候Mock测试就派上用场了。我选用的是目前使用最广泛的mockito库，步骤如下：

1. 模拟出该对象：XXX mock = mock(XXX.class);
2. 设置XXX.class的xx()方法返回值：when(mock.xx()).thenReturn(xx);

  这样就可以在测试代码中，很方便的调用mock.xx()了，非常简单。

  以后在每次Maven打包时，就会自动查找test路径下所有类中以@Test注解的所有public void xxx测试方法，如果测试结果与预期不符，就会终止打包，我们便可以在打包阶段就解决掉一些Bug，降低产品的风险。

流程图

  类似于之前采用工具的持续集成方式，jenkins的集成流程也分为了程序版与策划版。

程序版流程图

策划版流程图

优化

  可以看到，两条流水线的区别就是持续集成不同，程序是检出的代码，并且需要编译、测试、打包；而策划检出的是配置表。

  两条流水线采用的是独立运行的方式，程序负责代码的持续集成，策划负责配置表的持续集成。使用了一段时间之后，我们发现，这种方式在开发版本（主线分支）没有问题，因为开发版本只有测试服没有正式服。而到了生产版本（线上分支），如果按照这种方式，程序、策划流水线独立运行，会导致在正式服维护期间，需要重启2次正式服。而正式服的重启又比较重度（需要备份线上数据），这就导致服务器更新时间翻倍（10分钟增加到20分钟）。为了解决上述问题，我对上述方案进行了改进，将两条流水线合并。

流程图如下：

  在该优化版本中，将之前的两条流水线相同的部分合并，不同的部分保留（SVN检出配置表、代码）。配置表和代码的产物生成之后，将两份产物合并作为新的产物，再生成版本号，用该版本产物来更新测试服与正式服。

服务器更新操作流程对比

• 原有更新流程（每一步均为手动操作）：

• Existing update process (only one step):

Comparison details

The following are yes / no continuous integration, delivery, compared to results in the deployment:

• The official service maintenance time
  • No: 15 minutes + Developers operation and maintenance personnel for 15 minutes.
  • They are: Developer 1 seconds (no need to participate in the operation and maintenance personnel).
• The official service maintenance server update
  • No: about 30 minutes.
  • They are: about 10 minutes.
• Operational complexity
  • No: about 20 steps, very complex.
  • There: just point the mouse, it is very simple.
• Probability of error
  • None: high, 20 steps performed manually, if an error, it will update error (on several occasions arise because the code is not compiled, are not planning to use the latest configuration table appears, operation and maintenance personnel missing a feature update service and other issues ).
  • They are: almost zero, using a tested and validated procedure for updating the script.

to sum up

1. Continuous Integration and Deployment (CI / CD) should be established early in the project, can greatly improve development efficiency, improve product quality, make the product even further.
2. Custom-style continuous integration tool during the ongoing deployment would be more difficult to achieve the recommended or the use of Jenkins continuous integration, delivery and deployment.
3. Practice has proved that custom tools can be useful in the testing of code or configuration tables, because in this way will not worry about updating or testing of the code configuration tables to the official service.
4. Today, the idea of ​​DevOps has been popular in the world, it is the emphasis on software development (Dev) a cultural communication and cooperation between the operation and maintenance technology (Ops) and "software delivery" and "schema changes," the process through automation, makes building , testing, release the software can be more efficient, frequent and reliable.

参考资料：The Product Managers’ Guide to Continuous Delivery and DevOps