Java Architecture Patterns and Design Patterns (11)-Application and Practice of 12-Factor in the Cloud Native Era

Original link

About the author: Huang Qingbing, chief technical evangelist of NetEase Honeycomb, graduated from Zhejiang University, engaged in cloud computing, Docker, Go and other related development and technical evangelism work; likes open source, willing to share, diligent in evangelism, tossed open source gadgets, made Docker Course, shared Gopher Meetup. Welcome to discuss technology together! Personal homepage: http://bingohuang.com/

The following is the text:

1 Introduction

In the cloud era, applications will migrate more to the cloud. Cloud-based architecture design and development models require a new set of concepts to carry, so cloud-native ideas came into being, and best practice principles for cloud-native application development , 12-Factor stands out, and it also brings a new interpretation. This article will introduce how to practice the 12-Factor principle one by one in the Cloud Native era, combined with Docker and other technologies.

2. Cloud Native

Cloud Native is a concept proposed by Matt Stine of Pivotal in 2013. It is a collection of ideas summarized by his many years of architecture and consulting.

So how to understand cloud native applications? I think it can be explained from three angles, which coincides with the three levels of the cloud computing platform, as shown in the figure below:

 

1.png

• IaaS is regarded as a basic cloud facility to provide various basic resources (Infrastructure)
• PaaS is used as a development platform to provide various platform services (Platform)
• SaaS delivers applications or services, faces users directly, and provides application value (Application)

Cloud-native applications happen to fit the cloud, platform, and services, and are built layer by layer, so I usually understand it as designing our applications for the cloud (platform). Chen Xiaohui, the architect of [NetEase Sanshi Crowdfunding] [3], also gave it a small fresh name-Born to the Cloud, I think it is very appropriate, and in layman's terms, it can also be called a cloud platform application.

3. 12-Factor

12-Factor, first proposed and open sourced by Adam Wiggins, founder of Heroku, and jointly improved by many experienced developers. This combines almost all their experience and wisdom on SaaS applications, and is an ideal practice standard for developing such applications .

The full name of 12-Factor is The Twelve-Factor App. It defines some basic principles that need to be followed in the design process of an elegant Internet application, which is similar to Cloud-Native. There are a lot of translations in the text. I think the "12 elements" or "12 principles" are more appropriate.

What are the specific twelve principles, see the figure below:

 

2.png

In the following application and practice, we will practice each principle one by one.

Note: Although the original books of 12-Factor are published on its official website, it is not very convenient to read due to network problems and format problems. I converted it to GitBook format and set it up on the NetEase honeycomb platform. At the same time, it is open sourced in GitBook format. On GitHub, it is convenient for everyone to read and download:

Online reading address: http://12.bingohuang.com/zh_cn/index.html
GitHub open source address: https://github.com/bingohuang/12factor-gitbook
pdf/epub Download address: https://github.com/ bingohuang/12factor-gitbook/download
GitBook address: https://www.gitbook.com/book/bingohuang/12factor/details

1. Application and practice

Since 12-factor is the best practice principle for SaaS development, of course, it cannot be separated from practice. Next, we will design a cloud-native application, and follow the 12-factor to verify and upgrade our application step by step. From it, we will explain the main points of each factor and how to practice the factor in our application.

0. Application preparation

This is a simple web application designed for the cloud platform. It will expose an HTTP REST style interface, which can realize the addition, deletion, modification, and checking of users. The following technology stacks will be used:

1. Based on Node.js, it is very convenient to write web applications with Node.js, and it is one of the most popular programming platforms today

Download and install Node.js (including npm): https://nodejs.org/zh-cn/download/

• Note: As long as the Node version is v4.4 or higher, the latest stable version is v6.9.5, and my local version is v5.12.0

2. Based on Sails, similar to Rails framework, for rapid development of Node.js applications: http://sailsjs.com/

Install the latest version of the Sails framework:
<code>npm install sails -g</code>

3. Based on mongo 3.2: https://docs.mongodb.org/manual/installation/

4. Based on Docker, it fits well with the 12-Factor concept, as our tool for packaging, publishing and running

安装 Docker：https://docs.docker.com/engine/installation/

5. After the above environment is installed, we will initialize our application and run it. The name of the application is: 12factor-app

<code>
$ sails new 12factor-app
info: Created a new Sails app 12factor-app!
$ cd 12factor-app
$ sails generate api user
info: Created a new api!
$ npm start
</code>

Only 4 commands are required to complete the application framework code and automatically generate a user-based CRUD interface. We have started the application and can debug the interface locally in the following ways:

The console output is normal. Visit the link below in the browser to see the home page of the Sails application

 

5.PNG

Then, you can use the local curl command or http tool to do interface debugging, here is an example of regular addition, deletion, modification and checking:

1.增加一个新用户
<code>
$ curl -XPOST http://localhost:1337/user?name=bingo
{
"name": "bingo",
"createdAt": "2017-02-13T06:13:53.791Z",
"updatedAt": "2017-02-13T06:13:53.791Z",
"id": 58a41d952f53291200b9e065
}
</code>

2.获取用户列表
<code>
$ curl http://localhost:1337/user
[
{
"name": "bingo",
"createdAt": "2017-02-13T06:13:53.791Z",
"updatedAt": "2017-02-13T06:13:53.791Z",
"id": 58a41d952f53291200b9e065
}
]
</code>

3.修改一个用户
<code>
curl -XPUT http://localhost:1337/user/58a41d952f53291200b9e065?name=bingohuang
{
"name": "bingohuang",
"createdAt": "2017-02-13T06:13:53.791Z",
"updatedAt": "2017-02-13T06:14:13.460Z",
"id": 58a41d952f53291200b9e065
}
</code>

4. Delete a user

<code>
curl -XDELETE http://localhost:1337/user/58a41d952f53291200b9e065
</code>

I have deployed this application to the NetEase Hive online platform. If you are interested in this application, you can directly replace localhost with 59.111.110.95, you can also experience CRUD operations, as shown below, just replace yourname with your name ( It is recommended to operate on the PC side):

<code>
Register yourself
curl -XPOST http://59.111.110.95:1337/user?name=yourname to
view all registered users
curl http://59.111.110.95:1337/user
or PC browser to directly visit http: //59.111.110.95:1337/user
</code>

Next, let's start practicing each principle in 12-Factor one by one. In each principle, we will be divided into Factor explanation and Factor practice.

I. Benchmark code

Factor commentary:

The 12-Factor application has only one codebase (Codebase) and can be deployed in multiple copies (deploy). This means that an application has only one code repository for tracking all revisions, and there is always a one-to-one correspondence between the benchmark code and the application, because:

• Once there are multiple benchmark codes, it cannot be called an application, but a distributed system. Each component in the distributed system is an application, and each application can be developed separately using 12-Factor.
• Sharing a benchmark code for multiple applications is contrary to the 12-Factor principle. The solution is to split the shared code into independent class libraries, and then use the dependency management (second principle) strategy to load them.
• Multiple deployments are equivalent to running multiple instances of the application, such as one instance in the development environment, one instance in the test environment and production environment
• A code repository ensures a single source of trust, thereby ensuring fewer configuration errors and stronger fault tolerance and resilience
• 

 

11.png

Factor practice:

Use Git as the application version management system, and use our online repository
on GitHub to execute in the newly created application directory:

<code>
$ echo "# 12factor-app" >> README.md
$ git init
$ git add .
$ git commit -m "first commit"
$ git remote add origin [email protected]:bingohuang/12factor-app.git
$ git push -u origin master
</code>

II. Dependence

Factor commentary:

Applications under the 12-Factor rule will not implicitly rely on system-level class libraries. This means that all dependencies are declared through the dependency list, and the dependency isolation tool is used to ensure that the program does not call dependencies that exist in the system but are not declared in the list. And unified application to the production and development environment.

The cloud platform manages dependencies based on these statements to ensure the libraries and services required by cloud applications.

Factor practice:

package.json 就是我们的 依赖清单，所有应用程序的依赖都声明在此。
<code>
{
"name": "12factor-app",
"private": true,
"version": "0.0.0",
"description": "a Sails application",
"keywords": [],
"dependencies": {
"ejs": "2.3.4",
"grunt": "1.0.1",
"grunt-contrib-clean": "1.0.0",
"grunt-contrib-coffee": "1.0.0",
"grunt-contrib-concat": "1.0.1",
"grunt-contrib-copy": "1.0.0",
"grunt-contrib-cssmin": "1.0.1",
"grunt-contrib-jst": "1.0.0",
"grunt-contrib-less": "1.3.0",
"grunt-contrib-uglify": "1.0.1",
"grunt-contrib-watch": "1.0.0",
"grunt-sails-linker": "~0.10.1",
"grunt-sync": "0.5.2",
"include-all": "^1.0.0",
"rc": "1.0.1",
"sails": "~0.12.11",
"sails-disk": "~0.10.9"
},
"scripts": {
"debug": "node debug app.js",
"start": "node app.js"
},
"main": "app.js",
"repository": {
"type": "git",
"url": "git://github.com/bingo/12factor-app.git"
},
"author": "bingo",
"license": ""
}
</code>

Next we add the library dependency of mongodb (will be used later), just execute:
<code>
npm install sails-mongo --save
</code>

At the same time, there will be corresponding changes in package.json
<code>
{ ... "dependencies": { ... "sails-mongo": "^0.12.2" //Latest dependencies added } ... } </ code> The dependency libraries needed by the application are installed in the node_modules folder, which is used as the dependency isolation of the application, and is isolated from the system libraries

III. Configuration

Factor commentary:

12-Factor recommends storing the configuration of the application in environment variables to ensure that the configuration is excluded from the code, which has the following benefits:

• Environment variables are a clear, easy-to-understand and standardized configuration method
• Environment variables can be easily modified between different deployments without changing a single line of code
• Unlike configuration files, the probability of accidentally checking them into the code base is extremely small
• Compared with some traditional mechanisms for solving configuration problems (such as Java property configuration files), environment variables have nothing to do with language and system
• Another advantage of storing in environment variables is that it is convenient to use with Docker

A trick: to determine whether an application correctly excludes the configuration from the code, a simple way is to see that the application can be open source immediately, without worrying about exposing any sensitive information:)

Factor practice:

In the application's config/connections.js file, we use the MONGO_URL environment variable to define the mongo connection method
<code>
module.exports.connections = { mongo: { adapter:'sails-mongo', url: process.env .MONGO_URL } }; </code> In the file, specify the connection used by the module <code> module.exports.models = { connection: mongo, migrate:'safe' }; </code>

If you have started a mongodb test service locally, you can use this command to verify whether the application is properly configured
<code>
MONGO_URL= mongodb://localhost:27017/12factor-app npm start
</code>

IV. Backend Service

Factor commentary:

The 12-Factor application does not treat local or third-party services differently, and treats backing services as additional resources or remote resources.

• The so-called back-end services refer to the various services that are required by the program to run through the network, such as database (MySQL), message/queue system (RabbitMQ), SMTP mail delivery service (Postfix), and cache system (Memcached), etc.
• In addition to local services, the application may use third-party publishing and management services, such as SMTP (such as Postmark), data collection services, data storage services (such as Amazon S3), and services that use API access (such as Twitter) Wait

For applications, local or third-party services are additional resources. To obtain data through a URL or other settings stored in the configuration, you only need to modify the resource address in the configuration.

The application is therefore fault-tolerant and resilient, because it requires the unavailability of resources to be considered when coding, and it also enhances the benefits of the microservice method

 

15.PNG

Factor practice:

• For our application, the backend service used is the MongoDB database. It is through MONGO_URL that we pass the resource address of MongoDB, thus realizing the decoupling of back-end services and applications.
• If there is a problem with the current MongoDB instance, we can easily switch to a new instance by setting the MONGO_URL environment variable

V. Build, release, and run

Factor commentary:

The 12-facfor application strictly distinguishes the three steps of building, releasing, and running:

• The construction process of the Cloud Native application moves most of the release configuration to the development stage, including the actual code construction and the production environment configuration required to run the application
• For example, it is very undesirable to directly modify the code in the running state, because these modifications are difficult to synchronize back to the build step.
• The released version is like a ledger that can only be added. Once released, it cannot be modified. Any changes should produce a new release version.

 

15.PNG

Factor practice:

In view of this principle, it is strongly recommended to use Docker and its components (Compose). Its core concept is: Build, Ship and Run, which will be suitable for the entire build, release and run process. We will also explain from these three aspects .

Construct:

Write the build script: Dockerfile
<code>
FROM hub.c.163.com/library/node:5.12.0
MAINTAINER bingohuang [email protected]

Copy dependency list

COPY package.json /tmp/package.json

Install dependencies

RUN cd /tmp && npm install --registry=https://registry.npm.taobao.org

Copy the dependent package to the application directory

RUN mkdir /app && cp -a /tmp/node_modules /app/

Change working directory

WORKDIR /app

Copy application code

COPY . /app

Set application startup port

ENV PORT 1337

Expose application port

EXPOSE 1337

Start the app

CMD ["npm","start"]
</code>

Docker 构建
<code>$ docker build -t message-app:v1.0 .
</code>

Docker image push: you can push it to a specified mirror warehouse, such as the mirror warehouse of NetEase Honeycomb

<code>docker push hub.c.163.com/bingohuang/12factor-app:1.0
</code>

release:

Write the release script: docker-compose.yml

<code>version: '2'
services:
mongo:
image: hub.c.163.com/library/mongo:3.2
volumes:
- mongo-data:/data/db
ports:
- "27017:27017"
app:
image: hub.c.163.com/bingohuang/12factor-app:1.0
ports:
-"1337:1337"
links:
- mongo
depends_on:
- mongo
environment:
- MONGO_URL=mongodb://mongo/12factor-app
volumes:
mongo-data:
</code>

Based on the built image, a publishing process is defined above, and the configuration (MONGO_URL) is injected through environment variables.
<code>
MONGO_URL= mongodb://mongo/12factor-app
</code>

run:

It can be run locally through Docker Compose, or online orchestration through the cloud platform (NetEase Honeycomb will support service orchestration soon)
<code>
docker-compose up -d
</code>
and then view the logs
<code>
docker-compose logs -f
</code>
Note: In order to facilitate quick running of programs and local debugging for those who are not familiar with docker and docker-compose commands, I also provide the docker.sh script in the source code to facilitate the construction, release and operation of the application (see the follow-up for the source code) Information link).

VI. Process

Factor commentary:

• The process of 12-Factor application must be stateless and shareless.
• Any data that needs to be persisted must be stored in a back-end service, such as a database.
• The data in the session should be stored in a cache with an expiration time such as Memcached or Redis.
• In the runtime environment, an application usually runs in one or more processes.
• In the simplest scenario, the code is an independent script, the running environment is the developer's own laptop, and the process consists of a command line (for example, python my_script.py). Another extreme case is that complex applications may use many process types, that is, zero or more process instances.
• This is done to ensure the speed and efficiency of the Cloud Native infrastructure.

Factor practice:

Although this is a simple demo application, looking at the running process in the docker container, we found that there are also 4 processes running. Among them, npm is our startup process, which node app.jsis the process that actually runs the application.
<code>
$ docker exec eaaa922abf08 ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 2.0 1076204 42024? Ssl 18:22 0:00 npm
root 17 0.0 0.0 4340 724? S 18:22 0:00 sh -c node app.js
root 18 0.9 4.5 1253808 93808? Sl 18:22 0:01 node app.js
root 27 1.1 3.7 962884 77076? Sl 18:22 0:01 grunt
</code>
Here, our application process is stateless and persistent All of the data is stored in the back-end service MongoDB. So far
, we have shared 6 Factors and successfully implemented them in our application, Yes!

Next, we continue to share and practice the follow-up 6 factors.

VII. Port Binding

Factor commentary:

The 12-Factor application can create a network-oriented service by self-loading without relying on any web server.

• This means: Web applications provide services through port binding and listen for requests sent to that port.
• The service interface of Cloud Native application prefers HTTP API as the general integration framework.
  It should also be pointed out that this method of port binding also means that one application can become a back-end service of another application, and the caller will use the corresponding URL provided by the server as a resource and store it in the configuration for future calls.

Factor practice:

The docker-compose file defines the port binding for us.
<code>
ports:
-"1337:1337" // The application container exposes port 1337 in the container, and the host maps it to port 1337
</code>
Need to pay attention However, if you deploy multiple application instances on a host, you cannot map a host port to multiple container ports (port conflicts). The solution is to add a load balancer on top of this, and the load hosts are different. Different containers corresponding to the port service.

VIII. Concurrency

Factor commentary:

The 12-factor application is extended through the process model, which treats the process as a first-class citizen, and has the characteristics of no sharing and horizontal partitioning.

• This means that horizontal expansion can be achieved by relying on the underlying platform, without the need for technically difficult multithreaded coding.

For example, HTTP requests can be handed over to the web process for processing, while the permanent background work is handed over to the worker process, and the timing tasks are handed over to the clock to process, so it is very convenient to extend each type of process, as shown in the following figure. :

 

18.png

Factor practice:

As described in the sixth principle, our application has multiple processes, the most important being the Node.js http server process. The processes are stateless and shareless, so we can scale the application horizontally very easily.

IX. Easy to handle

Factor commentary:

The process of 12-Factor application is disposable, which means that any process can be quickly started and terminated gracefully. The advantages of this are:

• This is conducive to rapid and elastic scaling of applications, rapid deployment of changed codes or configurations, and improved robustness
• The process should pursue the minimum startup time, which can provide a more agile release and expansion process
• Once the process receives the termination signal (SIGTERM), it will gracefully terminate

As shown in the figure below, it is an elegant application startup and termination process

 

19.png

Factor practice:

Docker's inherent lightweight and isolation is very suitable for fast startup and graceful termination. Docker is very suitable for practicing this principle. In our application, we have added Docker and Compose practice.

For the online environment, it is recommended to build on a container cloud platform (such as the NetEase honeycomb platform), which can handle the start and stop of the process more elegantly.

X. Environmental Equivalence

Factor commentary:

For 12-Factor applications to achieve continuous deployment, the differences between local and online must be narrowed, including the following three differences:

• Reduce the time gap: developers can deploy code in hours or even minutes
• Narrowing the difference between personnel: developers should not only write code, but also closely participate in the deployment process and the performance of the code online
• Reduce tool differences: try to ensure the consistency of the development environment and the online environment

12-Factor application developers should oppose the use of different back-end services between different environments

This is because different back-end services mean sudden incompatibility, which leads to online problems with code that is normal for testing and pre-release.

Factor practice:

In our application, docker-compose is used as our publishing script, which allows the application to run locally or on any cloud platform that supports Docker. The application does not need to be changed, only the configuration file needs to be modified, which is very good. Eliminates the differentiation of different environments
From past experience, traditional applications and 12-Factor applications will have the following differences:

 

20.png

XI. Log

Factor commentary:

• The 12-factor application itself never considers storing its own output stream. Instead, every running process will have a direct standard output (stdout) event stream.
• When the log is processed by the cloud platform rather than the library included in the application, the log processing mechanism must be kept simple.

Factor practice:

• Many services can provide centralized log management, such as ELK, Splunk, Logentries, and most of them can be easily integrated with Docker
• Here we take Logentries as an example to integrate the log service for the application. You need to add the log service to the docker-compose file, as follows:
  <code>
  log:
  command:'-t a80277ea-4233-7785203ae328'
  image:'logentries/docker-logentries'
  restart: always
  tags:
  • development
    volumes:
  • '/var/run/docker.sock:/var/run/docker.sock'
    </code>
    A typical Logentries panel interface is as follows:

 

21.png

XII. Management Process

Factor commentary:

Developers often want to perform some one-time tasks for managing or maintaining applications, such as:

• Run data migration

• Running a console is also called REPL shell) to execute some code or do some checks against online databases.

• Run some one-time scripts submitted to the code repository.

• In the 12-Factor application, the one-time management process should use the same environment as the normal resident process (application process), and use the same code and configuration, run based on a certain release version, and be released together with other applications

• In Cloud Native, the management task is also a process, not a special tool; it is also important that the process of the management task should not use secret APIs or internal mechanisms.

Factor practice:

• We can define the management service in the docker-compose file and execute it with the program
• We can perform some management tasks through the docker exec command, such as:
  <code>docker exec -ti ADMIN_CONTAINER_ID bash
  </code>
• If multiple containers are on the same network, one container can be used to manage other containers

5. Summary

So far, 12-Factor has been practiced one by one. It can be seen that 12-Factor is not independent of each other, but a whole. Some involve code and frameworks (Node and Rails), some involve tools (Docker and Compose), and some Involving architecture and platform. In the cloud-native era, 12-Factor still has strong vitality. Every principle is the jewel of application development, and every principle is not static. With the emergence of new concepts, the original Factor will be extended and developed. There will also be new principles. If you are interested, you may wish to read the book "Beyond the 12 Factor App", and you will get even greater gains. Finally, I hope this sharing will help you understand cloud-native applications and practice 12-Factor.

6. Reference link:

• 12-Factor online books: http://12.bingohuang.com/zh_cn/index.html
• 12-Factor open source address: https://github.com/bingohuang/12factor-gitbook
• Article sharing link: http://talks.bingohuang.com/2017/cloud-native-12factor.article
• Demo open source address: https://github.com/bingohuang/12factor-app

Author: Netease cloud-based service
link: https: //www.jianshu.com/p/bbdccd020a1d
Source: Jane books
are copyrighted by the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.