[Project] Full-stack development of medical applets Chapter 1 Course introduction 1.5 Build HBase+Phoenix big data platform

[Project] Full-stack development of medical applets

Chapter 1 Course Introduction

1.5 Building HBase+Phoenix Big Data Platform

Usually, the MySQL database is the most used in our development programs, but the MySQL database has an unavoidable hurdle: the data volume of a single table exceeds 20 million
, and the performance of CRUD will drop rapidly.

Therefore, we need to arrange DBA or automated scripts to regularly migrate the half-year or one-year data to the archive database, so that the purpose of shrinking the table can be achieved. If you are developing an ordinary MIS system, there is not so much business data, and a MySQL can be supported without regular archiving. However, projects similar to e-commerce platforms generate a large amount of business data every day, and can accumulate tens of millions of data in a month or two. In this case, either do frequent data archiving, or choose a big data platform that can accommodate hundreds of terabytes without archiving. Here I choose the latter, which can not only reduce the difficulty of development, but also reduce the operation and maintenance cost of hiring a DBA. Why not do it.

1.5.1 Technology selection of big data platform

1 HBase technology

HBase is a distributed, versioned, column-oriented open source database built on top of Apache Hadoop and Apache ZooKeeper.

HBase is a high-reliability, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server_.

HBase is different from general relational databases. It is a database suitable for unstructured data storage. HBase is column-based rather than row-based.

Features of HBase

1. Massive data storage, the tables in HBase can hold tens of billions of rows x millions of columns of data.
2. Columnar storage, data in HBase is stored based on columns, and columns can be added and deleted dynamically.
3. Quasi-sale query, HBase can approach quasi-real-time query (within 100 milliseconds) under the massive amount of data
4. Multiple versions, each column of data in HBaser has multiple versions.
5. High reliability, the data in HBase is stored in HDFS and relies on Zookeeper for the coordinated management of Master and RegionServer.

Although HBase supports table structure, it is not convenient to operate. For example, the following is a student table, which contains a number of fields.

Example: Student data table

If we add a new record to the student table, we need to execute the following command:

put 'Student','0001','StuInfo:Name','Tom Green',1

When we want to get data, we just execute the get command.

get 'Student','0001'

For those of us who are accustomed to SQL statements, if the above commands are simple, it is okay, but if they are superimposed with complex conditions and functions, then the readability will be very poor
, so we need to introduce Phoenix technology.

scan 'mytest:test01',FILTER=>"RowFilter(=,'substring:202006')",LIMIT=>1

count 'mytest:test01',FILTER=>"RowFilter(=,'substring:202004')",INTERVAL=>10000

2 Phoenix Technology

Phoenix adds a syntax presentation layer to HBase, allowing us to use SQL statements to read and write data in HBase. It can do online transaction processing and has
low-latency features, which is much more convenient.

Phoenix: It will compile SQL into a series of Hbasel scan operations, and then generate a standard JDBC result set from the scan results. It only takes milliseconds or seconds to process tens of millions of rows of data. And Phoenix also supports the MyBatis framework, which can be integrated with our project.

1.5.2 Deploying HBase and Phoenix

1 Import image file

First, we find the phoenix.tar.gz image file in the resources shared by the course, then upload this file to the CentOS system, execute the command, and import the image file into Docker.

Order:docker load < phoenix.tar.gz

Excuting an order

Remember to start docker first

View existing mirrors

2 Create the container

Because the image already contains HBase and Phoenix, we only need to create a container.

Since HBase needs to use a large amount of memory, I do not specify a specific memory size here, and the container will automatically use free memory. The data directory in the container is /tmp/hbase-root/hbase/data, I map this directory to /root/hbase/datathe directory of the host machine.

docker run -it -d -p 2181:2181 -p 8765:8765 -p 15165:15165 \
-p 16000:16000 -p 16010:16010 -p 16020:16020 \
-v /root/hbase/data:/tmp/hbase-root/hbase/data \
--name phoenix \
boostport/hbase-phoenix-all-in-one:2.0-5.0

currently running container

3 open ports

The author directly turned off the firewall here

4 Initialize Phoenix

Run the command, enter the Phoenix container, and then execute the command to set HBASE_CONF_DIRthe environment variable.

docker exec -it phoenix bash

export HBASE_CONF_DIR=/opt/hbase/conf/

Next we'll connect to Phoenix's command-line client. Although IDEA also has a built-in Phoenix client, there are many bugs and it is not convenient to use
, so I suggest that you use the command line client that comes with Phoenix better. And we don't have many SQL statements to execute, the command line client
is enough.

/opt/phoenix-server/bin/sqlline.py localhost

Please wait for the first connection

out, so connected

1.5.3 Create logic library and data table

1 Create a logic library

In order to store data, we need to create a logic library first, and then define a data table, just like operating MySQL. In Phoenix's command line client, let's first execute the command to create a logic library.

CREATE SCHEMA hospital;

USE hospital;

2 Create a data table and import data

There are database scripts and SQL files on the git of this course. You can copy and paste the statements in this SQL file to the Phoenix command line to execute.

Just paste it all in one go

1350 lines

Wait for it to finish [Creating an index is time-consuming]

OK, it's done, exit

1.5.4 Configuring JDBC connection information

Here to import the project

Here the teacher used the api to demonstrate

Copy the project to your own directory

Import and open with IDEA

Use our own maven configuration

Quite new, SpringBoot version 2.7.2

Wait for the project build to complete

Note that the JDK version used is the 15 we installed earlier

build complete

Here we need to change it to our server (virtual machine IP)

There is another Java project, now open this is the interface background of the WeChat applet, and there is also a web-side interface background

Also use IDEA to import and open

Also use our local maven configuration

Just change the above place in the same way

The springboot version of this project is two points higher

OK, that's it.