Notes on the first day of "Seven Days to Play PolarDB-X Open Source Training Camp"
Course address: PolarDB-X Open Source Talent Primary Certification Training Course-Learning Center-Alibaba Cloud Developer Community (aliyun.com)
1. Introduction to PolarDB-X
1. Development of PolarDB-X
PolarDB-X is a cloud-native distributed database system designed for ultra-high concurrency, massive storage, and complex query scenarios. It adopts a shared-nothing and storage computing separation architecture, supports horizontal expansion, distributed transactions, mixed load and other capabilities, and has the characteristics of enterprise level, cloud native, high availability, and high compatibility with MySQL system and ecology.
PolarDB-X was originally born to solve the database scalability bottleneck of Alibaba's Tmall "Double Eleven" core transaction system, and has since grown along with Alibaba Cloud. It is a mature and stable database system that has been verified by various core business scenarios.
2. Evolution of PolarDB-X system architecture
1. At the beginning of the system, it used a simple single DB and a local disk.
2. With the development of technology, there are higher requirements for system capacity, support for concurrent reading and writing, and expansion of storage scale. Therefore, the middleware of sub-database and sub-table is used to solve the distributed 1.0 generation of system scalability.
3. Since the middleware will introduce some new problems, the continuous iteration of distributed has produced the distributed 2.0 architecture, which integrates the upper distributed layer and the lower storage layer to form an integrated form.
3. The system architecture of PolarDB-X
PolarDB-X is designed with Shared-nothing and storage separation computing architecture. The system consists of 4 core components:
- Compute Node (CN, Compute Node)
The computing node is the entrance of the system, adopts a stateless design, and includes modules such as SQL parser, optimizer, and executor. Responsible for data distributed routing, calculation and dynamic scheduling, responsible for distributed transaction 2PC coordination, global secondary index maintenance, etc., and provides enterprise-level features such as SQL current limiting and separation of three rights.
- Storage node (DN, Data Node)
Storage nodes are responsible for data persistence, providing high data reliability and strong consistency guarantees based on the majority Paxos protocol, and maintaining distributed transaction visibility through MVCC.
- Metadata service (GMS, Global Meta Service)
The metadata service is responsible for maintaining globally consistent Table/Schema, Statistics and other system Meta information, maintaining security information such as accounts and permissions, and providing global timing services (ie TSO).
- Log node (CDC, Change Data Capture)
The log node provides an incremental subscription capability fully compatible with the MySQL Binlog format and protocol, and a master-slave replication capability compatible with the MySQL Replication protocol.
4. The core features of PolarDB-X are as follows:
- Horizontal expansion
PolarDB-X is designed with Shared-nothing architecture, supports multiple Hash and Range data splitting algorithms, realizes transparent horizontal expansion of the system through implicit primary key splitting and dynamic scheduling of data fragmentation.
- distributed transaction
PolarDB-X adopts MVCC + TSO scheme and 2PC protocol to realize distributed transactions. Transactions meet ACID characteristics, support RC/RR isolation level, and achieve high performance of transactions through optimizations such as one-phase commit, read-only transaction, and asynchronous commit.
- mixed load
PolarDB-X supports analytical queries through native MPP capabilities, and realizes strong isolation of OLTP and OLAP traffic through CPU quota constraints, memory pooling, and storage resource separation.
- Enterprise
PolarDB-X has designed many core capabilities for enterprise scenarios, such as SQL current limiting, SQL Advisor, TDE, separation of powers, Flashback Query, etc.
- cloud native
PolarDB-X has many years of cloud-native practice on Alibaba Cloud, supports management of cluster resources through K8S Operator, supports deployment in various forms such as public cloud, hybrid cloud, and proprietary cloud, and supports localized operating systems and chips.
- high availability
Strong data consistency is achieved through the majority Paxos protocol, and multiple disaster recovery methods such as three centers in two places and five replicas in three places are supported. At the same time, system availability is improved through Table Group and Geo-locality.
- Compatible with MySQL system and ecology
The goal of PolarDB-X is to be fully compatible with MySQL. Currently, the compatible content includes MySQL protocol, most of MySQL syntax, Collation, transaction isolation level, Binlog, etc.
Two, four ways to deploy PolarDB-X
Deployment environment and practical address: follow me to learn PolarDB-X - Yunqi Lab
Preparatory work:
Step 1. Before starting the experiment, you need to create an ECS instance resource.
Step 2, install and start dependent Docker
- Install Docker:
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun - Execute the command to start Docker:
systemctl start docker
Method 1: Docker image to install PolarDB-X.
1. Pull the PolarDB-X container image:
docker pull polardbx/polardb-x
2. Run the PolarDB-X container:
docker run -d --name some-polardb-x -p 8527:8527 polardbx/polardb-x
Method 2, use the PXD tool to install PolarDB-X with one click.
This method is to install the PXD tool in a virtual environment.
1. Create and activate a virtual scene.
python3 -m venv venv
source venv/bin/activate
2. Upgrade pip.
pip install --upgrade pip
3. Install PXD.
pip install pxd
4. Deploy PolarDB-X.
Method 1: Create a PolarDB-X database, where the number of GMS, CN, DN, and CDC nodes is 1.
pxd tryout
Method 2: Create a PolarDB-X database, specify the number of CN, DN and CDC nodes as 1 and the version as latest.
pxd tryout -cn_replica 1 -cn_version latest -dn_replica 1 -dn_version latest -cdc_replica 1 -cdc_version latest
5. Log in to the PolarDB-X database.
mysql -h127.0.0.1 -P8527 -upolardbx_root -p123456
Method 3, use Kubernetes to install PolarDB-X.
Create a simple Kubernetes cluster and deploy PolarDB-X Operator, use Operator to deploy a complete PolarDB-X cluster:
1. Install kubectl.
a. Download the kubectl file.
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
b, grant executable permissions.
chmod +x ./kubectl
c, move to the system directory.
mv ./kubectl /usr/local/bin/kubectl
2. Install minikube
Download and install minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
3. Install Helm3
a. Download Helm3.
wget https://labfileapp.oss-cn-hangzhou.aliyuncs.com/helm-v3.9.0-linux-amd64.tar.gz
b. Unzip Helm3
tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
c, move to the system directory
mv linux-amd64/helm /usr/local/bin/helm
4. Use minikube to create a Kubernetes cluster
a. Create a new account galaxykube and add galaxykube to the docker group. minikube requires a non-root account for deployment, so you need to create a new account.
``useradd -ms /bin/bash galaxykube usermod -aG docker galaxykube
b. Execute the following command to switch to account galaxykube.
su galaxykube
c. Execute the following command to enter the home/galaxykube directory.
cd
d. Execute the following command to start a minikube.
minikube start --cpus 4 --memory 7168 --image-mirror-country cn --registry-mirror=https://docker.mirrors.ustc.edu.cn --kubernetes-version 1.23.3
e. Execute the following command and use kubectl to view the cluster information.
minikube kubectl -- cluster-info
5. Deploy PolarDB-X Operator.
a, Create a namespace called polardbx-operator-system
kubectl create namespace polardbx-operator-system
b. Execute the following command to install PolarDB-X Operator
helm install --namespace polardbx-operator-system polardbx-operator https://github.com/ApsaraDB/galaxykube/releases/download/v1.2.1/polardbx-operator-1.2.1.tgz
c. Execute the following command to check the running status of the PolarDB-X Operator component
kubectl get pods --namespace polardbx-operator-system
6. Deploy PolarDB-X cluster
a. Deploy a PolarDB-X cluster, which includes 1 GMS node, 1 CN node, 1 DN node and 1 CDC node.
echo "apiVersion: polardbx.aliyun.com/v1
kind: PolarDBXCluster
metadata:
name: quick-start
annotations:
polardbx/topology-mode-guide: quick-start" | kubectl apply -f -
b, View the creation status.
kubectl get polardbxcluster -w
c. Delete the PolarDB-X cluster.
kubectl delete polardbxCluster <name>
Method 4: Compile and install PolarDB-X from source code.
1. Install git.
yum -y install git
2. Download the source code.
git clone https://github.com/ApsaraDB/PolarDB-X.git
3. Enter the PolarDB-X directory.
cd PolarDB-X
4. Compile
make
5. Run PolarDB-X.
./build/run/bin/polardb-x.sh start
6. Execute the following command to stop PolarDB-X.
./build/run/bin/polardb-x.sh stop
PS:
Command statement:
docker images: Display all current images of the machine.
docker ps:Display the currently running containers.
docker ps -a :show all containers
How to use and experience the specific PolarDB-X database, you can go to the learning website and refer to the experiment manual to complete it.