1. What is Amoeba?
The Amoeba project focuses on distributed database proxy development. Located between Client and DB Server(s). Transparent to the client. It has load balancing, high availability, SQL filtering, read-write separation, can route related queries to the target database, and can concurrently request multiple databases to merge results.
Main solutions:
• Reduce the complex multi-database structure caused by data segmentation
• Provide segmentation rules and reduce the impact of data segmentation rules on applications
• Reduce the number of connections between db and clients
• Separation of reading and writing
2. Why use Amoeba?
Currently, there are several main solutions to realize the master-slave reading and writing separation of MySQL:
1. Implemented through programs. There are many ready-made codes on the Internet, which are relatively complicated. If you add slave servers, you need to change the codes of multiple servers.
2. Implemented through mysql-proxy. Since the master-slave reading and writing separation of mysql-proxy is implemented through lua scripts, the current development of lua scripts cannot keep up with the pace, and there are no perfect ready-made scripts, so it is used. In a production environment, the risk is relatively high. According to many people on the Internet, the performance of mysql-proxy is not high.
3. Develop your own interface implementation. This solution has high threshold and high development cost, which cannot be afforded by ordinary small companies.
4. Use Alibaba's open source project Amoeba to implement it. It has load balancing, high availability, SQL filtering, read and write separation, can route related queries to the target database, and the installation and configuration is very simple. Domestic open source software should be supported and is currently in use. I won’t publish too many conclusions. I’ll wait until the test is completed before publishing any conclusions, haha!
3. Installation of Amoeba
First introduce the deployment environment:
amoeba:192.168.121.10 (The machine where amoeba is installed is the mysql5.5 version, which is not supported by mysql8.0)
masterDB: 192.168.121.20 (mysql8.0 installed)
slaveDB: 192.168.121.30 (mysql8.0 installed)
The above systems are all centos7.8
The Amoeba framework is developed based on JDK1.5 and adopts the features of JDK1.5, so it also needs to install a java environment. It is recommended to use a JDK version of javaSE1.5 or above (the jdk1.8.0_242 version installed here)
1. Install java environment
First go to the official website to download: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Installation: After downloading the jdk compressed package, just decompress it and it can be used. Here I decompress it to the /amoeba/jdk directory.
tar -zxf jdk-8u251-linux-x64.tar.gz -C /amoeba/jdk
Then set the java environment variables: modify the configuration file /etc/profile
#set java environment
JAVA_HOME=/amoeba/jdk
JRE_HOME=/amoeba/jdk/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH
After modifying the configuration file, you need to reload it:
[root@hadoop1 amoeba]# source /etc/profile
Test whether the installation is successful
[root@hadoop1 amoeba]# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
The jdk environment is configured successfully.
2. Install Amoeba
You can download the latest version of Amoeba from https://sourceforge.net/projects/amoeba/. What I downloaded here is amoeba-mysql-binary-2.1.0-RC5.tar.gz. The installation of Amoeba is very simple. Just unzip it and use it. Here, unzip Amoeba to the /usr/local/amoeba directory. The installation is complete. First create the amoeba directory under the /usr/local/ directory.
[root@hadoop1 opt]# tar -zxf amoeba-mysql-binary-2.1.0-RC5.tar.gz -C /usr/local/amoeba/
You can see that amoeba is decompressed as follows:
[root@hadoop1 opt]# cd /usr/local/amoeba/
[root@hadoop1 amoeba]# pwd
/usr/local/amoeba
[root@hadoop1 amoeba]# ll
总用量 64
drwxr-xr-x. 2 root root 4096 11月 24 18:28 benchmark
drwxr-xr-x. 2 root root 4096 11月 24 18:40 bin
-rw-r--r--. 1 root root 3983 5月 18 2011 changelogs.txt
drwxr-xr-x. 2 root root 4096 11月 25 14:49 conf
drwxr-xr-x. 3 root root 4096 11月 24 18:28 lib
-rw-r--r--. 1 root root 34520 5月 18 2011 LICENSE.txt
drwxr-xr-x. 2 root root 4096 11月 25 14:50 logs
-rw-r--r--. 1 root root 2031 5月 18 2011 README.html
success.
3. Configure Amoeba
Amoeba's configuration file is located in the /usr/local/amoeba/conf directory in this environment. There are many configuration files, but if you only use the read-write separation function, you only need to configure two files, namely dbServers.xml and amoeba.xml. If you need to configure IP access control, you also need to modify the access_list.conf file. Let’s introduce it first. dbServers.xml
[root@hadoop1 amoeba]# cd conf/
[root@hadoop1 conf]# cat dbServers.xml
<?xml version="1.0" encoding="gbk"?>
<!DOCTYPE amoeba:dbServers SYSTEM "dbserver.dtd">
<amoeba:dbServers xmlns:amoeba="http://amoeba.meidusa.com/">
<!--
Each dbServer needs to be configured into a Pool,
If you need to configure multiple dbServer with load balancing that can be simplified by the following configuration:
add attribute with name virtual = "true" in dbServer, but the configuration does not allow the element with name factoryConfig
such as 'multiPool' dbServer
-->
<dbServer name="abstractServer" abstractive="true">
<factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory">
<property name="manager">${defaultManager}</property>
<property name="sendBufferSize">64</property>
<property name="receiveBufferSize">128</property>
<!-- mysql port -->
<property name="port">3306</property> #设置Amoeba要连接的mysql数据库的端口，默认是3306
<!-- mysql schema -->
<property name="schema">testdb</property> #设置缺省的数据库，当连接amoeba时，操作表必须显式的指定数据库名，即采用dbname.tablename的方式，不支持 use dbname指定缺省库，因为操作会调度到各个后端dbserver。注意检查你的数据库里面一定要有这个数据库
<!-- mysql user -->
<property name="user">test1</property> #设置amoeba连接后端数据库服务器的账号和密码，因此需要在所有后端数据库上创建该用户，并授权amoeba服务器可连接
<property name="password">111111</property> #设置amoeba连接后端数据库服务器的账号的密码，注意这里原来是注释的状态，注意把注释去掉，否则无效
</factoryConfig>
<poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool">
<property name="maxActive">500</property> #最大连接数，默认500
<property name="maxIdle">500</property> #最大空闲连接数
<property name="minIdle">10</property> #最新空闲连接数
<property name="minEvictableIdleTimeMillis">600000</property>
<property name="timeBetweenEvictionRunsMillis">600000</property>
<property name="testOnBorrow">true</property>
<property name="testWhileIdle">true</property>
</poolConfig>
</dbServer>
<dbServer name="writedb" parent="abstractServer"> #writedb名称可以根据自己需求改变
<factoryConfig>
<!-- mysql ip -->
<property name="ipAddress">192.168.121.20</property> #这里写你预设的想写入数据的主服务器IP
</factoryConfig>
</dbServer>
<dbServer name="slave" parent="abstractServer"> #slave名称可以根据自己需求改变
<factoryConfig>
<!-- mysql ip -->
<property name="ipAddress">192.168.121.30</property> #这里写你预设的想读取数据的从服务器IP
</factoryConfig>
</dbServer>
<dbServer name="myslave" virtual="true"> #写你的连接池，myslave池名称可以改，这个池提供客户端读取数据
<poolConfig class="com.meidusa.amoeba.server.MultipleServerPool">
<!-- Load balancing strategy: 1=ROUNDROBIN , 2=WEIGHTBASED , 3=HA-->
<property name="loadbalance">1</property>
<!-- Separated by commas,such as: server1,server2,server1 -->
<property name="poolNames">slave</property> #指定读取数据时从从服务器上读取。也可以指定读取数据时从主服务器和从服务器上读取，则改成writedb,slave
</poolConfig>
</dbServer>
</amoeba:dbServers>
[root@hadoop1 conf]#
另一个配置文件amoeba.xml[root@hadoop1 conf]# cat amoeba.xml
<?xml version="1.0" encoding="gbk"?>
<!DOCTYPE amoeba:configuration SYSTEM "amoeba.dtd">
<amoeba:configuration xmlns:amoeba="http://amoeba.meidusa.com/">
<proxy>
<!-- service class must implements com.meidusa.amoeba.service.Service -->
<service name="Amoeba for Mysql" class="com.meidusa.amoeba.net.ServerableConnectionManager">
<!-- port -->
<property name="port">8066</property> #设置amoeba监听的端口，默认是8066
<!-- bind ipAddress -->
<property name="ipAddress">192.168.121.10</property> #配置监听的接口，此处写amoeba安装所在的服务器IP，注意把原本的注释状态取消，否则无效
<property name="manager">${clientConnectioneManager}</property>
<property name="connectionFactory">
<bean class="com.meidusa.amoeba.mysql.net.MysqlClientConnectionFactory">
<property name="sendBufferSize">128</property>
<property name="receiveBufferSize">64</property>
</bean>
</property>
<property name="authenticator">
<bean class="com.meidusa.amoeba.mysql.server.MysqlClientAuthenticator">
<property name="user">amoeba</property> # 提供客户端连接amoeba时需要使用这里设定的账号 (这里的账号密码和amoeba连接后端数据库服务器的密码无关)
<property name="password">123456</property> #客户端连接amoeba时的账户的密码
<property name="filter">
<bean class="com.meidusa.amoeba.server.IPAccessController">
<property name="ipFile">${amoeba.home}/conf/access_list.conf</property>
</bean>
</property>
</bean>
</property>
</service>
<!-- server class must implements com.meidusa.amoeba.service.Service -->
<service name="Amoeba Monitor Server" class="com.meidusa.amoeba.monitor.MonitorServer">
<!-- port -->
<!-- default value: random number
<property name="port">9066</property>
-->
<!-- bind ipAddress -->
<property name="ipAddress">127.0.0.1</property>
<property name="daemon">true</property>
<property name="manager">${clientConnectioneManager}</property>
<property name="connectionFactory">
<bean class="com.meidusa.amoeba.monitor.net.MonitorClientConnectionFactory"></bean>
</property>
</service>
<runtime class="com.meidusa.amoeba.mysql.context.MysqlRuntimeContext">
<!-- proxy server net IO Read thread size -->
<property name="readThreadPoolSize">20</property>
<!-- proxy server client process thread size -->
<property name="clientSideThreadPoolSize">30</property>
<!-- mysql server data packet process thread size -->
<property name="serverSideThreadPoolSize">30</property>
<!-- per connection cache prepared statement size -->
<property name="statementCacheSize">500</property>
<!-- query timeout( default: 60 second , TimeUnit:second) -->
<property name="queryTimeout">60</property>
</runtime>
</proxy>
<!--
Each ConnectionManager will start as thread
manager responsible for the Connection IO read , Death Detection
-->
<connectionManagerList>
<connectionManager name="clientConnectioneManager" class="com.meidusa.amoeba.net.MultiConnectionManagerWrapper">
<property name="subManagerClassName">com.meidusa.amoeba.net.ConnectionManager</property>
<!--
default value is avaliable Processors
<property name="processors">5</property>
-->
</connectionManager>
<connectionManager name="defaultManager" class="com.meidusa.amoeba.net.MultiConnectionManagerWrapper">
<property name="subManagerClassName">com.meidusa.amoeba.net.AuthingableConnectionManager</property>
<!--
default value is avaliable Processors
<property name="processors">5</property>
-->
</connectionManager>
</connectionManagerList>
<!-- default using file loader -->
<dbServerLoader class="com.meidusa.amoeba.context.DBServerConfigFileLoader">
<property name="configFile">${amoeba.home}/conf/dbServers.xml</property>
</dbServerLoader>
<queryRouter class="com.meidusa.amoeba.mysql.parser.MysqlQueryRouter">
<property name="ruleLoader">
<bean class="com.meidusa.amoeba.route.TableRuleFileLoader">
<property name="ruleFile">${amoeba.home}/conf/rule.xml</property>
<property name="functionFile">${amoeba.home}/conf/ruleFunctionMap.xml</property>
</bean>
</property>
<property name="sqlFunctionFile">${amoeba.home}/conf/functionMap.xml</property>
<property name="LRUMapSize">1500</property>
<property name="defaultPool">writedb</property> #设置amoeba默认连接的池，这里设置为writedb
<property name="writePool">writedb</property> #这两个选项默认是注销掉的，需要取消注释，这里用来指定前面定义好的俩个读写池。写入数据到刚才dbServer.xml中配置的write池
<property name="readPool">myslave</property> #写入数据到刚才dbServer.xml中配置的write池
<property name="needParse">true</property>
</queryRouter>
</amoeba:configuration>
[root@hadoop1 conf]#
Modify the amoeba script, otherwise the JVM startup will fail due to the stack size being too small. You need to make the following modifications:
[root@hadoop1 bin]# vim /usr/local/amoeba/bin/amoeba
58 DEFAULT_OPTS="-server -Xms256m -Xmx256m -Xss256k"
4. Create the database testdb on masterdb (verify whether the master-slave synchronization is normal. The configuration of master-slave synchronization will not be explained here. You can refer to the previous configuration process)
Check whether slavedb is copied successfully:
Above, the master-slave synchronization configuration is successful.
Authorize amoedb on masterdb and slavedb respectively (note that because master-slave synchronization is configured at this time, the authorization operation is done here. Just do it once on the master server, and the slave server will automatically synchronize)
mysql8.0如下方式授权：
mysql>create user 'test1'@'192.168.121.10' identified with mysql_native_password by '111111';
mysql>GRANT ALL ON testdb.* TO 'test1'@'192.168.121.10';
mysql>flush privileges;mysql> GRANT ALL ON testdb.* TO 'test1'@'192.168.121.10' IDENTIFIED BY '111111';
Query OK, 0 rows affected (0.05 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)Start amoeba
[root@hadoop1 bin]# pwd
/usr/local/amoeba/bin
[root@hadoop1 bin]#
[root@hadoop1 bin]# ./amoeba start
log4j:WARN log4j config load completed from file:/usr/local/amoeba/conf/log4j.xml
2020-11-25 16:17:49,825 INFO context.MysqlRuntimeContext - Amoeba for Mysql current versoin=5.1.45-mysql-amoeba-proxy-2.1.0-RC5
log4j:WARN ip access config load completed from file:/usr/local/amoeba/conf/access_list.conf
2020-11-25 16:17:50,491 INFO net.ServerableConnectionManager - Amoeba for Mysql listening on /192.168.121.10:8066.
2020-11-25 16:17:50,506 INFO net.ServerableConnectionManager - Amoeba Monitor Server listening on /127.0.0.1:28750.
Started successfully.
Check the port to see the startup status of amoeba:
From this we can see that Amoeba starts normally.
5. Test
Remotely log in to the mysql client to connect to the mysql database by specifying the user name, password, and port specified in the amoeba configuration file and the amoeba server IP address. Note that -h writes the server where amoeba is located.
[root@hadoop1 bin]# mysql -uamoeba -p -h192.168.121.10 -P8066
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1899284416
Server version: 5.1.45-mysql-amoeba-proxy-2.1.0-RC5 MySQL Community Server - GPL
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> The current data is as follows:
In the client database testdb table a and insert data:
Above, it is verified that the master server is configured to write data successfully.
Verify whether reading data configuration is successful:
We now insert a piece of data from the server
- Because the master-slave server is configured, the slave server inserts data and does not synchronize the data to the master server.
- But the client we configured reads data from the slave server
- Therefore, the client will read the newly added data from the server
Above, the read-write separation configuration is successful.