Nowadays, large-scale e-commerce systems mostly use read-write separation technology at the database level, that is, a Master database and multiple Slave databases. The Master library is responsible for data update and real-time data query, and the Slave library is of course responsible for non-real-time data query. Because in practical applications, the database reads more and writes less (the frequency of reading data is high, and the frequency of updating data is relatively low), while reading data usually takes a long time and occupies more CPU of the database server, thus affect the user experience. Our usual practice is to extract the query from the main library, use multiple slave libraries, and use load balancing to reduce the query pressure of each slave library.
The goal of using the read-write separation technology is to effectively reduce the pressure on the Master library, and distribute user data query requests to different Slave libraries, thereby ensuring the robustness of the system. Let's look at the background of using read-write separation .
As the business of the website continues to expand, the data continues to increase, the number of users increases, and the pressure on the database increases. Traditional methods, such as database or SQL optimization, have basically failed to meet the requirements. At this time, you can Adopt the strategy of read-write separation to change the status quo.
Specifically in the development, how to easily realize the separation of reading and writing? There are two commonly used methods:
1 The first method is the most commonly used method, which is to define two database connections, one is MasterDataSource and the other is SlaveDataSource. We read MasterDataSource when updating data, and SlaveDataSource when querying data . This method is very simple, I will not go into details.
2 The second way, dynamic data source switching, is to dynamically weave the data source into the program when the program is running, so as to choose whether to read the master library or the slave library. The main technologies used are: annotation , Spring AOP, reflection . The implementation will be described in detail below.
Before introducing the implementation method, let's prepare some necessary knowledge, spring's AbstractRoutingDataSource class
The class AbstractRoutingDataSource was added after spring 2.0. Let's first look at the definition of AbstractRoutingDataSource :
public abstract class AbstractRoutingDataSource extends AbstractDataSource implements InitializingBean {}
AbstractRoutingDataSource inherits AbstractDataSource, and AbstractDataSource is a subclass of DataSource. DataSource is the data source interface of javax.sql, defined as follows:
public interface DataSource extends CommonDataSource,Wrapper {
/**
* <p>Attempts to establish a connection with the data source that
* this <code>DataSource</code> object represents.
*
* @return a connection to the data source
* @exception SQLException if a database access error occurs
*/
Connection getConnection() throws SQLException;
/**
* <p>Attempts to establish a connection with the data source that
* this <code>DataSource</code> object represents.
*
* @param username the database user on whose behalf the connection is
* being made
* @param password the user's password
* @return a connection to the data source
* @exception SQLException if a database access error occurs
* @since 1.4
*/
Connection getConnection(String username, String password)
throws SQLException;
}
The DataSource interface defines two methods, both of which are used to obtain database connections. We are looking at how AbstractRoutingDataSource implements the DataSource interface:
public Connection getConnection() throws SQLException {
return determineTargetDataSource().getConnection();
}
public Connection getConnection(String username, String password) throws SQLException {
return determineTargetDataSource().getConnection(username, password);
}
Obviously, call your own determineTargetDataSource() method to get the connection. The determineTargetDataSource method is defined as follows:
protected DataSource determineTargetDataSource() {
Assert.notNull(this.resolvedDataSources, "DataSource router not initialized");
Object lookupKey = determineCurrentLookupKey();
DataSource dataSource = this.resolvedDataSources.get(lookupKey);
if (dataSource == null && (this.lenientFallback || lookupKey == null)) {
dataSource = this.resolvedDefaultDataSource;
}
if (dataSource == null) {
throw new IllegalStateException("Cannot determine target DataSource for lookup key [" + lookupKey + "]");
}
return dataSource;
}
We are most concerned about the following two sentences:
Object lookupKey = determineCurrentLookupKey();
DataSource dataSource = this.resolvedDataSources.get(lookupKey);
The determineCurrentLookupKey method returns the lookupKey, and the resolvedDataSources method obtains the data source from the Map according to the lookupKey. resolvedDataSources and determineCurrentLookupKey are defined as follows:
private Map<Object, DataSource> resolvedDataSources;
protected abstract Object determineCurrentLookupKey()
Seeing the above definition, do we have some ideas? resolvedDataSources is a Map type. We can store MasterDataSource and SlaveDataSource in Map, as follows:
key value
master MasterDataSource
slave SlaveDataSource
We are writing a class DynamicDataSource that inherits AbstractRoutingDataSource and implements its determineCurrentLookupKey() method, which returns the map's key, master or slave.
Well, having said so much, it's a bit annoying, let's see how to achieve it.
The technology we want to use has been mentioned above, let's first look at the definition of annotation:
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface DataSource { String value(); } 我们还需要实现spring的抽象类AbstractRoutingDataSource，就是实现determineCurrentLookupKey方法：
public class DynamicDataSource extends AbstractRoutingDataSource {
@Override
protected Object determineCurrentLookupKey() {
// TODO Auto-generated method stub
return DynamicDataSourceHolder.getDataSouce();
}
}
public class DynamicDataSourceHolder {
public static final ThreadLocal<String> holder = new ThreadLocal<String>();
public static void putDataSource(String name) {
holder.set(name);
}
public static String getDataSouce() {
return holder.get();
}
} 从DynamicDataSource 的定义看出，他返回的是DynamicDataSourceHolder.getDataSouce()值，我们需要在程序运行时调用DynamicDataSourceHolder.putDataSource()方法，对其赋值。下面是我们实现的核心部分，也就是AOP部分，DataSourceAspect定义如下:
public class DataSourceAspect {
public void before(JoinPoint point)
{
Object target = point.getTarget();
String method = point.getSignature().getName();
Class<?>[] classz = target.getClass().getInterfaces();
Class<?>[] parameterTypes = ((MethodSignature) point.getSignature())
.getMethod().getParameterTypes();
try {
Method m = classz[0].getMethod(method, parameterTypes);
if (m != null && m.isAnnotationPresent(DataSource.class)) {
DataSource data = m
.getAnnotation(DataSource.class);
DynamicDataSourceHolder.putDataSource(data.value());
System.out.println(data.value());
}
} catch (Exception e) {
// TODO: handle exception
}
}
} 为了方便测试，我定义了2个数据库，shop模拟Master库，test模拟Slave库，shop和test的表结构一致，但数据不同，数据库配置如下：
<bean id="masterdataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://127.0.0.1:3306/shop" />
<property name="username" value="root" />
<property name="password" value="yangyanping0615" />
</bean>
<bean id="slavedataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://127.0.0.1:3306/test" />
<property name="username" value="root" />
<property name="password" value="yangyanping0615" />
</bean>
<beans:bean id="dataSource" class="com.air.shop.common.db.DynamicDataSource">
<property name="targetDataSources">
<map key-type="java.lang.String">
<!-- write -->
<entry key="master" value-ref="masterdataSource"/>
<!-- read -->
<entry key="slave" value-ref="slavedataSource"/>
</map>
</property>
<property name="defaultTargetDataSource" ref="masterdataSource"/>
</beans:bean>
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<!-- 配置SqlSessionFactoryBean -->
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation" value="classpath:config/mybatis-config.xml" />
</bean>
在spring的配置中增加aop配置
<!-- 配置数据库注解aop --> <aop:aspectj-autoproxy></aop:aspectj-autoproxy> <beans:bean id="manyDataSourceAspect" class="com.air.shop.proxy.DataSourceAspect" /> <aop:config> <aop:aspect id="c" ref="manyDataSourceAspect"> <aop:pointcut id="tx" expression="execution(* com.air.shop.mapper.*.*(..))"/> <aop:before pointcut-ref="tx" method="before"/> </aop:aspect> </aop:config> <!-- 配置数据库注解aop -->
下面是MyBatis的UserMapper的定义，为了方便测试，登录读取的是Master库，用户列表读取Slave库：
public interface UserMapper { @DataSource("master") public void add(User user); @DataSource("master") public void update(User user); @DataSource("master") public void delete(int id); @DataSource("slave") public User loadbyid(int id); @DataSource("master") public User loadbyname(String name); @DataSource("slave") public List<User> list(); }
好了，运行我们的Eclipse看看效果，输入用户名admin 登录看看效果
从图中可以看出，登录的用户和用户列表的数据是不同的，也验证了我们的实现，登录读取Master库，用户列表读取Slave库。