NoSQL Architecture Practice (1) - Supplemented by NoSQL

How to introduce NoSQL into our system architecture design needs to be analyzed according to the business scenario of our system, what type of data is suitable for storing in NoSQL database, and what type of data must be stored in relational database. Clearly introduce the role of the NoSQL database to the system, what problems it can solve, and the new problems it may bring. Below we analyze several common NoSQL architectures.

(1) NoSQL as a mirror

It does not change the original architecture with MySQL as the storage, uses NoSQL as the auxiliary mirror storage, and uses the advantages of NoSQL to help improve performance.

Figure 1 - NoSQL as Mirror (Code Completion Mode)

// Example pseudocode for writing data

//data is the data object we want to store
data.title=”title”;
data.name=”name”;
data.time=”2009-12-01 10:10:01”;
data.from=”1”;
id=DB.Insert(data);//Write to MySQL database
NoSQL.Add(id, data);//Write the NoSQL database with the self-increment id generated by writing to MySQL as the primary key

If there are data consistency requirements, it can be used as follows

// Example pseudocode for writing data
//data is the data object we want to store
bool status=false;
DB.startTransaction();//Start transaction
id=DB.Insert(data);//Write to MySQL database
if(id>0){
    status=NoSQL.Add(id, data);//Write the NoSQL database with the self-increment id generated by writing to MySQL as the primary key
}
if(id>0 && status==true){
    DB.commit();//Commit the transaction
}else{
    DB.rollback();//Unsuccessful, rollback
}

The above code may seem a little troublesome, but it only needs to do a unified encapsulation in the DB class or ORM layer to achieve reuse, and other codes do not need to be modified.

This architecture adds a layer of auxiliary NoSQL storage to the original MySQL database-based architecture. The amount of code is small and the technical difficulty is small, but it plays a very important role in scalability and performance. It is only necessary for the program to write to the NoSQL database after writing to the MySQL database, so that MySQL and NoSQL have the same mirror data. In some places that can be queried based on the primary key, use an efficient NoSQL database query, which saves MySQL's Queries, and fend off those queries with the high performance of NoSQL.

Figure 2 - NoSQL as Mirroring (Synchronous Mode)

This is not through program code, but through MySQL to synchronize data into NoSQL. This mode is a variant of the above one, which is a mode that is transparent to writing but has higher technical difficulty. This mode is suitable for the existing relatively complex old system, and it is not easy to implement by modifying the code, which may cause new problems. It is also suitable for synchronizing data to multiple types of storage.

The implementation of MySQL to NoSQL synchronization can be implemented using MySQL UDF functions and MySQL binlog parsing. It can be implemented using existing open source projects, such as:

MySQL memcached UDFs : from the Memcached protocol for manipulating through UDFs.
The open source mysql-udf-http of Zhang Yan in China : operates the http protocol through UDF.

With these two MySQL UDF function libraries, we can transparently process Memcached or Http protocol through MySQL, so as long as there is a NoSQL database compatible with Memcached or Http protocol, then we can operate through MySQL to synchronize data. Combined with lib_mysqludf_json , the automatic synchronization of data can be realized through the combination of UDF and MySQL trigger functions.

(2) MySQL and NoSQL combination

In MySQL, only small fields that need to be queried are stored, and NoSQL stores all data.

Figure 3 - MySQL and NoSQL combination

// Example pseudocode for writing data

//data is the data object we want to store
data.title=”title”;
data.name=”name”;
data.time=”2009-12-01 10:10:01”;
data.from=”1”;
bool status=false;
DB.startTransaction();//Start transaction
id=DB.Insert("INSERT INTO table (from) VALUES(data.from)");//Write to MySQL database, only write from fields that need where to query
if(id>0){
    status=NoSQL.Add(id, data);//Write the NoSQL database with the self-increment id generated by writing to MySQL as the primary key
}
if(id>0 && status==true){
    DB.commit();//Commit the transaction
}else{
    DB.rollback();//Unsuccessful, rollback
}

The fields that need to be queried, generally small fields such as numbers and time, are stored in MySQL, and corresponding indexes are established according to the query. Other unnecessary fields, including large text fields, are stored in NoSQL. When querying, we first query the primary key of the data from MySQL, and then directly extract the corresponding data from NoSQL.

This architectural model integrates the roles of MySQL and NoSQL, and each performs its own duties. MySQL is specially responsible for handling relational storage, which is good at, and NoSQL is used as data storage. It has the following advantages:

Save MySQL IO overhead. Since MySQL only stores small fields that need to be queried, it is no longer responsible for storing large text fields, which saves the space overhead of MySQL storage, thereby saving MySQL's disk IO. We once reduced a 40G table in MySQL to several hundred M through this optimization.
Improve the cache hit rate of MySQl Query Cache. We know that query cache cache invalidation is at the table level. Once the MySQL table is updated, it will be invalidated. After this field separation, if the updated field is not stored in MySQL, it will have no effect on the query cache. NoSQL caches are often row-level, and only caches of updated records are invalidated.
Improve MySQL master-slave synchronization efficiency. Due to the reduction of MySQL storage space, the number of synchronized data records is also reduced, and some data updates fall in NoSQL instead of MySQL, which also reduces the number of times that MySQL data needs to be synchronized.
Improve the speed of MySQL data backup and restore. As the data stored in the MySQL database is reduced, it is easy to see that the speed of data backup and recovery will also be greatly improved.
Easier to scale than ever. NoSQL is inherently easy to scale. After this optimization, MySQL performance is also improved.

For example, the mobile phone Phoenix network is this architecture http://www.cnblogs.com/sunli/archive/2010/12/20/imcp.html

Summarize

The architecture supplemented by NoSQL is still centered on the idea of MySQL architecture, but NoSQL is added to the previous architecture to improve its performance and scalability. This architecture is relatively easy to implement, but can achieve good results. If you are thinking of introducing NoSQL in your project, or your MySQL-based system is currently experiencing related bottlenecks, I hope this article can help you.