Stand-alone storage engine
Hash storage engine: array + linked list. readO(1)
B-number storage engine: sequential scan, range lookup innoDB
LSM storage engine: Incremental data is stored in memory and stored on hard disk when certain conditions are met. When querying, the hard disk content and memory merge are returned to the caller. Through CommitLog, memory data can be guaranteed to be recoverable after loss.
Data model classification: relational mysql. Key-value Memcached. Column-stored Hbase. Documentary MongoDB.
Principle and Design of Multi-machine Storage
Data is distributed on multiple nodes, and load balancing needs to be implemented among multiple nodes. Data distribution consistency hash. Multi-machine storage exists, node a cannot be used, the request is transferred to b, and then a is available again. At this time, if the data of ab is inconsistent, there will be dirty data. A timestamp can be stored on each record.
Use replication : Distributed storage of multiple copies.
Failure detection : heartbeat mechanism, data migration, failure recovery.
FLP Theorem and Design : FLP Impossibility FLP is impossible. In the asynchronous message communication scenario, even if only one process fails, there is no way to ensure that non-failed processes reach consistency.
The FLP system model is based on the following assumptions: Asynchronous communication: The biggest difference from synchronous communication is that there is no clock, no time synchronization, no timeout, no detection failure, messages can be delayed arbitrarily, and messages can be out of order. Communication robustness: As long as the process does not fail, the message will be wirelessly delayed, but will eventually be delivered, and the message will only be delivered once (repeat guarantee). fail-stop model: process failure no longer processes any messages. Number of failed processes: At most one process fails.
http://blog.csdn.net/chen77716/article/details/27963079
CAP. Consistency: read data is always up-to-date (the result of writing), strong consistency. Availability: A machine or part of a system fails and still provides read and write services. Partition tolerance: people can meet the consistency and availability under abnormal conditions such as machine failure, network failure, and computer room failure.
Partition tolerance must ensure P, strong synchronization when guaranteeing C, the master replica network is abnormal, write operations are blocked, and availability cannot be guaranteed. When ensuring availability A, the asynchronous replication mechanism ensures the availability of the distributed storage system, but strong consistency cannot be guaranteed.
Consistency and availability need to be compromised, no data loss allowed, strong consistency, financial. Small probability of loss, availability, IM messages.
2PC protocol, one coordinator and multiple transaction participants. 2pc is blocking, with a timeout of 3 seconds.
3PC
Paxos: Refer to the video of Zhixing Society.
2PC guarantees the atomicity of operations on multiple data shards. Paxos guarantees data consistency among multiple copies of a data shard.
Data storage redundancy
mysql, mongoDB. Master-Slave can never become the master if he hangs up.
mongoDB. ReplicSet. Lord can be elected.
Double-write: The storage layer lives in a peer-to-peer structure, double-write, and distributed consistency.
Data backup (hot backup)
Asynchronous hot backup, multiple data copies are written asynchronously and deflected, and the application returns after successfully writing one copy of the data. There are storage systems that asynchronously write to other replicas. mysql, mongoDB are both
Synchronous hot backup, multiple data copies are written at the same time. The application received write success for all replicas. No master-slave is completely equal.
Data backup landed
RDSMS master-slave read-write separation. When there are too many backup machines, a data backup management system is used, and the central control machine sends a backup command to the node. If the backup is not returned successfully within the specified time, it is considered that the failure requires an alarm and manual intervention.
data consistency
MS structure, M asynchronously syncs to S
The Proxy application writes the Proxy, and the proxy is responsible for updating multiple data.
For example, use the same data submission, delay deletion, and delete the cache twice at an interval of 5 seconds.
data migration
1. Mongo-s is removed, copied to mysql, and the update is written to the message queue, and then mysql reads from the message queue and executes it.