MongoDB3.2 - 4.2 New Features Reading

MongoDB is a distributed file storage based database. Written by C ++ language. Designed to provide scalable, high-performance data storage solution for WEB applications.

MongoDB is a product with function between relational databases and non-relational databases, non-relational database functions among the richest and most like a relational database.

First, the replica set

Version 3.2

1. Election mechanism is introduced raft protocol (pv1), an early agreement bully (campaign coordinator simple algorithm pv0)

With respect to bully agreement, the agreement benefits raft is: the elimination of the electoral lock, introducing term, weaken priority.

Early bully agreement is the addition of a global lock, as the case may produce multiple administrations screen after completing the vote from the library, the global lock to prevent duplicate voting 30s; raft introduced a term such an election cycle (similar to an auto-incremented such a concept cycle), if the election does not come out of the main during this period from libraries, but also random next cycle voting began, term will add 1; raft weakened priority, the main library failure will be switched to high priority will be selected as the main priority, raft would weaken this weight. Efficiency raft of agreements will be higher.

2. Increase electionTimeoutMillis replication configuration parameters, determines the timeout of the master repository can not communicate. Heartbeat is detected by the master-slave is normal, if the library Rom from the main library, and will not initiate immediate elections, but to wait for a while, and so this time expires, the election was made after launch over, reducing the false elections frequency

Version 3.4

No longer maintained after 2020 January 31.

1. Improved initialization total amount of data synchronized with the index creation order and the data copy process while pulling oplog (similar mysql inside binlog), oplog default 5% of disk capacity (minimum 1G, maximum 50G)
   
   improved reasons: If the copy set of data this process takes a long time, and write operations is very high, coupled with the need to create an index for a long time. oplog may be overwritten, oplog of this collection can be circular, if filled will overwrite the old one. Online problem: When faced with a large instance when adding the node may not have been on the increase, because at that point stretch incremental recording oplog time has been covered. From 3.4 to solve library plus not into question, while reducing the process of creating an index. If there is data of 10GB, comprising 64, each set comprising two index fields, document average 1KB, 3.4 version of the synchronization performance of the whole amount of about 20% increase, if the data set is large, and there are in the synchronization process write, to enhance the effect will be more obvious.

2. Increasing the total amount of data copying process retry logic, reducing the number of rework. If the network jitter, considered the copy interrupted, and that this is very painful for operation and maintenance. Therefore, the addition of retry logic.

3. Encountered during the initial synchronization can lead to rename the set of synchronization failure directly redo

4. New catchUpTimeoutMillis replication configuration parameters, the new master decided to get maximum time to update the data. As a candidate node when the time got me to the missing data, if more than this time still did not get the latest data, then enter a new round of election cycles

5. Increase writeConcernMajorityJournalDefault parameter determines the majority to write is to write memory or disk. When you configure replication set to increase writeConcernMajorityJournalDefault option, the default is true, that is, when designated WriteConcern as majority, and journal writing data to the node most successful brush-hours, only to confirm that the client successfully; if false, write data to most node memory, it confirms to the client.

Version 3.6

1. Abandoned early bully protocol (simple algorithm campaign coordinator pv0)
2. Increase replSetResizeOplog command and oplogInitialFindMaxSeconds timeout parameters, need to restart the early and switch the main library, it is very prone to failure, after 3.6 can be edited directly online. For a large oplog, if a streaming backup, without this time-out mechanism will be stuck over there, it is generally time-out set of 60s.
3. Version 3.4 fixes the full amount of the data synchronization process rename collections cause problems redo,

Version 4.0

1. Delete bully protocol (pv0) and master-slave mode cluster structure
2. Deleted directly using the replica set node wt engines are not allowed storage.journal.enabled: false. Must open journal
3. Increase rollbackTimeLimitSecs and createRollbackDataFiles parameter control oplog rollback maximum
   difference and whether to create rollback files locally between. Delayed 30 minutes from libraries and the main library, then you do not execute a rollback.
4. Solved by reading the snapshot from the library add global lock application oplog cause problems with reading business application oplog mutual obstruction.

Version 4.2

1. 4.2 from the beginning, the administrator can limit the rate of the primary database application written to the maximum value held in the delay configurable less. By default, flow control is enabled. (FCV if not to enable or disable the read concern majority rights, the flow control is not enabled.)

Second, the slice cluster

Version 3.2

config node discarding mirror mode (SCCC), the introduction of the replica set configuration mode (CSRS) engine must wt. Before and after versions of metadata stored in the metadata information is stored by the mirror 3, config node configuration changes to update the three nodes. This mirror mode biggest problem is that when a node is hung up or requires maintenance, need to stop taking the entire cluster. Instead of a replica set in 3.2 mode, easy to maintain.

Version 3.4

1. Slice each cluster can sense the presence of all the components of the entire fragmented cluster configuration information that is added to avoid the mistakes of fragmentation (fragmentation must configure the cluster configuration sharding.clusterRole). Prior versions online issue may arise: a slice belong to two clusters, resulting in inconsistent data.
2. Config node is no longer supported mirror mode (SCCC), Zones partitioning strategy is introduced to replace the Tag mode. Some data can be deployed across the room assigned to the specified one or more shard, this feature will greatly facilitate sharding cluster of
3. The default setting for wiredTiger engine secondaryThrottle false, greatly improving chunk migration rate. 1 fragment prior to release to migrate slice 2, have occurred in the master library, but also to synchronize data is written from the library, greatly reducing the speed of the chunk. Version 3.4 accelerates migration.
4. Equilibrium data routing node to initiate the transfer from the master library node config replica set, increasing parallelism mechanisms balance (N / 2) and the balancer startup, shutdown, and status view commands. Earlier versions, if you deploy multiple nodes, each node will initiate a balance task, first of all get distributed lock node can initiate. Previous version requires migration from A- to C-> D after completion of the migration B>, while the parallel version 3.4, A-> B and C-> D can be carried out at the same time, increase the migration rate.

Version 3.6

1. Slices must be forcibly set copy mode does not allow single-node mode
2. Increase the connection node ShardingTaskExecutorPoolMaxConnecting frequency slicing control routing node to monos
3. Increase orphanCleanupDelaySecs parameter controls the original document clean-up time after the latest chunk migration,

Version 4.0

1. Chunk migration data fetch and apply concurrent operation have been achieved performance gains of nearly 40%
2. Create and delete collections or delete database operations such as add a slice start writing the majority support (majority)

Version 4.2

1. Fragments can update the key of the document, unless the fragment key field _id field are immutable. Before version 4.2, fragment key field value document is immutable.
2. You can use the Multiple Document Services cluster fragmentation

Third, improve read consistency

Version 3.2

First introduced in reading-related characteristics (readConcern), copy version must be pv1, support local and majority of two strategies, which reads majority (wt engine supports only) can achieve read not rolled back. After this time t3 node can be achieved does not roll back.

Version 3.4

Added support linearizable (linear consistency) of

• 只能用于读primary及读单个文档
• 使用时建议增加maxTimeMS参数

读主库的单个文档的时候，读到的数据一定是在读操作之前写入的数据，这个数据也是写入到大多数节点的，加入这个超时参数，直接给主库发送线性读的情况下，如果从节点挂了，在没有加超时参数的情况下，这个读就可能会一直挂起，就可能引发故障。

3.6版本

增加“available”（可用性）的支持

• 用于分片集合读增加数据容错性但可能读到orphan (孤立)文档
• 读非分片集合的表现和local模式一样

分片集合的读：更宽松的一个隔离级别，甚至可以读到孤立文档，就是chunk迁移之后还留在原来的分片一些文档，理论上对业务来说是不可见的，但是如果你想要看到，那么可以配置这个参数。available用的不多，除非对一致性要求非常低。

4.0版本

1. 增加“snapshot”（快照读）模式，以支持多文档事务读

• 读可见性区分是否关联因果一致性( causally consistent session)

快照读的示例：
假设 x:1 在读事务T1开启前已经写入
事务+snapshot读+写大多数+非因果一致性会话= 读已写大多数的数据快照 (x:2)
事务+snapshot读+写大多数+因果一致性会话= 读T1开启前已写majority快照 (x:1)

4.2版本

1. MongoDB 增加了可重试读功能，对于一些临时的网络问题，用户无需自己实现重试逻辑，MongoDB 会自动重试处理，保证用户业务的连续性。

四、引擎相关的改进

3.2版本

• WiredTiger 核心特性：

• 压缩比：对于mongodb来说主要还是存储文档，所以大多数都是文档型数据，如果加入压缩特性，那么可以节约很多的空间，2~7倍。
  锁粒度：早期的基本上是全局的库就别或表集合级别的粒度， WiredTiger是记录级别的粒度，并发性大大的提高
  空洞率：早期的空洞率会很高，更新记录时如果原来分配的空间不够，会迁移到新的空间，空间分配是2ⁿ。新版的空洞率20%左右。例如一个400G的数据，导出后再导入，可能就占用只有250G了。
  Cache：早期的内存是叫给操作系统来控制的，WiredTiger 通过cacheSize参数来控制

1. 自3.0引入WiredTiger引擎后3.2版本在企业版本引入了WT加密配置项
2. 3.2版本WiredTiger作为默认引擎且已支持fsyncLock 创建一致性备份
3. 增加ephemeralForTest（测试引擎）及inMemory （内存引擎仅用于企业版）

3.4~4.0 版本

4.0开始WiredTiger引擎的副本集及分片集群不允许关闭 journal
4.0开始废弃MMAP V1引擎

4.2版本

4.2删除不推荐使用的MMAPv1存储引擎。

五、重要改进

ChangeStreams

3.6版本首次引入集合粒度的”变更流”，早期如果想获取某个集合的变更情况，需要去抓取oplog，而新版本中可以直接用变更流进行抓取。
4.0版本添加库及实例粒度的增强支持
4.0版本真正实现了只捕获“写大多数”
主要特点：
• 可只捕获变更或完整文档、可灵活指定过滤指定操作
• 分片集群中事件完全有序（通过一个全局逻辑锁来实现的）、非mongo shell 下断线重连。
• 只发布”写大多数”的变更 、变更流遵从权限控制系统，

多文档事务

MongoDB 4.0 支持副本集事务，极大的丰富了应用场景；4.0 的事务存在最大修改 16MB、事务执行时间不能过长的限制，在 4.2 支持分布式事务的这些问题都解决了。分布式事务的支持也意味用户修改分片key的内容成为可能，因为修改分片key的内容，可能会导致key要迁移到其他shard，而在4.2之前，无法保证这个迁移动作（目标上新写、源上删掉）的原子性，而借助分布式事务，这个问题也就迎刃而解。MongoDB 采用二阶段提交的方式，实现在多个 Shard 间发生的修改，要么同时发生，要么都不发生，保证事务的 ACID 特性。

• session.startTransaction() // 开启事务
• session.commitTransaction() // 提交事务
• session.abortTransaction() // 回滚事务

事务使用需要注意的问题：
只有副本集集群模式才支持；系统库/集合是不支持多文档事务；
事务部支持catalog操作（例如创建一个集合，删除一个索引），及explain（执行计划）在事务中都是不支持的；
读优先级策略值只支持读主库。

认证安全

权限管理

3.0版本首次具备完整的基于角色的权限管理体系， 后期版本持续增强
鉴权开启
3.4版本引入了transitionToAuth配置，确保集群可在线开通鉴权

白名单

3.6版本为了安全起见bindIp 默认绑定是的127.0.0.1 、 ipv6默认绑定::1
3.6版本自定义用户及角色引入authenticationRestrictions限制客户端及服务端的ip

认证策略

3.6版本开始废弃MONGODB-CR认证策略、 需要注意的是4.0版本彻底删除
自从3.0开始引入SCRAM-SHA-1策略(3.0-3.4都有性能问题)以来、4.0又新增SCRAM-SHA-256

SCRAM-SHA-1如果在高并发端连接的情况下，如果要获取一个随机的加密盐，如果要保证返回的不同，需要加一个spinlock的锁，所以可能会有性能问题。

优点：
可调迭代数 (默认15000) 客户及服务端随机加盐
比md5更强的hash函数 客户及服务端双向认证

4.2版本 引入「字段级加密」的支持，实现对用户JSON文档的Value 进行自动加密。整个过程在 Driver 层完成，传输、存储到服务端的文档Value都是密文，MongoDB 4.2 Drvier 支持丰富的加密策略，可以针对集合、字段维度开启加密，加密过程对开发者完全透明。

模型校验

3.2开始引入validation功能，因为同一个字段可能存储的数据会用混，可能存的时间可能存的数字。
3.6版本支持$jsonSchema

定义字段校验

db.createCollection( "test",
   {
      validator: { $or:
         [
            { phone: { $type: "long" } },
            { email: { $regex: /@huolala\.cn$/ } },
            { status: { $in: [ "bad", "good" ] } }
         ]
      },
      validationAction: "error"
   }
)

不符合校验规则的插入：

db.test.insert( { phone: 123456,email:"[email protected]", status: "bad" } )

符合校验规则的插入：

db.test.insert( { phone: NumberLong(123456),email:"test@huolala,cn", status: "good" } )

聚合框架

3.2版本首次引入 $ lookup 支持关联查询
3.4增加$grahpLookup支持层次递归查询
3.6版本进一步增强(支持多个关联字段)
4.2增加merge操作符，又称按需式物化视图，能不断的将增量分析结果与原来的结果进行汇总（老的版本只支持out，把当次分析结果写到某个集合）。

其他改进

• 3.4版本开始支持视图（view）
  语法：

db.createView(<view>, <source>, <pipeline>, <options>)

用例：

db.createView( 'test_view', // 视图名称
'sourceCollection', // 原集合
[ { $project: {'vField': '$name', age: 1 } } ] // 定义视图字段
) 

规定：
索引、只读、 视图上的视图、自然排序、shard集合、关联查询

• 3.6版本引入可重试写（Retryable Writes）
  场景：
  网络错误或遇到副本集切换
  规定：
  • 只重试一次解决暂时的网络错误
  • 只支持副本集和shard集群
  • 要求WiredTiger或in-memory存储引擎,mmapv1 不支持
  • 对于4.0版本多文档事务按整个事务进行
  • 无可以写primary 则等待serverSelectionTimeoutMS (默认30秒)（主库切换或选举过程中）
  用例：
  uri = “mongodb://example.com:7007/?retryWrites=true”

• 4.0版本引入Free Cloud Monitoring (仅支持单节点及副本集）
  db.enableFreeMonitoring() // 开启云监控
  {
  “state” : “enabled”,
  “message” : “To see your monitoring data, navigate………”,
  “url” : “https://cloud.mongodb.com/freemonitoring/…”,
  “userReminder” : “”,
  “ok” : 1
  }
  db.disableFreeMonitoring() // 关闭云监控
  db.getFreeMonitoringStatus() // 查看监控状态

3.2版本引入部分索引
• 与稀疏索引包含关系
• 唯一性索引约束范围
3.6版本嵌套字段查询可使用”覆盖索引”
3.6版本支持因果一致性
一个操作逻辑上依赖它之前的操作即认为两操作具备因果关系
因果一致性可为读写提供如下保障： • 读自己的写
• 单调性读 (w1->w2,r1->r2 如果r1 读到w2,那r2必然不会读w1)
• 单调性写(删除之前的插入) • 读然后写(先查询然后更新或插入)

4.2 version introduces Wildcard Index, can be automatically indexed for a range of fields, to meet the needs of rich query. Previous versions often encounter some scenarios, a field containing a number of properties, many properties are likely to need for the inquiry, present solutions for each property, you must know its behavior in advance, to establish the necessary index of.

Version 3.6 listdatabase improve
listDatabases command parameters add nameOnly execute the command or request an unlocked library-level locking
version 3.6 dropDatabase improve
dropDatabase wait drop all finished the set command is executed after spread to a large number of members of the replica set
4.0 listCollections improve
listCollections command to execute the command parameters add nameOnly Unlock or request set level locking
version 4.0 killOp improve
killOp kill mongos cluster query operations can be performed directly in the mongos routing node
version 4.0 show tables improvements
in safety considerations show tables are no longer displayed by default system tables system. *
4.2 version of the new wildcard index, also rather special, it means a field of uncertainty, some fields determined, but may need to do a quick search for a matching field of uncertainty, and the full-text index difference is relatively large, should be to provide for the Document-level data more flexibility index.