Remember a middle-synchronized data transmission Elasticsearch failed accident scene

Welcome attention to personal micro-channel public number: Patricia learn the Java , the first time to push high-quality paper yo!

Website: www.exception.site/essay/elast...

One, background
Second, digression
Third, start troubleshooting
Fourth, why the index is read-only state?
Fifth, how to solve

One, background

A few days ago Patricia receive feedback in the line of business in Chongqing nail the group, said that the station synchronous data transmission in the middleware Mysql incremental data to Elasticsearch always fail.

Second, digression

You said that data transmission and data transmission DTS Ali cloud provider is a thing?

Not! The above said data transmission is in Taiwan where the R & D department Patricia middleware independent research and development, the purpose is to replace each line of business dependence on Ali DTS synchronization function!

Currently, the data transmission still have to rely on the open source Ali Canal, or Ali DTS, dependent on the purpose is to achieve Mysql database binlog incremental subscription.

Network architecture diagram above example, the data transmission units act as a binlog event consumers through custom rules mapping, data processing, distribution and final synchronization to the target source Elasticsearch in.

Third, start troubleshooting

Return to the topic, a problem, and quickly flew through the springboard machine connected to the server where the data transmission, start looking at the log:

See a lot of logs exist [FORBIDDEN/12/index read-only / allow delete (api)]errors! !

An error is also evident: ES index is read-only ! ! After communication and business groups, finding the target with two indexes need to be synchronized, a commodity index (acting as the master table), a product attribute index (acting as merchandise from the table), the table synchronization is ok, that product attributes index non-read-only, write normal, just a commodity index is read-only, ultimately it failed to properly synchronize data.

Fourth, why the index is read-only state?

What causes the index read-only? Patricia began to read Elasticsearch official document , reads as follows:

Elasticsearch considers the available disk space on a node before deciding whether to allocate new shards to that node or to actively relocate shards away from that node.

Elasticsearch in deciding whether to assign a new slice to the node, the node or repositioned before slicing, the node will first determine whether enough storage space, if you use disk space more than 95%, ES will automatically index indexset for the read-onlystate.

So, let the machine's operation and maintenance facie ES disk space is sufficient, operation and maintenance, said feedback: two days ago because of lack of disk alarm, just expand the capacity is certainly enough!

The truth is out!

Two days before low disk space, at that time, commodity indexes have just written to the operation, due to the protection of ES, the index is set to read-only status.

Fifth, how to solve

The reason found! How to solve it?

The index is read-only and can only be queried or deleted. The ES also does not automatically switch back to the indexing status, we need to manually switch the:

PUT /<yourindex>/_settings
{
  "index.blocks.read_only_allow_delete": null
}
复制代码

After executing the above command commodity index. Let business group synchronous data again, everything is normal.