28. logstash-out-mongodb realizes data synchronization from elasticsearch to Mongodb (ES and non-relational database synchronization)

This article mainly implements the synchronization of the index data Index in Elasticsearch to the collection collection in Mongodb.

0. Premise

1) The source database has been installed: elasticsearch V2.X; 
2) The destination database has been installed: Mongodb; 
3) The logstash and related plugins logstash-output-mongodb have been installed

Google and Statckoverflow are full of articles and questions about mongodb to elasticsearch synchronization, and in turn, the operation of elasticsearch to mongodb synchronization is very little. 
This also shows that a good architecture design should store source data in Mongodb, and synchronize it to ES for retrieval when full-text retrieval is required.

But obviously the purpose of the logstash-out-mongodb plugin is to write data to mongodb, so it is natural to set the input to the address and index information of ES.

1. Detailed explanation of synchronization conf configuration

[root@la logstash_output_mongo]# cat mongo_out.conf
input {
  stdin {
  }
  elasticsearch {
 ‘#ESIP地址与端口
  hosts => "100.10.1.35:9200"
  ’#ES索引名称（自己定义的）
  index => "weibo"
  ’#自增ID编号
  ’# document_id => "%{id}"
  ’#定时字段 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
  ’#schedule => "* * * * *"
  ’#设定ES索引类型
  type => "message"
  }
}

filter {
json {
  source => "message"
  remove_field => ["message"]
  }
}

'#目标mongodb地址信息
  output {
  stdout { codec => rubydebug }
  mongodb {
’#目标mongodb集合
  collection => "weibo"
’#目标库名称
  database => "data"
  uri => "mongodb://100.20.12.45:27017"
  }
}

2. Summary

The above implements a full synchronization operation between an index Index in ES and a collection collection in Mongodb. 
Full implementation: by configuring source input and target output address information; 
incremental implementation: to be verified. 
When synchronizing, you don't need to do anything in mongodb. After synchronization, you can see the new collection collection after synchronization on the windows client of mongodb: Robomongo.

3. How to synchronize ES to Mongo in turn?

For details, see: mongo-connector realizes real-time synchronization between MongoDB and elasticsearch for in-depth details