Suppose you have an existing elasticsearch cluster A and, for any reason, you want to migrate the data to cluster B.
In this write up, My cluster A is on ELK 5.4 and cluster B is on ELK 6.7. This procedure is very handy to upgrade and migrate ELK on from existing to new servers.
On cluster A first you have to enable elasticsearch on network interface. Migration requests through apache/nginx proxy will not work. To do so edit elasticsearch.yml and add following line:
http.host: 0.0.0.0And restart elasticsearch on cluster A.
Now, on cluster B, you have to whitelist cluster A’s IP.
Edit elasticsearch.yml and add following line:
reindex.remote.whitelist: ClusterA_IP:9200Restart elasticsearch on cluster B.
Now, list down indices on cluster A and select desired index to be migrated to cluster B.
On Cluster A, do following.
curl -X GET "localhost:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2019.04.05 vVBX9Za2Q7iEODDWPCtT2g 5 1 1970918 0 709.1mb 709.1mb
yellow open logstash-2019.03.30 F4JIh-5sQuOjPJakucltBA 5 1 3787369 0 2.2gb 2.2gb
yellow open logstash-2019.04.02 Rq0hwtC4TrWe-KWHTCKG_g 5 1 3346499 0 702.8mb 702.8mb
yellow open logstash-2019.03.31 xnmPxs4ESzqJSUwM8BdRMQ 5 1 4909132 0 2.7gb 2.7gb
yellow open logstash-2019.04.04 KSZnfe31Qp6JirjgHk-wrA 5 1 4907046 0 1.8gb 1.8gb
yellow open logstash-2019.03.29 ZgngCWk6RByw3RZOb6pRUw 5 1 2632343 0 1.4gb 1.4gb
yellow open .kibana y404YXzhQVqy3Auysph1wQ 1 1 6 0 31.9kb 31.9kbLet us try to migrate logstash-2019.03.30 index on cluster B. On cluster B, do following:
curl -X POST "localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
{
"source": {
"remote": {
"host": "http://ClusterA_IP:9200"
},
"index": "logstash-2019.03.30"
},
"dest": {
"index": "logstash-2019.03.30"
}
}'
[2019-04-05T17:38:48,363][INFO ][o.e.c.m.MetaDataCreateIndexService] [elk6] [logstash-2019.03.30] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]
[2019-04-05T17:38:48,968][INFO ][o.e.c.m.MetaDataMappingService] [elk6] [logstash-2019.03.30/uyV4qsFHTuKvmRJgSD5t6Q] create_mapping [syslog]
[2019-04-05T17:38:49,336][INFO ][o.e.c.m.MetaDataMappingService] [elk6] [logstash-2019.03.30/uyV4qsFHTuKvmRJgSD5t6Q] update_mapping [syslog]
[2019-04-05T17:38:49,696][INFO ][o.e.c.m.MetaDataMappingService] [elk6] [logstash-2019.03.30/uyV4qsFHTuKvmRJgSD5t6Q] update_mapping [syslog]
[2019-04-05T17:53:02,770][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elk6] adding template [.management-beats] for index patterns [.management-beats]
[2019-04-05T17:54:03,147][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elk6] adding template [.management-beats] for index patterns [.management-beats]
{
"took":779734,
"timed_out":false,
"total":2632343,
"updated":0,
"created":2632343,
"deleted":0,
"batches":2633,
"version_conflicts":0,
"noops":0,
"retries":{
"bulk":0,
"search":0
},
"throttled_millis":0,
"requests_per_second":-1.0,
"throttled_until_millis":0,
"failures":[
]
}To confirm list indices on cluster B and you will see logstash-2019.03.30 is now available on cluster B.
curl -X GET "localhost:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 FpjlQCtASg2B7rM9jMqd0w 1 0 4 0 17.2kb 17.2kb
yellow open logstash-2019.03.30 uyV4qsFHTuKvmRJgSD5t6Q 5 1 3787369 0 1.7gb 1.7gb
green open .kibana_task_manager odbrnDVfQWuIg_G6K3p6QQ 1 0 2 0 12.5kb 12.5kbDon’t forget to disable elasticsearch on network interface of cluster A by commenting “http.host: 0.0.0.0” in elasticsearch.yml and restarting elasticsearch.