Search

Sort:

ElasticSearch
mail # user
[Elasticsearch] Recommended way to delete an index

From: Animageofmine 2017-06-25, 20:48

If you delete the index using DELETE API, does it clear the fielddata / cache? Or is it suggested that we stop the index before deleting it?

I want to make sure that we don't keep increasing memory footprint (JVM/ OS Memory) forever. Thanks so much.

---

From: Mark Walkom 2017-06-25, 23:37

If you delete the index you delete anything else related to it, including any caches.

---

From: Animageofmine 2017-06-25, 23:38

Thanks for the confirmation.

If you close the index, does it get rid of all resources that it would consume other than the disk space?

---

From: Mark Walkom 2017-06-25, 23:39

What do you mean by stop the index?

---

From: Animageofmine 2017-06-25, 23:40

My bad, I meant close the index. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-open-close.html

---

From: Mark Walkom 2017-06-25, 23:44

If you close an index it removes any caches, it still takes some resources because the cluster maintains the index info in cluster state (in memory).

---

From: Animageofmine 2017-06-25, 23:47

I am guessing that is purely metadata like mentioned in the documentation and doesn't occupy too much space. We have observed that having too many indices/shards slows down the cluster and there isn't any easy way to recovery from that state. For example, we have 47k shards on a cluster of 8 nodes (5 data + 3 dedicated master nodes).

We also have dedicated master nodes (3 to be specific), so suppose the that master nodes take care keeping index information.

---

From: Mark Walkom 2017-06-26, 01:37

Each shard needs a certain amount of resources to exist.

[quote="animageofmine, post:7, topic:90769"]
47k shards on a cluster of 8 nodes (5 data + 3 dedicated master nodes)
[/quote]

Yeah, that's a little overboard.

What version are you on.

---

From: Animageofmine 2017-06-26, 01:43

version 5.3

---

From: Mark Walkom 2017-06-26, 05:54

Use the `_shrink` API to reduce the count for old indices.

---

From: Mark Walkom 2017-07-17, 21:22

A quick question @animageofmine, are you using the default number of shards or have you customised them?

---

From: Animageofmine 2017-07-18, 00:02

customized. It is based on the number of expected documents for index. Why do you ask?

---

From: Mark Walkom 2017-07-18, 00:09

Just wondering if you were using defaults or not :)

---

From: Animageofmine 2017-07-18, 00:47

ok :-). Default is way to high for most of the cases for our system.

If you get a chance to look at this question, I would really appreciate that.

https://discuss.elastic.co/t/does-transport-client-cache-nodes-in-the-cluster/93243

---

NEW: Monitor These Apps!

elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB