Graylog: ERROR [IndexRotationThread] couldn’t point deflector to a new index

Graylog: ERROR [IndexRotationThread] couldn’t point deflector to a new index

Along with the subject line error you’ll get following errors as well and elastic search would not be able deflect indices. Eventually it will fill up Journal and output messages would not be written to disk.

[2016-05-04 16:48:37,178][WARN ][cluster.routing.allocation.decider] [Alcmena] high disk watermark [90%] exceeded on [3jSNwaVWSp6BtPvQfvRqNw][Alcmena]

free: 139.8gb[9.8%], shards will be relocated away from this node

[2016-05-04 16:23:02,411][WARN ][cluster.routing.allocation.decider] [Shocker] high disk watermark [90%] exceeded on [I-E4meU_SmyfOThxlFygwg][Shocker] free: 139.3gb[9.8%], shards will be relocated away from this node

[2016-05-04 16:23:02,411][INFO ][cluster.routing.allocation.decider] [Shocker] high disk watermark exceeded on one or more nodes, rerouting shards

Cause:

This happens due to the default settings in elasticsearch. Elasticsearch factors in the available disk space on a node before deciding whether to allocate new shards to that node or to actively relocate shards away from that node.

The default warning is set to 85%, meaning ES will not allocate new shards to nodes once they have more than 85% disk used. The default critical is set to 90%, meaning ES will attempt to relocate shards to another node if the node disk usage rises above 90%. So this problem happens when your indices FS is low on space.

Solution:

You can configure your elastic search not to set any threshold value. To achieve this you need to change the “/etc/elasticsearch/elasticsearch.yml” and restart the elasticsearch.

cluster.routing.allocation.disk.threshold_enabled: false

Or, you can define the threshold value.

cluster.routing.allocation.disk.watermark.low à Warning Value

cluster.routing.allocation.disk.watermark.high à Critical value

Restart the elasticsearch.

/etc/init.d/elasticsearch restart

You are done. You’ll see the following logs in elasticsearch logs.

INFO  [IndexRetentionThread] Number of indices (51) higher than limit (50). Running retention for 1 indices.

INFO  [IndexRetentionThread] Running retention strategy [org.testserver.indexer.retention.strategies.DeletionRetentionStrategy]

for index <graylog_232>

INFO  [RetentionStrategy] Strategy is deleting.

INFO  [RetentionStrategy] Finished index retention strategy [org.testserver.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog_232> in 1276ms.