Too many open files warning from elasticsearch

How to increase the maximum number of allowed open files depends slightly on your linux distribution. Here are some instructions for ubuntu and centos:

http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/ http://pro.benjaminste.in/post/318453669/increase-the-number-of-file-descriptors-on-centos-and

The elasticsearch documentation recommends setting the maximum file limit to 32k or 64k. Since you are at 16k and are already hitting a limit, I'd probably set it higher; something like 128k. See: http://www.elasticsearch.org/guide/reference/setup/installation/

After upping the number of open files and restarting elasticsearch you will want to verify that it worked by re-running the curl command you mentioned:

curl -XGET 'http://localhost:9200/_nodes?os=true&process=true&pretty=true'

As you add more indices (along with more documents), you will also see the number of files elasticsearch keeps track of increase. If you notice performance degradation with all of the indicies and documents, you can try adding a new node to your cluster: http://www.elasticsearch.org/guide/reference/setup/configuration/ - since you already have a sharded, replicated configuration, this should be a relatively painless process.

1. Stop ElasticSearch. if you start from command like (bin/elasticsearch) then please specific this to set up heap while starting. For ex, I use a 16GB box so my command is

   a. bin/elasticsearch -Xmx8g -Xms8g

   b. Go to config (elasticsearch/config/elasticsearch.yml) and ensure that

   bootstrap.mlockall: true

   c. Increase ulimits -Hn and ulimits -Sn to more than 200000

2. If you start as a service, then do the following

   a. export ES_HEAP_SIZE=10g b. Go to config (/etc/elasticsearch/elasticsearch.yml) and ensure that

   bootstrap.mlockall: true c. Increase ulimits -Hn and ulimits -Sn to more than 200000

Make sure that the size you enter is not more than 50% of the heap whether you start it as a service or from command line

Note that changing ulimits via /etc/security/limits.conf won't have any effect if elasticsearch is a systemd service.

To increase the elasticsearch limit under systemd create a file /etc/systemd/system/elasticsearch.service.d/override.conf with the following content:

[Service]
LimitNOFILE=infinity

Then run systemctl daemon-reload && systemctl restart elasticsearch.