Get the number of fields on an index
To build a bit further upon what the other answer provided, you can get the mapping and then simply count the number of times the keyword type
appears in the output, which gives the number of fields since each field needs a type:
curl -s -XGET localhost:9200/index/_mapping?pretty | grep type | wc -l
You can try this:
curl -s -XGET "http://localhost:9200/index/_field_caps?fields=*" | jq '.fields|length'
Just a quick way to get a relative estimate in Kibana without writing a script (I don't believe this is 100% precise, but it's an easy way to tell if your dynamic fields are blowing up to huge numbers for some reason).
Run this query in the Kibana dev tools
GET /index_name/_mapping
Inside the Kibana output, perform a search for all instances of "type"
(including quotes). This will count the instances and get you the answer. (In this example, 804)
This can be helpful if you scratching your head as to why you're getting the [remote_transport_exception]
error of
Limit of total fields [1000] in index [index_name] has been exceeded
The first answer by Val solves the problem for me too. But I just wanted to list out some corner cases which can lead to misleading numbers.
- The document has fields with the word "type" in them.
For example
"content_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
}
}
},
This will match grep type
thrice while it should do that only twice i.e. it should not match "content_type". This scenario has an easy fix.
Instead of
curl -s -XGET localhost:9200/index/_mapping?pretty | grep type
Use
curl -s -XGET localhost:9200/index/_mapping?pretty | grep '"type"'
to get an exact match of '"type"'
- The document has a field of the exact name "type"
For example
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
},
In this case also the match is thrice instead of twice. But using
curl -s -XGET localhost:9200/index/_mapping?pretty | grep '"type"'
is not going to cut it. We will have to skip fields with the "type" keyword as substring as well as an exact match. In this case we can add an additional filter like so:
curl -s -XGET localhost:9200/index/_mapping?pretty |\
grep '"type"' | grep -v "{"
In addition to the above 2 scenarios, if you are using the api programmatically to push numbers for tracking i.e. into something like AWS cloudwatch or Graphite, you can use the following code to call the API - get the data, and recursively search for the keyword "type" - while skipping any fuzzy matches and resolving deeper into fields with the exact name "type".
import sys
import json
import requests
# The following find function is a minor edit of the function posted here
# https://stackoverflow.com/questions/9807634/find-all-occurrences-of-a-key-in-nested-python-dictionaries-and-lists
def find(key, value):
for k, v in value.iteritems():
if k == key and not isinstance(v, dict) and not isinstance(v, list):
yield v
elif isinstance(v, dict):
for result in find(key, v):
yield result
elif isinstance(v, list):
for d in v:
for result in find(key, d):
yield result
def get_index_type_count(es_host):
try:
response = requests.get('https://%s/_mapping/' % es_host)
except Exception as ex:
print('Failed to get response - %s' % ex)
sys.exit(1)
indices_mapping_data = response.json()
output = {}
for index, mapping_data in indices_mapping_data.iteritems():
output[index] = len(list(find('type', mapping_data)))
return output
if __name__ == '__main__':
print json.dumps(get_index_type_count(sys.argv[1]), indent=2)
The above code is also posted as a gist here - https://gist.github.com/saurabh-hirani/e8cbc96844307a41ff4bc8aa8ebd7459