Elasticsearch read_only_allow_delete auto setting

In an attempt to add a sprinkling of value to the accepted answer (and because i'll google this and come back in future), for my case the read_only_allow_delete flag was set because of the default settings for disk watermark being percentage based - which on my large disk did not make as much sense. So I changed these settings to be "size remaining" based as the documentation explains.

So before setting read_only_allow_delete back to false, I first set the watermark values based on disk space:

(using Kibana UI):

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "20gb",
    "cluster.routing.allocation.disk.watermark.high": "15gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"
  }
}

PUT your_index_name/_settings
{
 "index": {
   "blocks": {
     "read_only_allow_delete": "false"
    }
  }
}

OR (through the terminal):

$ curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_cluster/_settings \
   -d '{"cluster.routing.allocation.disk.watermark.low": "20gb", 
     "cluster.routing.allocation.disk.watermark.high": "15gb", 
     "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"}'

$ curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_all/_settings \
   -d '{"index.blocks.read_only_allow_delete": false}'

Background

We maintain a cluster where we have filebeat, metricbeat, packetbeat, etc. shippers pushing data into the cluster. Invariably some index would become hot and we'd want to either disable writing to it for a time or do clean up and reenable indices which had breached their low watermark thresholds and had automatically gone into read_only_allow_delete: true.

Bash Functions

To ease the management of our clusters for the rest of my team I wrote the following Bash functions to help perform these tasks without having to fumble around with curl or through Kibana's UI.

$ cat es_funcs.bash

### es wrapper cmd inventory
declare -A escmd
escmd[l]="./esl"
escmd[p]="./esp"

### es data node naming conventions
nodeBaseName="rdu-es-data-0"
declare -A esnode
esnode[l]="lab-${nodeBaseName}"
esnode[p]="${nodeBaseName}"

usage_chk1 () {
    # usage msg for cmds w/ 1 arg
    local env="$1"

    [[ $env =~ [lp] ]] && return 0 || \
        printf "\nUSAGE: ${FUNCNAME[1]} [l|p]\n\n" && return 1
}

enable_readonly_idxs () {
    # set read_only_allow_delete flag
    local env="$1"
    usage_chk1 "$env" || return 1
    DISALLOWDEL=$(cat <<-EOM
        {
         "index": {
           "blocks": {
             "read_only_allow_delete": "true"
            }
          }
        }
    EOM
    )
    ${escmd[$env]} PUT '_all/_settings' -d "$DISALLOWDEL"
}

disable_readonly_idxs () {
    # clear read_only_allow_delete flag
    local env="$1"
    usage_chk1 "$env" || return 1
    ALLOWDEL=$(cat <<-EOM
        {
         "index": {
           "blocks": {
             "read_only_allow_delete": "false"
            }
          }
        }
    EOM
    )
    ${escmd[$env]} PUT '_all/_settings' -d "$ALLOWDEL"
}

Example Run

The above functions can be sourced in your shell like so:

$ . es_funcs.bash

NOTE: The arrays at the top of the file map short names for clusters if you happen to have multiple. We have 2, one for our lab and one for our production. So I represented those as l and p.

You can then run them like this to enable the read_only_allow_delete attribute (true) on your l cluster:

$ enable_readonly_idxs l
{"acknowledged":true}

or p:

$ enable_readonly_idxs p
{"acknowledged":true}

Helper Script Overview

There's one additional script that contains the curl commands which I use to interact with the clusters. This script is referenced in the escmd array at the top of the es_func.bash file. The array contains names of symlinks to a single shell script, escli.bash. The links are called esl and esp.

$ ll
-rw-r--r-- 1 smingolelli staff  9035 Apr 10 23:38 es_funcs.bash
-rwxr-xr-x 1 smingolelli staff  1626 Apr 10 23:02 escli.bash
-rw-r--r-- 1 smingolelli staff   338 Apr  5 00:27 escli.conf
lrwxr-xr-x 1 smingolelli staff    10 Jan 23 08:12 esl -> escli.bash
lrwxr-xr-x 1 smingolelli staff    10 Jan 23 08:12 esp -> escli.bash

The escli.bash script:

$ cat escli.bash
#!/bin/bash

#------------------------------------------------
# Detect how we were called [l|p]
#------------------------------------------------
[[ $(basename $0) == "esl" ]] && env="lab1" || env="rdu1"

#------------------------------------------------
# source escli.conf variables
#------------------------------------------------
# g* tools via brew install coreutils
[ $(uname) == "Darwin" ] && readlink=greadlink || readlink=readlink
. $(dirname $($readlink -f $0))/escli.conf


usage () {
    cat <<-EOF

    USAGE: $0 [HEAD|GET|PUT|POST] '...ES REST CALL...'

    EXAMPLES:

        $0 GET  '_cat/shards?pretty'
        $0 GET  '_cat/indices?pretty&v&human'
        $0 GET  '_cat'
        $0 GET  ''
        $0 PUT  '_all/_settings'   -d "\$DATA"
        $0 POST '_cluster/reroute' -d "\$DATA"


    EOF
    exit 1
}

[ "$1" == "" ] && usage

#------------------------------------------------
# ...ways to call curl.....
#------------------------------------------------
if [ "${1}" == "HEAD" ]; then
    curl -I -skK \
        <(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
        "${esBaseUrl}/$2"
elif [ "${1}" == "PUT" ]; then
    curl -skK \
        <(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
        -X$1 -H "${contType}" "${esBaseUrl}/$2" "$3" "$4"
elif [ "${1}" == "POST" ]; then
    curl -skK \
        <(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
        -X$1 -H "${contType}" "${esBaseUrl}/$2" "$3" "$4"
else
    curl -skK \
        <(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
        -X$1 "${esBaseUrl}/$2" "$3" "$4" "$5"
fi

This script takes a single property file, escli.conf. In this file you specify the commands to retrieve your username + password from whereever, I use LastPass for that so retrieve them via lpass as well as setting the base URL to use for accessing your clusters REST API.

$ cat escli.conf
#################################################
### props used by escli.bash
#################################################

usernameCmd='lpass show --username somedom.com'
passwordCmd='lpass show --password somedom.com'

esBaseUrl="https://es-data-01a.${env}.somdom.com:9200"
contType="Content-Type: application/json"

I've put all this together in a Github repo (linked below) which also includes additional functions beyond the above 2 that I'm showing as examples for this question.

References

  • https://github.com/slmingol/escli

Elasticsearch automatically sets "read_only_allow_delete": "true" when hard disk space is low.

Find the files which are filling up your storage and delete/move them. Once you have sufficient storage available run the following command through the Dev Tool in Kibana:

PUT your_index_name/_settings
{
 "index": {
   "blocks": {
     "read_only_allow_delete": "false"
    }
  }
}

OR (through the terminal):

$ curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_all/_settings \
     -d '{"index.blocks.read_only_allow_delete": false}'

as mentioned in your question.