Logstash with Elasticsearch

As Adam said, the thing was the protocol setting, so only for testing I did:

logstash -e 'input { stdin { } } output { elasticsearch { host => localhost protocol => "http" port => "9200" } }'

And that seems to be working on OSX. Issue here.

Following is tested on

elasticsearch:5.4.0

and

logstash:5.4.0

(I have use docker container on OpenStack)

For Elasticsearch :

/usr/share/elasticsearch/config/elasticsearch.yml should look like as follows -

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

No change in any other files of /usr/share/elasticsearch/config/ is required

Run Elasticsearch using simple command -

sudo docker run --name elasticsearch -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:5.4.0

For Logstash :

/usr/share/logstash/config/logstash.yml should look like as follows -

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline

# http://111.*.*.11:9200 is the IP & Port of Elasticsearch's server 
xpack.monitoring.elasticsearch.url: http://111.*.*.11:9200

# "elastic" is the user name of Elasticsearch's account
xpack.monitoring.elasticsearch.username: elastic 

# "changeme" is the password of Elasticsearch's "elastic" user 
xpack.monitoring.elasticsearch.password: changeme

No change in any other files of /usr/share/logstash/config/ is required

/usr/share/logstash/pipeline/logstash.conf should look like as follows -

input {
        file {
                path => "/usr/share/logstash/test_i.log"

        }
}


output {

        elasticsearch {
                # http://111.*.*.11:9200 is the IP & Port of Elasticsearch's server
                hosts => ["http://111.*.*.11:9200"]

                # "elastic" is the user name of Elasticsearch's account
                user => "elastic"

                # "changeme" is the password of Elasticsearch's "elastic" user
                password => "changeme"
        }
}

Run Logstash using simple command -

sudo docker run --name logstash --expose 25826 -p 25826:25826 docker.elastic.co/logstash/logstash:5.4.0 --debug

NOTE : Need not to do any configuration before running Docker containers. At first run the container using simple commands as mentioned above. Then go to corresponding dir, make the required changes, save it, exit container & restart the container, changes will be reflected.

First, I suggest matching your "type" attributes up. In your input you have 2 different types, and in your output you have a type that doesn't exists in any of your inputs.

For testing, change your output to:

output {
stdout { }
elasticsearch{
    type => "stdin-type"
    embedded => false
    host => "192.168.0.23"
    port => "9300"
    cluster => "logstash-cluster"
    node_name => "logstash"
    }
}

Then,have you created an index on your ES instance?

From the guides I've used, and my own experience (others may have another way that works) I've always used an index so that when I push something into ES, I can use the ES API and quickly check if the data has gone in or not.

Another suggestion would be to simply run your Logstash forwarder and indexer with debug flags to see what is going on behind the scenes.

Can you connect to your ES instance on 127.0.0.1? Also, try to experiment with the port and host. As a rather new user of the Logstash system, I found that my understanding at the start went against the reality of the setup. Sometimes the host IP isn't what you think it is, as well as the port. If you are willing to check your network and identify listening ports and IPs, then you can sort this out, otherwise do some intelligent trial and error.

I highly recommend this guide as a comprehensive starting point. Both points I've mentioned are (in)directly touched upon in the guide. While the guide has a slightly more complex starting point, the ideas and concepts are thorough.

I could not make Logstash connect to ES

This happened to me when my logstash and elasticsearch versions were out of sync

from the docs:

VERSION NOTE: Your Elasticsearch cluster must be running Elasticsearch 1.1.1. If you use any other version of Elasticsearch, you should set protocol => http in this plugin.

Setting protocol => http explicitly as outlined above fixed it for me.