无法使用elasticsearch运行kibana和logstash
弹性的serach在9201端口上运行良好。 但无法与docker-compose一起运行kibana和logstash。
对于logstash，它会抛出错误：尝试重新连接到死ES实例的连接，但出现错误。
对于kibana它会发出警告：“warning”，“elasticsearch”，“admin”]，“pid”：1，“message”：“没有活的连接”
以下是docker-compose.yml文件：
version: '2' services: # Service 1 : elasticsearch elasticsearch-5-6: image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3 container_name: elasticsearch-5-6 ports: - "9201:9200" volumes: - /etc/elasticsearch/elasticsearch-5-6.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /var/elasticsearch/data/immunedata-5-6/:/usr/share/elasticsearch/data/ #- /etc/elasticsearch/logging.yml:/usr/share/elasticsearch/config/logging.yml #- /var/log/elasticsearch/:/usr/share/elasticsearch/logs/ environment: - cluster.name=docker-cluster-elasticsearch-5-6 #- bootstrap.memory_lock=true - "ES_JAVA_OPTS: -Xmx2048m -Xms2048m" # Disabling the xpack security as it costs after one month of free trail. - xpack.security.enabled=false # Service 2 : logstash logstash-5-6: image: docker.elastic.co/logstash/logstash:5.6.3 container_name: logstash-5-6 ports: #- "5044:5044" - "5001:5001" volumes: - /etc/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml - /etc/logstash/pipeline:/usr/share/logstash/pipeline #- /etc/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml #- /var/logstash/pipeline:/usr/share/logstash/pipeline environment: - "ES_JAVA_OPTS: -Xmx2048m -Xms2048m" depends_on: - elasticsearch-5-6 # Service 3 : kibana kibana-5-6: image: docker.elastic.co/kibana/kibana:5.6.3 container_name: kibana-5-6 ports: - "5601:5601" volumes: - /etc/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml #- /var/kibana/immunedata-5-6/:/usr/share/kibana/data/ environment: - xpack.security.enabled=false - xpack.graph.enabled = false - xpack.ml.enabled = false - xpack.monitoring.enabled = false - xpack.watcher.enabled = false - xpack.reporting.enabled = false depends_on: - elasticsearch-5-6 # Service 4 : elasticseach-head elasticsearch-head: image: mobz/elasticsearch-head:5 container_name: elasticsearch-head # will not wait for elasticsearch to be ready. ports: - "9100:9100" elasticserach.yml
cluster.name: immunedata-cluster-5.6 node.name: "immunedata-cluster-5-6.node-1" # Elasticsearch in docker access different data directory, defined mapping directory in docker-compose.yml #path.data: /var/elasticsearch/data/immunedata-5-6/ path.data: /usr/share/elasticsearch/data/ #path.data: /var/elasticsearch/data # NOTE : Since elasticsearch 5.x index level settings can NOT be set on the nodes configuration like the elasticsearch.yaml #index.number_of_shards: 1 #index.number_of_replicas: 0 # Allow all host access network.bind_host: 0.0.0.0 http.port: 9200 # To enable cross-origin resource sharing (Accessing on browser) http.cors.enabled: true http.cors.allow-origin : "*"
logstash.yml文件
http.host: "0.0.0.0" path.config: /usr/share/logstash/pipeline #xpack.monitoring.elasticsearch.url: http://localhost:9201 ##xpack.monitoring.elasticsearch.url: http://elasticsearch:9201 #xpack.monitoring.elasticsearch.username: logstash_system #xpack.monitoring.elasticsearch.password: changeme xpack.monitoring.enabled: false
kibana.yml文件
server.name: kibana server.host: "0" elasticsearch.url: http://192.168.56.10:9201 xpack.monitoring.ui.container.elasticsearch.enabled: false #elasticsearch.url: http://elasticsearch:9201 xpack.security.enabled: false ## Above I tired this - not working #elasticsearch.username: elastic #elasticsearch.password: changeme #xpack.monitoring.ui.container.elasticsearch.enabled: false #xpack.monitoring.ui.container.elasticsearch.enabled: true # Extra: ssl.verificationMode: false
日志：
[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141] elasticsearch-5-6 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141] elasticsearch-5-6 | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141] elasticsearch-5-6 | [2017-11-26T06:07:57,084][WARN ][oemjJvmGcMonitorService] [immunedata-cluster-5-6.node-1] [gc][young][14][6] duration [18.2s], collections [1]/[18.5s], total [18.2s]/[23.5s], memory [178.2mb]->[79.5mb]/[1.9gb], all_pools {[young] [132.1mb]->[964kb]/[133.1mb]}{[survivor] [16.6mb]->[12.5mb]/[16.6mb]}{[old] [29.4mb]->[66.5mb]/[1.8gb]} elasticsearch-5-6 | [2017-11-26T06:07:57,085][WARN ][oemjJvmGcMonitorService] [immunedata-cluster-5-6.node-1] [gc][14] overhead, spent [18.2s] collecting in the last [18.5s] elasticsearch-5-6 | [2017-11-26T06:07:57,298][ERROR][oexmciIndexRecoveryCollector] [immunedata-cluster-5-6.node-1] collector [index-recovery] failed to collect data elasticsearch-5-6 | org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; elasticsearch-5-6 | at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:114) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:52) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:256) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.3.jar:5.6.3] elasticsearch-5-6 | at elasticsearch-5-6 | [2017-11-26T06:08:45,238][WARN ][oexweExecutionService] [immunedata-cluster-5-6.node-1] Failed to execute watch [XYNCje-TQzKm9OLdiH60gQ_elasticsearch_cluster_status_60e3c208-acca-4462-ba47-0711279d8f5e-2017-11-26T06:08:35.573Z] elasticsearch-5-6 | [2017-11-26T06:08:54,886][WARN ][oemjJvmGcMonitorService] [immunedata-cluster-5-6.node-1] [gc][young][63][9] duration [3.6s], collections [1]/[4.6s], total [3.6s]/[30.2s], memory [226.9mb]->[103.5mb]/[1.9gb], all_pools {[young] [127.5mb]->[1mb]/[133.1mb]}{[survivor] [16.6mb]->[11.3mb]/[16.6mb]}{[old] [82.7mb]->[91.2mb]/[1.8gb]} elasticsearch-5-6 | [2017-11-26T06:08:54,886][WARN ][oemjJvmGcMonitorService] [immunedata-cluster-5-6.node-1] [gc][63] overhead, spent [3.6s] collecting in the last [4.6s] logstash-5-6 | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties elasticsearch-5-6 | [2017-11-26T06:08:55,988][INFO ][oecraAllocationService] [immunedata-cluster-5-6.node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.watcher-history-6-2017.11.20][0], [.monitoring-es-6-2017.11.20][0]] ...]). logstash-5-6 | [2017-11-26T06:08:56,786][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} logstash-5-6 | [2017-11-26T06:08:56,891][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} logstash-5-6 | [2017-11-26T06:08:57,558][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.6.3-java/modules/arcsight/configuration"} logstash-5-6 | [2017-11-26T06:09:04,121][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch-5-6:9201/]}} logstash-5-6 | [2017-11-26T06:09:04,123][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} elasticsearch-5-6 | [2017-11-26T06:09:04,687][WARN ][oecraDiskThresholdMonitor] [immunedata-cluster-5-6.node-1] high disk watermark [90%] exceeded on [eAlcHBJ2QVG58e0HJsgrdQ][immunedata-cluster-5-6.node-1][/usr/share/elasticsearch/data/nodes/0] free: 1.9gb[7.4%], shards will be relocated away from this node elasticsearch-5-6 | [2017-11-26T06:09:04,687][INFO ][oecraDiskThresholdMonitor] [immunedata-cluster-5-6.node-1] rerouting shards: [high disk watermark exceeded on one or more nodes] logstash-5-6 | [2017-11-26T06:09:06,450][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} logstash-5-6 | [2017-11-26T06:09:06,452][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil} logstash-5-6 | [2017-11-26T06:09:06,455][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!", :class=>"ArgumentError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:37:in `read_template_file'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:23:in `get_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:58:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:25:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:398:in `start_pipeline'"]} logstash-5-6 | [2017-11-26T06:09:06,455][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch-5-6:9201"]} logstash-5-6 | [2017-11-26T06:09:06,462][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250} logstash-5-6 | [2017-11-26T06:09:09,818][INFO ][logstash.pipeline ] Pipeline main started logstash-5-6 | [2017-11-26T06:09:10,341][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} logstash-5-6 | [2017-11-26T06:09:11,460][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} logstash-5-6 | [2017-11-26T06:09:11,484][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} logstash-5-6 | [2017-11-26T06:09:16,491][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} logstash-5-6 | [2017-11-26T06:09:16,500][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:21Z","tags":["warning","elasticsearch","config","deprecation"],"pid":1,"message":"Config key \"ssl.verify\" is deprecated. It has been replaced with \"ssl.verificationMode\""} logstash-5-6 | [2017-11-26T06:09:21,513][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} logstash-5-6 | [2017-11-26T06:09:21,523][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:26Z","tags":["status","plugin:kibana@5.6.3","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} logstash-5-6 | [2017-11-26T06:09:26,536][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} logstash-5-6 | [2017-11-26T06:09:26,570][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:26Z","tags":["status","plugin:elasticsearch@5.6.3","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:26Z","tags":["status","plugin:xpack_main@5.6.3","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:29Z","tags":["status","plugin:graph@5.6.3","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:29Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch-5-6:9201/ => connect ECONNREFUSED 172.21.0.2:9201"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:29Z","tags":["status","plugin:monitoring@5.6.3","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:29Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-5-6:9201/"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:29Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} logstash-5-6 | [2017-11-26T06:09:31,585][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} logstash-5-6 | [2017-11-26T06:09:31,603][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["reporting","warning"],"pid":1,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["status","plugin:reporting@5.6.3","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["status","plugin:xpack_main@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["status","plugin:graph@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["status","plugin:reporting@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["status","plugin:elasticsearch@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:33Z","tags":["status","plugin:searchprofiler@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201.","prevState":"uninitialized","prevMsg":"uninitialized"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-5-6:9201/"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:34Z","tags":["status","plugin:ml@5.6.3","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201.","prevState":"uninitialized","prevMsg":"uninitialized"} elasticsearch-5-6 | [2017-11-26T06:09:34,750][WARN ][oecraDiskThresholdMonitor] [immunedata-cluster-5-6.node-1] high disk watermark [90%] exceeded on [eAlcHBJ2QVG58e0HJsgrdQ][immunedata-cluster-5-6.node-1][/usr/share/elasticsearch/data/nodes/0] free: 1.9gb[7.4%], shards will be relocated away from this node logstash-5-6 | [2017-11-26T06:09:36,692][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@elasticsearch-5-6:9201/, :path=>"/"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:37Z","tags":["status","plugin:ml@5.6.3","info"],"pid":1,"state":"yellow","message":"Status changed from red to yellow - Waiting for Elasticsearch","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch-5-6:9201."} logstash-5-6 | [2017-11-26T06:09:37,366][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch-5-6:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch-5-6:9201/][Manticore::SocketException] Connection refused (Connection refused)"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-5-6:9201/"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-5-6:9201/"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} kibana-5-6 | {"type":"log","@timestamp":"2017-11-26T06:09:37Z","tags":