卡夫卡在docker工人不工作

我试图用wurstmeister\kafka-docker docker-compose来使用wurstmeister\kafka-docker图片,但是我在连接所有东西时遇到了实际的问题。

我查的所有post或者问题似乎没有任何问题,但是我坦白的说是输了。 (至less有两个问题试图解决这个问题)

我相信问题是我对docker的networking了解docker 。 所以问题是:

我可以从同一个kafka容器中进行消费和生产,但是当我尝试创build另一个容器时(或者使用我的笔记本电脑和python客户端),我得到了一些与advertised.host.name参数有关的错误(在图像中,这个参数是KAFKA_ADVERTISED_HOST_NAME

我已经尝试以很多方式设置这个variables,但它不起作用。

所以我正在寻找一个authorative答案(即如何自动设置这些参数,这是什么意思)如何设置docker-compose.yml

这是我的:

 zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka # hostname: kafka ports: - "9092" links: - zookeeper:zk environment: KAFKA_ADVERTISED_HOST_NAME: "kafka" KAFKA_ADVERTISED_PORT: "9092" KAFKA_ZOOKEEPER_CONNECT: "zk:2181" 

UPDATE

遵循@dnephin的build议,我修改了以下几行中的start-kafka.sh

 ... if [[ -z "$KAFKA_ADVERTISED_PORT" ]]; then export KAFKA_ADVERTISED_PORT=$(hostname -i) fi ... 

并从KAFKA_ADVERTISED_HOST_NAME: "kafka" docker-compose.yml移除KAFKA_ADVERTISED_HOST_NAME: "kafka"

我以规范的方式启动容器:

 docker-compose up -d 

两个容器都在运行:

 $ docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------------------------------- infraestructura_kafka_1 start-kafka.sh Up 0.0.0.0:32768->9092/tcp infraestructura_zookeeper_1 /opt/zookeeper/bin/zkServe ... Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp 

之后我做了:

 docker-compose logs 

一切运行顺利。

为了检查IP地址:

 $ KAFKA_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' infraestructura_kafka_1) $ echo $KAFKA_IP 172.17.0.4 and $ ZK_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' infraestructura_zookeeper_1) $ echo $ZK_IP 172.17.0.3 

然后我执行两个不同的游戏机:

生产者:

 $ docker run --rm --interactive wurstmeister/kafka /opt/kafka_2.11-0.9.0.1/bin/kafka-console-producer.sh --topic grillo --broker-list 171.17.0.4:9092 

消费者:

 $ docker run --rm --interactive wurstmeister/kafka /opt/kafka_2.11-0.9.0.1/bin/kafka-console-consumer.sh --topic grillo --from-beginning --zookeeper 172.17.0.3:2181 

几乎立刻,警告开始飞过屏幕:

 [2016-03-11 00:39:17,010] WARN Fetching topic metadata with correlation id 0 for topics [Set(grillo)] from broker [BrokerEndPoint(1001,ba53d4fd7595,9092)] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74) at kafka.producer.SyncProducer.send(SyncProducer.scala:119) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2016-03-11 00:39:17,013] WARN [console-consumer-79688_9dd5f575d557-1457656747003-f1ed369d-leader-finder-thread], Failed to find leader for Set([grillo,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFin derThread) kafka.common.KafkaException: fetching topic metadata for topics [Set(grillo)] from broker [ArrayBuffer(BrokerEndPoint(1001,ba53d4fd7595,9092))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74) at kafka.producer.SyncProducer.send(SyncProducer.scala:119) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) ... 3 more 

等等

在制作人的控制台里,我写了一些句子:

 $ docker run --rm --interactive klustera/kafka /opt/kafka_2.11-0.9.0.1/bin/kafka-console-producer.sh --topic grillo --broker-list 171.17.0.4:9092 Hola ¿Cómo estáń? ¿Todo bien? 

过了一会儿,我得到了这个回应:

 [2016-03-11 00:39:28,955] ERROR Error when sending message to topic grillo with key: null, value: 4 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) [2016-03-11 00:40:28,956] ERROR Error when sending message to topic grillo with key: null, value: 16 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) [2016-03-11 00:41:28,956] ERROR Error when sending message to topic grillo with key: null, value: 12 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) 

docker-compose logs

 ... zookeeper_1 | 2016-03-11 00:39:07,072 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x153631368b1000b type:create c xid:0x2 zxid:0x47 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers zookeeper_1 | 2016-03-11 00:39:07,243 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x153631368b1000b type:create c xid:0x19 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-79688/owners/grillo Error:KeeperErrorCode = NoNode for /consumers/console-consumer-79688/owners/grillo zookeeper_1 | 2016-03-11 00:39:07,247 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x153631368b1000b type:create $xid:0x1a zxid:0x4c txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-79688/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-79688/owners ... 

更新2

至less在docker-machine ,我做到了这一点:

首先,我使用docker-machine的名字定义了一个variables:

 DOCKER_VM=kafka_test 

然后,我修改docker-compose.yml ,如下所示:

 KAFKA_ADVERTISED_HOST_NAME: "${DOCKER_MACHINE_IP}" 

最后,在docker-machine的环境中,我执行:

 DOCKER_MACHINE_IP=$(docker-machine ip $DOCKER_VM) docker-compose up -d 

但在笔记本电脑(我的意思是,不使用虚拟机,它不工作)

为了在本地主机上开发应用程序,在文档中有一个解决scheme:“HOSTNAME_COMMAND”

 kafka: image: wurstmeister/kafka ports: - 9092:9092 environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'" 

希望这可以帮助别人…

我对这个问题的解决方法稍有不同。 我configurationKafka在kafka主机上做广告,因为它暴露在localhost:9092的主机上,我在/etc/hosts添加了一个条目来parsing为localhost 。 通过这样做可以从其他Docker容器和本地主机访问Kafka。

泊坞窗,compose.yml:

  my-web-service: build: ./my-web-service ports: - "8000:8000" links: - kafka kafka: image: "wurstmeister/kafka:0.10.2.0" ports: - "9092:9092" hostname: kafka links: - zookeeper environment: - KAFKA_ADVERTISED_HOST_NAME=kafka - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 - KAFKA_ADVERTISED_PORT=9092 zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" 

更新主机文件:

 more /etc/hosts 127.0.0.1 localhost kafka 

我相信您使用KAFKA_ADVERTISED_HOST_NAME的价值将会根据容器的到达方式而改变。

如果您尝试从另一个容器进行连接,则使用kafka应该是正确的(只要您使用该名称作为链接别名)。

如果您尝试从主机连接,则该名称不起作用。 您需要使用容器IP地址,您可以使用docker inspect 。 但是容器的IP地址将会改变,所以最好使用$(hostname -i)来从容器中设置它来检索它。

只要尝试以下服务发现,例如这一个 。

 zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: build: . ports: - "9092:9092" links: - zookeeper:zk environment: KAFKA_ADVERTISED_HOST_NAME: 192.168.59.103 KAFKA_ADVERTISED_PORT: 9092 KAFKA_CREATE_TOPICS: "test:1:1" volumes: - /var/run/docker.sock:/var/run/docker.sock 

或者你使用这个:

 zookeeper: image: wurstmeister/zookeeper ports: - "2181" kafka: build: . ports: - "9092" links: - zookeeper:zk environment: KAFKA_ADVERTISED_HOST_NAME: 192.168.59.103 DOCKER_HOST: 192.168.59.103:2375 volumes: - /var/run/docker.sock:/var/run/docker.sock