在Java中通过spark连接到Cassandra时出现问题
我有服务器与docker,并创build3 Cassandra节点,2个工人火花节点和一个主火花节点。 现在我想通过Java应用程序连接到我的笔记本电脑的火花。 我的java代码是:
public SparkTestPanel(String id, User user) { super(id); form = new Form("form"); form.setOutputMarkupId(true); this.add(form); SparkConf conf = new SparkConf(true); conf.setAppName("Spark Test"); conf.setMaster("spark://172.11.100.156:9050"); conf.set("spark.cassandra.connection.host", "cassandra-0"); conf.set("spark.cassandra.connection.port", "9042"); conf.set("spark.cassandra.auth.username", "cassandra"); conf.set("spark.cassandra.auth.password", "cassandra"); JavaSparkContext sc = null; try { sc = new JavaSparkContext(conf); CassandraTableScanJavaRDD<com.datastax.spark.connector.japi.CassandraRow> cassandraTable = javaFunctions(sc).cassandraTable("test", "test_table"); List<com.datastax.spark.connector.japi.CassandraRow> collect = cassandraTable.collect(); for(com.datastax.spark.connector.japi.CassandraRow cassandraRow : collect){ Logger.getLogger(SparkTestPanel.class).error(cassandraRow.toString()); } } finally { sc.stop(); } }
我知道应用程序连接到火花大师,因为我看到我的应用程序在web ui的火花,但在线:
CassandraTableScanJavaRDD<com.datastax.spark.connector.japi.CassandraRow> cassandraTable = javaFunctions(sc).cassandraTable("test", "test_table");
我得到下面的错误:
2017-08-17 12:14:31,906 ERROR CassandraConnectorConf:72 - Unknown host 'cassandra-0' java.net.UnknownHostException: cassandra-0: nodename nor servname provided, or not known ...
和其他错误:
Caused by: java.lang.IllegalArgumentException: Cannot build a cluster without contact points at com.datastax.driver.core.Cluster.checkNotEmpty(Cluster.java:119) at com.datastax.driver.core.Cluster.<init>(Cluster.java:112) at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178) at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1335) at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:131) at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:159) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32) at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69) at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57) at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:79) at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111) at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:122) at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:330) at com.datastax.spark.connector.cql.Schema$.tableFromCassandra(Schema.scala:350) at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:50) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:62) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:62) at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:137) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:62) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:262) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121)
我去服务器(“172.11.100.156”),并得到火花主容器ping cassandra-0,并看到:
root@708d210056af:/# ping cassandra-0 PING cassandra-0 (21.1.0.21): 56 data bytes 64 bytes from 21.1.0.21: icmp_seq=0 ttl=64 time=0.554 ms 64 bytes from 21.1.0.21: icmp_seq=1 ttl=64 time=0.117 ms 64 bytes from 21.1.0.21: icmp_seq=2 ttl=64 time=0.116 ms 64 bytes from 21.1.0.21: icmp_seq=3 ttl=64 time=0.093 ms
我的应用程序中发生了什么,可以得到这个错误? 任何人都可以帮忙吗?
- 如何将Spark集群与其他Hadoop2.x集群集成到Docker上
- 使用Spark处理通过kafka进行的数据stream并使用Python进行编程
- Spark Job Server是否必须与Spark Master部署在同一台主机上?
- docker停止从退出火花容器
- 我怎样才能在DockerFile中定义我的ENVvariables,并将其传递给由supervisord托pipe脚本提交的spark图像?
- pthread_create失败:MongoDB上的资源暂时不可用
- 使用Hadoop和Docker进行代码开发的过程
- 在Windows 7上运行Hydrosphere Mist Docker
- 在火花集群模式下运行齐柏林飞艇