Vendrad Vendrad - 1 year ago 170
Scala Question

error: value cassandraFormat is not a member of org.apache.spark.sql.DataFrameWriter

Checking the repo on github I see cassandraFormat here

My import statement is not throwing an exception

import org.apache.spark.sql.cassandra._

.cassandraFormat("keyspace", "table")

<console>:34: error: value cassandraFormat is not a member of org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row]

cassandraFormat is not available under
but it is under

I am using Spark 2.1.1
Spark-Shell invoked by...

spark-shell --master spark:// --packages com.databricks:spark-csv_2.11:1.5.0,datastax:spark-cassandra-connector:1.6.6-s_2.11


Just to add... I did realise that cassandraFormat was basically an alias for
However a different error was returned:

.options(Map("table" -> "standard_feed", "keyspace" -> "testing"))

java.lang.AbstractMethodError: org.apache.spark.sql.cassandra.DefaultSource.createRelation
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:518)

Answer Source

I'm not so sure about cassandraFormat but saving a dataframe to a cassandra table can easiest be done in the following way:

  .options(Map("table" -> "table_name", "keyspace" -> "keyspace_name"))

About the error message you received:

I think the error message you get is due to a version mismatch. You use spark 2.1.1 and cassandra-connector version 1.6.6. For spark 2.1.x you need to use version 2.0 of the cassandra-connector, see table here for a full list of the version compatibilities.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download