I am using Apache Spark DataFrames to join two data sources and get the result as another DataFrame. I want to write the result to another Postgres table. I see this option :
myDataFrame.write.jdbc(url, table, connectionProperties)
It is not supported.
DataFrameWriter can either append to or overwrite existing table. If your application requires more complex logic you'll have to deal with this manually.
One option is to use an action (
foreachPartition) with standard JDBC connection. Another one is to write to a temporary and handle the rest directly in the database.