PBL PBL - 4 months ago 32
MySQL Question

In Apache Spark 2.0.0, is it possible to fetch a query from an external database (rather than grab the whole table)?

Using pyspark:

from pyspark.sql import SparkSession

spark = SparkSession\
.builder\
.appName("spark play")\
.getOrCreate()

df = spark.read\
.format("jdbc")\
.option("url", "jdbc:mysql://localhost:port")\
.option("dbtable", "schema.tablename")\
.option("user", "username")\
.option("password", "password")\
.load()


Rather than fetch "schema.tablename", I would prefer to grab the result set of a query.

Answer

Same as in 1.x you can pass valid subquery as dbtable argument for example:

...
.option("dbtable", "(SELECT foo, bar FROM schema.tablename) AS tmp")
...
Comments