Chaitanya Bapat Chaitanya Bapat - 1 year ago 1008
Python Question

SparkContext Error - File not found /tmp/spark-events does not exist

Running a Python Spark Application via API call -
On submitting the Application - response - Failed
SSH into the Worker

My python application exists in


Error can be found in


Show the following error -

Traceback (most recent call last):
File "/root/", line 34, in <module>
File "/root/", line 18, in main
sc = SparkContext(conf=conf)
File "/root/spark/python/lib/", line 115, in __init__
File "/root/spark/python/lib/", line 172, in _do_init
File "/root/spark/python/lib/", line 235, in _initialize_context
File "/root/spark/python/lib/", line 1064, in __call__
File "/root/spark/python/lib/", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
: File file:/tmp/spark-events does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:100)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:549)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
at java.lang.reflect.Constructor.newInstance(
at py4j.reflection.MethodInvoker.invoke(
at py4j.reflection.ReflectionEngine.invoke(
at py4j.Gateway.invoke(
at py4j.commands.ConstructorCommand.invokeConstructor(
at py4j.commands.ConstructorCommand.execute(

It indicates - /tmp/spark-events Does not exist - which is true
However, in

from pyspark import SparkContext, SparkConf

... few more lines ...

def main():
conf = SparkConf().setAppName("MyApp").setMaster("spark://")
sc = SparkContext(conf=conf)

if __name__ == "__main__":

Answer Source

/tmp/spark-events is the location that Spark store the events logs. Just create this directory in the master machine and you're set.

$mkdir /tmp/spark-events
$ sudo /root/spark-ec2/copy-dir /tmp/spark-events/
RSYNC'ing /tmp/spark-events to slaves...
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download