Vishwas Vishwas - 1 year ago 191
Scala Question

How to stop messages displaying on spark console?

I am using Spark 0.9 and Scala 2.10.

Everything is working fine. I just want to stop various messages that are coming on spark shell.

I tried to edit the
file in order to stop these message.

Here are the contents of

# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose$exprTyper=INFO$SparkILoopInterpreter=INFO

But messages are still getting displayed on the console.

Here are some example messages

15/01/05 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
15/01/05 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
15/01/05 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
15/01/05 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(,44728)
15/01/05 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager
15/01/05 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager with 0.0 B RAM
15/01/05 15:11:45 INFO BlockManagerMaster: Registered BlockManager
15/01/05 15:11:45 INFO HttpServer: Starting HTTP Server
15/01/05 15:11:45 INFO HttpBroadcast: Broadcast server star

How do I stop these?

Answer Source

Thanks @AkhlD and @Sachin Janani for suggesting changes in .conf file.

Following code solved my issue:

1) Added import org.apache.log4j.{Level, Logger} in import section

2) Added following line after creation of spark context object i.e. after val sc = new SparkContext(conf):

val rootLogger = Logger.getRootLogger()
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download