HughMungus HughMungus - 5 days ago 5
Java Question

I/O time when reading from HDFS in Hadoop

I would like to measure the time taken for map and reduce when performing I/O (reading from HDFS) in Hadoop. I am using Yarn.
Hadoop 2.6.0.
What are the options for that?

Answer

If you need exact measurments - you could use btrace, add it as a javaagent to your tasks via mapreduce.{map,reduce}.java.opts - and then write script which measures whatever you like. Sample of btrace scripts are here.

Also there is HTrace - that might also be helpful.

Comments