Ije Ije - 1 month ago 18
Java Question

java.lang.ArrayIndexOutOfBoundsException: MapReduce

Beginner level - can't get my head around this.

I wrote a MapReduce program that will result in a list of words and which line number they appear in, e.g.


  • and: 4

  • if: 1, 2, 3



etc. (see Sample Text below).

My code works perfectly well when there is no blank line as in line 2. But it throws an Exception 1 error for the below sample text.

From what I understand, this error indicates that there is an element in the array I am trying to access that doesn't exist - in this case there is just no element in the array for line 2. But how can I edit my code to ignore the blank line?

Here's the Mapper Code (with sample text):

import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;


// SampleText:
1: if you prick us do we not bleed
2:
3: if you tickle us do we not laugh
4: if you poison us do we not_ die and
5: ***if you wrong us shall we not revenge


public class IIndexMapper extends Mapper<LongWritable, Text, Text, Text>
{
private final static Text listing = new Text();
private Text wordText = new Text();

@Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException
{
//Single out line number by splitting each line by colon - first part being lineID
String[] line = value.toString().split(": ");
String lineID = line[0];
listing.set(lineID);

//Further split second part of the line by spaces
String textStr = line[1];

//Create an array of words contained in each line
String [] tokens = textStr.split(" ");
int count = tokens.length;

for (int i = 0; i < count; i++)
{
wordText.set(tokens[i]);
context.write(wordText, listing);
}




}
}


Any help is much appreciated!

Thanks!

Answer

I'd rewrite the above code something like this because we are accessing Elements in String Array with checking the existence.

String[] line = value.toString().split(": ");
if(line.length >= 2){

    listing.set(line[0]);

    //Create an array of words contained in each line
    String [] tokens = line[1].split(" ");
    int count = tokens.length;

    for (String token : tokens){
        wordText.set(token);
        context.write(wordText, listing);
    }
} else {
    // increase the counter for tacking bad lines
    context.getCounter("INVALID_LINES").increment(1);
}
Comments