Ahmed Talha Ahmed Talha - 6 months ago 9
HTML Question

How to avoid HTML spaces in td tags during web scraping through java

I am scraping a website which stores the data in the csv. The code is working fine but the problem is all those place in HTML where &nbsp is using, code is putting 'Â'. To avoid this I have tried something (mentioned below in the code) but because of that code rewrites that data again. Now, what I actually want that 'Â' this should be removed without any disturbance in the data.

for (Iterator<Element> it = tdElements.iterator(); it.hasNext();) {
if (it.hasNext()) {
sb.append(" , ");
sb.append(" \r\n ");
}

for (Iterator<Element> it2 = tdElements.iterator(); it.hasNext();) {
Element tdElement2 = it.next();
final String content;
content = tdElement2.text().replace(",", "");
//I added the following code to avoid that 'Â'
final String content2;
content2 = tdElement2.text().replace("\u00a0","");
if (it2.hasNext()) {

sb.append(formatData(content));
sb.append(formatData(content2));
sb.append(" , ");
}
}

System.out.println(sb.toString());
sb.flush();
sb.close();
}


What I understood that why it rewrites the data because of the
sb.append(formatData(content));
I am appending the whole data for two times that's why it is creating problem. So how can I manage both ',' and 'Â' at the same time.

Answer

You should do the replacements on the modified string, then you only have to add content2.

String content = tdElement2.text().replace(",", "");
//I added the following code to avoid that 'Â'
String content2 = content.replace("\u00a0","");
if (it2.hasNext()) {
 sb.append(formatData(content2));
 sb.append("   ,   ");
}

You can also rewrite it as

String content2 = tdElement2.text().replace(",", "").replace("\u00a0","");
if (it2.hasNext()) {
 sb.append(formatData(content2)).append("   ,   ");
}