Wiggy A. Wiggy A. - 5 months ago 67
HTML Question

Xpath, selecting text from B inside div while also taking normal text

Basically I have html similar to this:

<b>1</b> Communication
<b>2</b> Errors

What I'm trying (with Scrapy) is something like:


However this returns a list such as


I want to have something like:

"1 Communication",
"2 Errors"

Any help here would be greatly appreciated. I was trying to figure out a way to ignore the b tags, but I couldn't find anything that actually worked. The reason I can't just join the list indexes by two's is because not every html I need to parse works like this. I want to use something that could ignore the b tags if they exist, while just getting the text in p in any case. Thanks!

Answer Source

If your general pattern is to ignore <b> tags, you could use w3lib to remove those tags and construct new response from the result. Something like:

import w3lib
import scrapy

new_body = w3lib.html.remove_tags(response.body, which_ones=('b'))
new_response = scrapy.http.HtmlResponse(url=response.url, body=new_body)

new_response now contains the original response but with <b> tags removed. You can then use extraction logic without the need to consider them.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download