HTML Question
Xpath, selecting text from B inside div while also taking normal text
Basically I have html similar to this:
<div>
<p>
<b>1</b> Communication
</p>
<p>
<b>2</b> Errors
</p>
...
</div>
What I'm trying (with Scrapy) is something like:
response.xpath("//div//p//text()")
However this returns a list such as
[
"1",
"Communication",
"2",
"Errors"
]
I want to have something like:
[
"1 Communication",
"2 Errors"
]
Any help here would be greatly appreciated. I was trying to figure out a way to ignore the b tags, but I couldn't find anything that actually worked. The reason I can't just join the list indexes by two's is because not every html I need to parse works like this. I want to use something that could ignore the b tags if they exist, while just getting the text in p in any case. Thanks!
Recommended for you: Get network issues from WhatsUp Gold. Not end users.
Answer Source
If your general pattern is to ignore <b> tags, you could use w3lib to remove those tags and construct new response from the result. Something like:
import w3lib
import scrapy
new_body = w3lib.html.remove_tags(response.body, which_ones=('b'))
new_response = scrapy.http.HtmlResponse(url=response.url, body=new_body)
new_response now contains the original response but with <b> tags removed. You can then use extraction logic without the need to consider them.
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download