There is an anchor tag, sometimes followed by one or two span tags. I have to select anchor's href based upon an equality comparison with the text found in
<table class="table table-striped" width="95%">
<td ><span class="badge">P</span>
<a href="/abc" title="Title of anchor">some text</a>
I would recommend checking them individually, as the xpath could be very complicated and could even make your program slower.
Another tip would be to just create a selector with just the part you know contains the necessary information (if the whole document is big enought, this would help a lot):
from scrapy import Selector ... sel = Selector(text=response.css('table.table').extract_first()) anchor_selector = sel.css('a') anchor_text = anchor_selector.css('::text').extract_first() span_siblings = anchor_selector.xpath('./following-sibling::span/text()').extract() # now play with anchor_text and the list of span_siblings