I am working with response in scrapy and keep on getting this message.
I only gave the snippet where the error is occuring. I am trying to go through different webpages and need get the # of pages in that particular webpage. So I created A response object where I get the href for the next button but keep on getting
AttributeError: 'Response' object has no attribute 'body_as_unicode'
from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.http import Request
from scrapingtest.items import ScrapingTestingItem
from collections import OrderedDict
from scrapy.selector.lxmlsel import HtmlXPathSelector
from scrapy.http import Response
name = "scrapytesting"
allowed_domains = ["tripadvisor.in"]
# base_uri = ["tripadvisor.in"]
for i in range(len(site_array)):
response = Response(url=site_array[i])
sites = Selector(response).xpath('//a[contains(text(), "Next")]/@href').extract()
# sites = response.selector.xpath('//a[contains(text(), "Next")]/@href').extract()
for site in sites:
In this case the line where your error occurs expects a
TextResponse object not a normal response. Try to create a
TextResponse instead of the normal
Response to resolve the error.
The missing method is documented here.
More specifically use an
HtmlResponse because your response would be some HTML and not plain text.
HtmlResponse is a subclass of
TextResponse so it inherits the missing method.
One more thing: where do you set the body of your
Response? Without any body your
xpath query will return nothing. As far as in the example in your question you only set the URL but no body. This is why your
xpath returns nothing.