Attributeerror: 'response' Object Has No Attribute 'body_as_unicode' Scrapy For Python

I am working with response in scrapy and keep on getting this message. I only gave the snippet where the error is occuring. I am trying to go through different webpages and need g

Solution 1:

In this case the line where your error occurs expects a TextResponse object not a normal response. Try to create a TextResponse instead of the normal Response to resolve the error.

The missing method is documented here.

More specifically use an HtmlResponse because your response would be some HTML and not plain text. HtmlResponse is a subclass of TextResponse so it inherits the missing method.

One more thing: where do you set the body of your Response? Without any body your xpath query will return nothing. As far as in the example in your question you only set the URL but no body. This is why your xpath returns nothing.

Solution 2:

This does not really answer to this question but can be used to find the problem with the response object returned. I am adding it as an answer so that it might help someone debug the problem they are facing.

I had encountered a similar error: AttributeError: 'HtmlResponse' object has no attribute 'text' when I did:

scrapy shell 'http://example.com'
>>>response.text

To find out what was the problem I checked out the attributes present in the response object returned using:

response.__dict__

However, __dict__ does not return attributes that are attached due to an object's parent class.

The response object that I received had the attribute _body which contained the html for that page.