Cristian Calin Cristian Calin - 4 months ago 46
JSON Question

Extract from dynamic JSON response with Scrapy

I want to extract the 'avail' value from the JSON output that look like this.

{
"result": {
"code": 100,
"message": "Command Successful"
},
"domains": {
"yolotaxpayers.com": {
"avail": false,
"tld": "com",
"price": "49.95",
"premium": false,
"backorder": true
}
}
}


The problem is that the
['avail']
value is under
["domains"]["domain_name"]
and I can't figure out how to get the domain name.

You have my spider below. The first part works fine, but not the second one.

import scrapy
import json
from whois.items import WhoisItem

class whoislistSpider(scrapy.Spider):
name = "whois_list"
start_urls = []
f = open('test.txt', 'r')
global lines
lines = f.read().splitlines()
f.close()
def __init__(self):
for line in lines:
self.start_urls.append('http://www.example.com/api/domain/check/%s/com' % line)

def parse(self, response):
for line in lines:
jsonresponse = json.loads(response.body_as_unicode())
item = WhoisItem()
domain_name = list(jsonresponse['domains'].keys())[0]
item["avail"] = jsonresponse["domains"][domain_name]["avail"]
item["domain"] = domain_name
yield item


Thank you in advance for your replies.

Answer

Assuming you are only expecting one result per response:

domain_name = list(jsonresponse['domains'].keys())[0]
item["avail"] = jsonresponse["domains"][domain_name]["avail"]

This will work even if there is a mismatch between the domain in the file "test.txt" and the domain in the result.

Comments