wnj wnj - 9 months ago 31
Python Question

How can I put the the list type of item into database?

I use Scrapy to write a spider to get something from a website.And I want to put the item into database.In my code,there are five items ,two of items are unicode type,so I can put it into database directily,but two of items are list type,How can I put it into database?Here is my code about the items whose type are list:

descr = sel.xpath(
'//*[@id="root"]/div/main/div/div[2]/div[1]/div[2]/div/div/div/div[2]/div/div/span/p[1]/text()').extract()
print 'type is:', type(descr)
answer_time = sel.xpath(
'//*[@id="root"]/div/main/div/div[2]/div[1]/div[2]/div/div/div/div[2]/div/div/div/a/span/@data-tooltip').extract()
print 'type is:', type(answer_time)

Answer Source

It should depend strongly on how the data will be used once in the db and what kind of db you're using.

If you're using something like MongoDB, it fully supports just adding the list as part of the record. Some relational dbs such as postgresql have support for JSON column types.

Beyond that there are two main options. 1) Cast the list to a string using something like JSON and save it in a text column along with your other unicode columns. 2) Take full advantage of a relational db and use a second table to store a one:many relationship.

Casting it a string is much easier/faster to dev. Great for rapid prototyping. However, using multiple tables has huge efficiency perks if the data is going to be read for analysis on any data set that is not small.