Yancheng Zeng Yancheng Zeng - 3 months ago 28
Python Question

Can scrapy yield diffenent kinds of items?

I have two kinds of Item:

class MovieItem(scrapy.Item):
id = scrapy.Field()
image_urls=scrapy.Field()
image_paths =scrapy.Field()
torrents = scrapy.Field()
#...other fields


class TorrentItem(scrapy.Item):
id = scrapy.Field()
movie_id = scrapy.Field()
image_urls=scrapy.Field()
image_paths =scrapy.Field()


I want to use
ImagePipeline
and
FilePipeline
to download images and torrents in a movie. How should I yeild the two items in the
*parse*
method? And how should I define the corresponding pipeline?

Answer

The answer is yes, you can. Here's an example on how to do it. Here's an example.py spider:

# -*- coding: utf-8 -*-
import scrapy


class MovieItem(scrapy.Item):
    id = scrapy.Field()
    image_urls=scrapy.Field()
    images =scrapy.Field()
    torrents = scrapy.Field()
    itemtype = scrapy.Field()


class TorrentItem(scrapy.Item):
    id = scrapy.Field()
    movie_id = scrapy.Field()
    image_urls=scrapy.Field()
    images =scrapy.Field()


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = (
        'http://www.example.com/',
    )

    def parse(self, response):
        image_urls = [
            "http://...-Miles.jpg",
            "https:/.../58832_300x300",
            "http://...-Circuit-Tests.png"
        ]

        torent_ids = []
        for i in xrange(3):
            t = TorrentItem()
            t["id"] = "#id%d" % i
            t["movie_id"] = 143
            t["image_urls"] = [image_urls[i]]
            # ...
            torent_ids.append(t["id"])
            yield t

        m = MovieItem()
        m['id'] = 143
        m['image_urls'] = ['http://...test.png']
        m['torrents'] = torent_ids
        m['itemtype'] = ['movie']
        # ...
        yield m

On your settings.py add the following two lines:

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '.'

Run the spider:

scrapy crawl example -o test.jl

Your test.jl file will contain (after quite some formatting):

{
    "images": [
        {
            "url": "http://.../Stuart-Miles.jpg",
            "path": "full/27c8d5099f8785e8fbc2370249a0260e216ee2dd.jpg",
            "checksum": "dba2fc121610b328448dc37084f31dac"
        }
    ],
    "movie_id": 143,
    "id": "#id0",
    "image_urls": [
        "http://...ter-Key-by-Stuart-Miles.jpg"
    ]
}
{
    "images": [
        {
            "url": "https://i....t/58832_300x300",
            "path": "full/b11276eb5b64b5ec7f40eedf4c6fcc6d6d9072ac.jpg",
            "checksum": "a9b47ecbb2de9dcb6a61a159120f1bd2"
        }
    ],
    "movie_id": 143,
    "id": "#id1",
    "image_urls": [
        "https://i.vi..._300x300"
    ]
}
{
    "images": [
        {
            "url": "http://www.ej...rt-Circuit-Tests.png",
            "path": "full/a68282eb533d35a0aa8732a872277933db8951c5.jpg",
            "checksum": "24c0907e3ef610dc355e930f2535c0c4"
        }
    ],
    "movie_id": 143,
    "id": "#id2",
    "image_urls": [
        "http://www.ejob...nsformer-Open-and-Short-Circuit-Tests.png"
    ]
}
{
    "images": [
        {
            "url": "http://...est.png",
            "path": "full/1e3e0f775cd40aaa5ea081278957f4d49e39f610.jpg",
            "checksum": "50a57a6263b9640ee47e913deadaff7c"
        }
    ]
    "torrents": [
        "#id0",
        "#id1",
        "#id2"
    ],
    "itemtype": [
        "movie"
    ],
    "image_urls": [
        "http://xi.../10/test.png"
    ],
    "id": 143
}

This works nicely with .jl files as output. It won't work well with .csv but this shouldn't be a problem in your case.

Comments