user124697 user124697 - 1 year ago 223
Python Question

Scrapy: AttributeError: 'list' object has no attribute 'iteritems'

This is my first question on stack overflow. Recently I want to use linked-in-scraper, so I downloaded and instruct "scrapy crawl" and get the below error message. For your information, I use anaconda 2.3.0 and python 2.7.11. All the related packages, including scrapy and six, are updated by pip before executing the program.

Traceback (most recent call last):
File "/Users/byeongsuyu/anaconda/bin/scrapy", line 11, in <module>
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/", line 108, in execute
settings = get_project_settings()
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/utils/", line 60, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/settings/", line 285, in setmodule
self.set(key, getattr(module, key), priority)
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/settings/", line 260, in set
self.attributes[name].set(value, priority)
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/settings/", line 55, in set
value = BaseSettings(value, priority=priority)
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/settings/", line 91, in __init__
self.update(values, priority)
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/scrapy/settings/", line 317, in update
for name, value in six.iteritems(values):
File "/Users/byeongsuyu/anaconda/lib/python2.7/site-packages/", line 599, in iteritems
return d.iteritems(**kw)

AttributeError: 'list' object has no attribute 'iteritems'

I understand that this error stems from d is not the dictionary type but list type. And since the error is from the code on scrapy, maybe it is problem on scrapy package or six package. How can I try to fix this error?

EDIT: This is code from scrapy.cfg

# Automatically created by: scrapy start project
# For more information about the [deploy] section see:
default = linkedIn.settings

#url = http://localhost:6800/
project = linkedIn

Answer Source

This is caused by the linked-in scraper's settings:

ITEM_PIPELINES = ['linkedIn.pipelines.LinkedinPipeline']

However, ITEM_PIPELINES is supposed to be a link, according to the doc:

To activate an Item Pipeline component you must add its class to the ITEM_PIPELINES setting, like in the following example:

    'myproject.pipelines.PricePipeline': 300,
    'myproject.pipelines.JsonWriterPipeline': 800,

The integer values you assign to classes in this setting determine the order in which they run: items go through from lower valued to higher valued classes. It’s customary to define these numbers in the 0-1000 range.

According to this question, it used to be a list, which explains why this scraper uses a list. So you will have to either ask your the developer of the scraper to update their code, or to set ITEM_PIPELINES yourself.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download