Auginis Auginis - 1 year ago 227
Python Question

Python bs4 module

import requests
from bs4 import BeautifulSoup

It's a web crawler working in ebay, collecting every single item data

def ebay_spider(max_pages):
page = 1
while page <= max_pages:
url = '' \
+ str(page)
source_code = requests.get(url)

plain_text = source_code.text
soup = BeautifulSoup(plain_text)
for link in soup.findAll('a', {'class': 'vip'}):
href = '' + link.get('href')
title = link.string
page += 1

def get_single_item_data(item_url):
source_code = requests.get(item_url)
plain_text = source_code.text
soup = BeautifulSoup(plain_text)
for item_name in soup.findAll('h1', {'id': "itemTitle"}):


Blockquote And the error say that :

I tried to fix it but it seems not to work, so any tips/answers how to fix it?

Answer Source

When you're trying to make a BeatifulSoup object in line, do instead this:

soup = BeautifulSoup(plain_text)


soup = BeautifulSoup(plain_text, 'html.parser', markup_type=markup_ty‌​pe)`

Note: your problem refers to bs4 module, not requests.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download