Malik Malik - 7 months ago 81
Python Question

python2.7.8: TypeError: expected string or buffer with bs4 and re

I am unable to figure out why I am getting this error. I am following this tutorial to extract actual text. But I don’t understand the error.

Could someone have a look at my code?

import urllib
from bs4 import BeautifulSoup
import re


url = "https://en.wikipedia.org/wiki/Python_(programming_language)" # link of website
html = urllib.urlopen(url).read() # reading and opening link
soup = BeautifulSoup(html) #parsing


for script in soup(["script", "style","a","<div id=\"bottom\" >"]): # all tags
script.extract() # clear out


for p in soup.find_all('p'): # loop for printing text
r = re.sub("<.*?>", "", p) # expression to get rid from <p> <b> etc
print r


Error:

Traceback (most recent call last):
File "C:/Users/DELL/Desktop/python/s/fyp/textextractioon.py", line 16, in <module>
r = re.sub("<.*?>", "", p)
File "C:\Python27\lib\re.py", line 151, in sub
return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

Answer

change your final loop to :

for p in soup.find_all('p'): # loop for printing text
    r = re.sub("<.*?>", "", p.text) # expression to get rid from <p> <b> etc
    print r

every p is from type class 'bs4.element.Tag': and it has some built in methods, take a look so it would be clear for you