M.Huntz M.Huntz - 10 months ago 51
Python Question

Save body text on csv file | Python 3

I am trying to create a database with several articles for Text mining purposes.
I am extracting the body via web scraping and then save the body of these articles on a csv file. However, I couldn't manage to save all the body texts.
The code that I came up with saves only the text the last URL (article) while if I print what I am scraping (and what I am supposed to save) I obtain the body of all the articles.

I just included some of the URL from the list (which contains a larger number of URLs) just to give you an idea:

import requests
from bs4 import BeautifulSoup
import csv

"http://www.nytimes.com/2013/06/16/magazine/the-effort-to-stop-the- attack.html",

for url in r:
t= requests.get(url)
t.encoding = "ISO-8859-1"
soup = BeautifulSoup(t.content, 'lxml')
text = soup.find_all(("p",{"class": "story-body-text story-content"}))
with open('newdb30.csv', 'w', newline='') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=' ',quotechar='|', quoting=csv.QUOTE_MINIMAL)

Answer Source

Try declaring variable such as all_text = "" before the for loop and adding text to all_text by all_text += text + "\n" at the end of the for loop (the \n creates a new line).

Then, in the last row, instead of writing text, you write all_text.