Kainesplain Kainesplain - 2 months ago 7
Python Question

How to save a scraped list in a CSV file?

I wrote this code below, which scrapes from the OED.com website words by subject and date and prints them out in a list.

import requests
import re
import urllib2
import os
import csv

year_search = 1550
subject_search = ['Law']

path = '/Applications/Python 3.5/Economic'
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
header = {'User-Agent':user_agent}
request = urllib2.Request('http://www.oed.com/', None, header)
f = opener.open(request)
data = f.read()
f.close()
print 'database first access was successful'

resultPath = os.path.join(path, 'OED_table.csv')
htmlPath = os.path.join(path, 'OED.html')
outputw = open(resultPath, 'w')
outputh = open(htmlPath, 'w')
request = urllib2.Request(
'http://www.oed.com/search?browseType=sortAlpha&case-insensitive=true'
'&dateFilter='+str(year_search)+'&nearDistance=1&ordered=false&page=1'
'&pageSize=100&scope=ENTRY&sort=entry&subjectClass='
+ str(subject_search) + '&type=dictionarysearch', None, header)
page = opener.open(request)
urlpage = page.read()
outputh.write(urlpage)
new_word = re.findall(
r'<span class=\"hwSect\"><span class=\"hw\">(.*?)</span>', urlpage)
print str(new_word)
outputw.write(str(new_word))
page.close()
outputw.close()


Now I want to print them to a CSV file but in such a way that every year I input will be placed as a row, and the words will all fall in the line of the row.

Sort of like:

1550| word1| word2| etc.|
1551| word1| word2| etc.|


Does anyone have any ideas?

B B B B
Answer

I suggest using the csv.writer method. Here's the sample code:

`

with open('/Applications/Python 3.5/Economic/OED_table.csv', 'w') as csv_file:
    csv_writer = csv.writer(csv_file)
    year = ["1550"]
    new_word = ["apple", "banana"]
    complete_row = year + new_word
    csv_writer.writerow(complete_row)
    # writes 1550, apple, banana to OED_table.csv

`

You can modify it with a for loop to insert multiple rows.

Comments