nachomasterCR nachomasterCR - 3 months ago 8
Python Question

iterate over list of dicts to create different strings

I have a pandas file with 3 different columns that I turn into a dictionary with to_dict, the result is a list of dictionaries:

df = [
{'HEADER1': 'col1-row1', 'HEADER2: 'col2-row1', 'HEADER3': 'col3-row1'},
{'HEADER1': 'col1-row2', 'HEADER2: 'col2-row2', 'HEADER3': 'col3-row2'}
]


Now my problem is that I need the value of 'col2-rowX' and 'col3-rowX' to build an URL and use requests and bs4 to scrape the websties.

I need my result to be something like the following:

requests.get("'http://www.website.com/' + row1-col2 + 'another-string' + row1-col3 + 'another-string'")


And i need to do that for every dictionary in the list.

I have tried iterating over the dictionaries using for-loops.
something like:

import pandas as pd
import os

os.chdir('C://Users/myuser/Desktop')

df = pd.DataFrame.from_csv('C://Users/myuser/Downloads/export.csv')

#Remove 'Code' column
df = df.drop('Code', axis=1)

#Remove 'Code2' as index
df = df.reset_index()

#Rename columns for easier manipulation
df.columns = ['CB', 'FC', 'PO']

#Convert to dictionary for easy URL iteration and creation
df = df.to_dict('records')

for row in df:
for key in row:
print(key)


1) i can't find a way to print the values i need
2) i'm not sure how to assign those values to a variable to be able to use each dictionary to create the URL

Answer

You only ever iterate twice, and short-circuit out of the nested for loop every time it is executed by having a return statement there. Looking up the necessary information from dictionary will allow you to build up your url's. One possible example:

def get_urls(l_d):
    l=[]
    for d in l_d:
        l.append('http://www.website.com/' +  d['HEADER2'] + 'another-string' + d['HEADER3'] + 'another-string')
    return l

df = [{'HEADER1': 'col1-row1', 'HEADER2': 'col2-row1', 'HEADER3': 'col3-row1'},{'HEADER1': 'col1-row2', 'HEADER2': 'col2-row2', 'HEADER3': 'col3-row2'}]
print get_urls(df)
>>> ['http://www.website.com/col2-row1another-stringcol3-row1another-string', 'http://www.website.com/col2-row2another-stringcol3-row2another-string']
Comments