nachomasterCR nachomasterCR - 1 year ago 169
Python Question

iterate over list of dicts to create different strings

I have a pandas file with 3 different columns that I turn into a dictionary with to_dict, the result is a list of dictionaries:

df = [
{'HEADER1': 'col1-row1', 'HEADER2: 'col2-row1', 'HEADER3': 'col3-row1'},
{'HEADER1': 'col1-row2', 'HEADER2: 'col2-row2', 'HEADER3': 'col3-row2'}

Now my problem is that I need the value of 'col2-rowX' and 'col3-rowX' to build an URL and use requests and bs4 to scrape the websties.

I need my result to be something like the following:

requests.get("'' + row1-col2 + 'another-string' + row1-col3 + 'another-string'")

And i need to do that for every dictionary in the list.

I have tried iterating over the dictionaries using for-loops.
something like:

import pandas as pd
import os


df = pd.DataFrame.from_csv('C://Users/myuser/Downloads/export.csv')

#Remove 'Code' column
df = df.drop('Code', axis=1)

#Remove 'Code2' as index
df = df.reset_index()

#Rename columns for easier manipulation
df.columns = ['CB', 'FC', 'PO']

#Convert to dictionary for easy URL iteration and creation
df = df.to_dict('records')

for row in df:
for key in row:

1) i can't find a way to print the values i need
2) i'm not sure how to assign those values to a variable to be able to use each dictionary to create the URL

Answer Source

You only ever iterate twice, and short-circuit out of the nested for loop every time it is executed by having a return statement there. Looking up the necessary information from dictionary will allow you to build up your url's. One possible example:

def get_urls(l_d):
    for d in l_d:
        l.append('' +  d['HEADER2'] + 'another-string' + d['HEADER3'] + 'another-string')
    return l

df = [{'HEADER1': 'col1-row1', 'HEADER2': 'col2-row1', 'HEADER3': 'col3-row1'},{'HEADER1': 'col1-row2', 'HEADER2': 'col2-row2', 'HEADER3': 'col3-row2'}]
print get_urls(df)
>>> ['', '']