dubbbdan dubbbdan - 3 years ago 160
Python Question

Decoded text file to pandas data frame

I am getting a tab separated data file using

requests
and I want to convert it to a pandas data frame. However, I can't seem to figure out how to convert the decoded data file into a pandas data frame object.

import requests
import pandas as pd
from datetime import date, timedelta


def build_url(site,yesterday):
url = 'https://waterdata.usgs.gov/az/nwis/dv?cb_00060=on&format=rdb&site_no=' + gc + '&referred_module=sw&period=&begin_date=1989-01-01&end_date=' + yesterday
return url

yesterday = date.today() - timedelta(1)

yesterday=yesterday.strftime('%Y-%m-%d')

url = build_url(site,yesterday)
t = requests.get(url)
decoded = t.content.decode('utf-8')
tmp_df = pd.read_csv(decoded,sep='\t',encoding = 'utf8')


My understanding is that
decoded
is a text file living in memory, but when I pass it to
pd.read_csv
with the specified delimiter it begins to print out the data frame and ends with:

USGS 09402500 2017-07-19 15200 P
USGS 09402500 2017-07-20 15200 P
USGS 09402500 2017-07-21 15100 P
USGS 09402500 2017-07-22 15000 P
USGS 09402500 2017-07-23 14100 P
USGS 09402500 2017-07-24 13700 P
does not exist


How can I get pandas to convert
decoded
into a dataframe?

Answer Source

read_csv wants a filename or a buffer. You can either save decoded to a file, or use a StringIO object:

import StringIO    
tmp_df = pd.read_csv(StringIO.StringIO(decoded), sep='\t')
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download