user5727 user5727 - 6 months ago 75
Python Question

Python erro- "too many values to unpack using numpy.loadtxt" large data file

I am using numpy.loadtxt to read in dates from a csv file. I am getting the error that there are too many values to unpack. This is likely true in the literal sense because there are over 46,000 rows of dates. I would like to know if there is a way to read in the data without further breaking up the file. It has been broken up from one large zip file into site locations which I would like to keep together. The code is as follows:

import csv
import numpy as np
import matplotlib.pylab as plt
import matplotlib.dates as mdates

def getColumn(filename, column):
results = csv.reader(open(filename), dialect='excel')
return [result[column] for result in results]

time = getColumn("C:\Users\dfmcg\Documents\Thesis files\ALL_GPMP_O3_Met\ASIS-MA.csv",1)
for i in range (0,len(time)):
x=time[i]
x=x[0:(len(x)-3)]
time[i]=x
print time[0:10]

time = time[1:]
from datetime import date

NewData = np.loadtxt(time, delimiter=',',dtype=str, unpack=True,
converters = {0: mdates.strpdate2num('%m/%d/%Y')})

print (NewData)

print NewData.shape()

date, ozone = NewData


Error is as follows:

c:\users\dfmcg\appdata\local\temp\tmpkbgiq_.py in <module>()
19 #time = [date.datetime.strptime(item,"%Y/%m/%d"), float(time) for item in data
20 date, ozone = np.loadtxt(time, delimiter=',',dtype=str, unpack=True,


---> 21 converters = {0: mdates.strpdate2num('%m/%d/%Y')})

ValueError: too many values to unpack


Here is one row of data:

ABBR DATE O3 SWS VWS SWD VWD SDWD TMP RH RNF SOL
ASIS-MA 06/01/2005 00 -999 -999 -999 -999 -999

Answer

Instead of this:

date, ozone = np.loadtxt(time, delimiter=',',dtype=str, unpack=True,
    converters = {0: mdates.strpdate2num('%m/%d/%Y')})

Try using this:

NewData = np.loadtxt(time, delimiter=',',dtype=str, unpack=True,
    converters = {0: mdates.strpdate2num('%m/%d/%Y')})

or try since the error is in the converter

 NewData = np.loadtxt(time, delimiter=',',dtype=str )

#to view array 
print NewData.shape

#Then:
date, ozone = NewData

When you know the shape then, build your tuple.

#Also:
for i in range (0,len(time)):
#try 
for i,e in enumerate(time):
Comments