jstack jstack - 3 months ago 48
Python Question

How to save multiple data sets with the sio.savemat function?

I have created a code in python that reads multiple files (reads specific parts of them) and my goal is to save all these data points to matlab format. This is my 'main program' (all the functions are defined before):

file_location = glob.glob(os.path.expanduser('~/Dropbox/Argo/Data/*.nc'))
ifl=0
start_date = Start_Date(year=2016,month=1,day=1)
end_date = End_Date(year=2016,month=2,day=1)
for fl in file_location:
ifl=ifl+1
do_plot_this_profile, temp, lon, lat, juld, data=Read_Data(fl,start_date,end_date)
if do_plot_this_profile==1:
sio.savemat('Argo_Trajectories.mat', {'data':data})
z = sio.loadmat('Argo_Trajectories.mat')
print(z)
else:
pass
print(z)


I put to
print(z)
in the loop and out of it to see if the data is the same after the loop finishes running, and it's not: the final
z
is just the data from the last file. So my question, how do I save data from multiple files into one single file without having the files write over each other? Thanks, and please let me know if I can make this question clearer.

More Code (my
Read_Data()
function):

#imports file
fname = (file_location)
ncfile = netcdf_file(fname,'r');
#reads file and prepares it for usage
juld = np.array(ncfile.variables['JULD'][:])
#sorts data points to see whether they would be saved
if max(juld) < start_date or min(juld) > end_date :
do_plot_this_profile=0
print("not plotting, max jud=",max(juld))
else:
do_plot_this_profile=1
print("plotting, max jud=",max(juld))

#if data points meet the criteria, the rest of the file is read
if do_plot_this_profile:
temp = np.array(ncfile.variables['TEMP'][:])
temp[temp>45]=float('nan') #eliminates inaccurate data
temp[temp<-10]=float('nan') #eliminates inaccurate data
lon = np.array(ncfile.variables['LONGITUDE'][:])
lat = np.array(ncfile.variables['LATITUDE'][:])
data = lon, lat, juld
ncfile.close()

#if data points do not meet the criteria, the file is ignored
else:
temp='nan'
lon='nan'
lat='nan'
data='nan'

return do_plot_this_profile, temp, lon, lat, juld, data

Answer

This question is about how to append data using savemat. It looks like that asker says you can only append data to the existing dictionary key though, so it may not solve your problem.

What I recommend is adding the data to a dictionary and then saving it all to a mat file once, at the end.

Initialize a dictionary outside the loop

myDictionary = {}

You have one line of code that creates the dictionary and saves it.

sio.savemat('Argo_Trajectories.mat', {'data':data})

Replace that line with two: create a unique key rather than naming them all data and add the data to the dictionary

newkey = 'data%d' % ifl
myDictionary[newkey] = data

After the loop is finished, save the dictionary (this can be before or after the 2nd print(z) in your function

sio.savemat('savename.mat',myDictionary)
Comments