cJc cJc - 20 days ago 7
Python Question

Appending incomming api data to pandas dataframe, what is wrong with my function?

I'm receiving a feed of api data 'nftp' and I have a function handling the incoming data. I receive the data ok, I can print the data (as per function below).

How can I store the data in a pandas dataframe?

The data 'nftp' is streamed i.e. every time a new line or sometimes several lines at once comes in, the 'nftpGeneralHandler' shoud append the data to the dataframe and not start over with an empty dataframe, only an empty list (I guess).

Incoming data format:
[386, 3, 375, 'OSE', '?', 94520, 'STL', 94520, 800, 36.2, 'A', '?', '?', 0.0, 'S']
Print command in my function returns: ('STL', 94520, 36.2, 1145, 'A', 'S')

What am I doing wrong when attempting to store the data in a list, then dataframe?

graph_data = pd.DataFrame()
def nftpGeneralHandler(nftp, api):
print "Received ", nftp
data_list = []
if nftp[1] == 3 and nftp[10] == 'A':
print (nftp[6], nftp[7], nftp[9], nftp[8], nftp[10], nftp[14])
data_list.append([nftp[6], nftp[7], nftp[9], nftp[8], nftp[10], nftp[14]])
graph_data.append(data_list)

elif nftp[1] == 99:
nftpError(nftp, api)

Answer

I think you can return desired data in each loop of function and append them to graph_data. Last use DataFrame constructor:

def nftpGeneralHandler(nftp, api): 
   #print "Received ", nftp
   if nftp[1] == 3 and nftp[10] == 'A':
       print (nftp[6], nftp[7], nftp[9], nftp[8], nftp[10], nftp[14])
       out = [nftp[6], nftp[7], nftp[9], nftp[8], nftp[10], nftp[14]]
   elif nftp[1] == 99: 
      nftpError(nftp, api)
   return out

L1 = [386, 3, 375, 'OSE', '?', 94520, 'STL', 94520, 800, 36.2, 'A', '?', '?', 0.0, 'S'] 
L2 = [386, 3, 375, 'OSE', '?', 94520, 'STL', 94520, 800, 36.2, 'A', '?', '?', 0.0, 'S'] 
L3 = [386, 3, 375, 'OSE', '?', 94520, 'STL', 94520, 800, 36.2, 'A', '?', '?', 0.0, 'S'] 
api = ''    
graph_data = []

for L in [L1, L2, L3] :
    graph_data.append(nftpGeneralHandler(L,api))
print (graph_data)
[['STL', 94520, 36.2, 800, 'A', 'S'], 
 ['STL', 94520, 36.2, 800, 'A', 'S'], 
 ['STL', 94520, 36.2, 800, 'A', 'S']]

print (pd.DataFrame(graph_data))
     0      1     2    3  4  5
0  STL  94520  36.2  800  A  S
1  STL  94520  36.2  800  A  S
2  STL  94520  36.2  800  A  S