T.Y T.Y - 2 months ago 14
Python Question

Pandas Python- read_csv not reading complete data on each row

I'm using read_csv to read a fairly big csv in chunks (just reading the first line to test).

data = read_csv('VOD_Properties.csv', nrows=1, low_memory=False)
print(data)


Result:

PROPERTIES
0 {"video_id":241338,"play_uuid":"0d293b16-566a-...


Original data in excel:

{"video_id":241338,"play_uuid":"0d293b16-566a-46e7-ac90-e3caa602a527","seconds":0.116,"current_state":"PLAY","total_seconds":100.032}


To test if the data's just simply not showing up, I converted into a string:

string_data = data.to_string()


and print the last few characters of the string to see if they're '...':

6-566a-...


I've tried playing around with the parameters but not a single improvement.

Thanks in advance.

Answer

The full line was in fact read correctly, just when printing pandas truncates wide columns with ellipsis '...'.

You can change pandas' column width display threshold as per: How to remove ellipsis from a row in a Python Pandas series or data frame, shown when long lines/wide columns are truncated?

pandas.set_option('display.max_colwidth', 1000) # or whatever width
Comments