TravisVOX TravisVOX - 6 months ago 34
Python Question

UnicodeDecodeError when reading CSV file in Pandas with Python

I'm running a program which is processing 30,000 similar files. A random number of them are stopping and producing this error...

File "C:\Importer\src\dfman\", line 26, in import_chr
data = pd.read_csv(filepath, names=fields)
File "C:\Python33\lib\site-packages\pandas\io\", line 400, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Python33\lib\site-packages\pandas\io\", line 205, in _read
File "C:\Python33\lib\site-packages\pandas\io\", line 608, in read
ret =
File "C:\Python33\lib\site-packages\pandas\io\", line 1028, in read
data =
File "parser.pyx", line 706, in (pandas\parser.c:6745)
File "parser.pyx", line 728, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:6964)
File "parser.pyx", line 804, in pandas.parser.TextReader._read_rows (pandas\parser.c:7780)
File "parser.pyx", line 890, in pandas.parser.TextReader._convert_column_data (pandas\parser.c:8793)
File "parser.pyx", line 950, in pandas.parser.TextReader._convert_tokens (pandas\parser.c:9484)
File "parser.pyx", line 1026, in pandas.parser.TextReader._convert_with_dtype (pandas\parser.c:10642)
File "parser.pyx", line 1046, in pandas.parser.TextReader._string_convert (pandas\parser.c:10853)
File "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandas\parser.c:15657)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 6: invalid continuation byte

The source/creation of these files all come from the same place. What's the best way to correct this to proceed with the import?


read_csv takes an encoding option to deal with files in different formats. I mostly use ``read_csv('file', encoding = "ISO-8859-1"), or alternativelyencoding = utf8for reading, and generallyutf-8forto_csv`.

You can also use the alias 'latin1' instead of 'ISO-8859-1'.

See relevant Pandas documentation, python docs examples on csv files, and plenty of related questions here on SO.