epo3 epo3 - 10 months ago 66
R Question

Error: Invalid: File is too small to be a well-formed file - error when using feather in R

I'm trying to use feather (v. 0.0.1) in R to read a fairly large (3.5 GB) csv file with 21178665 rows and 16 columns.

I use the following lines to load the file:

path <- "pp-complete.csv"
df <- read_feather(path)

But I get the following error:

Error: Invalid: File is too small to be a well-formed file

There's no explanation in the documentation of
so I'm not sure what's the problem. I guess this function expects a different file form but I'm not sure what that would be.

Btw, I can read the file with
library but it takes a while.

Answer Source

The feather file format is distinct from a CSV file format. They are not interchangeable. The read_feather function cannot read simple CSV files.

If you want to read CSV files quickly, your best bets are probably readr::read_csv or data.table::fread. For large files, it will still usually take a while just to read it from disc.

After you've loaded the data into R, you can create a file in the feather format with write_feather so you can read it with read_feather the next time.