epo3 epo3 - 2 months ago 9
R Question

Error: Invalid: File is too small to be a well-formed file - error when using feather in R

I'm trying to use feather (v. 0.0.1) in R to read a fairly large (3.5 GB) csv file with 21178665 rows and 16 columns.

I use the following lines to load the file:

library(feather)
path <- "pp-complete.csv"
df <- read_feather(path)


But I get the following error:


Error: Invalid: File is too small to be a well-formed file


There's no explanation in the documentation of
read_feather
so I'm not sure what's the problem. I guess this function expects a different file form but I'm not sure what that would be.

Btw, I can read the file with
read_csv
in
readr
library but it takes a while.

Answer

The feather file format is distinct from a CSV file format. They are not interchangeable. The read_feather function cannot read simple CSV files.

If you want to read CSV files quickly, your best bets are probably readr::read_csv or data.table::fread. For large files, it will still usually take a while just to read it from disc.

After you've loaded the data into R, you can create a file in the feather format with write_feather so you can read it with read_feather the next time.