Pyxel Pyxel - 23 days ago 14
R Question

R - Reading unknown number of files

I'm new to R. I try to process some experimental data and I'm stuck with a bug when reading files.
I want to read some data files in a folder but I don't know how many are there. I just know that all files are named "Subject1ManualX.log", X being 1 or higher. Since I didn't find any possibility to count how many files are in a folder, I try to open files in a while loop until an exception is raised (i.e. I try to open "Subject1Manual1.log", then "Subject1Manual2.log" etc).

Here is my code (prints are for debug):

# Script to work on ET data
temp <- 0

while (temp == 0){
tryCatch({
subjectData <- read.delim("D:/Doctorat/XPs/XP1-2_LIPSMWA/SmartEye/Subject1/Subject1Manual3.log")

}, warning = function(w){
print(paste("warning", i))
temp <- 1

}, error = function(e){
print(paste("error", i))
temp <- 1

}, finally = {
print("finished")
})
}


Unfortunately it doesn't work (that's why I'm here...). I know I will have warnings and errors. If I handle warnings, R crashes because errors aren't handled ("all connection are in use" crash). If I only handle errors, it doesn't go through it and the while loop continues at each iteration.

Any idea on this matter would be highly appreciated (including better ways to read an unknown number of files). Thank you for your time!

Pyxel

EDIT: ok some nice persons answered how to import multiple data files, but for curiosity I would like to know how to deal with a try catch within a while loop. Any idea?

Answer
# Here we read path to all '*.log' files in folder
path_to_files = dir("D:/Doctorat/XPs/XP1-2_LIPSMWA/SmartEye/Subject1/", 
                    full.names = TRUE, 
                    recursive = TRUE, 
                    pattern = "\\.log$")

# further we read all files to list
files = lapply(path_to_files, read.delim)

# and finaly we combine files to single data.frame
# headers of files should be identical
res = do.call(rbind, files)

UPDATE Code with tryCatch and while. It is not safe and it grows data.frame in the loop - it is bad practice.

subjectData = TRUE
i = 1
res = NULL
while (!is.null(subjectData)){
    subjectData <-  tryCatch({
        read.delim(sprintf("D:/Doctorat/XPs/XP1-2_LIPSMWA/SmartEye/Subject1/Subject1Manual%s.log", i))

    }, warning = function(w){
        print(paste("warning", i))
        NULL

    }, error = function(e){
        print(paste("error", i))
        NULL

    })
    if(is.null(res)){
        res = subjectData
    } else {
        res = rbind(res, subjectData)
    }
    i = i + 1
}
Comments