horaceT horaceT - 2 months ago 16
R Question

How to make R foreach threads write to same log file

I have a number of lengthy jobs that I want to parallelize with foreach-dopar so that each thread works on a job independent of others. I want to track the status of each thread (some may fail while others succeed) by writing to a log file using

sink
. The following apparently doesn't work; the log file has only one entry.

library(foreach)
library(doParallel)
library(doSNOW)

cl = makeCluster(2, type="SOCK")
registerDoSNOW(cl)
dl = file("runlog.Rout", open="wt")
sink(dl, type="output", append=TRUE)
sink(dl, type="message", append=TRUE)
dump <- foreach(i=1:5,
.errorhandling = "stop",
.verbose=TRUE) %dopar%
{
beg.time = Sys.time()
cat(as.character(beg.time), " I am running....\n", file="mylog.txt")
# do something here.....
end.time = Sys.time()
del.tm = difftime(end.time, beg.time, units="mins")
cat("....saving output to file......\n\n", file="mylog.txt")
save(del.tm, file = paste("I:/Rhome/H", i, ".RData", sep=""))
return(i)
}
stopCluster(cl)
sink(type="output")
sink(type="message")


The log file has just one line :

....saving output to file......


What went wrong ?

Answer

Although I don't really trust having multiple processes write to the same file, you may have success by using the append=TRUE option:

cat("...\n", file="mylog.txt", append=TRUE)

Without setting this option, cat will overwrite the previous contents of "mylog.txt" each time it's called.

For other approaches, see my answer here.

Comments