horaceT horaceT - 11 months ago 66
R Question

How to make R foreach threads write to same log file

I have a number of lengthy jobs that I want to parallelize with foreach-dopar so that each thread works on a job independent of others. I want to track the status of each thread (some may fail while others succeed) by writing to a log file using

. The following apparently doesn't work; the log file has only one entry.


cl = makeCluster(2, type="SOCK")
dl = file("runlog.Rout", open="wt")
sink(dl, type="output", append=TRUE)
sink(dl, type="message", append=TRUE)
dump <- foreach(i=1:5,
.errorhandling = "stop",
.verbose=TRUE) %dopar%
beg.time = Sys.time()
cat(as.character(beg.time), " I am running....\n", file="mylog.txt")
# do something here.....
end.time = Sys.time()
del.tm = difftime(end.time, beg.time, units="mins")
cat("....saving output to file......\n\n", file="mylog.txt")
save(del.tm, file = paste("I:/Rhome/H", i, ".RData", sep=""))

The log file has just one line :

....saving output to file......

What went wrong ?

Answer Source

Although I don't really trust having multiple processes write to the same file, you may have success by using the append=TRUE option:

cat("...\n", file="mylog.txt", append=TRUE)

Without setting this option, cat will overwrite the previous contents of "mylog.txt" each time it's called.

For other approaches, see my answer here.