I'm working on training some neural networks. These have lots of hyperparameters, and typically you see how each set of hyperparameters performs, then pick your favorite. This is often done by (say) training a network with the given parameters for
epochs, then evaluate its performance, yielding a numerical score of each set of parameters and allowing you to pick the best.
There's a problem with this, though. Some sets of parameters let you go through more epochs more quickly, but benefit less from each epoch. Additionally, pretty much any set of parameters will always do better, given more epochs, so given infinite time, they would all do really well (to a point, but that's not the point right now).
What I would prefer to do is to let each process figure out how long it's been running, and cut itself off (gracefully) after a specified number of seconds. The problem is, I would like to multithread this, so just because the program has been running for 60 seconds doesn't mean the process has had 60 seconds of fair CPU time.
So how can I measure how much time the process has actually had available to it
, within the process itself?
method gives system time, which is problematic (as above).
module seems a bit better, but it's external to the script, so the process wouldn't know when to stop.
Is there a better way? Am I wrong about one of the above ways?
How can a python process see how many seconds it has been allocated so far? Not the amount of time that has passed, but how many seconds it itself has been allowed to execute for?