Chris Blake Chris Blake - 6 months ago 62
Linux Question

Is the UNIX `time` command accurate enough for benchmarks?

Let's say I wanted to benchmark two programs: foo.py and bar.py.

Are a couple thousand runs and the respective averages of

time python foo.py
and
time python bar.py
adequate enough for profiling and comparing their speed?




Edit: Additionally, if the execution of each program was sub-second (assume it wasn't for the above), would
time
still be okay to use?

Answer

time produces good enough times for benchmarks that run over one second otherwise the time it took exec()ing a process may be large compared to its run-time.

However, when benchmarking you should watch out for context switching. That is, another process may be using CPU thus contending for CPU with your benchmark and increasing its run time. To avoid contention with other processes you should run a benchmark like this:

sudo chrt -f 99 /usr/bin/time -f "\n***\ntime: %E\ncontext switches: %c\nwaits: %w" <benchmark>

sudo chrt -f 99 runs your benchmark in FIFO real-time class with priority 99, which makes your process the top priority process and avoids context switching (you can change your /etc/security/limits.conf so that it doesn't require a privileged process to use real-time priorities). It also makes time report the number of context switches your benchmark incurred which should normally be 0, otherwise you may like to rerun the benchmark.

And it is better to disable the CPU frequency scaling and boost, so that the CPU frequency stays constant during the benchmark to get consistent results.