Let's say I wanted to benchmark two programs: foo.py and bar.py.
Are a couple thousand runs and the respective averages of
time python foo.py
time python bar.py
time produces good enough times for benchmarks that run over one second otherwise the time it took
exec()ing a process may be large compared to its run-time.
However, when benchmarking you should watch out for context switching. That is, another process may be using CPU thus contending for CPU with your benchmark and increasing its run time. To avoid contention with other processes you should run a benchmark like this:
sudo chrt -f 99 /usr/bin/time -f "\n***\ntime: %E\ncontext switches: %c\nwaits: %w" <benchmark>
sudo chrt -f 99 runs your benchmark in FIFO real-time class with priority 99, which makes your process the top priority process and avoids context switching (you can change your
/etc/security/limits.conf so that it doesn't require a privileged process to use real-time priorities). It also makes
time report the number of context switches your benchmark incurred which should normally be 0, otherwise you may like to rerun the benchmark.
And it is better to disable the CPU frequency scaling and boost, so that the CPU frequency stays constant during the benchmark to get consistent results.