Hec Hec - 9 months ago 82
Python Question

How to capture non-blocking realtime output from fio using Python

>>>x = subprocess.Popen(["fio", "--filename=/dev/sdd", "--name=job1", "--numjobs=2"], stdout=subprocess.PIPE)

But I need to capture the fio progess in realtime. I'm though of doing a threaded subprocess.stdout.readline, but it doesn't seem to update the performance data:

>>> x = subprocess.Popen(["fio", "--filename=/dev/sdd", "--name=job1", "--numjobs=2"], stdout=subprocess.PIPE)

>>> x.stdout.readline()
'job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1\n'

>>> x.stdout.readline()

>>> x.stdout.readline()

>>> x.stdout.readline()
'Starting 2 processes\n'

>>> x.stdout.readline()

it just hangs after this.

Command line execution of fio looks like this:

[root@goblinbank tmp]# fio --filename=/dev/sdd --name=job1 --numjobs=2
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 2 processes
Jobs: 2 (f=2): [R(2)] [0.2% done] [246.3MB/0KB/0KB /s] [63.5K/0/0 iops] [eta 02h:26m:09s]

The text after "Jobs: 2 (f=2): [R(2)]" keeps refreshing with new values.

Hec Hec
Answer Source

This fixes the issue:

x = subprocess.Popen(["fio", "--filename=/dev/sdd", "--name=job1", "--numjobs=2", "--eta=always"], stdout=subprocess.PIPE, universal_newlines=True)

Notice the --eta=always flag in fio, and the universal_newlines=True argument in Popen.

More info here.