Thanasis Petsas Thanasis Petsas - 6 months ago 14
Python Question

How to avoid a Broken Pipe error when printing a large amount of formatted data?

I am trying to print a list of tuples formatted in my

stdout
. For this, I use the str.format method. Everything works fine, but when I pipe the output to see the
first lines using the
head
command a
IOError
occurs.

Here is my code:

# creating the data
data = []$
for i in range(0, 1000):
pid = 'pid%d' % i
uid = 'uid%d' % i
pname = 'pname%d' % i
data.append( (pid, uid, pname) )

# find max leghed string for each field
pids, uids, pnames = zip(*data)
max_pid = len("%s" % max( pids) )
max_uid = len("%s" % max( uids) )
max_pname = len("%s" % max( pnames) )

# my template for the formatted strings
template = "{0:%d}\t{1:%d}\t{2:%d}" % (max_pid, max_uid, max_pname)

# print the formatted output to stdout
for pid, uid, pname in data:
print template.format(pid, uid, pname)


And here is the error I get after running the command:
python myscript.py | head


Traceback (most recent call last):
File "lala.py", line 16, in <module>
print template.format(pid, uid, pname)
IOError: [Errno 32] Broken pipe


Can anyone help me on this?

I tried to put
print
in a
try-except
block to handle the error,
but after that there was another message in the console:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr


I also tried to flush immediately the data through a two consecutive
sys.stdout.write
and
sys.stdout.flush
calls, but nothing happend..

Answer

head reads from stdout then closes it. This causes print to fail, internally it writes to sys.stdout, now closed.

You can simply catch the IOError and exit silently:

try:
    for pid, uid, pname in data:
        print template.format(pid, uid, pname)
except IOError:
    # stdout is closed, no point in continuing
    # Attempt to close them explicitly to prevent cleanup problems:
    try:
        sys.stdout.close()
    except IOError:
        pass
    try:
        sys.stderr.close()
    except IOError:
        pass
Comments