buggy buggy - 4 months ago 14
Python Question

Executable called via subprocess.check_output prints on console but result is not returned

On a Windows machine, I'm trying to call an external executable from Python and gather its outputs for further processing. Because a local path variable has to be set before calling the executable, I created a batch script that


  • first calls another script to set %PATH% and

  • then calls the executable with the parameters given to it.



The *.bat file looks like this:

@echo off
call set_path.bat
@echo on
executable.exe %*


And the Python code like this:

print("before call");
result = subprocess.check_output([batfile, parameters], stderr=subprocess.STDOUT, shell=True);
print("after call");

print("------- ------- ------- printing result ------- ------- ------- ");
print(result);
print("------- ------- ------- /printing result ------- ------- ------- ");


Now, technically, this works. The executable is called with the intended parameters, runs, finishes and produces results. I know this, because they are mockingly displayed in the very console in which the Python script is running.

However, the result string only contains what the batch script returns, not the executables outputs:


before call

hello? yes, this is executable.exe

after call

------- ------- ------- printing result ------- ------- -------

C:\Users\me\Documents\pythonscript\execute\executable.exe "para1|para2|para3"

------- ------- ------- /printing result ------- ------- -------


The subprocess.check_output command itself somehow prints the intended output to the console, what it returns only contains the batch file's outputs after @echo is on again.

How can I access and save the executable's output to a string for further work?

Or do I have to somehow modify the batch file to catch and print the output, so that it will end upt in check_output's results? If so, how could I go about doing that?

Answer

If the program writes directly to the console (e.g. by opening the \\.\CONOUT$ device) instead of to the process standard handles, your only option is to scrape the console output. To make this simpler you can start with a new, empty screen buffer. Use ctypes to create, size, initialize, and activate it via the following functions:

If you wrap the screen buffer handle as a CRT file descriptor via msvcrt.open_osfhandle, then you can also pass it as the stdout or stderr argument of subprocess.Popen. But don't try to read the file descriptor directly. That won't work. Instead get the underlying file handle via msvcrt.get_osfhandle and call ReadConsoleOutputCharacter.

Note that you need to be attached to a console in order to be able to create and activate a console screen buffer. I added an allocate_console context manager to temporarily open a console, which is useful in a GUI application that normally doesn't have a console.

I tested the following code in Windows 10 only. Let me know how it works in Windows 7 if you test it.

ctypes definitions

import os
import msvcrt
import ctypes
from ctypes import wintypes
import contextlib

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

GENERIC_READ  = 0x80000000
GENERIC_WRITE = 0x40000000
FILE_SHARE_READ  = 1
FILE_SHARE_WRITE = 2
CONSOLE_TEXTMODE_BUFFER = 1
INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value
STD_OUTPUT_HANDLE = wintypes.DWORD(-11)
STD_ERROR_HANDLE = wintypes.DWORD(-12)

def _check_zero(result, func, args):
    if not result:
        raise ctypes.WinError(ctypes.get_last_error())
    return args

def _check_invalid(result, func, args):
    if result == INVALID_HANDLE_VALUE:
        raise ctypes.WinError(ctypes.get_last_error())
    return args

if not hasattr(wintypes, 'LPDWORD'): # Python 2
    wintypes.LPDWORD = ctypes.POINTER(wintypes.DWORD)

class COORD(ctypes.Structure):
    _fields_ = (('X', wintypes.SHORT),
                ('Y', wintypes.SHORT))

class CONSOLE_SCREEN_BUFFER_INFO(ctypes.Structure):
    _fields_ = (('dwSize',              COORD),
                ('dwCursorPosition',    COORD),
                ('wAttributes',         wintypes.WORD),
                ('srWindow',            wintypes.SMALL_RECT),
                ('dwMaximumWindowSize', COORD))

PCONSOLE_SCREEN_BUFFER_INFO = ctypes.POINTER(CONSOLE_SCREEN_BUFFER_INFO)
LPSECURITY_ATTRIBUTES = wintypes.LPVOID

kernel32.CreateConsoleScreenBuffer.errcheck = _check_invalid
kernel32.CreateConsoleScreenBuffer.restype = wintypes.HANDLE
kernel32.CreateConsoleScreenBuffer.argtypes = (
    wintypes.DWORD,        # _In_       dwDesiredAccess
    wintypes.DWORD,        # _In_       dwShareMode
    LPSECURITY_ATTRIBUTES, # _In_opt_   lpSecurityAttributes
    wintypes.DWORD,        # _In_       dwFlags
    wintypes.LPVOID)       # _Reserved_ lpScreenBufferData

kernel32.SetConsoleScreenBufferSize.errcheck = _check_zero
kernel32.SetConsoleScreenBufferSize.argtypes = (
    wintypes.HANDLE, # _In_ hConsoleOutput
    COORD)           # _In_ dwSize

kernel32.FillConsoleOutputCharacterW.errcheck = _check_zero
kernel32.FillConsoleOutputCharacterW.argtypes = (
    wintypes.HANDLE,  # _In_  hConsoleOutput
    wintypes.WCHAR,   # _In_  cCharacter
    wintypes.DWORD,   # _In_  nLength
    COORD,            # _In_  dwWriteCoord
    wintypes.LPDWORD) # _Out_ lpNumberOfCharsWritten

kernel32.GetConsoleScreenBufferInfo.errcheck = _check_zero
kernel32.GetConsoleScreenBufferInfo.argtypes = (
    wintypes.HANDLE,             # _In_  hConsoleOutput
    PCONSOLE_SCREEN_BUFFER_INFO) # _Out_ lpConsoleScreenBufferInfo

kernel32.ReadConsoleOutputCharacterW.errcheck = _check_zero
kernel32.ReadConsoleOutputCharacterW.argtypes = (
    wintypes.HANDLE,  # _In_  hConsoleOutput
    wintypes.LPWSTR,  # _Out_ lpCharacter
    wintypes.DWORD,   # _In_  nLength
    COORD,            # _In_  dwReadCoord
    wintypes.LPDWORD) # _Out_ lpNumberOfCharsRead

kernel32.SetConsoleActiveScreenBuffer.errcheck = _check_zero
kernel32.SetConsoleActiveScreenBuffer.argtypes = (
    wintypes.HANDLE,) # _In_ hConsoleOutput

kernel32.GetStdHandle.errcheck = _check_invalid
kernel32.GetStdHandle.restype = wintypes.HANDLE
kernel32.GetStdHandle.argtypes = (
    wintypes.DWORD,) # _In_ nStdHandle

kernel32.SetStdHandle.errcheck = _check_zero
kernel32.SetStdHandle.argtypes = (
   wintypes.DWORD,  # _In_ nStdHandle
   wintypes.HANDLE) # _In_ hHandle

functions

@contextlib.contextmanager
def allocate_console():
    allocated = kernel32.AllocConsole()
    try:
        yield allocated
    finally:
        if allocated:
            kernel32.FreeConsole()

@contextlib.contextmanager
def console_screen(ncols, nrows):
    hStdOut = kernel32.GetStdHandle(STD_OUTPUT_HANDLE)
    hScreen = kernel32.CreateConsoleScreenBuffer(
                GENERIC_READ | GENERIC_WRITE,
                FILE_SHARE_READ | FILE_SHARE_WRITE,
                None, CONSOLE_TEXTMODE_BUFFER, None)
    try:
        kernel32.SetConsoleScreenBufferSize(
                hScreen, COORD(ncols, nrows))
        n = (wintypes.DWORD * 1)()
        kernel32.FillConsoleOutputCharacterW(
                hScreen, u'\0', ncols * nrows, COORD(0,0), n)
        kernel32.SetConsoleActiveScreenBuffer(hScreen)
        try:
            yield msvcrt.open_osfhandle(
                        hScreen, os.O_RDWR | os.O_BINARY)
        finally:
            # TODO: error handling in case hStdOut isn't a console
            kernel32.SetConsoleActiveScreenBuffer(hStdOut)
    finally:
        kernel32.CloseHandle(hScreen)

def read_screen(fd):
    hScreen = msvcrt.get_osfhandle(fd)
    csbi = CONSOLE_SCREEN_BUFFER_INFO()
    kernel32.GetConsoleScreenBufferInfo(
        hScreen, ctypes.byref(csbi))
    ncols = csbi.dwSize.X
    pos = csbi.dwCursorPosition
    length = ncols * pos.Y + pos.X + 1
    buf = (ctypes.c_wchar * length)()
    n = (wintypes.DWORD * 1)()
    kernel32.ReadConsoleOutputCharacterW(
        hScreen, buf, length, COORD(0,0), n)
    lines = [buf[i:i+ncols].rstrip(u'\0')
                for i in range(0, n[0], ncols)]
    return u'\n'.join(lines)

example

if __name__ == '__main__':
    import io
    import textwrap
    import subprocess

    text = textwrap.dedent('''\
        Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
        eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut
        enim ad minim veniam, quis nostrud exercitation ullamco laboris
        nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor
        in reprehenderit in voluptate velit esse cillum dolore eu
        fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
        proident, sunt in culpa qui officia deserunt mollit anim id est
        laborum.''')

    cmd = ("python -c \""
           "print('piped output');"
           "conout = open(r'\\.\CONOUT$', 'r+');"
           "conout.write('''%s''')\"" % text)

    with allocate_console() as allocated:
        with console_screen(120, 1000) as fd_conout:
            stdout = subprocess.check_output(cmd).decode()
            conout = read_screen(fd_conout)
            with io.open('result.txt', 'w', encoding='utf-8') as f:
                f.write(u'stdout:\n' + stdout)
                f.write(u'\nconout:\n' + conout)

output

stdout:
piped output

conout:
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut
enim ad minim veniam, quis nostrud exercitation ullamco laboris
nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor
in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est
laborum.