rdm_ rdm_ - 5 months ago 38
Python Question

Python 2.6 : piping bash commands containing python variables(inside python script)

I want to run the below bash command from my python script:

stat --printf='%U%G%a' /tmp/file1.csv &&md5sum /tmp/file1.csv |awk '{print $1}'


I have done it using
subprocess.Popen
as below:

Command=subprocess.Popen(["stat --printf='%U%G%a' file1.csv &&md5sum file1.csv|awk '{print $1}'"],stdout=subprocess.PIPE,shell=True)


But instead of hard coding the filename I need to pass a python variable. I tried

filevar="/tmp/file.csv"
Command=subprocess.Popen(["stat --printf='%U%G%a' filevar &&md5sum filevar|awk '{print $1}'"],stdout=subprocess.PIPE,shell=True)


But the above code is not working.

I have been through all the answers related to
How to pass a python variable to subprocess


The best answer I got till now is piping python variable value to bash script (inside python script)

Based on this I tried:

Command=subprocess.Popen(["stat","--printf='%U%G%a'",filevar],stdout=subprocess.PIPE)


Which works great. But when I try to include more commands like
md5sum
it throws error.

Command=subprocess.Popen(["stat","--printf='%U%G%a'",filevar,"&&","md5sum",filevar],stdout=subprocess.PIPE)


Please suggest how this could be done.

Answer

To support spaces and other shell meta-characters, use pipes.quote():

#!/usr/bin/env python
import pipes
from subprocess import check_output

path = "/path/to/file.csv"
output = check_output("stat --printf='%U%G%a' {path} && md5sum {path}"
                      .format(path=pipes.quote(path))
                      + "|awk '{print $1}'", shell=True)

To get check_output() on Python 2.6, see What's a good equivalent to python's subprocess.check_call that returns the contents of stdout?

Note: pipes.quote() is not bullet-proof. Don't pass path to the shell unless it comes from a trusted source otherwise you risk an arbitrary shell command being executed (shell injection).

As an alternative, you could use plumbum to emulate the pipeline:

#!/usr/bin/env python
from plumbum.cmd import stat, md5sum, awk  # $ pip install plumbum

path = "/path/to/file.csv"
stat["--printf=%U%G%a", path]()
output = (md5sum[path] | awk['{print $1}'])()

See How do I use subprocess.Popen to connect multiple processes by pipes?

Depending on your case, it might make sense to implement the command in pure Python without external commands.