Smilzao Smilzao - 3 months ago 26
Python Question

grep No such file or directory with envoy.run

I try to build script that execute some grep search in my logs and print the results.
I try to use Envoy because is more easy than subprocess but when I execute grep command it gives me back an error of no such file o directory.

The dir structure is easy:


  • . # root of script

  • test.py # script file

  • web_logs/log/ # dir that contains log to search in



My test.py is easy:

import envoy

def test(value):

search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*"
print(search) #check of the search string
r = envoy.run(search)
print(r.status_code, r.std_out, r.std_err)#check of the command
response = r.std_out

if __name__ == "__main__":
test(2)


The output is:

grep 'cv=2' ./web_logs/log/log_*
(2, '', 'grep: ./web_logs/log/log_*: No such file or directory\n')


If i run the same command:

grep 'cv=2' ./web_logs/log/log_*


I can find the occurrence of the string "cv=2" in the log files.

Where is the error?

Update after the answers
The problem is in using of * that envoy cannot explode without use of glob module so I using the subprocess as it is and I try to study better the using of glob module to improve envoy.

The new code I used is:

import subprocess

def test(value):

search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*"
print(search) #check of the search string
proc = subprocess.check_output(search, shell=True)
print proc.split('\n')

if __name__ == "__main__":
test(2)

Answer

@baptistemm is actually right in that since you're not running bash as part of your process the globbing is not working.

However what's happening is a bit deeper.

When you run a sub process it can be done by one of several system services (system calls).

Short Answer (TLDR;)

Here's the correct way to do this:

import envoy

def test(value):

   search = "/bin/sh -c \"grep 'cv="+str(value)+"' ./web_logs/log/log_*\""
   print(search) #check of the search string
   r = envoy.run(search)
   print(r.status_code, r.std_out, r.std_err)#check of the command
   response = r.std_out

if __name__ == "__main__":
   test(2)

Running the command as a shell command will take care of globbing.

Long answer

Whenever a sub process is executed, it eventually gets translated into an execve system call (or equivalent).

In C library there're helper functions such as system(3) and popen(3) which wrap around execve(2) to provide easier ways of executing processes. system launches a shell and passes its argument as is to -c option of the shell. popen does extra magic, kinda like what envoy is doing in python.

In envoy, the argument is parsed for | (see def expand_args(command):) in the envoy code. and then uses the equivalent of popen to execute the processes. envoy is essentially what the shell does with the | marker (splits things up across the | and then uses popen).

What envoy is NOT doing is interpreting * as the shell does, as in expanding it to match files using a glob function of some sort. Bash does. Thus my answer.

A fun exercise would be for you to contribute code to envoy :-) and make it do the globbing.