aledj2 aledj2 - 1 month ago 13
Linux Question

Calling a shell script using subprocess doesn't run all the commands in shell script

I have a Python script which loops through a folder, creating a shell command for each file.

Each command is written to a shell script and this script is then run using subprocess.Popen. (I need to do this because I also need to set up the environment before for the commands to work).

Here is some pseudocode:

def create_shell_script(self):
'''loop through a folder, create a command for each file and write this to a shell script'''
# command to run
base_command="run this"

#list of files
command_list=[]

#loop through the files to create a folder
for file in my_folder:
command_list.append(base_command+" "+file)

# open the shell script
scriptname="shell.sh"
shellscript = open(scriptname,'w')
# set the environment using '.' as using bash below to run shell script
shellscript.write("#!/bin/bash\n. /etc/profile.d/set_environment\n")

#loop through commands and write to shellscript
for command in command_list:
shellscript.write(command+"\n")

# use subprocess to run the shell script. Use bash to interpret the environment
cmd="bash "+scriptname
proc = subprocess.Popen([cmd], stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)


When I run this python script only the first 6 commands within the shell script are executed. The error message from the command suggests the command is truncated as it is read by subprocess.

When I run the shell script manually all commands are executed as expected so I know the shell script is correct.

Each command is pretty instantaneous but I can't imagine the speed causing an issue.

I did try running a subprocess command for each file but I ran into difficulties setting the environment and I like the approach of creating a single sh script as it also serves as a log file.

I have read the subprocess docs but haven't spotted anything and google hasn't helped.

Answer

You should close the shellscript file object after writing the commands to it and before running it via Popen. Otherwise, the file might not be written completely before you execute it.

The most elegant solution is to use a context manager, which closes the file automatically:

with open(scriptname, "w") as f:
    f.write(...)