Sen Sen - 2 months ago 31
Python Question

Running background thread in GAE flexible environment with Python-compact

I am working on migrating an existing python GAE (Google App Engine) standard environment app to the flexible environment. I read through the guide and decided to try out the python-compact runtime, as it's always good to re-use as much code as possible.

In the standard environment app, we use background_thread.start_new_background_thread() to spawn a bunch of infinite-loop threads to work on some background work forever. However, I couldn't get start_new_background_thread working in the flexible environment, even for some really simple app. Like this sample app:
github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/background

I keep getting the following error while running the app in the cloud (it works fine locally though).

I debugged into it by using the cloud debugger, but there was no any error message available at all while the exception was raised in the background_thread.py

Any idea how I can run a long-live background thread in the flexible environment with python-compact runtime? Thanks!

Answer

One of the differences between App Engine standard and App Engine flexible is that with Flex we're really just running a docker container. I can think of 2 approaches to try out.

1. Just use Python multiprocessing

App Engine standard enforces a sandbox that mostly means no direct use of threads or processes. With Flex, you should be able to just use the standard Python lib for starting a new sub process:

https://docs.python.org/3/library/subprocess.html

2. Use supervisord and docker

If that doesn't work, another approach you could take here is to customize the docker image you're using in Flex, and use supervisord to start multiple processes. First, generate the dockerfile by cd-ing into folder with your sources and running:

gcloud preview app gen-config --custom

This will create a Dockerfile that you can customize. Now, you are going to want to start 2 processes - the process we were starting (I think for python-compat it's gunicorn) and your background process. The easiest way to do that with docker is to use supervisord:

https://docs.docker.com/engine/admin/using_supervisord/

After modifying your Dockerfile and adding a supervisord.conf, you can just deploy your app as you normally would with gcloud preview app deploy.

Hope this helps!