volatile volatile - 29 days ago 20
Python Question

push engine in tornado

I am trying to code a simple push engine in tornado.
Basically, I have a program running on my server, continuously producing an output that I process by Python to update a dictionary, and I want that dictionary published to web client, for example every minute.

I would be thankful if your answer contains links to documentation, or rephrasing of my own question. I am reading with a lot of pain tornado documentation, so any help would be appreciated.

Here is a skeleton of the code with comments inside explaining what I want to do:

import subprocess
import sys
import pprint

import tornado.ioloop
import tornado.web

# this is to run my bash process and continuously yiled its output
def runProcess(cmd):
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while True:
retcode = p.poll()
line = p.stdout.readline()
yield line
if retcode is not None:
break


class MainHandler(tornado.web.RequestHandler):
def get(self):
#What can I do here if I want to send the update data every minute?
self.write(data)


def get_data(self):
data = dict()
cmd = 'myProg --args'
# this program will produce a continuous stream of data
for line in runProcess(cmd.split()):
data[line.split()[0] = line.plit()[1]
#now dictionary is updated? yield result?
# even if I want to publish updates every minute?
yield all_data


def make_app():
return tornado.web.Application([
(r"/", MainHandler),
])

if __name__ == "__main__":
app = make_app()
app.listen(8888)
tornado.ioloop.IOLoop.current().start()

Answer

If your client is a web browser, this sounds like a good fit for websockets.

Below you'll find a very very simple example. you should add extra checks, as do an actual verification of the connection's origin, error handling if the web socket is closed (it can happen... very, very often)

1) Tornado server (file stack_073.py):

import datetime
import time

import tornado.ioloop
import tornado.web
import tornado.websocket


def get_data():
    return {
        "current_time": datetime.datetime.strftime(
            datetime.datetime.now(), "%Y-%m-%d %H:%M:%S"
        )
    }


class WebSocketHandler(tornado.websocket.WebSocketHandler):
    def check_origin(self, origin):
        return True

    def open(self):
        print("WebSocket opened")
        while True:
            data = get_data()
            self.write_message(data)
            time.sleep(1)


def make_app():
    return tornado.web.Application([
        (r"/websocket", WebSocketHandler),
    ])


if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

Now, the HTML (+Javascript) that connects to the websocket and receives the Pushes:

2) Web client (file stack_073.html):

<html>
    <header>
        <script type="text/javascript">
            var ws = new WebSocket("ws://localhost:8888/websocket");
            ws.onmessage = function (evt) {
                var current_time_str = JSON.parse(evt.data)['current_time'];
                document.getElementById("date").innerHTML = current_time_str;
            };
        </script>
    </header>

    <body>
        <p id="date"></p>
    </body>
</html>

If you launch your Tornado server on a terminal, and then open the stack_073.html file, you should see that the time getting updated every second. There's no Javascript timer or anything, as you can see (the update comes from the write_message performed by Tornado)

This will basically create an eternally pending request from the browser to the server:

enter image description here

EDIT (as per the OP's comment to this answer):

Does replacing the return statement by a yield statement there solve this problem?

Pretty much, yeah. Is just that in that case, your get_data returns a generator, but yep. Look at these two changes:

def get_data():
    while True:
        yield {
            "current_time": datetime.datetime.strftime(
                datetime.datetime.now(), "%Y-%m-%d %H:%M:%S"
            )
        }
        time.sleep(1)


class WebSocketHandler(tornado.websocket.WebSocketHandler):
    def check_origin(self, origin):
        return True

    def open(self):
        print("WebSocket opened")
        while True:
            data = next(get_data())
            self.write_message(data)
Comments