Zelong Zelong - 6 days ago 5
Python Question

How to use gevent for a task "read - search - write"?

I am trying deal with a task as following:


  1. read a line in a local file

  2. send the line to a web service for query

  3. write the results to a local file



`

def read(fin):
for query in fin:
return query

def search(query):
# Send the query to something like Google Search
result = google(query)
return result

def write(fout, result):
fout.write(result)


`

I refer to the Gevent tutorial but could not figure out how to put the three functions into gevent.

Answer

I have no idea about the point of your example related to gevent. Anyway, it still can be processed through example code below.

import gevent

def read(fin):
    for query in fin:
        return query

def search(query):
    # Send the query to something like Google Search
    result = google(query)
    return result

def write(fout, result):
    fout.write(result)

# read from different files
input_files = ["input1.json", "input2.json" ... ] 

# write result into output file
fout = open("output.json", 'wb')

# get queries from input files
queries = []
for input_file in input_files:
    with open(input_file, 'rb') as fin:
        queries.append(gevent.joinall([gevent.spawn(read, fin)]))

# store results of queries
results = []
for item in [query for query in queries]:
    results.append(gevent.joinall([gevent.spawn(search, item[0].value)]))

# save results into output file
for item in [result for result in results]:
    gevent.joinall([gevent.spawn(write, fout, item[0].value)])

The thing is both read-to-search and search-to-write have dependences, which is not able to use asynchronous way to achieve this. Thus in the example, it just can process each function and save return one by one.

Comments