Hassan Baig Hassan Baig - 1 year ago 71
Python Question

using pipeline for bulk processing in redis (python example)

I'm trying to clear my concept of pipelining implemented in redis, using a python client. Take the following example:

my_server = redis.Redis(connection_pool=POOL)
for obj_id in list_of_obj_ids:
sorted_set = "s:"+str(obj_id)
if my_server.exists(hash_name):
if my_server.zcard(sorted_set):
my_server.zadd(sorted_set, hash_name, time.time())

I.e. I'm updating multiple hashes via iterating through a for loop. How can I accomplish this kind of bulk update via pipelining? From what I've read, the following is what comes to my mind:

my_server = redis.Redis(connection_pool=POOL)
p = my_server.pipeline()
for obj_id in list_of_obj_ids:
if p.exists(hash_name):
if p.zcard(sorted_set):
p.zadd(sorted_set, hash_name, time.time())

Is this correct?

Answer Source

Read about what pipelining is/does and then you'll understand why this won't work - until you execute the pipeline, none of the commands in it will be sent to the server. That makes your conditional statements miss the purpose you had in mind.