Hassan Baig Hassan Baig - 1 year ago 76
Python Question

using pipeline for bulk processing in redis (python example)

I'm trying to clear my concept of pipelining implemented in redis, using a python client. Take the following example:

my_server = redis.Redis(connection_pool=POOL)
for obj_id in list_of_obj_ids:
hash_name="n:"+str(obj_id)
sorted_set = "s:"+str(obj_id)
if my_server.exists(hash_name):
my_server.hset(hash_name,'val',0)
if my_server.zcard(sorted_set):
my_server.zadd(sorted_set, hash_name, time.time())


I.e. I'm updating multiple hashes via iterating through a for loop. How can I accomplish this kind of bulk update via pipelining? From what I've read, the following is what comes to my mind:

my_server = redis.Redis(connection_pool=POOL)
p = my_server.pipeline()
for obj_id in list_of_obj_ids:
hash_name="n:"+str(obj_id)
sorted_set="s:"+str(obj_id)
if p.exists(hash_name):
p.hset(hash_name,'val',0)
if p.zcard(sorted_set):
p.zadd(sorted_set, hash_name, time.time())
p.execute()


Is this correct?

Answer Source

Read about what pipelining is/does and then you'll understand why this won't work - until you execute the pipeline, none of the commands in it will be sent to the server. That makes your conditional statements miss the purpose you had in mind.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download