Hassan Baig Hassan Baig - 18 days ago 7
Python Question

using pipeline for bulk processing in redis (python example)

I'm trying to clear my concept of pipelining implemented in redis, using a python client. Take the following example:

my_server = redis.Redis(connection_pool=POOL)
for obj_id in list_of_obj_ids:
hash_name="n:"+str(obj_id)
sorted_set = "s:"+str(obj_id)
if my_server.exists(hash_name):
my_server.hset(hash_name,'val',0)
if my_server.zcard(sorted_set):
my_server.zadd(sorted_set, hash_name, time.time())


I.e. I'm updating multiple hashes via iterating through a for loop. How can I accomplish this kind of bulk update via pipelining? From what I've read, the following is what comes to my mind:

my_server = redis.Redis(connection_pool=POOL)
p = my_server.pipeline()
for obj_id in list_of_obj_ids:
hash_name="n:"+str(obj_id)
sorted_set="s:"+str(obj_id)
if p.exists(hash_name):
p.hset(hash_name,'val',0)
if p.zcard(sorted_set):
p.zadd(sorted_set, hash_name, time.time())
p.execute()


Is this correct?

Answer

Read about what pipelining is/does and then you'll understand why this won't work - until you execute the pipeline, none of the commands in it will be sent to the server. That makes your conditional statements miss the purpose you had in mind.