John Leonardo John Leonardo - 3 years ago 60
Python Question

How would I post 100,000 unique messages to my users, the quickest?

I have an app with 3 million users, and I am trying to make ~100,000 POST requests through python-requests. I am still very green to Python, so here is what I'm doing currently. I have identified ~100,000 users that are to receive a personalized message, and have scraped their user IDs and stored them in a text file. Here is the code I am using to send them all a message through my own web API (where each line in the txt file is a user ID). In my API, the only thing needed to send a message is user ID and text body in the payload. The headers part is irrelevant, just ios/android device information for debugging.

with open("users.txt") as f:
ids = f.read().splitlines()
for uid in ids:
payload = {"uid":uid,"text":"my personalized message to each user"}

result = requests.post(base, data=json.dumps(payload), headers=headers)
t += 1
print(t, result)


The "t" value is so I can see the progress, and result is to make sure I'm getting 200 status. So, this may come across as a duplicate, but after googling and searching through here, I'm not finding a solution that would work best for me. At my current speed, it will take a day and a half to send these 100,000 reqs for some reason.

One solution was to use async. I'm extremely open to this, but the tricky part is the txt file. I have 100k UNIQUE ID's I need to post to. Sometimes they overlap and then multiple users get the same message. I just need to pointed in the right direction I suppose. Thank you very much for your time :)

Answer Source

Split this file of all users into 1000 files and then run requests code with one file each thread. So this will create 100 users per file so if since you are running this on a server 1K threads would not be much problem. Also this will cut down too much time cost almost 1/1000. You might want to look at Threads

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download