WeInThis WeInThis - 26 days ago 10
Python Question

Python Multithread/process combination of Json and Python?

So currently im trying to get a function right. Basically im trying to do a multiple (Concurrently) thread that takes information from a Json file into the program and then for every thread it should use each json object and then execute the code with those information.

What I have done so far - This code is only for Multiprocess which indeed works.:

#Read json File
with open('profileMulti.json', 'r', encoding='UTF-8') as json_data:
profiles_string = json.load(json_data)

def get_individual_profiles(config):
top_layer = config.get('Profiles')
if top_layer:
top_level_keys = ['profile_{}'.format(i) for i in range(len(top_layer))]
print(top_level_keys)
return [(key, top_layer.get(key)) for key in top_level_keys]
return []

def stringify(key, next_layer):
return [
' '.join(key.capitalize().split('_')),
next_layer.get('Name'),
next_layer.get('Last_Name'),
next_layer.get('Email'),
next_layer.get('Phone')
#etc etc...
]

config = profiles_string
profiles = get_individual_profiles(config)

pool = ThreadPool()

# Launch a process for each item
threads = [pool.apply_async(stringify, tuple(item)) for item in profiles]


# get() the results as each finishes
results = [res.get() for res in threads]
print('threaded results:')
for item in results:
print(item)


Output:

threaded results:
['Profile 0', 'Thrill', 'Ofit', 'Stack@hotmail.com', '123 412 123']

['Profile 1', 'Hellow', 'World', 'Stac321k@hotmail.com', '543 412 312']


Which is great. But the issues is:

That whenever I want to etc use those info

EXEMPLE

def checkoutNames(NameUrl, nameID):
payload = {
"shared": {
"challenge": {
"Name": item["Name"],
"Last_Name": item["Last_Name"],
"Email": item["Email"],
"Phone": item["Phone"],


Is that it wont recognize those attributes but what I need to do is that I need to call them by item1.. item[2] etc from that order that is from
def stringify(key, next_layer):
which I don't know by now how to make it so I don't need to do it.

The other problem is that whenever we say ETC that I use item1 in the code. Then it will only use the last thread and skip the rest. So if I do

print(item[1])


Then the only output that will come is the last one which is Hello

So the problems I need to get solved:

1. To make each thread to execute concurrently and execute the code with those information

2. Make a fix so I don't need to use item1 but to use instead item['Name'].

So the question is, is this possible and what is the ideas about this?

EDIT - This is the code i'm currently having with only one Json profile and that is working fine with just one profile. This is WITHOUT multiprocessing

Json file

{
"Profiles": {
"profile_0": {
"Url": "Myownwebsite.se",
"My-Note": "Helloworld",
"Email": "Stackoverflow@gmail.com"
"PersonNumber": "1234543",
"postal_code": "54123",
"given_name": "World",
"Last_name": "Hellow",
"street_address": "helloworld 123",
"city": "Stockholm",
"country": "Sweden",
"phone": "123456789",
"Color": "Red",
"house_number": "123",
"year": "2017"
},
"profile_1": {
"Url": "Myasdwfaesite.se",
"My-Note": "aasfase",
"Email": "fasfsef@gmail.com"
"PersonNumber": "5634543",
"postal_code": "123445",
"given_name": "Balling",
"Last_name": "Calling",
"street_address": "qwertr 123",
"city": "London",
"country": "UK",
"phone": "65412331",
"Color": "Blue",
"house_number": "321",
"year": "2018"
}

#Profile_2 etc etc
}
}


Code

with open('profileMulti.json', 'r', encoding='UTF-8') as json_data:
config = json.load(json_data)

NameUrl = config["Url"]

myNote = config["My-Note"]

def checkoutNames(NameUrl, nameID):

#Request & other codes - Removed to recude the code
#......
#......
headers = {
'Referer': '',
'Content-Type': ''
}
payload = {
"shared": {
"challenge": {
"email": config["Email"],
"PersonNumber": config["PersonNumber"],
"postal_code": config["ZipCode"],
"given_name": config["Name"],
"Last_name": config["LastName"],
"street_address": config["Address"],
"postal_code": config["ZipCode"],
"city": config["City"],
"country": config["Country"],
"email": config["Email"],
"phone": config["Phone"],
}

def checkoutNotes(NamesUrl, NamesPost):

#Request & other codes - Removed to recude the code
#......
#......

headers = {
'Accept': 'application/json, text/javascript, /; q=0.01',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': NameUrl,
'Connection': 'keep-alive'
}
payloadInfo = {
"Information": {
"Color": config["Color"],
"house_number": config["houseNumber"],
"year": config["Year"]
}
}
def wipe():
os.system('cls' if os.name == 'nt' else 'clear')

def main():
time.sleep(1)

FindName(myNote)

if _name_ == '_main_':
try: {
main()
}
except KeyboardInterrupt:
wipe()


EDIT

I just wanted to print out what Yaroslav code gives for output

enter image description here

Answer Source

You making list and expexcting it to work like dictionary. If order matters - use OrderedDict from collections. By the way, you could load you json using something like this:

from collections import OrderedDict
.... # your previous code
profiles_string = json.load(json_data, object_pairs_hook=OrderedDict)

Try with this code:

import json
from collections import OrderedDict
from multiprocessing.pool import ThreadPool
import threading
from time import sleep
import random

with open('profileMulti.json', 'r', encoding='UTF-8') as json_data:
  config = json.loads(json_data, object_pairs_hook=OrderedDict)


def stringify(key, next_layer):
  sleep(random.random()) # work emulation
  # `key` is profile name (profile_0, profile_1 etc)
  # `next_layer` is profile payload as OrderedDict:
  # {
  #    "Url": "Myasdwfaesite.se",
  #    "My-Note": "aasfase",
  #    "Email": "fasfsef@gmail.com",
  #    "PersonNumber": "5634543",
  #    "postal_code": "123445",
  #    "given_name": "Balling",
  #    "Last_name": "Calling",
  #    "street_address": "qwertr 123",
  #    "city": "London",
  #    "country": "UK",
  #    "phone": "65412331",
  #    "Color": "Blue",
  #    "house_number": "321",
  #    "year": "2018"
  #}

  # this print statement just for illustartion
  print(threading.current_thread().name, key)
  return { key: next_layer }  # still ordered

# how much workers should be started
profiles_number = len(config.get('Profiles', {}).items())

pool = ThreadPool(profiles_number)

# Launch a thread for each item and get() the results as each finishes
# `starmap_async` returns result object when all tasks are finished, so
# single get is called. Result object in thios case behave like list
results = pool.starmap_async(stringify, config.get('Profiles', {}).items()).get()

print('threaded results:')
for i in results:
    print(i, end='\n***\n')  # result separator when printed