Ilish Ilish - 7 months ago 62
Python Question

Python - Merge data from multiple thread instances

I am currently working on a project that involves connecting two devices to a python script, retrieving data from them and outputting the data.

Code outline:

• Scans for paired devices

• Paired device found creates thread instance (Two devices connected = two thread instances )

• Data is printed within the thread i.e. each instance has a separate
bundle of data

Basically when two devices are connected two instances of my thread class is created. Each thread instance returns a different bundle of data.

My question is: Is there a way I can combine the two bundles of data into one bundle of data?

Any help on this is appreciated :)


I assume you are using the threading module.

Threading in Python

Python is not multithreaded for CPU activity. The interpreter still uses a GIL (Global Interpreter Lock) for most operations and therefore linearizing operations in a python script. Threading is good to do IO however, as other threads can be woken up while a thread waits for IO.


Because of the GIL we can just use a standard list to combine our data. The idea is to pass the same list or dictionary to every Thread we create using the args parameter. See pydoc for threading.

Our simple implementation uses two Threads to show how it can be done. In real-world applications you probably use a Thread group or something similar..


def worker(data):
    # retrieve data from device
l = []
# Let's pass our list to the target via args.
a = Thread(target=worker, args=(l,))
b = Thread(target=worker, args=(l,))
# Start our threads
# Join them and print result

Further thoughts

If you want to be 100% correct and don't rely on the GIL to linearize access to your list, you can use a simple mutex to lock and unlock.

Depending on the nature of the data a dictionary might be more convenient to join data by certain keys.

Other considerations

Threads should be carefully considered. Alternatives such as asyncio, etc might be better suited.