H. Minear H. Minear - 3 months ago 11
Linux Question

Looping through Bytes in Python

I need to loop through two lists that each contain hundreds of thousands of 4 byte positive integers. The files that contain these integers are on a remote machine. Do I need to extract the files from the remote machine to write Python code that loops through them? What functions should I use to loop through bytes in 2 lists in Python? I am working on a linux machine. Thank you in advance.

Answer Source

Sounds like you aren't using scp properly - see https://unix.stackexchange.com/questions/188285/how-to-copy-a-file-from-a-remote-server-to-a-local-machine

Depending on what the remote machine makes available, you could run the script there and just get the results; this might be more efficient.

You are pretty vague about the actual operation you want to perform; if you want to deal with a lot of data quickly NumPy might really help - something like

import numpy as np

FILES = ["a.dat", "b.dat"]    # we assume that all files are the same length

data = np.stack(
    (np.fromfile(f, dtype=np.uint32) for f in FILES),   # endian-ness may be an issue!
    axis=1
)

# applying a Python function
def myfunc(row):
    return min(row)
result = np.apply_along_axis(myfunc, 1, data)

# but using a numpy function directly would be better!
result = np.min(data, axis=1)