user189035 user189035 - 1 month ago 8
Python Question

Multiprocessing: use **only** the physical cores?

I have a function

foo
which consumes a lot of memory and which I would like to run several instances of in parallel.

Suppose I have a CPU with 4 physical cores, each with two logical cores.

My system has enough memory to accommodate 4 instances of
foo
in parallel but not 8.

So I want to run
foo
on all 4 physical cores. In other words, I would like to ensure that doing
multiprocessing.Pool(4)
(4 being the maximum number of concurrent run of the function I can accommodate on this machine due to memory limitations) dispatches the job to the four physical cores (and not, for example, to a combo of two physical cores and their two logical offsprings).

How to do that in python?

Edit:



I removed the ' and all' part of the question which seem to have confused some of the readers. The question is really about how to dispatch a task to the physical cores only.

Edit 2:



I earlier used a code example from
multiprocessing
but I am library agnostic ,so to avoid confusion, I removed that.

Answer

I found a solution that doesn't involve changing the source code of a python module. It implements the solution suggested here. One can check that only the physical cores are active after running that command. Doing:

lscpu

in the bash returns:

CPU(s):                8
On-line CPU(s) list:   0,2,4,6
Off-line CPU(s) list:  1,3,5,7
Thread(s) per core:    1

[One can run that script from within python or before launching python]. In any case, after running the script above, typing these commands in python:

import multiprocessing
multiprocessing.cpu_count()

returns 4.

Comments