14,808 questions
0
votes
0
answers
56
views
How to share a large CustomObject to workers in Python multiprocessing on Windows (spawn)?
I'm trying to run calculations using multiple cores in Python on multiple platforms (Linux, macOS, Windows). I need to pass a large CustomClass Object and a dict (both readonly) to all workers. So far ...
- 449
2
votes
2
answers
87
views
How does multiprocessing.Pool() create its child processes?
Overview:
I am trying to use a Pool internally in a module that is not __main__ and make it invisible to main that this pool exists.
Because of this, if __name__ == "__main__": protection is ...
- 23
3
votes
1
answer
84
views
JAX crashes with `CUDNN_STATUS_INTERNAL_ERROR` when using `joblib` or `multiprocessing`, but works in a single process
I am running into a FAILED_PRECONDITION: DNN library initialization failed error when trying to parallelize a JAX function using either Python's multiprocessing library or joblib.
The strange part is ...
0
votes
2
answers
178
views
Process in parallel multiple .ndjson files inside a .tar.gz file
I have a file enwiktionary_namespace_0.tar.gz that contains 86 .ndjson files
enwiktionary_namespace_0_0.ndjson
enwiktionary_namespace_0_1.ndjson
enwiktionary_namespace_0_2.ndjson
...
...
- 2,820
1
vote
0
answers
80
views
Parent process unexpectedly exits during debugging when child receives SIGTERM (works fine outside debugger)
I'm experiencing an issue where the parent Python process terminates unexpectedly when debugging, but only when a child process is sent SIGTERM. The same code works perfectly when run normally (...
- 35
1
vote
1
answer
100
views
Brightway 'bw2io.import_ecoinvent_release()' failing unless I disable multiprocessing
I tried to run bw2io.import_ecoinvent_release() to import ecoinvent as I used to, but the function is now stuck in a loop and return error messages indefinitely (see below).
What I run:
import bw2data ...
0
votes
1
answer
94
views
Python gets stuck in an infinite loop restarting multiprocessing pool workers on error in initilization routine
I am trying to setup a multiprocessing Python task on Windows 10, Python 3.13. I have "main.py" module, containing the main entry, "orchestration.py" module with worker ...
- 1,869
0
votes
1
answer
29
views
Can I avoid setting-up and tearing down processes when using PyTorch DataLoader?
In my scenario I use multiple DataLoaders with multiple Datasets to evaluate models against each other (I want to test models with multiple resolutions, which means each dataset has a distinct ...
- 3,598
1
vote
1
answer
145
views
is jax really incompatible with python multiprocesses? [closed]
I have a simple app with a main controller process, and a child process that handles API calls. They communicate using Python queues.
The app looks (something) like this:
import multiprocessing as mp
...
- 39
1
vote
2
answers
113
views
Why does NumPy load() take longer in a larger script than in another?
I scan for a trigger and when I get it, load a .npy file and process it. It started to take almost 2 seconds to load the NumPy file from within the process but when I tried to load the same file from ...
- 21
3
votes
1
answer
132
views
Isolation of a custom multiprocessing manager and how to update internal state
I try to use a custom multiprocessing manager, mostly following the example from the docs. The main difference is that my class updates internal state. It looks like this:
class IdIndex:
def ...
- 15.7k
2
votes
1
answer
84
views
Python multiprocessing parallelization
I have a class with methods to simulate sources across 16 detectors using the Gelsa package. In my main script, I call the method generate.sources. I am trying to use multiprocessing to speed up the ...
- 21
4
votes
2
answers
213
views
Python multiprocessing shared memory seems to hang or crash when interacting with small numpy arrays
I've been trying to parallelize some code that I wrote in python. The actual work is embarrassingly parallel, but I don't have much experience with multiprocessing in Python.
The actual code I'm ...
- 116
0
votes
0
answers
88
views
XCB error when using Python with Matplotlib and Multiprocessing
I ran into a problem when I'm trying to plot figures with matplotlib in separate processes using multiprocessing. I tried it on OpenBSD 7.7 with Python 3.12.11 and on Debian 12 with Python 3.11.2 and ...
- 1
0
votes
1
answer
68
views
subprocess.run() eventually reaches python's global thread limit
I have a python script that pings another device every so often.
PING_SIZE = "1"
PING_RETRYS = "1"
while True:
try:
result = subprocess.run(
["ping&...
