0

I have a process in which an optimiser which runs thousands of iterations with different inputs.

optimizer = ng.optimizers. NGOpt (parametrization=instrum, budget=10000, num workers = 25)

with futures. ProcessPoolExecutor (max workers=optimizer.num_workers) as executor:

   recommendation optimizer.minimize (run_model, verbosity = 0, executor = executor, batch_mode=False)

Here I'm trying to minimise the return value of the function run model which is having the dataframes I want to log. The optimiser is using 25 workers as shown in the code

Currently multiprocessing is integrated with this process to reduce run time and also scale up the number of iterations.

The problem I have here is that, during each of the process I need to log the dataframes in a consolidated form so that I can monitor the performance of the optimiser in real time.

I keep running into process failures when if I use any I/o files

Created a sqllite3 database and append values to the tables in them as an interim solution .Although I'm aware this is not a full proof answer but is the only solution that I could think off.

Not able to incorporate multithreading as well due to compatibility issues and encountering GIL

Looking for any suggestions or methods to store these multiple dataframes during multipreocessing

Improve this question
5
  • Please provide enough code so others can better understand or reproduce the problem. Commented Feb 19, 2023 at 8:42
  • If you have one control process, which spawns children processes, you could pass a multiprocessing Queue to the children which they can use to post progress updates to the control process. Commented Feb 19, 2023 at 8:50
  • Understanding how optimizer.minimize is implemented is crucial here Commented Feb 19, 2023 at 8:54
  • @Pingu It's from the nevergrad package, facebookresearch.github.io/nevergrad/optimization.html Commented Feb 19, 2023 at 8:56
  • The python logging module supports writing to a single log file from different processes. See docs.python.org/3/howto/… Commented Feb 21, 2023 at 2:53
Related questions
4
Sum of cells with same row and column name in R
184
Sum across multiple columns with dplyr
2
Row-wise sum for columns with certain names
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.