I scan for a trigger and when I get it, load a .npy file and process it. It started to take almost 2 seconds to load the NumPy file from within the process but when I tried to load the same file from a different script it takes 0.015 seconds.
How can I diagnose this? It used to be instantaneous originally and I am not sure why it started to take longer since 3 weeks ago. The production script uses multiprocessing. I created a module containing several definitions and I call one of those definitions in the multiprocessing script:
from projectAI.projectAI import dataProcessor
.
.
.
initializations
.
.
.
processes = []
process1 = Process(target = dataProcessor,
args = (genModelName, dataPath, DB, Max, Min),
daemon = True)
processes.append(process1)
for i in range(Count):
process = Process(target = appEstimator,
args = (Data, str(i+1), city),
daemon = True)
processes.append(process)
for process in processes:
process.start()
for process in processes:
process.join()
The definition "dataProcessor" within the module "projectAI":
dataProcessor(genModelName, dataPath, DB, Max, Min):
while True:
# Check for a new entry in DB using queries
if newEntry:
data = np.load(dataPath + newEntry["fileName"] + ".npy")
# Further processing
This has been in production for months and never had issues. Suddenly it started to take 1.8 to 2 seconds to load this file. I executed the following script in the same environment:
import numpy as np, time
T1 = time.time()
data = np.load(dataPath + newEntry["fileName"] + ".npy")
T2 = time.time()
print(T2-T1)
Which takes only 0.015 seconds. What might be the cause?
while Trueloop in yourdataProcessoris hammering your system polling for a new database entry... Maybe the process that puts the entry in there can signal you more economically?sleep()of 0.05s immediately after thewhile True.