0

I guess the issue is not with the python specifically but with overall RAM management in Linux. So here's the code:

!pip install numpy opencv-python pandas matplotlib tensorflow scikit-learn

import numpy as np
import cv2 as cv
import pandas as pd
import matplotlib.pyplot as plt
import os
import gc

gc.enable()
train_dir = 'fruits-360/Training'
classes = os.listdir(train_dir)
classes = classes[:30]

all_arrays=[]
img_size=100
for i in classes:
    path=os.path.join(train_dir, i)
    class_num=classes.index(i)
    for img in os.listdir(path):
    img_array=cv.imread(os.path.join(path, img))
    mig_array=cv.cvtColor(img_array, cv.COLOR_BGR2RGB)
    all_arrays.append([img_array, class_num])

test_dir = 'fruits-360/Test'
classes2 = os.listdir(test_dir)
classes2 = classes2[:30]

all_arrays2=[]
img_size=100
for i in classes2:
    path=os.path.join(test_dir, i)
    class_num2=classes.index(i)
    for img in os.listdir(path):
    img_array=cv.imread(os.path.join(path, img))
    mig_array=cv.cvtColor(img_array, cv.COLOR_BGR2RGB)
    all_arrays.append([img_array, class_num2])

import random
random.shuffle(all_arrays)

X_train=[]
Y_train=[]
for features, label in all_arrays:
    X_train.append(features)
    Y_train.append(label)
X_train=np.array(X_train)

random.shuffle(all_arrays2)

X_test=[]
Y_test=[]
for features, label in all_arrays:
    X_test.append(features)
    Y_test.append(label)
X_test=np.array(X_test)

X_train=X_train.reshape(-1, img_size, img_size, 3)
X_train=X_train/255
X_test=X_test.reshape(-1, img_size, img_size, 3)
X_test=X_test/255

from keras.utils import to_categorical
Y_train=to_categorical(Y_train, num_classes=30)
Y_test=to_categorical(Y_test, num_classes=30)

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten, MaxPool2D, Dropout
from keras.callbacks import ReduceLROnPlateau
from keras.optimizers import Adam

x_train, x_val, y_train, y_val = train_test_split(X_train, Y_train, test_size=0.3, random_state=42)

model=Sequential()
model.add(Conv2D(filters=16, kernel_size=(5,5), padding='Same', activation='relu', input_shape=(100, 100, 3)))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Conv2D(filters=32, kernel_size=(5,5), padding='Same', activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Conv2D(filters=64, kernel_size=(5,5), padding='Same', activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.6))
model.add(Dense(30, activation='softmax'))

model.compile(optimizer=Adam(learning_rate=0.001), 
          loss='categorical_crossentropy', metrics=['accuracy'])
epochs=10
batch_size = 32

history=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs)

y_pred = model.predict(x_train)
y_pred_classes = np.argmax(y_pred, axis=1)
y_true = np.argmax(y_train, axis=1)
conf_mat = confusion_matrix(y_true, y_pred_classes)

disp = ConfusionMatrixDisplay(conf_mat, display_labels=classes)
fig, ax = plt.subplots(figsize=(15,15))
disp.plot(ax=ax)
plt.xticks(rotation=90)
plt.show()

model.summary()

The issue is not with the code itself, provided it just in case. When i run this code on jupyter notebook on windows, overall RAM usage is at about 9/16GB and the code runs fine, however if i run the code on Linux, it consumes all available RAM and swap partition, then jupyter crashes. If i run jupyter notebook with command:

systemd-run --scope -p MemoryMax=8192M jupyter-notebook

Jupyter still crashes after reaching 8 gigs and using the entire swap.

Is there a way to fix it somehow?

Improve this question
2
  • What distro is this and how much total memory is on the machine? Commented Apr 19, 2023 at 21:45
  • @NasirRiley, the distro is Endeavour os, 16 GB ram total + 8 GB swap Commented Apr 19, 2023 at 22:46

1 Answer 1

Reset to default
1

I notice you're enabling garbage collection right at the start, but gc is only going to release memory for things that are no longer in scope and the way you have this laid out, everything seems to stay in scope. A couple of approaches that might help you track the problem down would be to:

  • Compose your code into more discrete functions. i.e. break out the all_array builder logic and the model builder logic into separate functions that are then called from a main. ex: 
    def parse_classes(dir_path, ref_classes):
  out_arrays = []
  for i in ref_classes:
    ...
  return out_arrays

train_arrays = parse_classes(train_dir, classes)
test_arrays = parse_classes(test_dir, classes)

def shuffle(arr):
  random.shuffle(arr)
  for features, labels, in arr:
    ...
  return X_train, Y_train
# etc...
  • And use tracemalloc to help you identify how much memory is being used in each area so that you can better understand what's happening. This can help you jump start that process.
Improve this answer
2
  • Thanks for the answer, but the main issue is that my code works on windows without using all RAM, while in Linux it does use it fully Commented Apr 19, 2023 at 23:08
  • Yep, and that's an aspect of how the 2 OS's manage memory differently and how that impacts the GIL in python. In Windows, python will optimistically release memory back to the Heap but Linux will hold that space until the process has terminated iirc. Refactoring and profiling can help you target where in your app mem is being consumed/held so you can better manage app performance. Here's two articles that help break it down a bit more. Commented Apr 20, 2023 at 0:13

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.