Numpy Reshape Memory Error

I want to reshape list of images and the train_data size is 639976. this is how I am importing images, train_data=[] for img in tqdm(os.listdir('Images/train/images')):

Solution 1:

Looks like train_data is a large list of small arrays. I'm not familiar with cv2, so I'm guessing that the

image=cv2.resize(image, (28,28)).astype('float32')/255

creates (28,28) or (28,28,3) array of floats. By itself, not very big. Apparently that works.

The error is in:

np.reshape(train_data,(-1,28,28,3))

Since train_data is list, reshape has to first create an array, probably with np.array(train_data). If the all the components are (28,28,3) this array will already be (n,28,28,3) shape. But that's where the memory error occurs. Apparently there are some of these small(ish) arrays, that it doesn't have memory to assemble them into one big array.

I'd experiment with a subset of the files.

In[1]: 639976*28*28*3Out[1]: 1505223552  # floatsIn[2]: _*8Out[2]: 12041788416   # bytes

What's that, 12gb array? I'm not surprise you get a memory error. The list of arrays takes up more than that space, but they can be scattered in small blocks through out memory and swap. Make an array from the list and you double the memory usage.

Just for fun, try to make a blank array of that size:

np.ones((639976,28,28,3), 'float32')

If that works, try to make two.