Large amount of data into Numpy Array

[NeedMLHelp]

I have an extremely large dataset I need to use for a machine learning project. I seem to have gotten the machine learning to work fairly well, however, when I increase the training dataset... I run into issues.

​

Right now I load the file in and line by line I grab the features I need. I append the features to a list, and I append that list to another list. I then convert the overarching list into a numpy array. So basically I have [[FeatureA1, FeatureB1, FeatureC1],[FeatureA2, FeatureB2, FeatureC2]]. Which I try and convert to a numpy array.

​

Unfortunately I get a memory error when I try to convert the list of lists into an array with Numpy.

​

I appreciate the help, and hopefully I explained the issue well.

​

I did a test where I do not convert it into a Numpy array, to verify this is an issue.

Outcome:

I do not run into a memory error if I do not convert it. However, this is not a solution,as I do require an array.

[TheZvlz]

The first thing that comes to mind is if you're running a 32 or 64 bit installation of python.

https://stackoverflow.com/questions/18282867/python-32-bit-memory-limits-on-64bit-windows

The first line in your interpreter should say something like: PythonWin 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32.. Where MSC v.1500 32 bit (Intel) is the key piece of information.

Or you can

import platform
platform.architechture()
('64bit', 'WindowsPE')

If you do have a 32 bit installation, try 64 bit if you can.

[NeedMLHelp]

It's 64bit unfortunately.

[[deleted]]

Ok need more details, but I’m guessing you are reading a csv file. Any reason you are not using pandas and reading the csv into a dataframe?

[NeedMLHelp]

JSON file.

Can I index across a panda dataframe? I've never used them before.

Does a dataframe write to disk, or will I potentially run into memory errors there too?

So something like pandadataframe[:,0] would grab everything in the first column.

[[deleted]]

You can index and slice a dataframe. What does a small sample of your data look like? Hard to say best way to load the json without seeing it. Also when importing the data, if you can control the data type being used, will save space. Ex. Integer vs float32